Proposed new core library: pl.el

Proposed new core library: pl.el

[John Wiegley]

pl.el (standing for "parser library") is a combinator parsing library for
Emacs, similar to Haskell's Parsec. You can see how it works at the following
README:

    https://github.com/jwiegley/emacs-pl

PL offers a way to write very compact parsers of structured text. For example
(from the README):

  (pl-parse
    (delete-region (pl-str "<xml>" :beg)
                   (pl-until
                     (pl-str "</xml>" :end))))

The idea being applicative parsers is that the result of `pl-parse' is the
FORM you pass in, where every sub-parser becomes a value of the type you
intended to parse. If a sub-parse fails, either the whole parse fails, or it
returns nil if you wrap the parser in `pl-try'.

There is room for improving performance, but the API is complete enough to
start using it. Giving the unproven status, though, perhaps it should start
out in ELPA, and move to core after it has solidified and gained some users?

John

Re: Proposed new core library: pl.el

[Dmitry Gutov]

On 11/05/2015 04:14 AM, John Wiegley wrote:

> There is room for improving performance, but the API is complete enough to
> start using it. Giving the unproven status, though, perhaps it should start
> out in ELPA, and move to core after it has solidified and gained some users?

I think the rule for moving new stuff out of ELPA should be whether it's 
used by the core. That's for libraries.

For utility packages (ui/commands/etc), the question should be asked: if 
it works fine in ELPA, why move? Popularizing ELPA would be better, IMHO.

ELPA policy (was: Proposed new core library: pl.el)

[John Wiegley]

>>>>> Dmitry Gutov <dgutov@yandex.ru> writes:

> I think the rule for moving new stuff out of ELPA should be whether it's
> used by the core. That's for libraries.

An exception to this rule is when a certain service (say, streams) should
always be available, without requiring further installation of libraries.
Emacs acts as a sort of "standard library" for Emacs Lisp, so the same kinds
of things we'd like to have in such a meta-library, should be in core.

I do think that applications using such libraries should almost always go in
ELPA, except for those that have been grandfathered in, like Emacs Calc. There
are some things you should always be able to reach for, no matter whose
machine you are on, or if the Internet is currently available.

John

Re: ELPA policy

[Dmitry Gutov]

On 11/05/2015 04:41 AM, John Wiegley wrote:

> An exception to this rule is when a certain service (say, streams) should
> always be available, without requiring further installation of libraries.
> Emacs acts as a sort of "standard library" for Emacs Lisp, so the same kinds
> of things we'd like to have in such a meta-library, should be in core.

Why not consider ELPA a part of the "standard library", too?

As long as all code that uses streams.el is not in Emacs core, I don't 
see any harm in requiring the user to install streams from ELPA (which 
would probably happen automatically, via a declared dependency).

When the core starts using it, sure, we can move it there.

> I do think that applications using such libraries should almost always go in
> ELPA, except for those that have been grandfathered in, like Emacs Calc. There
> are some things you should always be able to reach for, no matter whose
> machine you are on, or if the Internet is currently available.

Yes, some pieces of functionality that we consider a part of the 
standard set (Calc is not that, for me, but apparently it's popular 
enough). The criteria for applications to include them might be that 
Emacs has dedicated menu items, or uses the application's commands in 
some global bindings (that's why xref, for example, can't be moved to ELPA).

Re: ELPA policy

[Artur Malabarba]

2015-11-05 3:00 GMT+00:00 Dmitry Gutov <dgutov@yandex.ru>:
> On 11/05/2015 04:41 AM, John Wiegley wrote:
>
>> An exception to this rule is when a certain service (say, streams) should
>> always be available, without requiring further installation of libraries.
>> Emacs acts as a sort of "standard library" for Emacs Lisp, so the same
>> kinds
>> of things we'd like to have in such a meta-library, should be in core.
>
>
> Why not consider ELPA a part of the "standard library", too?
>
>[...]
> When the core starts using it, sure, we can move it there.

I agree in a general sense. But case-by-case discretion will have to be applied.

> The
> criteria for applications to include them might be that Emacs has dedicated
> menu items, or uses the application's commands in some global bindings
> (that's why xref, for example, can't be moved to ELPA).

IMO, xref can't be moved to Gelpa because users expect this "find
definition" functionality out-of-the box. By which I don't mean "Emacs
users have come to expect it", I mean "users coming to Emacs will
expect it".

Re: ELPA policy

[Michael Welsh Duggan]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 11/05/2015 04:41 AM, John Wiegley wrote:
>
>> An exception to this rule is when a certain service (say, streams) should
>> always be available, without requiring further installation of libraries.
>> Emacs acts as a sort of "standard library" for Emacs Lisp, so the same kinds
>> of things we'd like to have in such a meta-library, should be in core.
>
> Why not consider ELPA a part of the "standard library", too?

Because it is not.  I have often had to use Emacs on computers that did
not have access to an Internet connection.  To consider something part
of the "standard library," I believe you should have access to it when
Emacs is installed.  This is why I dislike the move towards systems that
only provide easy installation via network channels.  (Python, this
means you!)

-- 
Michael Welsh Duggan
(md5i@md5i.com)

Re: ELPA policy

[Dmitry Gutov]

On 11/05/2015 02:51 PM, Michael Welsh Duggan wrote:

>> Why not consider ELPA a part of the "standard library", too?
>
> Because it is not.  I have often had to use Emacs on computers that did
> not have access to an Internet connection.  To consider something part
> of the "standard library," I believe you should have access to it when
> Emacs is installed.

We do intend to bundle some packages from ELPA to be distributed with 
Emacs releases.

Anyway, if you don't have Internet access, you also can't install any 
package that depends on the library in question. Right?

Re: ELPA policy

[Michael Welsh Duggan]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 11/05/2015 02:51 PM, Michael Welsh Duggan wrote:
>
>>> Why not consider ELPA a part of the "standard library", too?
>>
>> Because it is not.  I have often had to use Emacs on computers that did
>> not have access to an Internet connection.  To consider something part
>> of the "standard library," I believe you should have access to it when
>> Emacs is installed.
>
> We do intend to bundle some packages from ELPA to be distributed with
> Emacs releases.

In this case, I can consider these part of a standard library.

> Anyway, if you don't have Internet access, you also can't install any
> package that depends on the library in question. Right?

Not so.  Packages can still be installed via sneakernet, and elisp can
be written that wants to take advantage of a "standard library."  I have
no objection to ELPA in general.  I just have an issue with the idea
that all of ELPA (non-bundled) can be considered "standard."

-- 
Michael Welsh Duggan
(mwd@cert.org)

Re: ELPA policy

[John Wiegley]

>>>>> Dmitry Gutov <dgutov@yandex.ru> writes:

> We do intend to bundle some packages from ELPA to be distributed with Emacs
> releases.

I'd like to know more about this. How does the user access these bundled
packages; by installing them with M-x package-install?

John

Re: ELPA policy

[Dmitry Gutov]

On 11/05/2015 05:09 PM, John Wiegley wrote:

> I'd like to know more about this. How does the user access these bundled
> packages; by installing them with M-x package-install?

By having them in the default load-path somewhere already, I imagine.

The user would be able to M-x package-install, however, to install newer 
a version of the same package from ELPA, that was released in the meantime.

I think Fabian was looking into making that happen, but I can't find the 
respective thread right now.

Re: ELPA policy

[Artur Malabarba]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 11/05/2015 05:09 PM, John Wiegley wrote:
>
>> I'd like to know more about this. How does the user access these bundled
>> packages; by installing them with M-x package-install?
>
> By having them in the default load-path somewhere already, I imagine.

There are a few possible levels this could go.
But yes, the basic idea is that these bundled packages would be pulled
from Gelpa into some subdir of emacs/lisp when making the tarball and
autoloading/byte-compilation would be done. So they'd be available like
any other built-in package.

The advantage over just keeping the packages there in the first place is
that packages offered on GElpa can be updated more frequently.

A second level would be to do this in such a way that allows Emacs
packages to depend on these pulled-in Gelpa packages. This would require
changes to the build process, not just the tarball.

> I think Fabian was looking into making that happen, but I can't find
> the respective thread right now.

I think I asked about this a couple of months ago, and Stefan (?) wrote
a very nice explanation of what his thoughts were on this possible
future.

Re: ELPA policy

[Dmitry Gutov]

On 11/05/2015 06:58 PM, Artur Malabarba wrote:

>> I think Fabian was looking into making that happen, but I can't find
>> the respective thread right now.
>
> I think I asked about this a couple of months ago, and Stefan (?) wrote
> a very nice explanation of what his thoughts were on this possible
> future.

Stefan, most likely.  Too bad it wasn't in a dedicated bug discussion.

Re: ELPA policy

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I'd like to know more about this. How does the user access these bundled
  > packages; by installing them with M-x package-install?

I think they should appear, to the user, like any other standard part of Emacs.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: ELPA policy

[John Wiegley]

>>>>> Richard Stallman <rms@gnu.org> writes:

>> I'd like to know more about this. How does the user access these bundled
>> packages; by installing them with M-x package-install?

> I think they should appear, to the user, like any other standard part of
> Emacs.

If that's the case, it changes my thoughts on what needs to be in core, and
what should be in ELPA. Until now I was thinking ELPA required Internet
access; but if there are parts of ELPA that "come in the box", then I'd like
to see more packages there.

John

Re: ELPA policy

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Date: Sun, 08 Nov 2015 08:30:17 -0800
> Cc: mwd@md5i.com, emacs-devel@gnu.org, dgutov@yandex.ru
> 
> If that's the case, it changes my thoughts on what needs to be in core, and
> what should be in ELPA. Until now I was thinking ELPA required Internet
> access; but if there are parts of ELPA that "come in the box", then I'd like
> to see more packages there.

ELPA requires Internet access if you are building from the repository.

My greatest fears are that packages that move from core to ELPA will
get much less attention from the core maintenance team, which will
eventually degrade their quality.

Re: ELPA policy

[Wolfgang Jenkner]

On Sun, Nov 08 2015, Eli Zaretskii wrote:

> My greatest fears are that packages that move from core to ELPA will
> get much less attention from the core maintenance team, which will
> eventually degrade their quality.

But this is not what actually happens with completely separately
developed complex packages like SLIME, Evil...

On the other hand, IIUC, moving CEDET to emacs core has not been
completely without problems for the CEDET project and its users.

Re: ELPA policy

[Eli Zaretskii]

> From: Wolfgang Jenkner <wjenkner@inode.at>
> Cc: John Wiegley <jwiegley@gmail.com>,  mwd@md5i.com,  emacs-devel@gnu.org,  rms@gnu.org,  dgutov@yandex.ru
> Date: Sun, 08 Nov 2015 19:00:06 +0100
> 
> On Sun, Nov 08 2015, Eli Zaretskii wrote:
> 
> > My greatest fears are that packages that move from core to ELPA will
> > get much less attention from the core maintenance team, which will
> > eventually degrade their quality.
> 
> But this is not what actually happens with completely separately
> developed complex packages like SLIME, Evil...

Most packages in core are not of this kind.

> On the other hand, IIUC, moving CEDET to emacs core has not been
> completely without problems for the CEDET project and its users.

Such moves are never without problems, which is exactly my point.  In
this case, I think the benefits far outweighed the problems: until
CEDET was incorporated into Emacs core, we couldn't even dream about
using it as infrastructure for IDEs and other similar features.
AFAIR, there was also lots of cleanup during the move.

But moving ELPA packages into the core is not being discussed or
suggested.  So what happened with CEDET, even if you disagree with my
assessment above, has little relevance to the issue at hand.

Re: ELPA policy

[Wolfgang Jenkner]

On Sun, Nov 08 2015, Eli Zaretskii wrote:

>> On the other hand, IIUC, moving CEDET to emacs core has not been
>> completely without problems for the CEDET project and its users.
>
> Such moves are never without problems, which is exactly my point.

I was using the wrong word here as it's more a fork than a move:

http://sourceforge.net/p/cedet/git/ci/master/tree/INSTALL

> In
> this case, I think the benefits far outweighed the problems: until
> CEDET was incorporated into Emacs core, we couldn't even dream about
> using it as infrastructure for IDEs and other similar features.

CEDET being a separate project did not stop various people elsewhere
from realizing their dream about an "IDE" according to their own ideas:

https://www.youtube.com/results?search_query=emacs+cedet

(sorry for giving a link to a google service...)

> AFAIR, there was also lots of cleanup during the move.

> But moving ELPA packages into the core is not being discussed or
> suggested.  So what happened with CEDET, even if you disagree with my
> assessment above, has little relevance to the issue at hand.

To be clear: FWIW, I agree that CEDET is great and that using real
grammars for doing various stuff in emacs is an exciting idea.  I

Re: ELPA policy

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> If that's the case, it changes my thoughts on what needs to be in core, and
>> what should be in ELPA. Until now I was thinking ELPA required Internet
>> access; but if there are parts of ELPA that "come in the box", then I'd like
>> to see more packages there.
>
> ELPA requires Internet access if you are building from the repository.
>
> My greatest fears are that packages that move from core to ELPA will
> get much less attention from the core maintenance team, which will
> eventually degrade their quality.

Org-mode, for instance, gets regular bug fix releases on GNU ELPA while
it stagnates on core. I would venture that the contribution from the
"core maintenance team" to org-mode amounts to a negative effect,
causing more hassle than it saves.

For projects with their own development team and VCS repository, having
the project copied into the Emacs core VCS makes no sense, moreover when
they have to support multiple Emacs releases.

Re: ELPA policy

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Sun, 08 Nov 2015 19:26:20 +0100
> 
> For projects with their own development team and VCS repository, having
> the project copied into the Emacs core VCS makes no sense, moreover when
> they have to support multiple Emacs releases.

I could probably agree, as soon as we find a good way of getting the
"right" version of those when an Emacs tarball is being prepared (or
maybe even when Emacs is built from the repository).

But most packages in core are not of this variety, and it's them that
I worry about, not Org or Gnus.

Re: ELPA policy

[Artur Malabarba]

Eli Zaretskii <eliz@gnu.org> writes:

> My greatest fears are that packages that move from core to ELPA will
> get much less attention from the core maintenance team, which will
> eventually degrade their quality.

We could bring the Elpa and Emacs repositories closer together somehow.
Maybe as a submodule or something like that.

This way the core maintenance team would also have the Elpa sources
"right there" to do bug fixes.

Re: ELPA policy

[Eli Zaretskii]

> From: Artur Malabarba <bruce.connor.am@gmail.com>
> Cc: John Wiegley <jwiegley@gmail.com>,  emacs-devel@gnu.org,
> Date: Sun, 08 Nov 2015 19:27:08 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > My greatest fears are that packages that move from core to ELPA will
> > get much less attention from the core maintenance team, which will
> > eventually degrade their quality.
> 
> We could bring the Elpa and Emacs repositories closer together somehow.
> Maybe as a submodule or something like that.

How about just using a single repository? ;-)

Seriously, why should they be separate?  Submodules still complicate
things, compared to just having the stuff in the repo.

Re: ELPA policy

[Artur Malabarba]

Eli Zaretskii <eliz@gnu.org> writes:

> How about just using a single repository? ;-)
>
> Seriously, why should they be separate?  Submodules still complicate
> things, compared to just having the stuff in the repo.

I'm not strongly against that. But there are a couple of things that
might be annoying with that:

- Elpa commits would occasionally flood the git log

- Git operations in both repositories would become even slower than they
  are now (as they'd both increase in size).

Re: ELPA policy

[Eli Zaretskii]

> From: Artur Malabarba <bruce.connor.am@gmail.com>
> Cc: jwiegley@gmail.com,  emacs-devel@gnu.org
> Date: Sun, 08 Nov 2015 20:04:56 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > How about just using a single repository? ;-)
> >
> > Seriously, why should they be separate?  Submodules still complicate
> > things, compared to just having the stuff in the repo.
> 
> I'm not strongly against that. But there are a couple of things that
> might be annoying with that:
> 
> - Elpa commits would occasionally flood the git log

I'm sure no one actually looks at too many commits at once, only those
that are of interest.

> - Git operations in both repositories would become even slower than they
>   are now (as they'd both increase in size).

GCC is migrating to Git as we speak, even though it has something like
3-4 times more commits in its history than we do.  If they aren't
afraid, I don't think we should be.

Re: ELPA policy

[Dmitry Gutov]

On 11/08/2015 09:58 PM, Eli Zaretskii wrote:

> I'm sure no one actually looks at too many commits at once, only those
> that are of interest.

Filtering out the non-interesting commits is a hassle in itself, and I 
saying this as a emacs-diffs subscriber. With everyone committing to the 
same repo, it will become worse.

Re: ELPA policy

[Eli Zaretskii]

> Cc: jwiegley@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 8 Nov 2015 22:10:29 +0200
> 
> On 11/08/2015 09:58 PM, Eli Zaretskii wrote:
> 
> > I'm sure no one actually looks at too many commits at once, only those
> > that are of interest.
> 
> Filtering out the non-interesting commits is a hassle in itself, and I 
> saying this as a emacs-diffs subscriber. With everyone committing to the 
> same repo, it will become worse.

ELPA gets about 4 commits per day on average, so I don't think we
should worry about this aspect.

Re: ELPA policy

[Dmitry Gutov]

On 11/08/2015 10:26 PM, Eli Zaretskii wrote:

> ELPA gets about 4 commits per day on average, so I don't think we
> should worry about this aspect.

Emacs master has had ~82 commits in November, according to 'git log', 
which comes out to ~10 a day. With ELPA, this will be something like 40% 
increase?

And this is only while ELPA is relatively unpopular. If it ever grows to 
anything comparable to the size of MELPA (which we probably should 
aspire to), we will drown.

Re: ELPA policy

[Eli Zaretskii]

> Cc: bruce.connor.am@gmail.com, jwiegley@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 8 Nov 2015 22:36:00 +0200
> 
> On 11/08/2015 10:26 PM, Eli Zaretskii wrote:
> 
> > ELPA gets about 4 commits per day on average, so I don't think we
> > should worry about this aspect.
> 
> Emacs master has had ~82 commits in November, according to 'git log', 
> which comes out to ~10 a day. With ELPA, this will be something like 40% 
> increase?
> 
> And this is only while ELPA is relatively unpopular. If it ever grows to 
> anything comparable to the size of MELPA (which we probably should 
> aspire to), we will drown.

OK, then let's drop this idea.

Re: ELPA policy

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Filtering out the non-interesting commits is a hassle in itself, and I 
  > saying this as a emacs-diffs subscriber. With everyone committing to the 
  > same repo, it will become worse.

I am sure we could set up multiple mailing lists and send each commit
to a list according to what directory it is in.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: ELPA policy

[Dmitry Gutov]

On 11/09/2015 01:16 AM, Richard Stallman wrote:

> I am sure we could set up multiple mailing lists and send each commit
> to a list according to what directory it is in.

You have a point there, and if someone were to actually do that, it 
would help with making the notifications manageable.

Still, the more happens in the repository, the harder it is to grasp or 
manage. Right now you can call 'git log' and see the most recent things 
that happened to Emacs master. Or do 'git pull', and call 'git diff' on 
the revision range that was printed out, and see all changes since your 
last update.

Unless the Emacs sources move into a subdirectory, even listing all 
commits that were made to Emacs recently, would involve the contributor 
learning magic like this: 
http://stackoverflow.com/questions/5685007/making-git-log-ignore-changes-for-certain-paths

Re: ELPA policy

[Yuri Khan]

On Mon, Nov 9, 2015 at 7:45 AM, Dmitry Gutov <dgutov@yandex.ru> wrote:

> Still, the more happens in the repository, the harder it is to grasp or
> manage. Right now you can call 'git log' and see the most recent things that
> happened to Emacs master. Or do 'git pull', and call 'git diff' on the
> revision range that was printed out, and see all changes since your last
> update.

I almost wanted to mention that you can restrict “git log” and “git
diff” to a particular path in the repository, but realized it might
start a new git subthread.

(FWIW I also think separate repositories are better, as long as
development is mostly independent across the divide.)

Re: ELPA policy

[Artur Malabarba]

John Wiegley <jwiegley@gmail.com> writes:

> If that's the case, it changes my thoughts on what needs to be in core, and
> what should be in ELPA. Until now I was thinking ELPA required Internet
> access; but if there are parts of ELPA that "come in the box", then I'd like
> to see more packages there.

I think we ought to clarify what we're talking about here.

- There are packages that are offered only on Elpa. For these you need
  an internet connection and you need to `M-x package-install' them.

- There are packages that are offered both on Elpa and Emacs. These
  are just like any other core package (no connection and no
  package-install necessary).

  The difference is that users can get access to new versions from Elpa
  without having to wait for the next Emacs release. The disadvantage
  here is code duplication, as someone needs to make sure that any bug
  fixes applied on the Emacs side also get applied on the Elpa side (and
  vice versa).

  Current examples of this are seq.el and let-alist.

- Then there's a third option. I only found this out recently while
  browsing the `externals-list' file on Elpa, and it was apparently
  implemented by Fabián in September.

  You can configure Elpa to (while building) copy the most recent
  version of an packages from the Emacs repository. IIUC, this would
  have the same effect as the previous option, except no code
  duplication. The thing is: no packages seem to be using this, and it's
  only documented in the comments of that file (though I've verified
  that there's indeed code to implement this).

Fabián, is this feature complete?

Re: ELPA policy

[Stephen Leake]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> Richard Stallman <rms@gnu.org> writes:
>
>>> I'd like to know more about this. How does the user access these bundled
>>> packages; by installing them with M-x package-install?
>
>> I think they should appear, to the user, like any other standard part of
>> Emacs.
>
> If that's the case, it changes my thoughts on what needs to be in core, and
> what should be in ELPA. Until now I was thinking ELPA required Internet
> access; but if there are parts of ELPA that "come in the box", then I'd like
> to see more packages there.

It depends on what git repository the package is in, and how the ELPA
packages are included in core.

My impression is that the latest proposal for doing this is to keep
"core ELPA" packages in the emacs repository (not the Gnu ELPA repository),
and fix the Gnu ELPA server scripts to work with that.

Then all core Emacs developers see the core ELPA packages in their
current workflow, and they are released along with Emacs.

They are also released separately to Gnu ELPA, and they appear in
`list-packages' (as other core packages do now); that's what makes them
"ELPA core packages" as opposed to "core code".

That's an elegant solution.


The alternative is to figure out how to merge a partial checkout of the
Gnu ELPA repository into an Emacs git checkout without confusing git and
make; not at all easy.

-- 
-- Stephe

Re: ELPA policy

[David Kastrup]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> Dmitry Gutov <dgutov@yandex.ru> writes:
>
>> I think the rule for moving new stuff out of ELPA should be whether it's
>> used by the core. That's for libraries.
>
> An exception to this rule is when a certain service (say, streams)
> should always be available, without requiring further installation of
> libraries.

What does "service" and "should always be available" mean?  If it is not
end-user functionality useful for ad-hoc code entered with M-: or its
ilk (it would appear that streams is moving there), then I don't see a
necessity for including in the core: it makes more sense to put it as a
dependency in ELPA.  That way, other ELPA packages may easily require it
without depending on an upgrade of the core Emacs.

> Emacs acts as a sort of "standard library" for Emacs Lisp, so the same
> kinds of things we'd like to have in such a meta-library, should be in
> core.

Emacs cannot be upgraded using the package manager, so any dependencies
on Emacs core code are harder to cater for than a dependency on another
ELPA package.

-- 
David Kastrup

Re: Proposed new core library: pl.el

[Artur Malabarba]

2015-11-05 2:22 GMT+00:00 Dmitry Gutov <dgutov@yandex.ru>:
> On 11/05/2015 04:14 AM, John Wiegley wrote:
>
>> There is room for improving performance, but the API is complete enough to
>> start using it. Giving the unproven status, though, perhaps it should
>> start
>> out in ELPA, and move to core after it has solidified and gained some
>> users?
>
> I think the rule for moving new stuff out of ELPA should be whether it's
> used by the core. That's for libraries.
>
> For utility packages (ui/commands/etc), the question should be asked: if it
> works fine in ELPA, why move? Popularizing ELPA would be better, IMHO.

I think pl.el would be fine on Gelpa. I've personally never felt the
need for this in core (though I've definitely missed it when writing
some external packages).

Re: Proposed new core library: pl.el

[Ted Zlatanov]

On Wed, 04 Nov 2015 21:14:27 -0500 John Wiegley <jwiegley@gmail.com> wrote: 

JW> pl.el (standing for "parser library") is a combinator parsing library for
JW> Emacs, similar to Haskell's Parsec. You can see how it works at the following
JW> README:

JW>     https://github.com/jwiegley/emacs-pl
...
JW> There is room for improving performance, but the API is complete enough to
JW> start using it. Giving the unproven status, though, perhaps it should start
JW> out in ELPA, and move to core after it has solidified and gained some users?

My vote, after thinking about it, is to move it to the core.  That would
turn it into an Emacs facility, rather than an external package.  The
closest analogue is SMIE, which also lives in the core.

PL is a library for building other packages, so I think users don't
really care where it lives.

It's also unlikely existing packages will switch to use it, so new
packages can require it *and* a new Emacs. If a compatibility version is
needed, I would put the compatibility PL with version 24.999 in the GNU
ELPA or elsewhere, so the built-in PL with version 25.x or newer wins.
But I'd rather not maintain two versions.

Ted

Re: Proposed new core library: pl.el

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 535 bytes --]

On 5 Nov 2015 8:19 pm, "Ted Zlatanov" <tzz@lifelogs.com> wrote:
> My vote, after thinking about it, is to move it to the core.  That would
> turn it into an Emacs facility, rather than an external package.  The
> closest analogue is SMIE, which also lives in the core.
>
> PL is a library for building other packages, so I think users don't
> really care where it lives.

IMO, that's a reason to put it on Gelpa.
That way it can be used by such other packages that want to support Emacs <
25 without having to maintain duplicate code.

[-- Attachment #2: Type: text/html, Size: 692 bytes --]

Re: Proposed new core library: pl.el

[Ted Zlatanov]

On Thu, 5 Nov 2015 23:54:21 +0000 Artur Malabarba <bruce.connor.am@gmail.com> wrote: 

AM> On 5 Nov 2015 8:19 pm, "Ted Zlatanov" <tzz@lifelogs.com> wrote:
>> My vote, after thinking about it, is to move it to the core.  That would
>> turn it into an Emacs facility, rather than an external package.  The
>> closest analogue is SMIE, which also lives in the core.
>> 
>> PL is a library for building other packages, so I think users don't
>> really care where it lives.

AM> IMO, that's a reason to put it on Gelpa.
AM> That way it can be used by such other packages that want to support Emacs <
AM> 25 without having to maintain duplicate code.

I'm not convinced.

Ted

Re: Proposed new core library: pl.el

[Ted Zlatanov]

On Fri, 06 Nov 2015 10:35:47 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> On Thu, 5 Nov 2015 23:54:21 +0000 Artur Malabarba <bruce.connor.am@gmail.com> wrote: 
AM> On 5 Nov 2015 8:19 pm, "Ted Zlatanov" <tzz@lifelogs.com> wrote:
>>> My vote, after thinking about it, is to move it to the core.  That would
>>> turn it into an Emacs facility, rather than an external package.  The
>>> closest analogue is SMIE, which also lives in the core.
>>> 
>>> PL is a library for building other packages, so I think users don't
>>> really care where it lives.

AM> IMO, that's a reason to put it on Gelpa.
AM> That way it can be used by such other packages that want to support Emacs <
AM> 25 without having to maintain duplicate code.

TZ> I'm not convinced.

Artur, since there have been no further comments, maybe it would help if
I explained why I'm not convinced: because parsing libraries tend to be
very performance-sensitive and could take advantage of the core in ways
that most other libraries don't. They are also rare, so it makes sense
to treat them with special care instead of as just another library.

The only other example I know is SMIE, which again lives in the core.
So there are two things that would convince me in combination:

1) examples of other parsing libraries in ELPAs (GNU or otherwise)

2) examples of packages that would use PL *and* want to support Emacs 24
or older (please, let's not invent them, I want actual examples)

Thanks!
Ted

Re: Proposed new core library: pl.el

[Artur Malabarba]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Artur, since there have been no further comments, maybe it would help if
> I explained why I'm not convinced: because parsing libraries tend to be
> very performance-sensitive and could take advantage of the core in ways
> that most other libraries don't. They are also rare, so it makes sense
> to treat them with special care instead of as just another library.

Yes, this makes sense.

My comment was more about dependency libs in general, not really about parsing libs.

> The only other example I know is SMIE, which again lives in the core.
> So there are two things that would convince me in combination:
>
> 1) examples of other parsing libraries in ELPAs (GNU or otherwise)

That I'm aware there's edn.el: https://github.com/expez/edn.el

> 2) examples of packages that would use PL *and* want to support Emacs 24
> or older (please, let's not invent them, I want actual examples)

The only examples that come to mind are the ones I was involved in.
ham-mode and SX (both from Melpa) both manually parse html and would
benefit from a lib for that. (Am I correct in understanding pl helps
with html parsing?)

I'd be fine with dropping 24 support on ham-mode because it's rather
niche, but I wouldn't to drop it on SX (but then, I'm not even sure I'd
use `pl' on SX because its needs are very specific and fine-tuned).

Anyway, you've convinced me that this lib might be good in core. So I'm
OK with this now.

Re: Proposed new core library: pl.el

[John Wiegley]

>>>>> Ted Zlatanov <tzz@lifelogs.com> writes:

> Artur, since there have been no further comments, maybe it would help if I
> explained why I'm not convinced: because parsing libraries tend to be very
> performance-sensitive and could take advantage of the core in ways that most
> other libraries don't. They are also rare, so it makes sense to treat them
> with special care instead of as just another library.

Ted, what if pl.el is one of those "EPLA packages bundled with the tarball",
the way that seq.el is?

John

Re: Proposed new core library: pl.el

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 915 bytes --]

On 9 Nov 2015 10:02 pm, "John Wiegley" <jwiegley@gmail.com> wrote:
>
> >>>>> Ted Zlatanov <tzz@lifelogs.com> writes:
>
> > Artur, since there have been no further comments, maybe it would help
if I
> > explained why I'm not convinced: because parsing libraries tend to be
very
> > performance-sensitive and could take advantage of the core in ways that
most
> > other libraries don't. They are also rare, so it makes sense to treat
them
> > with special care instead of as just another library.
>
> Ted, what if pl.el is one of those "EPLA packages bundled with the
tarball",
> the way that seq.el is?

I'm fine either way now.
One thing to remember about these double packages is that they kinda need
someone who keeps an eye on both sides to ensure that any change made to
one is reflected on the other.
If a package is just going to be "maintained by emacs-devel", then it's
probably safer to keep it one-sided.

[-- Attachment #2: Type: text/html, Size: 1202 bytes --]

Re: Proposed new core library: pl.el

[John Wiegley]

>>>>> Artur Malabarba <bruce.connor.am@gmail.com> writes:

> If a package is just going to be "maintained by emacs-devel", then it's
> probably safer to keep it one-sided.

Fair enough. I think Ted was interested in helping to maintain this package,
and if so, then whichever is easiest for him. For me, having it in core is of
course easier.

John

Re: Proposed new core library: pl.el

[Ted Zlatanov]

On Mon, 09 Nov 2015 15:18:31 -0800 John Wiegley <jwiegley@gmail.com> wrote: 

>>>>>> Artur Malabarba <bruce.connor.am@gmail.com> writes:
>> If a package is just going to be "maintained by emacs-devel", then it's
>> probably safer to keep it one-sided.

JW> Fair enough. I think Ted was interested in helping to maintain this package,
JW> and if so, then whichever is easiest for him. For me, having it in core is of
JW> course easier.

I'll put it in the core when I have a chance to catch up. Before the
feature freeze, I hope.

Ted

Re: Proposed new core library: pl.el

[Ted Zlatanov]

On Mon, 09 Nov 2015 20:45:10 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> On Mon, 09 Nov 2015 15:18:31 -0800 John Wiegley <jwiegley@gmail.com> wrote: 
>>>>>>> Artur Malabarba <bruce.connor.am@gmail.com> writes:
>>> If a package is just going to be "maintained by emacs-devel", then it's
>>> probably safer to keep it one-sided.

JW> Fair enough. I think Ted was interested in helping to maintain this package,
JW> and if so, then whichever is easiest for him. For me, having it in core is of
JW> course easier.

TZ> I'll put it in the core when I have a chance to catch up. Before the
TZ> feature freeze, I hope.

John and others, can you please review the branch scratch/tzz/import-pl?

I've never imported a whole library before.  I did:

* split README.md into Commentary and the `pl-parse' docstring
* set Author to John Wiegley
* changed Keywords
* removed Version
* commit message is very brief, just "import library"

If that's OK, I'll push it.  Otherwise, please let me know what needs
adjustment.

Thanks
Ted

Re: Proposed new core library: pl.el

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 159 bytes --]

Ted Zlatanov <tzz@lifelogs.com> writes:

> * removed Version

I don't think you had to remove the version, many packages in Emacs have
a version number.

Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: Proposed new core library: pl.el

[Ted Zlatanov]

On Wed, 11 Nov 2015 16:08:24 +0100 Nicolas Petton <nicolas@petton.fr> wrote: 

NP> Ted Zlatanov <tzz@lifelogs.com> writes:
>> * removed Version

NP> I don't think you had to remove the version, many packages in Emacs have
NP> a version number.

I want the version in the core to be pinned to the Emacs version, which
will be automatic this way IIUC (e.g. 25.1).

Meanwhile the version in Github can stay at 1.x in MELPA or GNU ELPA for
pre-25 users, until it's killed off.  Did I plan this correctly?

Ted

Re: Proposed new core library: pl.el

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 899 bytes --]

On 11 Nov 2015 3:28 pm, "Ted Zlatanov" <tzz@lifelogs.com> wrote:
>
> On Wed, 11 Nov 2015 16:08:24 +0100 Nicolas Petton <nicolas@petton.fr>
wrote:
>
> NP> Ted Zlatanov <tzz@lifelogs.com> writes:
> >> * removed Version
>
> NP> I don't think you had to remove the version, many packages in Emacs
have
> NP> a version number.
>
> I want the version in the core to be pinned to the Emacs version, which
> will be automatic this way IIUC (e.g. 25.1).
>
> Meanwhile the version in Github can stay at 1.x in MELPA or GNU ELPA for
> pre-25 users, until it's killed off.  Did I plan this correctly?

Not quite. If a file inside emacs has no version number it is not
considered a package at all. So a user of emacs 25 will be able to use pl
just fine, but package.el will not see it and therfore will gladly shadow
it with pl installed from *elpa.

This is only relevant if pl is actually on one of the elpas.

[-- Attachment #2: Type: text/html, Size: 1244 bytes --]

Re: Proposed new core library: pl.el

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 838 bytes --]

On 11 Nov 2015 5:27 pm, "Artur Malabarba" <bruce.connor.am@gmail.com> wrote:
>
>
> On 11 Nov 2015 3:28 pm, "Ted Zlatanov" <tzz@lifelogs.com> wrote:
> >
> > On Wed, 11 Nov 2015 16:08:24 +0100 Nicolas Petton <nicolas@petton.fr>
wrote:
> >
> > NP> Ted Zlatanov <tzz@lifelogs.com> writes:
> > >> * removed Version
> >
> > NP> I don't think you had to remove the version, many packages in Emacs
have
> > NP> a version number.
> >
> > I want the version in the core to be pinned to the Emacs version, which
> > will be automatic this way IIUC (e.g. 25.1).
> >
> > Meanwhile the version in Github can stay at 1.x in MELPA or GNU ELPA for
> > pre-25 users, until it's killed off.  Did I plan this correctly?

> This is only relevant if pl is actually on one of the elpas.

I just checked and it's not on Melpa, so it's fine to remove the number.

[-- Attachment #2: Type: text/html, Size: 1317 bytes --]

Re: Proposed new core library: pl.el

[Ted Zlatanov]

After discussing with John, we'll put pl.el in the "tarball ELPA."
Please disregard my request for review, and sorry for the noise.

Ted

Re: Proposed new core library: pl.el

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

It was good to remove the version number.

Most of the Lisp files in Emacs don't need version numbers of their
own, and when there is one, people feel obliged to worry about when to
increment it.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

RE: Imports / inclusion of s.el into Emacs

[Drew Adams]

> >  > > So, if I would rename concat, it would be concat-to-string.
> >
> >   > But then people who need to concatenate strings will not find it,
> >   > because they will type string- TAB.
> >
> > My solution, extending apropos and its variants so it "finds"
> > 'string-' in the name 'concat', would deal with that.
> 
> The main motivation for "renaming" was to have completion find those
> names.  People who advance that proposal don't want to use apropos
> instead of completion.  So we will need to extend the completion to do
> the same trick you had in mind for apropos.

Don't forget that completion can now use substring
and other kinds of matching.

So depending on what you mean, perhaps there's no
need to extend completion - there's just a need to
let users get the kind of completion they want in
any context.

The question, for any given _use_ of completion,
is what kind of completion someone wants.  For
code-completion maybe prefix completion is great.
For `C-h f' maybe substring completion is better.

Re: Imports / inclusion of s.el into Emacs

[Philippe Vaucher]

> Don't forget that completion can now use substring
> and other kinds of matching.

You understand that substring completion fails as soon as the term is
too generic? e.g "string", "regexp" or "list", the list is just a lot
of noise. Also it doesn't quite work when the order is "reversed", e.g
C-h f "string TAB multibyte" would not return the desired function
because it has to be "multibyte TAB string".

With other topics where most functions are really about the topic I
agree it works reasonably well.

Re: Imports / inclusion of s.el into Emacs

[João Távora]

On Tue, May 5, 2020 at 8:26 AM Philippe Vaucher
<philippe.vaucher@gmail.com> wrote:
>
> > Don't forget that completion can now use substring
> > and other kinds of matching.
>
> You understand that substring completion fails as soon as the term is
> too generic? e.g "string", "regexp" or "list", the list is just a lot
> of noise.

Have you actually tried Drew's suggestion? Starts an Emacs -Q and (setq
completion-styles '(flex)), or just M-x fido-mode.  Then C-h f for
"string".  You get a _little_ noise but it's not "a lot", by far.  And
the little noise you _do_ get from internal functions in other packages
is pretty easy to remove if you boost things by their mentions in the
manual.  In fact there's a patch at the end of this message.

> Also it doesn't quite work when the order is "reversed", e.g> C-h f
> "string TAB multibyte" would not return the desired function
> because it has to be "multibyte TAB string".

But that's never going to work either way, unless you alias all the
multi-word symbols to all their possible permutations.  You'll always
get a group of people that really expected it to be multibyte-string and
group of people that expect string-multibyte.  Oh, maybe we should all
be searching for "multibyte" instead!  Bam.  First 5 results:

multibyte-string-p
string-to-multibyte
string-as-multibyte
set-buffer-multibyte
multibyte-char-to-unibyte

To be clear.  I'm not presenting these examples to be antagonistic.  I'm
kind of sympathetic to your laziness to read the manuals, I'm lazy
myself, a reasonably good trait in programmers.  So I really do use
these tools (and work on them) to get around an imperfect world instead
of demanding that the world adjust to my rigid way of thinking.

That said, it is possible to devise a completion style that will take
two words and match them against candidates in different ways, maybe
even matching them against other properties besides their names.  I
haven't found it really useful, yet.  But it's not absurd to think about
that and writing a smart completion style for Emacs is a much better
(and more interesting) contribution to it than the renaming
sledgehammer.

Also sometimes, I'll read the manual and be really impressed by its
quality, too.

João

Anyway here's the "use the manual to bump up flex score"
patch.

diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index f6e2b236f3..1590b954b7 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -3191,6 +3191,12 @@ flex-score-match-tightness
 than the latter (which has two \"holes\" and three
 one-letter-long matches).")

+(defvar flex-score-adjustment nil
+  "If non nil a function of to adjust score of a flex match.
+Function is passed three arguments: MATCH, PATTERN and
+preliminary SCORE.  It should return a factor by which to
+multiply SCORE to reach the final value.")
+
 (defun completion-pcm--hilit-commonality (pattern completions)
   (when completions
     (let* ((re (completion-pcm--pattern->regex pattern 'group))
@@ -3279,9 +3285,16 @@ completion-pcm--hilit-commonality
                 'completions-first-difference
                 nil str))
            (unless (zerop (length str))
-             (put-text-property
-              0 1 'completion-score
-              (/ score-numerator (* len (1+ score-denominator)) 1.0) str)))
+             (let ((prelim
+                    (/ score-numerator (* len (1+ score-denominator)) 1.0)))
+               (put-text-property
+                0 1 'completion-score
+                (*
+                 (if (functionp flex-score-adjustment)
+                     (funcall flex-score-adjustment str pattern prelim)
+                   1)
+                 prelim)
+                str))))
          str)p
        completions))))


Then do this:

(setq flex-score-adjustment
      (lambda (m _p _s)
        (if (assoc m (info-lookup->completions 'symbol 'emacs-lisp-mode))
            4
          1)))

Then try C-h f string,  C-h f process

Re: Imports / inclusion of s.el into Emacs

[Philippe Vaucher]

> > You understand that substring completion fails as soon as the term is
> > too generic? e.g "string", "regexp" or "list", the list is just a lot
> > of noise.
>
> Have you actually tried Drew's suggestion? Starts an Emacs -Q and (setq
> completion-styles '(flex)), or just M-x fido-mode.  Then C-h f for
> "string".  You get a _little_ noise but it's not "a lot", by far.  And
> the little noise you _do_ get from internal functions in other packages
> is pretty easy to remove if you boost things by their mentions in the
> manual.  In fact there's a patch at the end of this message.

I must admit I'm a bit of an extremist here, because even when trying
this the little noise annoys me. Also the noise is bigger with
"regexp". But I understand this solution seems good enough from your
perspective.

> > Also it doesn't quite work when the order is "reversed", e.g> C-h f
> > "string TAB multibyte" would not return the desired function
> > because it has to be "multibyte TAB string".
>
> But that's never going to work either way, unless you alias all the
> multi-word symbols to all their possible permutations.  You'll always
> get a group of people that really expected it to be multibyte-string and
> group of people that expect string-multibyte.

I think "all in string-" or "all in string- or multibyte-" would work
for me. Sure, the first time you try to find "multibyte" in in
"string-" you fail but you rapidly know that from now only multibyte
starts with multibyte. For example your list I'd do:

multibyte-string-p -> multibyte-string-p
string-to-multibyte -> multibyte-from-string
string-as-multibyte -> multibyte-from-string-unsafe
set-buffer-multibyte -> buffer-set-multibyte
multibyte-char-to-unibyte -> multibyte-char-to-unibyte

> To be clear.  I'm not presenting these examples to be antagonistic.  I'm
> kind of sympathetic to your laziness to read the manuals, I'm lazy
> myself, a reasonably good trait in programmers.  So I really do use
> these tools (and work on them) to get around an imperfect world instead
> of demanding that the world adjust to my rigid way of thinking.

I agree you have a point. But even without renaming as much as I
suggest, I think renaming the functions that does not follow the
"standard" of Emacs Lisp should be done (once it is defined). It's
much easier for the brain to think in terms of patterns, even if they
are not as straight forward as namespaces they should at least be
consistent.


> That said, it is possible to devise a completion style that will take
> two words and match them against candidates in different ways, maybe
> even matching them against other properties besides their names.  I
> haven't found it really useful, yet.  But it's not absurd to think about
> that and writing a smart completion style for Emacs is a much better
> (and more interesting) contribution to it than the renaming
> sledgehammer.

Yes, personally I'd be more enclined toward the proposal of grouping
functions according to
https://www.gnu.org/software/emacs/manual/html_node/elisp/index.html,
this should be fairly easy to implement.

Re: Imports / inclusion of s.el into Emacs

[João Távora]

On Tue, May 5, 2020 at 12:58 PM Philippe Vaucher
<philippe.vaucher@gmail.com> wrote:

> I think "all in string-" or "all in string- or multibyte-" would work
> for me. Sure, the first time you try to find "multibyte" in in
> "string-" you fail but you rapidly know that from now only multibyte
> starts with multibyte. For example your list I'd do:
>
> multibyte-string-p -> multibyte-string-p
> string-to-multibyte -> multibyte-from-string
> string-as-multibyte -> multibyte-from-string-unsafe
> set-buffer-multibyte -> buffer-set-multibyte
> multibyte-char-to-unibyte -> multibyte-char-to-unibyte

[1]

Do you know what this would make to fine completion list
I gave you?  It would make it suck.  So you, the "bit of an
extremist" who "is annoyed by a little noise", are prepared to
introduce  an unimaginable amount of it into my and other's
workflows at the slightest difficulty and resistance to read
a fine manual.  I like Emacs because it respects people's
established workflows, and allows for programmers
to build on it, so they can use whatever workflow they
prefer.

Unfortunately, you're finding it a bit hard to support
your -- absolutely legitimate, mind you -- ruby-esque
ways.  But you should be working to have proper
namespacing so you can work with a magnar-string.el
or ruby-regexp.el library the way you feel more
confortable.  That takes work, yes, but at least it's a
win-win, not a lose-lose. If proper namespacing takes a lot
of work, then work on a powerful completion tool. I can
help you with that.  You're French, right? Imagine if
Google  decided they'd to the complete works of Raymond
Roussell from French to modern, easy-to-search, French.

João

PS: BTW let a monkey bite me if for each of those
proposed renames you don't start a separate
100-mail long bike-shedding war.

Re: Imports / inclusion of s.el into Emacs

[Philippe Vaucher]

> > multibyte-string-p -> multibyte-string-p
> > string-to-multibyte -> multibyte-from-string
> > string-as-multibyte -> multibyte-from-string-unsafe
> > set-buffer-multibyte -> buffer-set-multibyte
> > multibyte-char-to-unibyte -> multibyte-char-to-unibyte
>
> [1]
>
> Do you know what this would make to fine completion list
> I gave you?  It would make it suck.  So you, the "bit of an
> extremist" who "is annoyed by a little noise", are prepared to
> introduce  an unimaginable amount of it into my and other's
> workflows at the slightest difficulty and resistance to read
> a fine manual.  I like Emacs because it respects people's
> established workflows, and allows for programmers
> to build on it, so they can use whatever workflow they
> prefer.

I think I already acknowledged that point in
https://lists.gnu.org/archive/html/emacs-devel/2020-05/msg00508.html
Which is why I don't think any of my renames proposal will be
accepted. Maybe one or two but so far I haven't found a single example
where everyone agreed. In the multibyte example I was just
illustrating how I think about things.


> Unfortunately, you're finding it a bit hard to support
> your -- absolutely legitimate, mind you -- ruby-esque
> ways.  But you should be working to have proper
> namespacing so you can work with a magnar-string.el
> or ruby-regexp.el library the way you feel more
> confortable.  That takes work, yes, but at least it's a
> win-win, not a lose-lose. If proper namespacing takes a lot
> of work, then work on a powerful completion tool. I can
> help you with that.  You're French, right? Imagine if
> Google  decided they'd to the complete works of Raymond
> Roussell from French to modern, easy-to-search, French.

Actually I'm swiss, from the french speaking part of Switzerland.
Funnily enough we think we speak a better french than the frenchs
themselves, given we say the more logical "seventy-five" instead of
"sixty-fifteen" like they do. Maybe that's why I think I know better
how to name things than the actual authors :-)

Anyway, at this point I'll make a "concrete proposal" like I was
asked, something very simple and hopefully very uncontroversial, but I
think the bikeshedding can stop. We obviously disagree how APIs should
be designed.

Basically I focus more on the advantages of putting the
discoverability/consistency inside the language itself instead of its
tools, while you focus more on how disruptive this approch will be and
how we could still get what I want with new tools instead of the
language.

I thought we had a golden opportunity to put s.el inside Emacs:

- The author is willing.
- The number of contributors is small, most of them already signed the
papers including the author.
- A lot of "github" Emacs users would see this as a good sign.

But I realise now that even if this was done, what would likely happen
is that s.el get stale because adding/modifying it is not as simple &
smooth as doing so on other platforms, and a new s.el will appear and
we can repeat what happened these past days in a few years.

On the topic of s.el inclusion here are my conclusions (please correct
me if they are wrong):

- I can try to suggest a few aliases, and maybe one or two will be
accepted, but certainly not a lot.
- I can try to suggest a few new functions, and maybe one or two will
be accepted.
- Whatever is introduced is likely to be "Emacsified" as not to look
too much like clojure.
- I can notify the author of s.el that only a tiny subset of s.el (if
any) is likely to be imported, but he should know he's very welcome to
put s.el on ELPA.

Kind regards,
Philippe

Re: Imports / inclusion of s.el into Emacs

[Stefan Kangas]

Philippe Vaucher <philippe.vaucher@gmail.com> writes:

> I thought we had a golden opportunity to put s.el inside Emacs:

We can still put it in GNU ELPA, which is just as good.  Frankly, it's
better.  We can pick and choose all the best ideas and include them in
a way that fits with the rest of Emacs.  And users who still prefer
s.el are free to use it, including in other GNU ELPA packages.

> - I can try to suggest a few aliases, and maybe one or two will be
> accepted, but certainly not a lot.
> - I can try to suggest a few new functions, and maybe one or two will
> be accepted.

Sounds good, thank you.  (I'm not sure why you think that most of your
suggestions would be rejected, though.)

> - Whatever is introduced is likely to be "Emacsified" as not to look
> too much like clojure.

The way I understand this discussion, I don't think the point is that
we just want to be different for the sake of being different.  It's
just that we can't import functions wholesale simply to be similar to
another language when we already have perfectly good alternatives in
ELisp.  Or at least we can't do that in core (OTOH it's fine if a
library like s.el wants to do that).

> - I can notify the author of s.el that only a tiny subset of s.el (if
> any) is likely to be imported, but he should know he's very welcome to
> put s.el on ELPA.

If I read you correctly, you're a bit disappointed.  Even so, please
focus on the positive, that we would like to add s.el to GNU ELPA,
rather than the negative "it will not be added to Emacs core".  I
believe adding it to GNU ELPA will send most of the positive and
encouraging signals that we all want to communicate to the
outside-of-ELPA package author crowd.

Best regards,
Stefan Kangas

Re: Imports / inclusion of s.el into Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I don't think we should distribute or recommend s.el in any way.
It would be self-defeating to do that.

This is not a moral issue.  s.el is free software; it is not evil.
But we have to exercize good technical judgment as well as moral judgment.
We don't want people to make their Emacs Lisp code depend on s.el,
because that would not fit into Emacs well.  So we should avoid
encouraging making Emacs Lisp code depend on s.el.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Imports / inclusion of s.el into Emacs

[Stefan Monnier]

> This is not a moral issue.  s.el is free software; it is not evil.
> But we have to exercize good technical judgment as well as moral judgment.
> We don't want people to make their Emacs Lisp code depend on s.el,
> because that would not fit into Emacs well.  So we should avoid
> encouraging making Emacs Lisp code depend on s.el.

I have the impression that you don't live in the same universe as mine:
in my world, `s.el` is already used by the majority of new packages even
tho it's neither part of Emacs nor of GNU ELPA.  Not including it in GNU
ELPA just increases the difficulty of accepting other packages in GNU ELPA.

We can try and provide something better, but as this long thread about
trying to have a more organized namespace has shown, I don't think this
is going to happen any time soon.


        Stefan

Re: Imports / inclusion of s.el into Emacs

[Philippe Vaucher]

> > This is not a moral issue.  s.el is free software; it is not evil.
> > But we have to exercize good technical judgment as well as moral judgment.
> > We don't want people to make their Emacs Lisp code depend on s.el,
> > because that would not fit into Emacs well.  So we should avoid
> > encouraging making Emacs Lisp code depend on s.el.
>
> I have the impression that you don't live in the same universe as mine:
> in my world, `s.el` is already used by the majority of new packages even
> tho it's neither part of Emacs nor of GNU ELPA.  Not including it in GNU
> ELPA just increases the difficulty of accepting other packages in GNU ELPA.

Richard, Eli: please decide wether you want s.el into ELPA or not.
This is a bit embarassing for me to have done the work of getting
magnars to agree to put it there just to be refused at the last
minute, especially since in the past days I receive nothing but
requests to at least get s.el into ELPA, if the namespace way lead to
nowhere.

I'm not sure why there's this sudden turnaround on this issue, maybe
I'm missing something.

Re: Imports / inclusion of s.el into Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Richard, Eli: please decide wether you want s.el into ELPA or not.

In its current form, I think it would be a grave mistake to include
s.el in ELPA.  Its purpose is to make Emacs Lisp mimic object-oriented
languages which are alien to Emacs Lisp.

See my message to Stefan for a change that would make s.sl ok to add.

  > I'm not sure why there's this sudden turnaround on this issue, maybe
  > I'm missing something.

I don't think there was a turnaround.  We never decided to include
s.el in GNU ELPA.

Before yesterday, we were talking about a different question: whether
to adopt the s.el functions and their names in Emacs Lisp.  When I saw
concretely what those actually were, I said no.

Then yesterday I saw a message proposing to include s.el in ELPA, and
_presuming_ that that was ok.  I responded no, saying that it would
send Emacs Lisp down the same wrong path.  We should not have code in
Emacs that uses string-prepend instead of concat.  We should fix that
code to use concat.

  > This is a bit embarassing for me to have done the work of getting
  > magnars to agree to put it there just to be refused at the last
  > minute,

I am sorry for your disappointment, because I feel for your eagerness
to make a change you consider an improvement.  But we have to make the
decision that is right.

There is no need for you to feel embarrassed.  The people you talked
with will understand that there is no reason to blame or criticize you
for this.

Meanwhile, maybe we could include it with some changes, as clostring.el.
Please see my message to Stefan.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Imports / inclusion of s.el into Emacs

[Stefan Monnier]

> See my message to Stefan for a change that would make s.sl ok to add.

There's no change under discussion: either it's in GNU ELPA or it's not
(in which case it will keep living happily ever after in MELPA).

>   > I'm not sure why there's this sudden turnaround on this issue, maybe
>   > I'm missing something.
> I don't think there was a turnaround.  We never decided to include
> s.el in GNU ELPA.

Yes, I did.


        Stefan

Re: Imports / inclusion of s.el into Emacs

[Philippe Vaucher]

> >   > I'm not sure why there's this sudden turnaround on this issue, maybe
> >   > I'm missing something.
> > I don't think there was a turnaround.  We never decided to include
> > s.el in GNU ELPA.
>
> Yes, I did.

I was sure that at least 3 people asked for it but now I can only find
you. I think I misremembered the following message from Richard to be
about `s.el`:

> Would someone with good social skills like to ask the developers of
> dash to move their development into GNU ELPA, or ask why they have not
> done so?  I am sure they can see the advantage of having their latest
> version available through Emacs.

There's something I don't quite understand tho, please explain it to
me: from my uneducated eye dash.el is very similar to s.el. If I
understood correctly having dash.el active in ELPA is something you'd
like but having s.el in ELPA would be "bad". I just don't understand,
almost all of the criticism that you did to s.el I can do to dash.el
as well ("useless" functions, clojure-like, already exists in Emacs
core but named differently, etc).

Philippe

Re: Imports / inclusion of s.el into Emacs

[Stefan Monnier]

>> Would someone with good social skills like to ask the developers of
>> dash to move their development into GNU ELPA, or ask why they have not
>> done so?  I am sure they can see the advantage of having their latest
>> version available through Emacs.
> There's something I don't quite understand tho, please explain it to
> me: from my uneducated eye dash.el is very similar to s.el.

My guess: he doesn't know dash.el (just like he doesn't know about most
of elpa.git, and of course even more so about the rest of the Emacs
world outside of emacs-devel and emacs.git: he doesn't have the time to
learn about those things).

With a bit of luck, he'll now ask for dash.el to be removed from GNU ELPA.


        Stefan "depressed"

Re: Imports / inclusion of s.el into Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > My guess: he doesn't know dash.el 

Of course I don't!  How would I?  With so many responsibilities, I
can't learn everything that it would be useful for me to know.  I
can't try everything in GNU ELPA in my copious spare time.

The only way I would find out about dash.el is if you tell me.
So why didn't you tell me?

It is not too late.  Would you please tell me what I need to know
about it?  It sounds like you know something problematic about it, or
that you think might be problematic.  What is that?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Imports / inclusion of s.el into Emacs

[Stefan Monnier]

> It is not too late.  Would you please tell me what I need to know
> about it?  It sounds like you know something problematic about it, or
> that you think might be problematic.  What is that?

I made the first efforts to get it integrated into GNU ELPA (and Phil
did the heavy lifting), so no I definitely don't think it's problematic.


        Stefan

ELPA policy (was: Imports / inclusion of s.el into Emacs)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: philippe.vaucher@gmail.com,  stefan@marxist.se,  emacs-devel@gnu.org,
>   joaotavora@gmail.com,  pcr910303@icloud.com,  dgutov@yandex.ru,
>   eliz@gnu.org,  drew.adams@oracle.com
> Date: Thu, 07 May 2020 23:38:23 -0400
> 
> I made the first efforts to get it integrated into GNU ELPA (and Phil
> did the heavy lifting), so no I definitely don't think it's problematic.

FWIW, I took a cursory look at dash.el in ELPA, and was surprised to
see that its doc strings are not according to our coding conventions,
and functions/macros that clearly aren't internal don't follow the
naming conventions.

So maybe it's time we defined the minimum requirements for packages to
be included in ELPA?

Re: ELPA policy

[Stefan Monnier]

> FWIW, I took a cursory look at dash.el in ELPA, and was surprised to
> see that its doc strings are not according to our coding conventions,
> and functions/macros that clearly aren't internal don't follow the
> naming conventions.

Then I suggest you file a bug report with the author.

> So maybe it's time we defined the minimum requirements for packages to
> be included in ELPA?

What good would it do?


        Stefan

Re: ELPA policy

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Fri, 08 May 2020 10:57:58 -0400
> 
> > FWIW, I took a cursory look at dash.el in ELPA, and was surprised to
> > see that its doc strings are not according to our coding conventions,
> > and functions/macros that clearly aren't internal don't follow the
> > naming conventions.
> 
> Then I suggest you file a bug report with the author.

Aren't ELPA bugs reported via our bug tracker?

> > So maybe it's time we defined the minimum requirements for packages to
> > be included in ELPA?
> 
> What good would it do?

<Shrug> It will get us on the same page when we are reviewing packages
and patches for ELPA.

Re: ELPA policy

[Stefan Monnier]

>> > FWIW, I took a cursory look at dash.el in ELPA, and was surprised to
>> > see that its doc strings are not according to our coding conventions,
>> > and functions/macros that clearly aren't internal don't follow the
>> > naming conventions.
>> Then I suggest you file a bug report with the author.
> Aren't ELPA bugs reported via our bug tracker?

We tell users that they can submit bug reports via our bug tracker.
And maintainers are free to use other bug trackers.
Ideally they should keep an eye on our tracker, or else someone needs to
forward our bugs to their tracker when such things happen.

IOW, as a user of dash.el you can submit your bug report via `M-x
report-emacs-bug` (and if needed, hopefully someone will forward it to
where the maintainer will see it).

>> > So maybe it's time we defined the minimum requirements for packages to
>> > be included in ELPA?
>> What good would it do?
> <Shrug> It will get us on the same page when we are reviewing packages
> and patches for ELPA.

That's not a very high concern for me, compared to the concern of keeping
GNU ELPA relevant and vaguely up-to-date.


        Stefan

Re: ELPA policy

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Fri, 08 May 2020 19:16:03 -0400
> 
> >> > So maybe it's time we defined the minimum requirements for packages to
> >> > be included in ELPA?
> >> What good would it do?
> > <Shrug> It will get us on the same page when we are reviewing packages
> > and patches for ELPA.
> 
> That's not a very high concern for me, compared to the concern of keeping
> GNU ELPA relevant and vaguely up-to-date.

Excuse me, but what do you mean by "for me"?  Isn't GNU ELPA an
important part of the Emacs project?  I'd expect something like "for
us" or "for Emacs", in which case not just you personally have a say.

And why do I have to submit bug reports against an ELPA package for
violation of our coding conventions?  I'd expect the maintainer of the
package to be asked to fix those as a precondition for accepting the
package in GNU ELPA, or at least as a long-term plan to which the
maintainer agreed in advance (in which case no bug report would have
been necessary).

What am I missing here?

Re: ELPA policy

[David Engster]

> And why do I have to submit bug reports against an ELPA package for
> violation of our coding conventions?  I'd expect the maintainer of the
> package to be asked to fix those as a precondition for accepting the
> package in GNU ELPA, or at least as a long-term plan to which the
> maintainer agreed in advance (in which case no bug report would have
> been necessary).
>
> What am I missing here?

The README for GNU ELPA states:

We do not impose a particular coding style on GNU ELPA packages, but of
course we recommend the coding style used in Emacs's own source code.
Furthermore we recommend the following:
- Use `cl-lib` rather than `cl` if it all possible.
- Use lexical-binding if it all possible.
- Try and fix the warnings emitted when compiling the package with a
  recent Emacs.

From: http://git.savannah.gnu.org/cgit/emacs/elpa.git/plain/README

So from my understanding: Following Emacs coding guidelines is a
recommendation, but not a precondition for getting packages into
GNU ELPA.

If we start bundling certain ELPA packages with Emacs proper, then of
course these special "core packages" would need to adhere to the Emacs
coding style. I don't see any difficulty in making this distinction
between core packages and the rest. And I also don't see any problem to
put s.el in ELPA and say: note that using this package is against the
Emacs coding style, so as long as you depend on this packages, it cannot
become a "core package" in the future (same for dash.el).

-David

Re: ELPA policy

[Eli Zaretskii]

> From: David Engster <deng@randomsample.de>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  emacs-devel@gnu.org
> Date: Sat, 09 May 2020 09:35:49 +0200
> 
> From: http://git.savannah.gnu.org/cgit/emacs/elpa.git/plain/README
> 
> So from my understanding: Following Emacs coding guidelines is a
> recommendation, but not a precondition for getting packages into
> GNU ELPA.
> 
> If we start bundling certain ELPA packages with Emacs proper, then of
> course these special "core packages" would need to adhere to the Emacs
> coding style. I don't see any difficulty in making this distinction
> between core packages and the rest. And I also don't see any problem to
> put s.el in ELPA and say: note that using this package is against the
> Emacs coding style, so as long as you depend on this packages, it cannot
> become a "core package" in the future (same for dash.el).

If we mark some ELPA packages up front as unsuitable for inclusion in
core, then I guess this could be acceptable.  It does mean that such a
decision would make it hard to change our minds later, though.  But at
least we should have an indication in each package whether it is or
isn't exempt from our conventions, something we don't have now.

Moreover, our conventions not being an absolute requirement, I think
_some_ requirements should still be non-negotiable, because we cannot
just accept anything.  Otherwise any request for a change could be met
with an argument like "but I'm not under any obligation to comply" or
somesuch.  I don't see any such requirements described in README, so
my question still stands, I think.

Re: ELPA policy

[David Engster]

>> From: David Engster <deng@randomsample.de>
>> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  emacs-devel@gnu.org
>> Date: Sat, 09 May 2020 09:35:49 +0200
>
>> 
>> From: http://git.savannah.gnu.org/cgit/emacs/elpa.git/plain/README
>> 
>> So from my understanding: Following Emacs coding guidelines is a
>> recommendation, but not a precondition for getting packages into
>> GNU ELPA.
>> 
>> If we start bundling certain ELPA packages with Emacs proper, then of
>> course these special "core packages" would need to adhere to the Emacs
>> coding style. I don't see any difficulty in making this distinction
>> between core packages and the rest. And I also don't see any problem to
>> put s.el in ELPA and say: note that using this package is against the
>> Emacs coding style, so as long as you depend on this packages, it cannot
>> become a "core package" in the future (same for dash.el).
>
> If we mark some ELPA packages up front as unsuitable for inclusion in
> core, then I guess this could be acceptable.  It does mean that such a
> decision would make it hard to change our minds later, though.  But at
> least we should have an indication in each package whether it is or
> isn't exempt from our conventions, something we don't have now.

If you look here

http://git.savannah.gnu.org/cgit/emacs/elpa.git/tree/externals-list

you'll see that there are already packages marked as ":core" which all
currently refer to a path in Emacs proper. This way, they could be
overriden through GNU ELPA, and they of course need to adhere to the
Emacs coding style.

You are right that it will be difficult if we decide a package should
become "core" and currently does not adhere to the Emacs coding style,
but frankly, we should be more worried that this ship has already sailed
far away. Many packages which are absolutely essential for a modern
programmer's editor are already only available through MELPA. If we want
to make GNU ELPA more relevant, we need to lower the hurdle for
inclusion. I also fully agree with Stefan that we should make it
possible for packages that have non-FSF copyright to be included. Of
course these packages could never become "core", but having them
installable throught GNU ELPA would be the next best thing.

> Moreover, our conventions not being an absolute requirement, I think
> _some_ requirements should still be non-negotiable, because we cannot
> just accept anything.  Otherwise any request for a change could be met
> with an argument like "but I'm not under any obligation to comply" or
> somesuch.  I don't see any such requirements described in README, so
> my question still stands, I think.

Yes. For instance, it is clear that a package in GNU ELPA must not
recommend or even depend on non-free software.

-David

Re: ELPA policy

[Eli Zaretskii]

> From: David Engster <deng@randomsample.de>
> Cc: monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Sat, 09 May 2020 10:16:38 +0200
> 
> You are right that it will be difficult if we decide a package should
> become "core" and currently does not adhere to the Emacs coding style,
> but frankly, we should be more worried that this ship has already sailed
> far away. Many packages which are absolutely essential for a modern
> programmer's editor are already only available through MELPA.

Why is that a problem?  We can never do anything to prevent people
from concocting whatever packages they want and making them available
for others.  Nor do I think we should: this is, after all, Free
Software: people should be free to choose whatever software they like
that does the job for them.  We can never control that, and we
shouldn't try.

> I also fully agree with Stefan that we should make it possible for
> packages that have non-FSF copyright to be included. Of course these
> packages could never become "core", but having them installable
> throught GNU ELPA would be the next best thing.

If we are going to drop requirements, then what will distinguish ELPA
from MELPA?  And what's the problem with having non-core packages
available through MELPA, anyway? why do we need to have them in ELPA?

This all goes back to my confusion about the relation between ELPA and
Emacs, something I thought I understood, but now I conclude that I
don't.

Re: ELPA policy

[David Engster]

>> From: David Engster <deng@randomsample.de>
>> Cc: monnier@iro.umontreal.ca,  emacs-devel@gnu.org
>> Date: Sat, 09 May 2020 10:16:38 +0200
>> 
>> You are right that it will be difficult if we decide a package should
>> become "core" and currently does not adhere to the Emacs coding style,
>> but frankly, we should be more worried that this ship has already sailed
>> far away. Many packages which are absolutely essential for a modern
>> programmer's editor are already only available through MELPA.
>
> Why is that a problem?  We can never do anything to prevent people
> from concocting whatever packages they want and making them available
> for others.  Nor do I think we should: this is, after all, Free
> Software: people should be free to choose whatever software they like
> that does the job for them.  We can never control that, and we
> shouldn't try.
>
>> I also fully agree with Stefan that we should make it possible for
>> packages that have non-FSF copyright to be included. Of course these
>> packages could never become "core", but having them installable
>> throught GNU ELPA would be the next best thing.
>
> If we are going to drop requirements, then what will distinguish ELPA
> from MELPA?  And what's the problem with having non-core packages
> available through MELPA, anyway? why do we need to have them in ELPA?

In principal, I agree with you. The problem is mainly Richard's stance
on this issue, which says that we must not recommend packages which are
not in Emacs or GNU ELPA, but that we should rather re-implement them. I
think that's a terrible waste.

-David

Re: ELPA policy

[Eli Zaretskii]

> From: David Engster <deng@randomsample.de>
> Cc: monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Sat, 09 May 2020 10:43:55 +0200
> 
> > If we are going to drop requirements, then what will distinguish ELPA
> > from MELPA?  And what's the problem with having non-core packages
> > available through MELPA, anyway? why do we need to have them in ELPA?
> 
> In principal, I agree with you. The problem is mainly Richard's stance
> on this issue, which says that we must not recommend packages which are
> not in Emacs or GNU ELPA, but that we should rather re-implement them. I
> think that's a terrible waste.

Is this only about "recommending" or not "recommending" a package?  Is
this why we created GNU ELPA and invest non-trivial amount of effort
in maintaining and developing it?  I very much hope there's more to
it than just that.

I could understand if you'd say "use" instead of "recommend",
i.e. have code in Emacs, which, if a package is installed, would use
it.  That'd actually have the package's name in our sources, and would
constitute some kind of "endorsement".  But as long as we don't use
any of those packages, why should we care what other people like or
don't like?

Re: ELPA policy

[David Engster]

>> From: David Engster <deng@randomsample.de>
>> Cc: monnier@iro.umontreal.ca,  emacs-devel@gnu.org
>> Date: Sat, 09 May 2020 10:43:55 +0200
>
>> 
>> > If we are going to drop requirements, then what will distinguish ELPA
>> > from MELPA?  And what's the problem with having non-core packages
>> > available through MELPA, anyway? why do we need to have them in ELPA?
>> 
>> In principal, I agree with you. The problem is mainly Richard's stance
>> on this issue, which says that we must not recommend packages which are
>> not in Emacs or GNU ELPA, but that we should rather re-implement them. I
>> think that's a terrible waste.
>
> Is this only about "recommending" or not "recommending" a package?  Is
> this why we created GNU ELPA and invest non-trivial amount of effort
> in maintaining and developing it?  I very much hope there's more to
> it than just that.

We created GNU ELPA because we wanted a package system in Emacs. It was
simply the first, then things evolved to the state we have now, where
MELPA is the dominant package repository.

> I could understand if you'd say "use" instead of "recommend",
> i.e. have code in Emacs, which, if a package is installed, would use
> it.  That'd actually have the package's name in our sources, and would
> constitute some kind of "endorsement".  But as long as we don't use
> any of those packages, why should we care what other people like or
> don't like?

Sorry, but you lost me there. All I'm saying is that there's a whole lot
of terrific packages out there but which we must not recommend to users,
although they are free software and often vastly superior to the things
that are built into Emacs. For instance, there's a discussion going on
about making a video showing Emacs' capabilities, but I assume we'd not
be allowed to show Magit. That's a huge loss.

-David

Re: ELPA policy

[Eli Zaretskii]

> From: David Engster <deng@randomsample.de>
> Cc: monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Sat, 09 May 2020 12:13:01 +0200
> 
> > I could understand if you'd say "use" instead of "recommend",
> > i.e. have code in Emacs, which, if a package is installed, would use
> > it.  That'd actually have the package's name in our sources, and would
> > constitute some kind of "endorsement".  But as long as we don't use
> > any of those packages, why should we care what other people like or
> > don't like?
> 
> Sorry, but you lost me there. All I'm saying is that there's a whole lot
> of terrific packages out there but which we must not recommend to users,
> although they are free software and often vastly superior to the things
> that are built into Emacs. For instance, there's a discussion going on
> about making a video showing Emacs' capabilities, but I assume we'd not
> be allowed to show Magit. That's a huge loss.

So package maintainers are supposed to want to be on ELPA so that they
could appear in a video, or in someone's message on a GNU mailing
list?  Really?

Once again, I wasn't asking whether it was okay or not to show off
Magit in a video or promote it on this list, I was asking why do we
need ELPA when MELPA is out there and has many more packages (and
always will)?

Re: ELPA policy

[David Engster]

>> From: David Engster <deng@randomsample.de>
>> Cc: monnier@iro.umontreal.ca,  emacs-devel@gnu.org
>> Date: Sat, 09 May 2020 12:13:01 +0200
>
>> 
>> > I could understand if you'd say "use" instead of "recommend",
>> > i.e. have code in Emacs, which, if a package is installed, would use
>> > it.  That'd actually have the package's name in our sources, and would
>> > constitute some kind of "endorsement".  But as long as we don't use
>> > any of those packages, why should we care what other people like or
>> > don't like?
>> 
>> Sorry, but you lost me there. All I'm saying is that there's a whole lot
>> of terrific packages out there but which we must not recommend to users,
>> although they are free software and often vastly superior to the things
>> that are built into Emacs. For instance, there's a discussion going on
>> about making a video showing Emacs' capabilities, but I assume we'd not
>> be allowed to show Magit. That's a huge loss.
>
> So package maintainers are supposed to want to be on ELPA so that they
> could appear in a video, or in someone's message on a GNU mailing
> list?  Really?

No, package maintainers usually don't care. I think this should be in
*our* interest.

> Once again, I wasn't asking whether it was okay or not to show off
> Magit in a video or promote it on this list, I was asking why do we
> need ELPA when MELPA is out there and has many more packages (and
> always will)?

Indeed, one possibility would be to simply close GNU ELPA for everything
but core or maybe-in-the-future-core packages. Is that what you're
proposing?

-David

Re: ELPA policy

[Eli Zaretskii]

> From: David Engster <deng@randomsample.de>
> Cc: monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Sat, 09 May 2020 12:29:22 +0200
> 
> > So package maintainers are supposed to want to be on ELPA so that they
> > could appear in a video, or in someone's message on a GNU mailing
> > list?  Really?
> 
> No, package maintainers usually don't care. I think this should be in
> *our* interest.

If it's in our interest, then why do we accept packages in exactly the
same form as they are on MELPA?  What do we gain from this?

> > Once again, I wasn't asking whether it was okay or not to show off
> > Magit in a video or promote it on this list, I was asking why do we
> > need ELPA when MELPA is out there and has many more packages (and
> > always will)?
> 
> Indeed, one possibility would be to simply close GNU ELPA for everything
> but core or maybe-in-the-future-core packages. Is that what you're
> proposing?

I don't have any proposals yet, because I still don't have a clear
idea of what is (or should be) the relation between Emacs and ELPA,
nor even what are the goals of adding packages to ELPA which clearly
couldn't be added to core even if we wanted.  And with Stefan's
proposal to drop the copyright assignment requirement, bringing them
into core will even harder.  But OTOH Stefan says that requesting the
assignment is harmful to the Emacs project.  So I'm utterly confused
regarding our goals in this regard.

Re: ELPA policy

[David Engster]

>> Indeed, one possibility would be to simply close GNU ELPA for everything
>> but core or maybe-in-the-future-core packages. Is that what you're
>> proposing?
>
> I don't have any proposals yet, because I still don't have a clear
> idea of what is (or should be) the relation between Emacs and ELPA,
> nor even what are the goals of adding packages to ELPA which clearly
> couldn't be added to core even if we wanted.  And with Stefan's
> proposal to drop the copyright assignment requirement, bringing them
> into core will even harder.  But OTOH Stefan says that requesting the
> assignment is harmful to the Emacs project.  So I'm utterly confused
> regarding our goals in this regard.

Even if we drop the FSF copyright requirement, we still would have
requirements that MELPA does not have, like not supporting/promoting
non-free software, GPLv3-or-later license (MELPA only requires
GPL-compatible), and probably (and unfortunately, IMHO) also the
requirement to not depend on free software that is seen critical by the
FSF, like LLVM/clang.

My hope is that if we could drop the FSF copyright requirement for
non-core packages, we at least could try to get popular and useful
packages into GNU ELPA, we could make them prominently visible and easy
to install without the user having to put anything into her dotemacs,
and we could show the power of Emacs that is visible in these packages
on our main web site, in videos, etc.

-David

Re: ELPA policy

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Indeed, one possibility would be to simply close GNU ELPA for everything
  > but core or maybe-in-the-future-core packages. Is that what you're
  > proposing?

I think that is unnecessarily strict.  It presumes that when adding a
package to GNU ELPA we always know, already, whether we will ever want
to put it in the core.  I expect there are lots of cases  where
that decision will depend on circumstances.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: ELPA policy

[Alfred M. Szmidt]

I think one of the reasons is simply that the GNU project doesn't want
to be in a situation where a third party (i.e., MELPA) is recommending
non-free software, or where the license is unclear, etc.

So ELPA was setup so that this cannot happen, since that is part
(AFAIK) of Emacs.

Re: ELPA policy

[Dmitry Gutov]

On 09.05.2020 09:22, Eli Zaretskii wrote:
> And why do I have to submit bug reports against an ELPA package for
> violation of our coding conventions?

Because it's the only way how things can work well. Once we have a bug 
tracker that most maintainers like to use, that could change.

> I'd expect the maintainer of the
> package to be asked to fix those as a precondition for accepting the
> package in GNU ELPA, or at least as a long-term plan to which the
> maintainer agreed in advance (in which case no bug report would have
> been necessary).
> 
> What am I missing here?

You're missing that the maintainers don't have any contractual 
obligations. The more conditions we add, and the less effort we want to 
expend ourselves [forwarding bug reports], the higher the odds are that 
we end up with fewer contributions/stale code/untended bug reports.

Re: ELPA policy

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 9 May 2020 18:06:05 +0300
> 
> On 09.05.2020 09:22, Eli Zaretskii wrote:
> > And why do I have to submit bug reports against an ELPA package for
> > violation of our coding conventions?
> 
> Because it's the only way how things can work well. Once we have a bug 
> tracker that most maintainers like to use, that could change.

There's no chance in the world someone will be serious about such bug
reports, given that ELPA's README basically tells that these
requirements are "recommendations".  Why would someone invest any
effort if they can avoid doing that?

> You're missing that the maintainers don't have any contractual 
> obligations. The more conditions we add, and the less effort we want to 
> expend ourselves [forwarding bug reports], the higher the odds are that 
> we end up with fewer contributions/stale code/untended bug reports.

And what's the problem with that?  These packages are available from
elsewhere anyway.

Our coding standards aren't arbitrary.  Either we think they are solid
and should apply to every code that could end up in Emacs, or we
should stop requiring adherence to them.  Anything else makes no
sense.

Re: ELPA policy

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Our coding standards aren't arbitrary.  Either we think they are solid
  > and should apply to every code that could end up in Emacs, or we
  > should stop requiring adherence to them.  Anything else makes no
  > sense.

In principle, they should apply to ELPA packages.  But it is ok, I think,
if we don't rush to bring all of them completely up to snuff.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: ELPA policy

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: dgutov@yandex.ru, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> Date: Mon, 11 May 2020 23:16:20 -0400
> 
>   > Our coding standards aren't arbitrary.  Either we think they are solid
>   > and should apply to every code that could end up in Emacs, or we
>   > should stop requiring adherence to them.  Anything else makes no
>   > sense.
> 
> In principle, they should apply to ELPA packages.  But it is ok, I think,
> if we don't rush to bring all of them completely up to snuff.

I could agree to that, but I wouldn't agree to drop all of the
requirements.  And even if we loosen some of them, it's already
problematic: the contributors of code for the core, which are
generally required to adhere to all of the requirements, could
rightfully feel they are treated unfairly, even though developing the
core benefits Emacs more than an unbundled package.

Re: ELPA policy

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Cc: philippe.vaucher@gmail.com,  stefan@marxist.se,  emacs-devel@gnu.org,
>>   joaotavora@gmail.com,  pcr910303@icloud.com,  dgutov@yandex.ru,
>>   eliz@gnu.org,  drew.adams@oracle.com
>> Date: Thu, 07 May 2020 23:38:23 -0400
>> 
>> I made the first efforts to get it integrated into GNU ELPA (and Phil
>> did the heavy lifting), so no I definitely don't think it's problematic.
>
> FWIW, I took a cursory look at dash.el in ELPA, and was surprised to
> see that its doc strings are not according to our coding conventions,
> and functions/macros that clearly aren't internal don't follow the
> naming conventions.
>
> So maybe it's time we defined the minimum requirements for packages to
> be included in ELPA?


dash.el has an extensive README with a very large number of
examples. The README is produced by combining the examples from a lisp
file which also generates a test file. I think Magnar values doc tests
more than he values Emacs doc string conventions.

Phil

streams are cool, you could stream virtually anything!

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 708 bytes --]

Hi guys,

Aren't you tired of writing the same for loops in various contexts?
I know I was, and there's a better way!

I just added a function `stream-regexp' to stream.el to stream the
search of a regexp in a buffer:


    (let ((str (stream-regexp buf ".*foo\\(.*\\)$")))
      (seq-map (lambda (match-data)
                 ...)
               str))

(I still have to document it and test it properly)

Because streams are lazy and immutable, you can reuse such a stream
forever or keep it for later, move around in a buffer or modify the
buffer and re-evaluate the stream (you'll get new match data), or map
the result, etc.

All the credit goes to Damien Cassou for giving me the idea :-)

Cheers,
Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: streams are cool, you could stream virtually anything!

[John Wiegley]

>>>>> Nicolas Petton <nicolas@petton.fr> writes:

> I just added a function `stream-regexp' to stream.el to stream the search of
> a regexp in a buffer:

This is amazing, Nicolas! I've wanted this exact construction so many times!

Do you know the memory behavior of doing it this way? That is, for 1000 hits,
how does the number of cons's allocated compare to the iterative approach?

John

Re: streams are cool, you could stream virtually anything!

[Michael Heerdegen]

John Wiegley <jwiegley@gmail.com> writes:

> Do you know the memory behavior of doing it this way? That is, for
> 1000 hits, how does the number of cons's allocated compare to the
> iterative approach?

That depends on whether you want/need to keep the matches so far.  If
you don't need to save prior matches, you can just

  (setq stream (stream-cdr stream))

after generating an element, and memory usage will be O(1).  (Note to
Nicolas: a `stream-pop' function would be nice in this context).

In that case, you can alternatively use generators (generator.el) or the
derived iterators.el.

If you keep the complete stream in memory, you get O(nbr-matches) conses
(of course).  Not much a problem IME, if you keep an eye on what you
want to keep and what you can throw away when using streams.


Regards,

Michael.

Re: streams are cool, you could stream virtually anything!

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 443 bytes --]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> That depends on whether you want/need to keep the matches so far.  If
> you don't need to save prior matches, you can just
>
>   (setq stream (stream-cdr stream))
>
> after generating an element, and memory usage will be O(1).  (Note to
> Nicolas: a `stream-pop' function would be nice in this context).

Indeed!  Could I by any chance get another pull request from you on
GitHub?

Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: streams are cool, you could stream virtually anything!

[T.V Raman]

As more of these patterns emerge, I'm even more convinced that  the
stream package belongs alongside thunk.el in core emacs.
-- 

Re: streams are cool, you could stream virtually anything!

[John Wiegley]

>>>>> T V Raman <raman@google.com> writes:

> As more of these patterns emerge, I'm even more convinced that the stream
> package belongs alongside thunk.el in core emacs.

Let's see it develop during these earlier stages on ELPA, so people can easily
update the package as it develops. Once it stabilizes, I agree we should move
it into core.

John

Re: streams are cool, you could stream virtually anything!

[T.V Raman]

SGTM.

While on the topic -- should we move things out of core into elpa --
there are a lot of elisp packages that are presently bundled -- mostly
because "there was no elpa around when they were bundled".

John Wiegley writes:
 > >>>>> T V Raman <raman@google.com> writes:
 > 
 > > As more of these patterns emerge, I'm even more convinced that the stream
 > > package belongs alongside thunk.el in core emacs.
 > 
 > Let's see it develop during these earlier stages on ELPA, so people can easily
 > update the package as it develops. Once it stabilizes, I agree we should move
 > it into core.
 > 
 > John

-- 

-- 

ELPA policy (Was: streams are cool, you could stream virtually anything!)

[John Wiegley]

>>>>> T V Raman <raman@google.com> writes:

> While on the topic -- should we move things out of core into elpa -- there
> are a lot of elisp packages that are presently bundled -- mostly because
> "there was no elpa around when they were bundled".

I would in fact like to see more packages move from core into ELPA, although
I'm not eager to do so willy nilly. Some packages now in Core are just fun, or
should be there because people are used to them being there, even if today we
wouldn't necessarily add that package to core.

If someone would be willing to go through every core package, and produce a
list of what should stay in core, what should move to ELPA, and why they think
so, that would be a great starting point.

I know from past experience that reading every core file is highly interesting
and educational, and I guarantee whoever volunteers for this will discover
many cool and interesting things. :)

John

Re: ELPA policy

[Achim Gratz]

John Wiegley writes:
> I would in fact like to see more packages move from core into ELPA, although
> I'm not eager to do so willy nilly. Some packages now in Core are just fun, or
> should be there because people are used to them being there, even if today we
> wouldn't necessarily add that package to core.

I've proposed this before, but I guess I should run it by you again (and
warn you it wasn't favorably received):

The current definition of "core" is that the sources live inside the
Emacs repository.  Several of the larger core packages like Org, CEDET
and Gnus are already largely developed outside Emacs, for instance
because the developers want to keep them compatible with different/older
Emacsen and need to be synced into Emacs sources to make them "core"
anyway.

I posit that the only thing that actually matters for something to be
considered "core" is that authors of other packages can rely on the
(stable) API provided by these packages to be available in an Emacs as
it gets distributed and no installation of further packages or software
is necessary, neither by the sysadmin nor the user.  If so, instead of
keeping the "core" sources all in Emacs, they could equally well be
living in ELPA and be pre-installed into the distribution, or installed
into the Emacs build tree as submodules or subtrees.  The most radical
(and likely most controversial) thing to do would be to move everything
to ELPA that isn't needed to bootstrap Emacs.

Doing this would need some as of yet non-existing infrastructure to get
the chosen ELPA version of each package built into the distribution, and
facilities for sysadmins and users to update (but not disable) the
"core" packages at the system level or in their private directories.
Emacs may eventually distribute some "non-core" packages also that
sysadmins or users could disable if they chose to not use them.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf Blofeld V1.15B11:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: ELPA policy

[Eli Zaretskii]

> From: Achim Gratz <Stromeko@nexgo.de>
> Date: Sun, 08 Nov 2015 18:18:26 +0100
> 
> I posit that the only thing that actually matters for something to be
> considered "core" is that authors of other packages can rely on the
> (stable) API provided by these packages to be available in an Emacs as
> it gets distributed and no installation of further packages or software
> is necessary, neither by the sysadmin nor the user.  If so, instead of
> keeping the "core" sources all in Emacs, they could equally well be
> living in ELPA and be pre-installed into the distribution, or installed
> into the Emacs build tree as submodules or subtrees.

IMO, no serious move such as this one should be argued for, let alone
attempted, without some minimal analysis of advantages and
disadvantages.  In particular, such an analysis cannot be limited to
the POV of maintainers of packages tat are currently not bundled, it
must first and foremost look at this from the POV of the Emacs
maintenance, definitely if the argument is to leave in the Emacs
repository only what's needed for bootstrap.

And any change in maintenance routine, small or large, should have
enough advantages to justify the energy that will certainly go into
the move itself and into cleaning up the resulting fallout.

I don't see how we can seriously discuss such suggestions when they
are not accompanied by anything like the analysis they need.

> The most radical (and likely most controversial) thing to do would
> be to move everything to ELPA that isn't needed to bootstrap Emacs.

Most such packages don't have any active maintainers, i.e. they are
maintained by the "FSF", which means us the core developers.  IMO, it
makes very little sense to spread the stuff we maintain between 2
separate repositories, because all this does is add overhead and
complexity without any clear benefits that I could see.

Another important aspect that this suggestion seems to overlook is
that Emacs packages rely on others not only via APIs, but also by
inheritance, like all the modes that derive from Text mode etc.

> Doing this would need some as of yet non-existing infrastructure to get
> the chosen ELPA version of each package built into the distribution, and
> facilities for sysadmins and users to update (but not disable) the
> "core" packages at the system level or in their private directories.

Yes, and the effort this will require is squarely in the disadvantages
camp.

Let me turn the table and ask: Are there any _advantages_ in moving
stuff like Dired, CC Mode, Shell Mode, Speedbar, and ps-print, to name
just a random few?

Re: ELPA policy

[Aaron Ecay]

Hi Eli,

2015ko azaroak 8an, Eli Zaretskii-ek idatzi zuen:

[...]

> 
> Let me turn the table and ask: Are there any _advantages_ in moving
> stuff like Dired, CC Mode, Shell Mode, Speedbar, and ps-print, to name
> just a random few?

Imagine someone implements an awesome new feature for dired.  Emacs
users the world over are amazed by this, and fill their blogs, twitter,
etc. with the news.  If dired is an ELPA package, everyone who hears
this news can get the new feature in their emacs instantly by upgrading
their ELPA packages.  No need to wait N months for a new release of
emacs, or compile a non-release version of emacs from git.

Whether this counts as an advantage or not is complicated, and partially
depends on one’s point of view.  It would spell the end of an era where
each emacs release’s features are explicated through a year of excited
blogposts (for example
<http://endlessparentheses.com/tags-expanded.html#emacs-25>).  Users
will be able to choose a new model where features trickle in rather than
coming in sudden major chunks.  This might make it more complicated to
advocate adoption of new emacs versions.  But I don’t think so actually:
we regularly get big features at the language level too, which are very
exciting (for emacs 24 lexical binding, now dynamic modules and
xwidgets, and one dares to hope for the concurrency branch in emacs26).

I also think that discussion around this topic is distorted by a
long-tailed distribution.  Most people probably have in mind the big
emacs packages with dynamic developer communities and independent
release cycles.  Org, Gnus, CEDET, and maybe a few others.  On the other
hand, it seems to me that you have in mind (in addition to these) all
the tiny packages that see very few commits and perhaps no new features
(in the extreme case take kermit.el, which has seen no substantive
changes since at least 1992, but still gets its copyright header
lovingly updated every year).  I don’t know how to reconcile these
viewpoints, or even if it’s necessary.  Just something to consider.

Speaking for myself, my primary interest in emacs development is working
on org-mode, and I heavily use org as well as third-party packages for
python (elpy), clojure (cider), R (ESS), and a few other random things.
I build emacs from git every month or so in order to pick up little
quality of life improvements, like with-eval-afer-load, some of the
subr-x stuff, seq.el, the overhauled package menu, etc.  But I have to
make sure to keep a couple months’ worth of old emacsen around, in case
my monthly pull happens to land on a buggy commit – one that causes
regular segfaults, which I have had happen more than once.  I use emacs
for work so I don’t have the luxury of just going without for a few days
until the problem is fixed.  I could go back to a previous released
version of course, but then my init.el breaks because it relies on new
features since the release.

I’d be much happier running released versions of emacs for stability, if
it were easier to pull in new versions of core elisp libraries as they
are developed.  I suspect there are some others out there like me, and
many more who don’t currently build emacs from source for whatever reason
but would enjoy more regular access to new features in core elisp
libraries.

My 2 cents FWIW,
Aaron

PS I also have the impression that life would be easier for org’s
maintainers under a system that freed them from merging code back and
forth from org’s repo to emacs.  This would have indirect positive
effects on me as I work on org development.  However, I’m sure the
maintainers themselves – and maintainers of other projects in the same
situation like CEDET and Gnus – can speak to the question better than I
can.

-- 
Aaron Ecay

Re: ELPA policy

[Eli Zaretskii]

> From: Aaron Ecay <aaronecay@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Sun, 08 Nov 2015 20:52:12 +0000
> 
> Imagine someone implements an awesome new feature for dired.  Emacs
> users the world over are amazed by this, and fill their blogs, twitter,
> etc. with the news.  If dired is an ELPA package, everyone who hears
> this news can get the new feature in their emacs instantly by upgrading
> their ELPA packages.  No need to wait N months for a new release of
> emacs, or compile a non-release version of emacs from git.

How is this different when Dired is in the Emacs repository?  The
Emacs repository is a public one, so anyone and everyone can get the
latest version from there and use it, if they want.

> I also think that discussion around this topic is distorted by a
> long-tailed distribution.  Most people probably have in mind the big
> emacs packages with dynamic developer communities and independent
> release cycles.  Org, Gnus, CEDET, and maybe a few others.  On the other
> hand, it seems to me that you have in mind (in addition to these) all
> the tiny packages that see very few commits and perhaps no new features
> (in the extreme case take kermit.el, which has seen no substantive
> changes since at least 1992, but still gets its copyright header
> lovingly updated every year).  I don’t know how to reconcile these
> viewpoints, or even if it’s necessary.  Just something to consider.

The suggestion was to move _all_ of them, except the few that are
needed for bootstrap, out of the Emacs repository.  Most of the
packages in that category are neither like Org nor like kermit.  They
are relatively small, but get quite a significant number of changes.

> Speaking for myself, my primary interest in emacs development is working
> on org-mode, and I heavily use org as well as third-party packages for
> python (elpy), clojure (cider), R (ESS), and a few other random things.
> I build emacs from git every month or so in order to pick up little
> quality of life improvements, like with-eval-afer-load, some of the
> subr-x stuff, seq.el, the overhauled package menu, etc.  But I have to
> make sure to keep a couple months’ worth of old emacsen around, in case
> my monthly pull happens to land on a buggy commit – one that causes
> regular segfaults, which I have had happen more than once.  I use emacs
> for work so I don’t have the luxury of just going without for a few days
> until the problem is fixed.  I could go back to a previous released
> version of course, but then my init.el breaks because it relies on new
> features since the release.

I don't see how the issue at hand can affect you, then.  Your usage is
very similar to mine, FWIW.

Re: ELPA policy

[Dmitry Gutov]

On 11/09/2015 05:42 AM, Eli Zaretskii wrote:

>> Imagine someone implements an awesome new feature for dired.  Emacs
>> users the world over are amazed by this, and fill their blogs, twitter,
>> etc. with the news.  If dired is an ELPA package, everyone who hears
>> this news can get the new feature in their emacs instantly by upgrading
>> their ELPA packages.  No need to wait N months for a new release of
>> emacs, or compile a non-release version of emacs from git.
>
> How is this different when Dired is in the Emacs repository?  The
> Emacs repository is a public one, so anyone and everyone can get the
> latest version from there and use it, if they want.

a) That's a more involved endeavor than installing a package from ELPA. 
And then you don't get the same conveniences, such as automatic updates.

b) There's a much higher probability that Dired depends on something 
only the current development version of Emacs has. ELPA packages declare 
their version requirements explicitly, and try not to break 
compatibility with earlier versions without sufficient reasons.

> The suggestion was to move _all_ of them, except the few that are
> needed for bootstrap, out of the Emacs repository.  Most of the
> packages in that category are neither like Org nor like kermit.  They
> are relatively small, but get quite a significant number of changes.

There were different suggestions, with different degrees between "let's 
move Org and Gnus out" and "let's move everything out".

Re: ELPA policy

[Eli Zaretskii]

> Cc: Stromeko@nexgo.de, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 9 Nov 2015 05:51:16 +0200
> 
> On 11/09/2015 05:42 AM, Eli Zaretskii wrote:
> 
> >> Imagine someone implements an awesome new feature for dired.  Emacs
> >> users the world over are amazed by this, and fill their blogs, twitter,
> >> etc. with the news.  If dired is an ELPA package, everyone who hears
> >> this news can get the new feature in their emacs instantly by upgrading
> >> their ELPA packages.  No need to wait N months for a new release of
> >> emacs, or compile a non-release version of emacs from git.
> >
> > How is this different when Dired is in the Emacs repository?  The
> > Emacs repository is a public one, so anyone and everyone can get the
> > latest version from there and use it, if they want.
> 
> a) That's a more involved endeavor than installing a package from ELPA. 
> And then you don't get the same conveniences, such as automatic updates.

Nothing prevents us from making the same arrangements in the Emacs
repository as we have in ELPA, and then users could add the Emacs
repository to their list of sites that package.el knows about.

Besides, most core packages don't need any elaborate setup, you just
drop them in and restart Emacs.

> b) There's a much higher probability that Dired depends on something 
> only the current development version of Emacs has. ELPA packages declare 
> their version requirements explicitly, and try not to break 
> compatibility with earlier versions without sufficient reasons.

How will that change if we move them outside Emacs?  It's not like
there's some magic at ELPA that just putting a package there magically
resolves all these issues.  And if someone will have to work on that,
she can do that without moving the package aside.

> > The suggestion was to move _all_ of them, except the few that are
> > needed for bootstrap, out of the Emacs repository.  Most of the
> > packages in that category are neither like Org nor like kermit.  They
> > are relatively small, but get quite a significant number of changes.
> 
> There were different suggestions, with different degrees between "let's 
> move Org and Gnus out" and "let's move everything out".

I didn't see any.

Re: ELPA policy

[Dmitry Gutov]

On 11/09/2015 06:05 PM, Eli Zaretskii wrote:

> Nothing prevents us from making the same arrangements in the Emacs
> repository as we have in ELPA, and then users could add the Emacs
> repository to their list of sites that package.el knows about.

Create yet another package archive? With a separate publishing mechanism?

> Besides, most core packages don't need any elaborate setup, you just
> drop them in and restart Emacs.

"Just drop them in" doesn't address the issue of package updates, not in 
the slightest.

> How will that change if we move them outside Emacs?  It's not like
> there's some magic at ELPA that just putting a package there magically
> resolves all these issues.  And if someone will have to work on that,
> she can do that without moving the package aside.

They will be forced to declare the required Emacs version, for one. And 
abide by it.

But yes, that's also possible to do inside the Emacs repo.

>> There were different suggestions, with different degrees between "let's
>> move Org and Gnus out" and "let's move everything out".
>
> I didn't see any.

All right, then. Let me make one right now: let's move out Org, Gnus and 
CEDET. And consider doing that with any other big codebase that's also 
maintained separately (Calc comes to mind).

There have been objections to doing that for each of these packages, but 
there you go. Suggestion made.

Re: ELPA policy

[Eli Zaretskii]

> Cc: aaronecay@gmail.com, Stromeko@nexgo.de, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 9 Nov 2015 18:15:28 +0200
> 
> On 11/09/2015 06:05 PM, Eli Zaretskii wrote:
> 
> > Nothing prevents us from making the same arrangements in the Emacs
> > repository as we have in ELPA, and then users could add the Emacs
> > repository to their list of sites that package.el knows about.
> 
> Create yet another package archive? With a separate publishing mechanism?

Why not?  There are several already, no?

> > Besides, most core packages don't need any elaborate setup, you just
> > drop them in and restart Emacs.
> 
> "Just drop them in" doesn't address the issue of package updates, not in 
> the slightest.

It does, for most of the core.  That's what you do each time you say
"git pull", I believe.

> >> There were different suggestions, with different degrees between "let's
> >> move Org and Gnus out" and "let's move everything out".
> >
> > I didn't see any.
> 
> All right, then. Let me make one right now: let's move out Org, Gnus and 
> CEDET. And consider doing that with any other big codebase that's also 
> maintained separately (Calc comes to mind).
> 
> There have been objections to doing that for each of these packages, but 
> there you go. Suggestion made.

It was made before -- that's the "let's move Org and Gnus out"
variant, to which I said I could easily agree.  And then there was the
"move everything" one, to which I object for the reasons stated.  What
I meant was that there was no 3rd variant, AFAIR.

Re: ELPA policy

[Dmitry Gutov]

On 11/09/2015 06:20 PM, Eli Zaretskii wrote:

>> Create yet another package archive? With a separate publishing mechanism?
>
> Why not?  There are several already, no?

So far, it seems we're short of manpower to manage even the current one. 
To manage the infrastructure, improve the scripts, etc.

> It does, for most of the core.  That's what you do each time you say
> "git pull", I believe.

I cannot "git pull" a new version of Dired into my stable installation 
of Emacs 24.5.

> It was made before -- that's the "let's move Org and Gnus out"
> variant, to which I said I could easily agree.  And then there was the
> "move everything" one, to which I object for the reasons stated.  What
> I meant was that there was no 3rd variant, AFAIR.

Well, I enumerated 4 packages instead of 2, didn't I? :)

I agree that "move everything" is a non-starter (or very hard, at 
least). So it seems more like a red herring, there's no real use arguing 
against it.

Re: ELPA policy

[Eli Zaretskii]

> Cc: aaronecay@gmail.com, Stromeko@nexgo.de, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 9 Nov 2015 18:26:58 +0200
> 
> On 11/09/2015 06:20 PM, Eli Zaretskii wrote:
> 
> >> Create yet another package archive? With a separate publishing mechanism?
> >
> > Why not?  There are several already, no?
> 
> So far, it seems we're short of manpower to manage even the current one. 
> To manage the infrastructure, improve the scripts, etc.

Is it really such a significant effort?  I wasn't aware of that.

> > It does, for most of the core.  That's what you do each time you say
> > "git pull", I believe.
> 
> I cannot "git pull" a new version of Dired into my stable installation 
> of Emacs 24.5.

Of course you can: just copy the file over.

> > It was made before -- that's the "let's move Org and Gnus out"
> > variant, to which I said I could easily agree.  And then there was the
> > "move everything" one, to which I object for the reasons stated.  What
> > I meant was that there was no 3rd variant, AFAIR.
> 
> Well, I enumerated 4 packages instead of 2, didn't I? :)

We could probably find a few more.  MH-E, for example.  (Of course,
their developers will have to agree to move.)

> I agree that "move everything" is a non-starter (or very hard, at 
> least). So it seems more like a red herring, there's no real use arguing 
> against it.

Agreed.

Re: ELPA policy

[Dmitry Gutov]

On 11/09/2015 06:44 PM, Eli Zaretskii wrote:

>> So far, it seems we're short of manpower to manage even the current one.
>> To manage the infrastructure, improve the scripts, etc.
>
> Is it really such a significant effort?  I wasn't aware of that.

There are things to be done, and just a couple months ago nobody was 
doing them. I see some movement now, e.g. the last commit by Thomas 
Fitzsimmons.

>> I cannot "git pull" a new version of Dired into my stable installation
>> of Emacs 24.5.
>
> Of course you can: just copy the file over.

Automatic update is a really handy feature when you have 50+ packages 
installed. See some user statistics at 
https://www.reddit.com/r/emacs/comments/3s4k03/how_many_packages_do_you_use/

> We could probably find a few more.  MH-E, for example.  (Of course,
> their developers will have to agree to move.)

Yeah, I bet the majority of our users are unaware of what MH-E is.

Re: ELPA policy

[Artur Malabarba]

Eli Zaretskii <eliz@gnu.org> writes:

>> > Nothing prevents us from making the same arrangements in the Emacs
>> > repository as we have in ELPA, and then users could add the Emacs
>> > repository to their list of sites that package.el knows about.
>> 
>> Create yet another package archive? With a separate publishing mechanism?
>
> Why not?  There are several already, no?

This sounds already quite similar to the system that Fabian implemented
a couple months ago (which I hear isn't quite perfect yet, but should be
almost there).

With this system, the daily script that Gelpa runs would copy some
packages straight from Emacs core. This way you could very well
configure Gelpa to always offer the lastest Dired (or maybe only when we
bump a version number, but that's a detail).

Re: ELPA policy

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> It was made before -- that's the "let's move Org and Gnus out" variant, to
> which I said I could easily agree. And then there was the "move everything"
> one, to which I object for the reasons stated. What I meant was that there
> was no 3rd variant, AFAIR.

Wow, a lot of discussion about ELPA policy, this is great. We definitely have
an opportunity here to bring clarity to an area that is on people's minds.

I agree with a lot of what I read, although it was a too spread out for me to
add specific quotes here. Let me just summarize a possible path forward:

 1. Things that are maintained by the core Emacs developers should be in core,
    and in the Emacs.git repository. This makes it easy for them to access and
    build, search history, read emacs-diffs, etc.

 2. Things that are maintained outside of Emacs, but contributed back for
    inclusion, should be in ELPA, and in the Elpa.git repository. This
    includes Gnus, Org, CEDET, and a few more. (We don't want to go crazy,
    so let's start with the big ones).

 3. There should be a defined subset of packages that get copied from ELPA
    into the Emacs tarball during release, and an easy way to manage this list
    for the core developers. That way, certain packages like seq.el and
    stream.el can feel like "core" packages to users, when they are really
    "external" packages from the point of view of the core developers.

 4. Everything else in ELPA is Internet-installation based, as it is today.

I would very much like for ELPA to be a way to:

  - give the core developers a smaller surface area to focus on;

  - allow ELPA package maintainers to "have their package in Emacs", while
    retaining nimble in their own development process and in delivering
    updates to users;

  - allow the core maintainers to decide what exactly is going into "Emacs":
    For example, I wouldn't want to pull Org releases into the distribution
    from a submodule; I really do want a version of that source code to be in
    ELPA, so we can separately patch if there are last minute problems.

ELPA should thus benefit core developers by reducing load, and benefit package
maintainers by being an easier platform for their contributions and their
users. If it causes extra work for anyone, that's something I'd want to
change.

There are three actions I'd like to take from here:

  a. That we clearly document an ELPA policy we all agree on;

  b. That we move "external" packages from core into ELPA, starting with the
     really big ones;

  c. That we assess any points of friction after doing so, and adjust as
     necessary.

John

Re: ELPA policy

[Achim Gratz]

John Wiegley writes:
>  3. There should be a defined subset of packages that get copied from ELPA
>     into the Emacs tarball during release, and an easy way to manage this list
>     for the core developers. That way, certain packages like seq.el and
>     stream.el can feel like "core" packages to users, when they are really
>     "external" packages from the point of view of the core developers.

They could (and probably should) effectively be in the build environment
as well.  In other words, I would try to not pull them into the release
tarball at the last possible moment.  One thing that would make
difficult is to test interactions between such "external core" packages
and here I have to agree with Eli that this has potential for
degradation of quality.

>   - allow the core maintainers to decide what exactly is going into "Emacs":
>     For example, I wouldn't want to pull Org releases into the distribution
>     from a submodule; I really do want a version of that source code to be in
>     ELPA, so we can separately patch if there are last minute problems.

You'd do that with branches if it really becomes necessary.  I don't
really see why submodules could not be used, except for the extra rope
they give you to hang yourself with.  Any other solution is going to
have the same basic complexity beneath and the potential to not work on
some platform or other.

> ELPA should thus benefit core developers by reducing load, and benefit package
> maintainers by being an easier platform for their contributions and their
> users. If it causes extra work for anyone, that's something I'd want to
> change.

I maintain that ELPA should benefit Emacs users first.

>   b. That we move "external" packages from core into ELPA, starting with the
>      really big ones;

...after the necessary changes to Emacs' build system and package.el are
architected and implemented.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

Re: ELPA policy

[John Wiegley]

>>>>> Achim Gratz <Stromeko@nexgo.de> writes:

> They could (and probably should) effectively be in the build environment as
> well. In other words, I would try to not pull them into the release tarball
> at the last possible moment. One thing that would make difficult is to test
> interactions between such "external core" packages and here I have to agree
> with Eli that this has potential for degradation of quality.

Could this be solved by branching ELPA leading up to a release -- this being
the "frozen ELPA" that we do integration against before cutting the tarball --
while cherry-picking fixes in from master? I think the way we proceed here
would be similar to what we do for Emacs itself.

> You'd do that with branches if it really becomes necessary. I don't really
> see why submodules could not be used, except for the extra rope they give
> you to hang yourself with. Any other solution is going to have the same
> basic complexity beneath and the potential to not work on some platform or
> other.

A submodule doesn't allow us to have "Org + some-patch-not-yet-in-Org".

> I maintain that ELPA should benefit Emacs users first.

I'm intrigued; could you clarify this a bit more? I saw what you wrote in
response to Eli, but that didn't really help. Specifically, do you have a
scenario in mind that is better for users and worse for developers?

>> b. That we move "external" packages from core into ELPA, starting with the
>> really big ones;

> ...after the necessary changes to Emacs' build system and package.el are
> architected and implemented.

Agreed.

John

Re: ELPA policy

[Achim Gratz]

John Wiegley writes:
>> You'd do that with branches if it really becomes necessary. I don't really
>> see why submodules could not be used, except for the extra rope they give
>> you to hang yourself with. Any other solution is going to have the same
>> basic complexity beneath and the potential to not work on some platform or
>> other.
>
> A submodule doesn't allow us to have "Org + some-patch-not-yet-in-Org".

You'd branch on ELPA to something like emacs-25.0.50, patch away and
point the submodule there if you really can't wait for the patch to be
committed in Org upstream and mirrored back to ELPA.

>> I maintain that ELPA should benefit Emacs users first.
>
> I'm intrigued; could you clarify this a bit more? I saw what you wrote in
> response to Eli, but that didn't really help. Specifically, do you have a
> scenario in mind that is better for users and worse for developers?

It was a long day and I'm afraid I'd not be too coherent, so I'll punt
for now.  But in short, look at CPAN, CRAN and CTAN for NIH examples of
what ELPA might possibly become.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Terratec KOMPLEXER:
http://Synth.Stromeko.net/Downloads.html#KomplexerWaves

Re: ELPA policy

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I agree with a lot of what I read, although it was a too spread out for me to
  > add specific quotes here. Let me just summarize a possible path forward:

This plan seems good, overall, But in the specific case of stream.el
and seq.el, shouldn't they be maintained in the core?  I don't see
stream.el in the sources I fetched, but seq.el is small, in emacs-lisp,
and it says Maintainer: emacs-devel.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: ELPA policy

[John Wiegley]

>>>>> Richard Stallman <rms@gnu.org> writes:

> This plan seems good, overall, But in the specific case of stream.el and
> seq.el, shouldn't they be maintained in the core? I don't see stream.el in
> the sources I fetched, but seq.el is small, in emacs-lisp, and it says
> Maintainer: emacs-devel.

If that's the case, then yes, it should be in core. I'm personally a bit
surprised that a library of this nature is in ELPA, unless it was being
provided and actively maintained by a non-core developer.

John

Re: ELPA policy

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 748 bytes --]

On 9 Nov 2015 10:06 pm, "John Wiegley" <jwiegley@gmail.com> wrote:
>
> >>>>> Richard Stallman <rms@gnu.org> writes:
>
> > This plan seems good, overall, But in the specific case of stream.el and
> > seq.el, shouldn't they be maintained in the core? I don't see stream.el
in
> > the sources I fetched, but seq.el is small, in emacs-lisp, and it says
> > Maintainer: emacs-devel.
>
> If that's the case, then yes, it should be in core. I'm personally a bit
> surprised that a library of this nature is in ELPA, unless it was being
> provided and actively maintained by a non-core developer.

Seq.el was put on Gelpa so that it can be used by emacsen < 25. I'm very
grateful for that as I use it as a dependency in quite a few of my packages
already.

[-- Attachment #2: Type: text/html, Size: 1009 bytes --]

Re: ELPA policy

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Seq.el was put on Gelpa so that it can be used by emacsen < 25.

It is fine to have it there, but this suggests we should maintain
it inside Emacs.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: ELPA policy

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 419 bytes --]

Richard Stallman <rms@gnu.org> writes:

> It is fine to have it there, but this suggests we should maintain
> it inside Emacs.

I'm not sure I understand what you mean Richard.  seq.el is maintained
inside Emacs (in lisp/emacs-lisp/seq.el and
test/automated/seq-tests.el).  The version published in ELPA is there
for backward-compatibility, and does not have all the features (some
features do require Emacs 25).

Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: ELPA policy

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > It is fine to have it there, but this suggests we should maintain
  > > it inside Emacs.

  > I'm not sure I understand what you mean Richard.

I mean that we should maintain the file inside Emacs, 
not inside ELPA.

  >   seq.el is maintained
  > inside Emacs (in lisp/emacs-lisp/seq.el and
  > test/automated/seq-tests.el).

That means people are already doing what I think we should do.
I didn't have this information when I wrote my previous message.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: ELPA policy

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 538 bytes --]

John Wiegley <jwiegley@gmail.com> writes:

> If that's the case, then yes, it should be in core. I'm personally a bit
> surprised that a library of this nature is in ELPA,

In the case of seq.el, it's for backward-compatibility.  It made it
possible for instance for CIDER devs to start using it while Emacs 25 is
still in progress.

> unless it was being
> provided and actively maintained by a non-core developer.

IIRC, Stefan thought that having stream.el in the ELPA meant more sense
at least until people start using it more.

Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: ELPA policy

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 424 bytes --]

Richard Stallman <rms@gnu.org> writes:

> This plan seems good, overall, But in the specific case of stream.el
> and seq.el, shouldn't they be maintained in the core?  I don't see
> stream.el in the sources I fetched, but seq.el is small, in emacs-lisp,
> and it says Maintainer: emacs-devel.

stream.el is currently in ELPA, which can make some sense, but regarding
seq.el, I think it should really stay in the core.

Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: ELPA policy

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 1047 bytes --]

John Wiegley <jwiegley@gmail.com> writes:

>  3. There should be a defined subset of packages that get copied from ELPA
>     into the Emacs tarball during release, and an easy way to manage this list
>     for the core developers. That way, certain packages like seq.el and
>     stream.el can feel like "core" packages to users, when they are really
>     "external" packages from the point of view of the core developers.

I only added seq.el to ELPA for backward compatibility with Emacs 24.5.
The development of seq.el (and map.el for that matter) happen only in
Emacs trunk.

Moving it out of lisp/emacs-lisp/, and only having it copied in the
release tarballs would be a step backward IMHO, it would make it hard to
use it in core packages, and would basically mean demoting the libraries
to external ones while I'm really working on seq.el to provide a good,
consistent and comprehensive *built-in* sequence library to
Emacs-Lisp. That's also why all the functions are documented in the
Sequences chapter of the Elisp documentation.

Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: ELPA policy

[Aaron Ecay]

Hi John,

Thanks for setting this out very concretely.

2015ko azaroak 9an, John Wiegley-ek idatzi zuen:
> 
>>>>>> Eli Zaretskii <eliz@gnu.org> writes:
> 
>> It was made before -- that's the "let's move Org and Gnus out" variant, to
>> which I said I could easily agree. And then there was the "move everything"
>> one, to which I object for the reasons stated. What I meant was that there
>> was no 3rd variant, AFAIR.
> 
> Wow, a lot of discussion about ELPA policy, this is great. We definitely have
> an opportunity here to bring clarity to an area that is on people's minds.
> 
> I agree with a lot of what I read, although it was a too spread out for me to
> add specific quotes here. Let me just summarize a possible path forward:
> 
>  1. Things that are maintained by the core Emacs developers should be in core,
>     and in the Emacs.git repository. This makes it easy for them to access and
>     build, search history, read emacs-diffs, etc.
> 
>  2. Things that are maintained outside of Emacs, but contributed back for
>     inclusion, should be in ELPA, and in the Elpa.git repository. This
>     includes Gnus, Org, CEDET, and a few more. (We don't want to go crazy,
>     so let's start with the big ones).
> 
>  3. There should be a defined subset of packages that get copied from ELPA
>     into the Emacs tarball during release, and an easy way to manage this list
>     for the core developers. That way, certain packages like seq.el and
>     stream.el can feel like "core" packages to users, when they are really
>     "external" packages from the point of view of the core developers.
> 
>  4. Everything else in ELPA is Internet-installation based, as it is
>     today.

The one thing that I see missing from this list is a way to take
packages that are developed in emacs.git and distribute them on an
ELPA.  (Either GNU ELPA, or a (g)new ELPA as was suggested by Eli.)
This would be the inverse of your 3: packages that feel like core
packages to emacs devs, but feel like external packages to users (at
least in the sense of receiving updates outside of the emacs release
cycle).

I’m not sure if that omission was intentional or not.  I hope it wasn’t:
this model of distribution has been chosen for several important
“low-level” packages, like seq, cl-lib, and let-alist.

If this addition is desired then:

[...]

> 
> There are three actions I'd like to take from here:
> 
>   a. That we clearly document an ELPA policy we all agree on;
> 
>   b. That we move "external" packages from core into ELPA, starting with the
>      really big ones;
> 
>   c. That we assess any points of friction after doing so, and adjust as
>      necessary.

This list needs an additional item, namely the development of the
script that publishes elisp from emacs.git to an ELPA.  I think people
have said that Fabian is already working on this.

Thanks,

-- 
Aaron Ecay

Re: ELPA policy

[John Wiegley]

>>>>> Aaron Ecay <aaronecay@gmail.com> writes:

> I’m not sure if that omission was intentional or not. I hope it wasn’t: this
> model of distribution has been chosen for several important “low-level”
> packages, like seq, cl-lib, and let-alist.

[...]

> This list needs an additional item, namely the development of the script
> that publishes elisp from emacs.git to an ELPA. I think people have said
> that Fabian is already working on this.

It was an unintentional omission, and I agree that we should support that
usage pattern as well. So let it be 4 modes of looking at "Emacs packages",
then.

John

Re: ELPA policy

[Stephen Leake]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> Eli Zaretskii <eliz@gnu.org> writes:
>
>> It was made before -- that's the "let's move Org and Gnus out" variant, to
>> which I said I could easily agree. And then there was the "move everything"
>> one, to which I object for the reasons stated. What I meant was that there
>> was no 3rd variant, AFAIR.
>
> Wow, a lot of discussion about ELPA policy, this is great. We definitely have
> an opportunity here to bring clarity to an area that is on people's minds.
>
> I agree with a lot of what I read, although it was a too spread out for me to
> add specific quotes here. Let me just summarize a possible path forward:
>
>  1. Things that are maintained by the core Emacs developers should be in core,
>     and in the Emacs.git repository. This makes it easy for them to access and
>     build, search history, read emacs-diffs, etc.
>
>  2. Things that are maintained outside of Emacs, but contributed back for
>     inclusion, should be in ELPA, and in the Elpa.git repository. This
>     includes Gnus, Org, CEDET, and a few more. (We don't want to go crazy,
>     so let's start with the big ones).
>
>  3. There should be a defined subset of packages that get copied from ELPA
>     into the Emacs tarball during release, and an easy way to manage this list
>     for the core developers. That way, certain packages like seq.el and
>     stream.el can feel like "core" packages to users, when they are really
>     "external" packages from the point of view of the core developers.

ELPA packages that other core code depends on (like CEDET; xref uses it
- called "core ELPA packages" hereinafter) have to be in every
developer's build environment everyday; the other core code has to see
the current version of the package, and it has to be the same for every
developer.

If core ELPA packages are in the ELPA git repository, you would copy
some subtrees of the ELPA git workspace into the Emacs git workspace.
People have looked into doing this, but it gets messy and confusing.
"git fetch" is dealing with two upstreams for one workspace. There has
been talk about git submodules, etc, but nothing concrete.

The alternative, which is being implemented by the :core external
package feature in Gnu ELPA, and has been shown to work (but has some
details to work out), is to keep core ELPA packages in the emacs
git repository. Then they are part of the developer's build environment
using the current git workflow, and they are included in both the emacs
tarball and the ELPA package server with the current release workflows.


There are two rationales for moving a package to ELPA:

1) Allow a release schedule independent of the core Emacs release
   schedule.

2) Reduce the size of the Emacs git workspace, which simplifies managing
   it.

But a _core_ ELPA package must remain in the Emacs git workspace, so the
only rationale for moving a core package to ELPA is for an independent
release schedule.

> There are three actions I'd like to take from here:
>
>   a. That we clearly document an ELPA policy we all agree on;

In admin/notes/elpa ?

>   b. That we move "external" packages from core into ELPA, starting with the
>      really big ones;

It's not clear that "really big" implies "independent release schedule".
"externally managed" implies that.

"not used by other core code" is also a good reason to move a package to
ELPA. 

>   c. That we assess any points of friction after doing so, and adjust as
>      necessary.

If we use the approach of keeping core ELPA packages in the Emacs git
repository, there is _zero_ friction for the current core Emacs
developers; the only change is in the ELPA config scripts. 

-- 
-- Stephe

Re: ELPA policy

[John Wiegley]

>>>>> Stephen Leake <stephen_leake@stephe-leake.org> writes:

> ELPA packages that other core code depends on (like CEDET; xref uses it -
> called "core ELPA packages" hereinafter) have to be in every developer's
> build environment everyday; the other core code has to see the current
> version of the package, and it has to be the same for every developer.
> 
> If core ELPA packages are in the ELPA git repository, you would copy some
> subtrees of the ELPA git workspace into the Emacs git workspace. People have
> looked into doing this, but it gets messy and confusing. "git fetch" is
> dealing with two upstreams for one workspace. There has been talk about git
> submodules, etc, but nothing concrete.

Elpa.git should be a submodule referenced from within Emacs.git (under "elpa").
This allows us to expressly state which "version" of Elpa is expected to work
with the current Emacs.git.

Large packages like CEDET should move outside of Emacs.git and into Elpa.git.
If xref.el depends on CEDET, it would move to Elpa.git as well.

> If we use the approach of keeping core ELPA packages in the Emacs git
> repository, there is _zero_ friction for the current core Emacs developers;
> the only change is in the ELPA config scripts.

I do not want this code replication. We will promote ELPA as an active
development space alongside core Emacs. If we use a submodule, and improve our
tooling slightly, it should become a seamless thing.

John

Re: ELPA policy

[Dmitry Gutov]

On 11/10/2015 08:28 PM, John Wiegley wrote:

> If xref.el depends on CEDET, it would move to Elpa.git as well.

I can cut out xref.el's dependency on CEDET easily enough.

Moving it into ELPA would be kinda problematic, because guess what M-. 
and M-, and bound to by default now.

Re: ELPA policy

[John Wiegley]

>>>>> Dmitry Gutov <dgutov@yandex.ru> writes:

> I can cut out xref.el's dependency on CEDET easily enough.

Great, let's do that. I'd prefer to have the functionality of xref.el in core.

Thanks,
  John

Re: ELPA policy

[David Engster]

John Wiegley writes:
>>>>>> Dmitry Gutov <dgutov@yandex.ru> writes:
>
>> I can cut out xref.el's dependency on CEDET easily enough.
>
> Great, let's do that. I'd prefer to have the functionality of xref.el in core.

So that means xref won't support CEDET out of the box? What is gained by
that, exactly?

-David

Re: ELPA policy

[John Wiegley]

>>>>> David Engster <deng@randomsample.de> writes:

> So that means xref won't support CEDET out of the box? What is gained by
> that, exactly?

It could always support it with customization hooks.

John

Re: ELPA policy

[Dmitry Gutov]

On 11/10/2015 08:44 PM, David Engster wrote:

> So that means xref won't support CEDET out of the box? What is gained by
> that, exactly?

It doesn't support CEDET even now. Nobody wrote the Semantic 
implementation of a xref backend, and it might be tool late now for the 
25.1 release. If CEDET were moved to ELPA, however, you can add that 
support at your leisure later.

And that's an orthogonal issue. xref has no need to support CEDET: CEDET 
should support xref (the support code will be in CEDET's tree).

Re: ELPA policy

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Date: Tue, 10 Nov 2015 10:35:13 -0800
> Cc: aaronecay@gmail.com, Eli Zaretskii <eliz@gnu.org>, Stromeko@nexgo.de,
> 	Stephen Leake <stephen_leake@stephe-leake.org>, emacs-devel@gnu.org
> 
> >>>>> Dmitry Gutov <dgutov@yandex.ru> writes:
> 
> > I can cut out xref.el's dependency on CEDET easily enough.
> 
> Great, let's do that.

Sounds like a step backward to me.  CEDET allows xref to be more
versatile and support more back-ends.

Re: ELPA policy

[Dmitry Gutov]

On 11/10/2015 08:35 PM, John Wiegley wrote:
>>>>>> Dmitry Gutov <dgutov@yandex.ru> writes:
>
>> I can cut out xref.el's dependency on CEDET easily enough.
>
> Great, let's do that. I'd prefer to have the functionality of xref.el in core.

Let me know when it becomes a blocker, and I'll do it in the simplest 
way. But I'd like to defer that for now, because it comes with making 
choices (see the other messages in this thread), and I'm running low on 
decision-making juice these days.

Re: ELPA policy

[David Engster]

Dmitry Gutov writes:
> On 11/10/2015 08:28 PM, John Wiegley wrote:
>
>> If xref.el depends on CEDET, it would move to Elpa.git as well.
>
> I can cut out xref.el's dependency on CEDET easily enough.

Which illustrates nicely the downside of moving packages to ELPA.

-David

Re: ELPA policy

[Dmitry Gutov]

On 11/10/2015 08:37 PM, David Engster wrote:

>> I can cut out xref.el's dependency on CEDET easily enough.
>
> Which illustrates nicely the downside of moving packages to ELPA.

Nope. It will be beneficial to the code in Emacs, because xref is 
currently depending on a minor feature of CEDET (the 
semantic-symref-tool infrastructure), which could use a rewrite, to be 
available for a more general usage.

Re: ELPA policy

[Eli Zaretskii]

> Cc: Stephen Leake <stephen_leake@stephe-leake.org>,
>  Eli Zaretskii <eliz@gnu.org>, aaronecay@gmail.com, Stromeko@nexgo.de,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 10 Nov 2015 21:57:25 +0200
> 
> On 11/10/2015 08:37 PM, David Engster wrote:
> 
> >> I can cut out xref.el's dependency on CEDET easily enough.
> >
> > Which illustrates nicely the downside of moving packages to ELPA.
> 
> Nope. It will be beneficial to the code in Emacs, because xref is 
> currently depending on a minor feature of CEDET (the 
> semantic-symref-tool infrastructure), which could use a rewrite, to be 
> available for a more general usage.

I think a point we should take from this is that the prerequisites (in
this case, the said rewrite) should be in place _before_ we seriously
consider such moves (pun intended ;-).

Re: ELPA policy

[Dmitry Gutov]

On 11/10/2015 10:01 PM, Eli Zaretskii wrote:

> I think a point we should take from this is that the prerequisites (in
> this case, the said rewrite) should be in place _before_ we seriously
> consider such moves (pun intended ;-).

The only prerequisite is to have this functionality (which is fairly 
self-contained) in the core. It would benefit from a rewrite (and moving 
away from CEDET would encourage that), but as the first step, we can 
just copy it and do some renaming.

However, like mentioned in the "project.el semantics" thread:

   I'm seriously considering to get away from it and always use Grep,
   because the other tools, while they can be faster, they're also
   entirely opaque to the user, can have outdated databases, non-indexed
   relevant directories, unsupported languages, and so on.

So this feature, which xref is depending on for the default 
xref-find-references implementation, requires certain amount of 
hand-holding from the user. And I don't know how to approach documenting 
it, and otherwise make it apparent to the user.

So someone else should probably start on taking care of that. Otherwise, 
I'd rather fall back on Grep, until such point that Emacs starts to 
manage the Global/id-utils/CScope datbases for the user, and learns to 
make sure that the database is up-to-date.

Re: ELPA policy

[Eli Zaretskii]

> Cc: deng@randomsample.de, stephen_leake@stephe-leake.org,
>  aaronecay@gmail.com, Stromeko@nexgo.de, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 10 Nov 2015 22:19:46 +0200
> 
> So someone else should probably start on taking care of that. Otherwise, 
> I'd rather fall back on Grep, until such point that Emacs starts to 
> manage the Global/id-utils/CScope datbases for the user, and learns to 
> make sure that the database is up-to-date.

Grep doesn't scale well to large projects, IME.  You get too many
false positives.

Outdated databases are easy to avoid with the likes of cron jobs.
Yes, that's hand-holding, but when you have to quickly find stuff in a
project with 3 million lines of code and thousands of classes, there
really is no other alternative.

Re: ELPA policy

[Dmitry Gutov]

On 11/10/2015 10:34 PM, Eli Zaretskii wrote:

> Grep doesn't scale well to large projects, IME.  You get too many
> false positives.

We can also check the "inside string or comment" status, as well as use 
the Emacs syntax tables to make sure that the match begins and ends at 
symbol boundaries appropriate for the file's language. Everything 
necessary for that is already written (and you can try it out by calling 
project-find-regexp and using something like \_<tool_bar_items\_> as the 
regexp). I don't think that id-utils does anything more to weed out 
false positives.

Grep is still probably going to be slower than at least some of the 
tools in question. Could you test, on a large project of your choice?

> Outdated databases are easy to avoid with the likes of cron jobs.
> Yes, that's hand-holding, but when you have to quickly find stuff in a
> project with 3 million lines of code and thousands of classes, there
> really is no other alternative.

Yes. But the xref commands should be easy to use. Even if the above is 
not rocket science, the user would still have to know what they need to 
do to get up-to-date results. (Believe it or not, I haven't created a 
single cron job for code writing purposes in my life, and I don't know 
its syntax for the time intervals. I'm likely not alone in that.)

Re: ELPA policy

[Dmitry Gutov]

On 11/10/2015 11:16 PM, Dmitry Gutov wrote:

> project-find-regexp and using something like \_<tool_bar_items\_> as the

Sorry, using the Emacs-only syntax there doesn't exactly work yet. But 
you can compare by calling xref-find-references when the project has an 
id-utils index file, and doing the same without it.

Re: ELPA policy

[Stephen Leake]

Dmitry Gutov <dgutov@yandex.ru> writes:

> However, like mentioned in the "project.el semantics" thread:

please change the thread subject.

-- 
-- Stephe

Re: ELPA policy

[David Engster]

Dmitry Gutov writes:
> On 11/10/2015 08:37 PM, David Engster wrote:
>
>>> I can cut out xref.el's dependency on CEDET easily enough.
>>
>> Which illustrates nicely the downside of moving packages to ELPA.
>
> Nope. It will be beneficial to the code in Emacs, because xref is
> currently depending on a minor feature of CEDET (the
> semantic-symref-tool infrastructure), which could use a rewrite, to be
> available for a more general usage.

As you know, we will drop our upstream repository. During the last
merge, I will also have to discuss with Eric what we can do to narrow
CEDET's scope. I agree with you that semantic-symref is probably one of
the packages that should be replaced with something more general. But as
long as it's there, I'd appreciate if you could leave whatever support
you have coded for it.

I'd also be happy to drop the completion GUI stuff, BTW. I've repeatedly
said that company should be in core for that.

-David

Re: ELPA policy

[Dmitry Gutov]

On 11/10/2015 10:45 PM, David Engster wrote:

> As you know, we will drop our upstream repository. During the last
> merge, I will also have to discuss with Eric what we can do to narrow
> CEDET's scope.
> I agree with you that semantic-symref is probably one of
> the packages that should be replaced with something more general.

In the end, I would leave only the "backend" part of semantic-symref 
(calling the tools + piping the results through Semantic), and use it in 
the Semantic xref backend. So the results will be displayed by xref.

Alas, xref is not up to feature parity yet (cannot group hits by 
containing functions, no checkboxes, simplistic "rename" command), and 
I'm having difficulties deciding on the changes to the xref data 
structures that would make that possible, as well as keep allowing 
certain other uses of xref, such as project-find-regexp.

I'll write that up in a separate thread later this week.

> But as
> long as it's there, I'd appreciate if you could leave whatever support
> you have coded for it.

Just to be clear, there's no "support" for semantic-symref in xref yet. 
We just use some low-level pieces of it as a "fast Grep".

> I'd also be happy to drop the completion GUI stuff, BTW. I've repeatedly
> said that company should be in core for that.

Also not exactly necessary. Semantic should make its 
completion-at-point-functions element the main entry point for 
completion, and Company will pick it up. You can even have argument list 
annotations, if you add the :annotation-function property.

I do want to have Company more widely available, but probably not in 
25.1. There are quite a few things I wanted to do with it first, and 
most are still untouched.

Re: ELPA policy

[David Engster]

John Wiegley writes:
>>>>>> Stephen Leake <stephen_leake@stephe-leake.org> writes:
>> ELPA packages that other core code depends on (like CEDET; xref uses it -
>> called "core ELPA packages" hereinafter) have to be in every developer's
>> build environment everyday; the other core code has to see the current
>> version of the package, and it has to be the same for every developer.
>> 
>> If core ELPA packages are in the ELPA git repository, you would copy some
>> subtrees of the ELPA git workspace into the Emacs git workspace. People have
>> looked into doing this, but it gets messy and confusing. "git fetch" is
>> dealing with two upstreams for one workspace. There has been talk about git
>> submodules, etc, but nothing concrete.
>
> Elpa.git should be a submodule referenced from within Emacs.git (under "elpa").
> This allows us to expressly state which "version" of Elpa is expected to work
> with the current Emacs.git.

Since ELPA comprises many packages, that simply cannot work.

> Large packages like CEDET should move outside of Emacs.git and into Elpa.git.

Why?

> I do not want this code replication. We will promote ELPA as an active
> development space alongside core Emacs. If we use a submodule, and improve our
> tooling slightly, it should become a seamless thing.

From my experience, git submodules are never seamless to work with.

-David

Re: ELPA policy

[John Wiegley]

>>>>> David Engster <deng@randomsample.de> writes:

>> Elpa.git should be a submodule referenced from within Emacs.git (under "elpa").
>> This allows us to expressly state which "version" of Elpa is expected to work
>> with the current Emacs.git.

> Since ELPA comprises many packages, that simply cannot work.

Perhaps I should say, "The version of Elpa.git, a subset of which will appear
in the release tarball of this commit".

>> Large packages like CEDET should move outside of Emacs.git and into
>> Elpa.git.

> Why?

There will never be 100% agreement on whether they should be in ELPA, or be in
Core, so I'm making the decision that they belong in ELPA. The big three that
will be moved first are: Gnus, Org and CEDET.

> From my experience, git submodules are never seamless to work with.

Even still, they work perfectly fine for this purpose.

John

Re: ELPA policy

[David Engster]

John Wiegley writes:
>>> Large packages like CEDET should move outside of Emacs.git and into
>>> Elpa.git.
>
>> Why?
>
> There will never be 100% agreement on whether they should be in ELPA, or be in
> Core, so I'm making the decision that they belong in ELPA.

So you simply don't want to give any reason?

-David

Re: ELPA policy

[John Wiegley]

>>>>> David Engster <deng@randomsample.de> writes:

> So you simply don't want to give any reason?

David, although I work by consensus as much as possible, I won't explain
everything I choose to do. My reason is that I think this will improve Emacs
development.

Part of my goal is to make the boundary between Emacs and ELPA thinner, so it
will actually be *easier* to move CEDET and friends back into core later, if
my reasoning is wrong.

John

Re: ELPA policy

[David Engster]

John Wiegley writes:
>>>>>> David Engster <deng@randomsample.de> writes:
>
>> So you simply don't want to give any reason?
>
> David, although I work by consensus as much as possible, I won't explain
> everything I choose to do. My reason is that I think this will improve Emacs
> development.

This is not about reaching a consensus. This is about you giving proper
reasons for such a big change.

-David

Re: ELPA policy

[John Wiegley]

>>>>> David Engster <deng@randomsample.de> writes:

> This is not about reaching a consensus. This is about you giving proper
> reasons for such a big change.

To be clear, I would also put Eshell (though not pcomplete) into the category
of "big things that should be in ELPA" -- albeit, the subset of ELPA that will
be in the release tarball.

It's hard to pin down why in a manner that fits in an e-mail. If Eshell were
in ELPA today, and we were talking about moving it into core, I would have
just as much trouble justifying that move too. Perhaps this simply underscores
the fact that we don't have an agreed upon ELPA policy we can all refer to.

OK, David, I won't decide this by fiat. Let us decide what our ELPA policy
will be, until it becomes clear, based on that document, what should go where,
and why. We'll defer discussions of package movement until we have that in
place.

John

Re: ELPA policy

[David Engster]

John Wiegley writes:
>>>>>> David Engster <deng@randomsample.de> writes:
>
>> This is not about reaching a consensus. This is about you giving proper
>> reasons for such a big change.
>
> To be clear, I would also put Eshell (though not pcomplete) into the category
> of "big things that should be in ELPA" -- albeit, the subset of ELPA that will
> be in the release tarball.
>
> It's hard to pin down why in a manner that fits in an e-mail. If Eshell were
> in ELPA today, and we were talking about moving it into core, I would have
> just as much trouble justifying that move too. Perhaps this simply underscores
> the fact that we don't have an agreed upon ELPA policy we can all refer to.

In my opinion, the main question is whether something provides
infrastructure for other packages to use. This is precisely what CEDET
tries to do. I wouldn't have much trouble with putting parts of CEDET in
ELPA, namely those parts that do not directly provide infrastructure,
like support for certain languages, project types, indexing tools, etc.

> OK, David, I won't decide this by fiat. Let us decide what our ELPA policy
> will be, until it becomes clear, based on that document, what should go where,
> and why. We'll defer discussions of package movement until we have that in
> place.

It is still not clear to me what exactly is gained by moving core
packages to ELPA. In my opinion, packages like Org, Gnus or CEDET didn't
create significant problems for Emacs development in the past. On the
contrary, it made sure that those packages were adapted quickly when
larger changes in core were made. Those changes got synced back upstream
by people like Bastien, Katsumi and me. So in my opinion, you are trying
to fix something that is not broken.

What actually *does* create problems are packages that don't have an
active maintainer. The 'big three' however are not among those.

-David

Re: ELPA policy

[John Wiegley]

>>>>> David Engster <deng@randomsample.de> writes:

> the main question is whether something provides infrastructure for other
> packages to use.

Sounds like a good sentence for an ELPA policy. :)

> It is still not clear to me what exactly is gained by moving core packages
> to ELPA.

Agility. What is appropriate. Knowing when a thing goes into core, and when in
ELPA.

Org is an application, it's not infrastructure; the same with Gnus. *Parts* of
Gnus might rightly be considered infrastructure, but the whole of Gnus just
doesn't belong there. Parts of CEDET probably do belong in Emacs core, but as
an application, I don't think the whole of it does.

Since we don't have an agreed upon basis by which to draw such lines, we're
both talking about what we feel is right. Let's agree to revisit after having
the ELPA discussion.

John

Re: ELPA policy

[Stephen Leake]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> David Engster <deng@randomsample.de> writes:
>
>> the main question is whether something provides infrastructure for other
>> packages to use.
>
> Sounds like a good sentence for an ELPA policy. :)
>
>> It is still not clear to me what exactly is gained by moving core packages
>> to ELPA.

Since no other core code depends on ELPA, CEDET is _not_ a "core package".

Rather, CEDET is a "tarball package"; it is in Emacs git solely to
ensure that it is included in the Emacs release tarball.

A better example of a possible "core ELPA package" is the "seq" package.

> Agility. 

I hear that as "easier to make small/frequent changes". That is what the
ELPA release policy gives you, yes. So this is one rationale for moving
packages to ELPA.

> What is appropriate. 

That's what we are trying to figure out :)

> Knowing when a thing goes into core, and when in ELPA.

Ditto.

> Org is an application, it's not infrastructure; the same with Gnus. *Parts* of
> Gnus might rightly be considered infrastructure, but the whole of Gnus just
> doesn't belong there. Parts of CEDET probably do belong in Emacs core, but as
> an application, I don't think the whole of it does.

I think the distinction between "tarball package" and "core package" is
helpful here.

I'm guessing that the main motivation for including Org and Gnus in
Emacs git (well, CVS back then?) was to include them in the release
tarball. If we have a mechanism to allow that for ELPA packages, moving
them to ELPA makes sense.

-- 
-- Stephe

Re: ELPA policy

[John Wiegley]

>>>>> Stephen Leake <stephen_leake@stephe-leake.org> writes:

> I think the distinction between "tarball package" and "core package" is
> helpful here.
> 
> I'm guessing that the main motivation for including Org and Gnus in Emacs
> git (well, CVS back then?) was to include them in the release tarball. If we
> have a mechanism to allow that for ELPA packages, moving them to ELPA makes
> sense.

I like this. I think we have a good striation:

  core
  tarball ELPA
  net ELPA

To the user, core and tarball ELPA should be indistinguishable. I think also
that some tarball ELPA packages should come "pre-installed", if that is not
already done. This would makes them accessible via autoload, rather than
requiring the package interface to opt-in.

If core == tarball ELPA for everyone but us, this makes the decision easy
whether something should be in core or not a lot: We put it in core if core
requires it. If nothing at all in core uses the package -- and if that package
doesn't define "essential" functionality, like isearch.el or grep.el -- it can
shift to tarball ELPA.

This still leaves open the meaning of "essential", but it does make the
picture clearer.

John

RE: ELPA policy

[Drew Adams]

> > I think the distinction between "tarball package" and "core package" is
> > helpful here.
> >
> > I'm guessing that the main motivation for including Org and Gnus in
> > Emacs git (well, CVS back then?) was to include them in the release tarball.
> > If we have a mechanism to allow that for ELPA packages, moving them to ELPA
> > makes sense.
> 
> I like this. I think we have a good striation:
>   core
>   tarball ELPA
>   net ELPA
> To the user, core and tarball ELPA should be indistinguishable.

I haven't yet received an answer to my question whether anything
will change for users, depending on where you happen to manage
the code wrt ELPA etc.

But it sounds like the answer is yes.  If some stuff that has
traditionally been part of the distribution gets moved to (net)
ELPA, it will no longer be distributed to users.  They will
need to pull it down using the package interface.  Is that right?

If so, I'm not crazy about that.  I don't particularly want to
go fishing in (net) ELPA for stuff that I've always been able to
simply grep from within the distribution `lisp' directory. 
Especially, but not only, when I am not on the Internet.

I hope you will continue to (also) distribute Emacs with all
of its (traditional) source code, and not just make users
request it from (net) ELPA.

Re: ELPA policy

[John Wiegley]

>>>>> Drew Adams <drew.adams@oracle.com> writes:

> I haven't yet received an answer to my question whether anything will change
> for users, depending on where you happen to manage the code wrt ELPA etc.
> 
> But it sounds like the answer is yes. If some stuff that has traditionally
> been part of the distribution gets moved to (net) ELPA, it will no longer be
> distributed to users. They will need to pull it down using the package
> interface. Is that right?

Correct. Sorry I assumed that the discussion had answered your question in
passing.

> If so, I'm not crazy about that. I don't particularly want to go fishing in
> (net) ELPA for stuff that I've always been able to simply grep from within
> the distribution `lisp' directory. Especially, but not only, when I am not
> on the Internet.

Those things could then be in tarball ELPA. The distinction between tarball
ELPA and net ELPA is much more plastic than core and tarball ELPA. In fact, I
have a feeling that, initially, tarball ELPA and net ELPA will just be the
same thing, until we have a reason to tease them apart.

> I hope you will continue to (also) distribute Emacs with all of its
> (traditional) source code, and not just make users request it from (net)
> ELPA.

This is my preference as well.

John

Re: ELPA policy

[Stephen Leake]

John Wiegley <jwiegley@gmail.com> writes:

> The distinction between tarball
> ELPA and net ELPA is much more plastic than core and tarball ELPA. In fact, I
> have a feeling that, initially, tarball ELPA and net ELPA will just be the
> same thing, until we have a reason to tease them apart.

Including a package in tarball ELPA means that the Emacs project has
decided that the package is sufficiently important to be made
immediately available to all Emacs users with no effort on their part
beyond installing Emacs from the tarball.

That implies a certain level of maturity and quality, and implies that
it should be favored over other similar packages.

On the other hand, we have in several cases recommended that someone put
a package in Gnu ELPA in order to give it more exposure, so that it
might eventually become higher quality and more mature.

In general, ELPA packages have far less oversight than core packages.
tarball ELPA packages should have the same oversight as core, or at
least more than normal ELPA packages.

So I disagree that all of Gnu ELPA should be included in the tarball.

Another choice would be to say "use MELPA for experimental packages",
but it is already too late for that.

-- 
-- Stephe

Re: ELPA policy

[Filipp Gunbin]

I do not consider myself competent enough in this area, but I'd like
to share some thoughts here:

For each package version there is a range (possibly sparse) of core
versions on which the package is supported (or just should work).

While the intent to move as much as possible in ELPA can be understood,
it leads to potentially more incompatibilities between important parts
which can provide API by themselves.

So, there should be some compromise between "latency" of new features
before they generally become available for use in packages and overall
core stability.  To me, it seems that the latter is more important and
it's better to keep infrastructure & library things in core, while
moving everything that uses them for a concrete purpose to ELPA.

If that in turn provides some API which others use, then we have
package interdependencies and that is probably OK (but can lead to
conflicts).  If sufficient amount of important packages use some API
and that API is rather mature, then the maintainer can decide to move
that in core to simplify dependency management.

So, I'd suggest that:

- Language features go straight into core (and people working on them
  / using them will have to use git version of Emacs until next
  release)

- New libraries which are not supposed to be in very common use go to
  ELPA first

- Then they are probably promoted to core as they get mature and more
  widely used - just to simplify their usage ("they will always be
  available").

- Major applications (like Gnus) and smaller ones always go to ELPA
  (most probably we should also bundle them in a tarball, but keep
  outside of the core).  A user can then decide whether to use git
  version of e.g. Gnus (from Elpa or private package repository) if she
  likes, or update to a released package version from Elpa (if core
  supports it), or just keep using the Elpa version she uses at the
  moment (which probably came together with current core).

- If such an application provides things useful for other
  applications, then it probably should extract that into a library to
  go through the cycle oulined above (elpa --maturity--> core).

The API / SPI notion can also be used to provide implementations
(backends) for different features which may have default simple
implementations in core and more advanced ones in packages.  A package
must somehow "register" itself as a candidate for being a service for
a concrete feature during installation.

Filipp

Re: ELPA policy

[Stephen Leake]

Filipp Gunbin <fgunbin@fastmail.fm> writes:

> For each package version there is a range (possibly sparse) of core
> versions on which the package is supported (or just should work).

For normal ELPA packages, that's true.

For core and tarball ELPA packages, they must work in each current Emacs
release (since they are in the release tarball). They may also support
some set of previous Emacs releases.

> While the intent to move as much as possible in ELPA can be understood,
> it leads to potentially more incompatibilities between important parts
> which can provide API by themselves.

Yes.

> So, there should be some compromise between "latency" of new features
> before they generally become available for use in packages and overall
> core stability.  To me, it seems that the latter is more important and
> it's better to keep infrastructure & library things in core, while
> moving everything that uses them for a concrete purpose to ELPA.

For normal ELPA packages, that's true. But this is partly why core and
tarball ELPA packages are being considered.

> If that in turn provides some API which others use, then we have
> package interdependencies and that is probably OK (but can lead to
> conflicts). 

A more flexible requirements spec may  be needed. For example, Debian
packages allow specifying a range of versions in the dependency; that
can reflect actual testing, and not rely on implicit promises of future
backward compatibility. Of course, it can also lead to false failures.

> If sufficient amount of important packages use some API and that API
> is rather mature, then the maintainer can decide to move that in core
> to simplify dependency management.

or to a tarball ELPA package, or to a core ELPA package.

Perhaps that is too much choice?

> So, I'd suggest that:
>
> - Language features go straight into core (and people working on them
>   / using them will have to use git version of Emacs until next
>   release)

By "language features" do you mean things like ada-mode.el (primary mode
for editing Ada source files)? If so, I strongly disagree; ada-mode.el
is very happy as a normal ELPA package. Similar modes for other
languages with a small audience could be in ELPA as well; I have not
looked to see what's actually there.

cc-mode is the core for a lot of similar C-like languages, so it
probably makes sense to keep that in core. But it would be interesting
to see what the consequences of moving it to a normal or tarball ELPA
package would be.

But if you mean things like parsers and xref support, then I agree.
Although ada-mode also provides its own parser ...

> - New libraries which are not supposed to be in very common use go to
>   ELPA first

> - Then they are probably promoted to core as they get mature and more
>   widely used - just to simplify their usage ("they will always be
>   available").

That's what tarball ELPA packages are for.

The only reason to move a package to actual core (ie emacs git, as
opposed to elpa git) is if some other core code wants to depend in it.

> - Major applications (like Gnus) and smaller ones always go to ELPA
>   (most probably we should also bundle them in a tarball, but keep
>   outside of the core).

Right; that's a tarball ELPA package.

> - If such an application provides things useful for other
>   applications, then it probably should extract that into a library to
>   go through the cycle oulined above (elpa --maturity--> core).

Yes.

> The API / SPI notion can also be used to provide implementations
> (backends) for different features which may have default simple
> implementations in core and more advanced ones in packages.  A package
> must somehow "register" itself as a candidate for being a service for
> a concrete feature during installation.

If we use cl-generic to provide dispatching, the cl-defmethod form does
the registration.

-- 
-- Stephe

Re: ELPA policy

[Filipp Gunbin]

Stephen,

On 11/11/2015 15:22 -0600, Stephen Leake wrote:

>> If sufficient amount of important packages use some API and that API
>> is rather mature, then the maintainer can decide to move that in core
>> to simplify dependency management.
>
> or to a tarball ELPA package, or to a core ELPA package.
>
> Perhaps that is too much choice?

Mm yes, that's one of my fears, that it will become too complex for a
package maintaner.  Why not just treat each ELPA package just as an ELPA
package and leave the option of bundling it to core maintainers which
are better in this area (they'll do it in agreement with package
maintainer, of course)?

A "tarball" ELPA package is a one thing (that's what I call "bundle into
tarball", if I understood right), but "core" ELPA package is another -
here I have another fear, that normal and tarball ELPA packages
depending on such "core" packages, will have to accurately define
versions of the core package they support, and then we have to check all
this during the installation.  We'll probably have to calculate and
offer to user which set of the multiple core packages' multiple versions
suits for multiple normal and tarball (perhaps overriden by version from
Internet package archive) packages, at the same time probably giving the
user to opportunity to specify her preferred version of each requested
package.  Is it worth the trouble?  Or do I understand something wrong?

That's why I wrote about the feature latency - maybe it would be enough
if a person willing to use latest core features in her package will have
to develop it on git emacs and wait for the next official release to
make her package available to the public.  This will be the same as with
new C level features, which we cannot "push quicky" as we can with ELPA
package.

>> So, I'd suggest that:
>>
>> - Language features go straight into core (and people working on them
>>   / using them will have to use git version of Emacs until next
>>   release)
>
> By "language features" do you mean things like ada-mode.el (primary mode
> for editing Ada source files)? If so, I strongly disagree; ada-mode.el
> is very happy as a normal ELPA package. Similar modes for other
> languages with a small audience could be in ELPA as well; I have not
> looked to see what's actually there.

No-no, what I meant were Emacs Lisp libraries extending language, like
seq.el.

>> The API / SPI notion can also be used to provide implementations
>> (backends) for different features which may have default simple
>> implementations in core and more advanced ones in packages.  A package
>> must somehow "register" itself as a candidate for being a service for
>> a concrete feature during installation.
>
> If we use cl-generic to provide dispatching, the cl-defmethod form does
> the registration.

Thanks, that's one more reason for me to become acquainted with Emacs CL
features.

Filipp

Re: ELPA policy

[John Wiegley]

>>>>> Filipp Gunbin <fgunbin@fastmail.fm> writes:

> A "tarball" ELPA package is a one thing (that's what I call "bundle into
> tarball", if I understood right), but "core" ELPA package is another

I must have missed this distinction. What is the difference between "tarball
ELPA" and "core ELPA"?  I was thinking of "core" as anything that is in
Emacs.git.

John

Re: ELPA policy

[Filipp Gunbin]

On 12/11/2015 09:11 -0800, John Wiegley wrote:

>>>>>> Filipp Gunbin <fgunbin@fastmail.fm> writes:
>
>> A "tarball" ELPA package is a one thing (that's what I call "bundle into
>> tarball", if I understood right), but "core" ELPA package is another
>
> I must have missed this distinction. What is the difference between "tarball
> ELPA" and "core ELPA"?  I was thinking of "core" as anything that is in
> Emacs.git.

If I got it right, a "tarball" package is copied from elpa.git into the
tarball at release time.  This is to bundle package so that Internet is
not required to install it (whether or not it is installed by default).

"core" package is developed in emacs.git (because core code depends on
it and we have to have full environment in the repo), but then can be
updated from elpa if new version is available.  This is to allow
frequent updates of such packages.

If I didn't get it right, then probably other package maintainers would
feel a bit lost, too, unless we provide concise explanation of this
separation.

Filipp

Re: ELPA policy

[John Wiegley]

>>>>> Filipp Gunbin <fgunbin@fastmail.fm> writes:

> "core" package is developed in emacs.git (because core code depends on it
> and we have to have full environment in the repo), but then can be updated
> from elpa if new version is available. This is to allow frequent updates of
> such packages.

Ah, that makes perfect sense, thanks.

John

Re: ELPA policy

[Stephen Leake]

Filipp Gunbin <fgunbin@fastmail.fm> writes:

> On 12/11/2015 09:11 -0800, John Wiegley wrote:
>
>>>>>>> Filipp Gunbin <fgunbin@fastmail.fm> writes:
>>
>>> A "tarball" ELPA package is a one thing (that's what I call "bundle into
>>> tarball", if I understood right), but "core" ELPA package is another
>>
>> I must have missed this distinction. What is the difference between "tarball
>> ELPA" and "core ELPA"?  I was thinking of "core" as anything that is in
>> Emacs.git.
>
> If I got it right, a "tarball" package is copied from elpa.git into the
> tarball at release time.  This is to bundle package so that Internet is
> not required to install it (whether or not it is installed by default).
>
> "core" package is developed in emacs.git (because core code depends on
> it and we have to have full environment in the repo), but then can be
> updated from elpa if new version is available.  This is to allow
> frequent updates of such packages.

Yes.

I'm working on writing this up for admin/notes/elpa.

-- 
-- Stephe

Re: ELPA policy

[Stephen Leake]

Filipp Gunbin <fgunbin@fastmail.fm> writes:

> Stephen,
>
> On 11/11/2015 15:22 -0600, Stephen Leake wrote:
>
>>> If sufficient amount of important packages use some API and that API
>>> is rather mature, then the maintainer can decide to move that in core
>>> to simplify dependency management.
>>
>> or to a tarball ELPA package, or to a core ELPA package.
>>
>> Perhaps that is too much choice?
>
> Mm yes, that's one of my fears, that it will become too complex for a
> package maintaner.  Why not just treat each ELPA package just as an ELPA
> package and leave the option of bundling it to core maintainers which
> are better in this area (they'll do it in agreement with package
> maintainer, of course)?
>
> A "tarball" ELPA package is a one thing (that's what I call "bundle into
> tarball", if I understood right), 

Yes. We have not implemented this yet, but I'm imagining there is a list
of these in the Gnu Emacs Makefile somewhere (probably in a separate
file read by the Makefile). I don't think there will need to be any
metadata in the package files for this.

Declaring an ELPA package to be a tarball package does impose some
restrictions on the package maintainer; they have to synchronize with
each Emacs release, and accept the same review/oversight as core code.

> but "core" ELPA package is another -
> here I have another fear, that normal and tarball ELPA packages
> depending on such "core" packages, will have to accurately define
> versions of the core package they support, and then we have to check all
> this during the installation.  

I don't see why that is different from depending on a normal ELPA
package. 

> We'll probably have to calculate and offer to user which set of the
> multiple core packages' multiple versions suits for multiple normal
> and tarball (perhaps overriden by version from Internet package
> archive) packages, at the same time probably giving the user to
> opportunity to specify her preferred version of each requested
> package. Is it worth the trouble? Or do I understand something wrong?

Emacs does not support multiple versions of packages available
simultaneously; there is only one instance of a package that is first in
load-path.

You can end up with one copy in the installed Emacs distribution, and
one in ~/.emacs.d/elpa. But the elpa one will take precedence; that is
the only one that other packages have to worry about. It either meets
their dependency requirement, or not.

> That's why I wrote about the feature latency - maybe it would be enough
> if a person willing to use latest core features in her package will have
> to develop it on git emacs and wait for the next official release to
> make her package available to the public.  This will be the same as with
> new C level features, which we cannot "push quicky" as we can with ELPA
> package.

That's the main reason to make a package available in both core and ELPA:
users of the released version of Emacs can use the latest version of the
core ELPA package from ELPA, overriding the copy in their installation.

>>> So, I'd suggest that:
>>>
>>> - Language features go straight into core (and people working on them
>>>   / using them will have to use git version of Emacs until next
>>>   release)
>
> No-no, what I meant were Emacs Lisp libraries extending language, like
> seq.el.

That's a good candidate for a core ELPA package.

-- 
-- Stephe

Re: ELPA policy

[Filipp Gunbin]

On 12/11/2015 13:52 -0600, Stephen Leake wrote:

>> A "tarball" ELPA package is a one thing (that's what I call "bundle into
>> tarball", if I understood right), 
>
> Yes. We have not implemented this yet, but I'm imagining there is a list
> of these in the Gnu Emacs Makefile somewhere (probably in a separate
> file read by the Makefile). I don't think there will need to be any
> metadata in the package files for this.
>
> Declaring an ELPA package to be a tarball package does impose some
> restrictions on the package maintainer; they have to synchronize with
> each Emacs release, and accept the same review/oversight as core code.

They'll just have to make sure that a single version (which is going to
be bundled in a tarball) works good with an Emacs release being
prepared.

>> but "core" ELPA package is another -
>> here I have another fear, that normal and tarball ELPA packages
>> depending on such "core" packages, will have to accurately define
>> versions of the core package they support, and then we have to check all
>> this during the installation.  
>
> I don't see why that is different from depending on a normal ELPA
> package.

I think it's their dependants what make them different from tarball and
normal packages.

Emacs core provides API, which others use.  A normal package declares
which versions of this API it is supposed to work against.

With core packages, Emacs still provides API, but it now consists of a
static part (C level & core) and dynamic part (core packages, which can
later be updated from ELPA - correct me if I'm wrong).  So, a package
should independently define which core version it works agains and which
core package(s) version(s) it works against.

Here's where I see the complexity with multiple packages installed on
user request with package manager, which I tried to describe below.

>> We'll probably have to calculate and offer to user which set of the
>> multiple core packages' multiple versions suits for multiple normal
>> and tarball (perhaps overriden by version from Internet package
>> archive) packages, at the same time probably giving the user to
>> opportunity to specify her preferred version of each requested
>> package. Is it worth the trouble? Or do I understand something wrong?
>
> Emacs does not support multiple versions of packages available
> simultaneously; there is only one instance of a package that is first in
> load-path.
>
> You can end up with one copy in the installed Emacs distribution, and
> one in ~/.emacs.d/elpa. But the elpa one will take precedence; that is
> the only one that other packages have to worry about. It either meets
> their dependency requirement, or not.

Of course, I was talking about the set of available versions which could
be installed and from which a user or a package manager should choose.

>> That's why I wrote about the feature latency - maybe it would be enough
>> if a person willing to use latest core features in her package will have
>> to develop it on git emacs and wait for the next official release to
>> make her package available to the public.  This will be the same as with
>> new C level features, which we cannot "push quicky" as we can with ELPA
>> package.
>
> That's the main reason to make a package available in both core and ELPA:
> users of the released version of Emacs can use the latest version of the
> core ELPA package from ELPA, overriding the copy in their installation.

Yes, but it can introduce complexities I wrote of above.

Filipp

Re: ELPA policy

[Stephen Leake]

Filipp Gunbin <fgunbin@fastmail.fm> writes:

> On 12/11/2015 13:52 -0600, Stephen Leake wrote:
>
>> I don't see why that is different from depending on a normal ELPA
>> package.
>
> I think it's their dependants what make them different from tarball and
> normal packages.
>
> Emacs core provides API, which others use.  A normal package declares
> which versions of this API it is supposed to work against.
>
> With core packages, Emacs still provides API, but it now consists of a
> static part (C level & core) and dynamic part (core packages, which can
> later be updated from ELPA - correct me if I'm wrong).  So, a package
> should independently define which core version it works agains and which
> core package(s) version(s) it works against.

package.el dependencies can only specify a minimum version, not a
maximum. there is no way that a normal ELPA package can declare that it
works with seq.el 1.0 but not seq.el 1.1

So if emacs 25 contains a core ELPA package seq.el 1.0, the declaration

;; Package-Requires: ((emacs "25.1"))

is equivalent to:

;; Package-Requires: ((emacs "25.1") (seq "1.0))

If seq.el 1.1 is later released via the Gnu ELPA server, it will be used
in either case.

Listing a core ELPA package explicitly is only necessary when the emacs
version is less than the first version that included the core ELPA
package version.

On the other hand, it doesn't hurt, so it's probably best to recommend
listing all ELPA packages explicitly.

>>> We'll probably have to calculate and offer to user which set of the
>>> multiple core packages' multiple versions suits for multiple normal
>>> and tarball (perhaps overriden by version from Internet package
>>> archive) packages, at the same time probably giving the user to
>>> opportunity to specify her preferred version of each requested
>>> package. Is it worth the trouble? Or do I understand something wrong?
>>
>> Emacs does not support multiple versions of packages available
>> simultaneously; there is only one instance of a package that is first in
>> load-path.
>>
>> You can end up with one copy in the installed Emacs distribution, and
>> one in ~/.emacs.d/elpa. But the elpa one will take precedence; that is
>> the only one that other packages have to worry about. It either meets
>> their dependency requirement, or not.
>
> Of course, I was talking about the set of available versions which could
> be installed and from which a user or a package manager should choose.

I still don't see a problem.

We have an example of a core ELPA package now; ada-mode. version 4.3 is
in emacs git; version 5.1.8 is in ELPA (if I ever get 5.x to be fast
enough on huge files, I'll delete 4.3 from core).

package.el has no issues with this.

Similar things can occur when there are different versions of the same
package in multiple repositories. In that sense, emacs git is just
another package repository.

Can you be more explicit about what problem you foresee? Give an example
of a package that would cause a problem.

-- 
-- Stephe

Re: ELPA policy

[Filipp Gunbin]

Stephen,

On 14/11/2015 04:30 -0600, Stephen Leake wrote:

> Filipp Gunbin <fgunbin@fastmail.fm> writes:
>
>> On 12/11/2015 13:52 -0600, Stephen Leake wrote:
>>
>>> I don't see why that is different from depending on a normal ELPA
>>> package.
>>
>> I think it's their dependants what make them different from tarball and
>> normal packages.
>>
>> Emacs core provides API, which others use.  A normal package declares
>> which versions of this API it is supposed to work against.
>>
>> With core packages, Emacs still provides API, but it now consists of a
>> static part (C level & core) and dynamic part (core packages, which can
>> later be updated from ELPA - correct me if I'm wrong).  So, a package
>> should independently define which core version it works agains and which
>> core package(s) version(s) it works against.
>
> package.el dependencies can only specify a minimum version, not a
> maximum. there is no way that a normal ELPA package can declare that it
> works with seq.el 1.0 but not seq.el 1.1

My words were more of a theoretical speculation rather than discussion
of the current state of package.el.

Some parts of API may be deprecated and removed, so it may be the case
that a current package is not updated instantly and need to use some
previous version of core package.  While having a "single snapshot" of
Emacs + core packages does not cause such issues (even if package does
not specify maximum version which author of course does not know in
advance, the package just doesn’t work and so the user can downgrade to
previous "snapshot", that is, previous Emacs version), separate core
packages update after installation of such a "snapshot" might move
forward and thus break some normal package (and downgrade will not help
- the user will not know what to downgrade - what package? or core
emacs?  And how can I downgrade a single (core) package?).  Just theory,
again.  Ignore if it is irrelevant and I’m complicating things too much.

> So if emacs 25 contains a core ELPA package seq.el 1.0, the declaration
>
> ;; Package-Requires: ((emacs "25.1"))
>
> is equivalent to:
>
> ;; Package-Requires: ((emacs "25.1") (seq "1.0))
>
> If seq.el 1.1 is later released via the Gnu ELPA server, it will be used
> in either case.

Will the user have the option to continue to use the tarball version
instead of newer ELPA version?  Or choosing which ELPA version to use?
This may be needed when she uses previous major Emacs release, and
current ELPA package version requires newer core APIs.

>>>> We'll probably have to calculate and offer to user which set of the
>>>> multiple core packages' multiple versions suits for multiple normal
>>>> and tarball (perhaps overriden by version from Internet package
>>>> archive) packages, at the same time probably giving the user to
>>>> opportunity to specify her preferred version of each requested
>>>> package. Is it worth the trouble? Or do I understand something wrong?
>>>
>>> Emacs does not support multiple versions of packages available
>>> simultaneously; there is only one instance of a package that is first in
>>> load-path.
>>>
>>> You can end up with one copy in the installed Emacs distribution, and
>>> one in ~/.emacs.d/elpa. But the elpa one will take precedence; that is
>>> the only one that other packages have to worry about. It either meets
>>> their dependency requirement, or not.
>>
>> Of course, I was talking about the set of available versions which could
>> be installed and from which a user or a package manager should choose.
>
> I still don't see a problem.
>
> We have an example of a core ELPA package now; ada-mode. version 4.3 is
> in emacs git; version 5.1.8 is in ELPA (if I ever get 5.x to be fast
> enough on huge files, I'll delete 4.3 from core).
>
> package.el has no issues with this.
>
> Similar things can occur when there are different versions of the same
> package in multiple repositories. In that sense, emacs git is just
> another package repository.
>
> Can you be more explicit about what problem you foresee? Give an example
> of a package that would cause a problem.

For now, I don’t know of any practical examples.  But thanks for your
time on this.

Filipp

Re: ELPA policy

[Stephen Leake]

Filipp Gunbin <fgunbin@fastmail.fm> writes:

>> So if emacs 25 contains a core ELPA package seq.el 1.0, the declaration
>>
>> ;; Package-Requires: ((emacs "25.1"))
>>
>> is equivalent to:
>>
>> ;; Package-Requires: ((emacs "25.1") (seq "1.0))
>>
>> If seq.el 1.1 is later released via the Gnu ELPA server, it will be used
>> in either case.
>
> Will the user have the option to continue to use the tarball version
> instead of newer ELPA version?  

Yes, they may install it or not; my statement above assumed the ELPA
package was installed and updated.

> Or choosing which ELPA version to use?

Technically, there is only one ELPA version; the one shown by
`list-packages'. However, after user installs one version, they may opt
to not install an update.

If they delete the installed version, they can only install the latest
version.

That's all just using the commands in package.el; users can of course
save code and copy it into and out of `load-path' manually.

> This may be needed when she uses previous major Emacs release, and
> current ELPA package version requires newer core APIs.

If there is a compatible ELPA package version that is more recent than
the version bundled with that Emacs version, but older than the latest
ELPA version, yes.

But that's not supported by the current ELPA server and package.el.

So at the moment, users of older Emacsen are stuck with either the
bundled package, or the latest package.

The maintainer of the ELPA package can provide compatibility code so it
is useful with older Emacsen. But that gets harder the farther back you
go; my current limit is 24.3.

-- 
-- Stephe

Re: ELPA policy

[Tom Tromey]

Stephen> package.el dependencies can only specify a minimum version, not a
Stephen> maximum. there is no way that a normal ELPA package can declare that it
Stephen> works with seq.el 1.0 but not seq.el 1.1

If I had to do it all over, I'd require semver for Emacs packages.
Maybe this could be done optionally somehow.

Tom

Re: ELPA policy

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Another choice would be to say "use MELPA for experimental packages",
  > but it is already too late for that.

If my memory serves, MELPA contains packages we would like to include
in Emacs, but we can't get legal papers for them.
We are seeking people to write replacements for those, though we wish
it were not necessary.

We would like developers of future useful packages to want to put
their packages into the GNU ELPA.

To recommend putting some packages -- whatever they might be -- into
MELPA would work against these goals.  It would have been a bad
approach.

  > Which raises the question; what is the rationale for moving from MELPA
  > to Gnu ELPA?

If a package is useful to recommend, and we CAN get papers for it,
we definitely want to move it to GNU ELPA.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: ELPA policy

[John Wiegley]

>>>>> Richard Stallman <rms@gnu.org> writes:

> If a package is useful to recommend, and we CAN get papers for it, we
> definitely want to move it to GNU ELPA.

Yes, this is why I want to clearly define ELPA policy, and streamline things
as much as possible for developers and users: so that it becomes a more
attractive means for distributing Emacs packages. I think many people may be
largely ignoring it right now, and so they reach for MELPA. Since so many
people contribute to MELPA, they consider it a more attractive distribution
platform.

John

Re: ELPA policy

[Phillip Lord]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> Richard Stallman <rms@gnu.org> writes:
>
>> If a package is useful to recommend, and we CAN get papers for it, we
>> definitely want to move it to GNU ELPA.
>
> Yes, this is why I want to clearly define ELPA policy, and streamline things
> as much as possible for developers and users: so that it becomes a more
> attractive means for distributing Emacs packages. I think many people may be
> largely ignoring it right now, and so they reach for MELPA. Since so many
> people contribute to MELPA, they consider it a more attractive distribution
> platform.


MELPA is also *easier* to contribute to. Aside from copyright issues, it
involves writing something that looks like lisp, testing on your local
fork, then a PR.

With GNU ELPA, it involves some fairly obscure git cleverness. GNU ELPA
could be made easier by allowing recipes, and by accepting PRs. This
avoids the necessity to have commit access to GNU ELPA to contribute.

Finally, I think that core devs should contribute to individual packages
by PR to their repos.

Phil

Re: ELPA policy

[John Wiegley]

>>>>> Phillip Lord <phillip.lord@russet.org.uk> writes:

> With GNU ELPA, it involves some fairly obscure git cleverness. GNU ELPA
> could be made easier by allowing recipes, and by accepting PRs. This avoids
> the necessity to have commit access to GNU ELPA to contribute.

We can always use the power of Git here: Keep a fork of ELPA.git on GitHub,
accept PRs there, and periodically merge back in.

John

Re: ELPA policy

[Stephen Leake]

phillip.lord@russet.org.uk (Phillip Lord) writes:

> John Wiegley <jwiegley@gmail.com> writes:
>
>>>>>>> Richard Stallman <rms@gnu.org> writes:
>>
>>> If a package is useful to recommend, and we CAN get papers for it, we
>>> definitely want to move it to GNU ELPA.
>>
>> Yes, this is why I want to clearly define ELPA policy, and streamline things
>> as much as possible for developers and users: so that it becomes a more
>> attractive means for distributing Emacs packages. I think many people may be
>> largely ignoring it right now, and so they reach for MELPA. Since so many
>> people contribute to MELPA, they consider it a more attractive distribution
>> platform.
>
>
> MELPA is also *easier* to contribute to. Aside from copyright issues, it
> involves writing something that looks like lisp, testing on your local
> fork, then a PR.
>
> With GNU ELPA, it involves some fairly obscure git cleverness. 

It's only normal git cleverness; there's nothing special about the Gnu
ELPA git. Unless you are using an "external" package, perhaps.

I can understand treating all of git as "obscure", if that's what you
meant. I much prefer monotone.

> GNU ELPA could be made easier by allowing recipes, and by accepting
> PRs. This avoids the necessity to have commit access to GNU ELPA to
> contribute.

"PR" is "Pull Request"? Doesn't that mean "pull from my git repository"?
Or is it more general than that?

> Finally, I think that core devs should contribute to individual packages
> by PR to their repos.

I'm guessing "their repos" is _not_ Gnu ELPA git? So this only applies
to packages that have a primary repo that is not Gnu ELPA git.

I agree that any one that edits a Gnu ELPA package in Gnu ELPA git
should also notify the upstream project if there is one. But they do not
have to wait for approval if it's a critical bug.

Normal ettiquette should apply of course.

This is more important for tarball and core ELPA packages, since they
are part of the blessed Emacs standard library; Emacs developers can
edit those as if they were in core.

-- 
-- Stephe

Re: ELPA policy

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > MELPA is also *easier* to contribute to. Aside from copyright issues, it
  > involves writing something that looks like lisp, testing on your local
  > fork, then a PR.

What is a PR?

  > With GNU ELPA, it involves some fairly obscure git cleverness. GNU ELPA
  > could be made easier by allowing recipes, and by accepting PRs. This
  > avoids the necessity to have commit access to GNU ELPA to contribute.

Maybe we should do this (but I don't know what a PR is).

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: ELPA policy

[JJ Asghar]

On 11/13/15 3:58 PM, Richard Stallman wrote:
>   > MELPA is also *easier* to contribute to. Aside from copyright issues, it
>   > involves writing something that looks like lisp, testing on your local
>   > fork, then a PR.
> 
> What is a PR?

A great detailed explanation of a PR is here[1]. A short explanation is
a request to put in a patch to a repository.

[1]: https://help.github.com/articles/using-pull-requests/

-JJ

Re: ELPA policy

[Jorge A. Alfaro-Murillo]

Stephen Leake writes:
 
> Including a package in tarball ELPA means that the Emacs project 
> has decided that the package is sufficiently important to be 
> made immediately available to all Emacs users with no effort on 
> their part beyond installing Emacs from the tarball. 
> 
> That implies a certain level of maturity and quality, and 
> implies that it should be favored over other similar packages.

Would that just be packages that are currently in core? Because 
AUCTeX definitely fits that description. I have always been 
curious as why it is not in core currently.

Best,
-- 
Jorge.

Re: ELPA policy

[John Wiegley]

>>>>> Jorge A Alfaro-Murillo <jorge.alfaro-murillo@yale.edu> writes:

>> Including a package in tarball ELPA means that the Emacs project has
>> decided that the package is sufficiently important to be made immediately
>> available to all Emacs users with no effort on their part beyond installing
>> Emacs from the tarball.
>> 
>> That implies a certain level of maturity and quality, and implies that it
>> should be favored over other similar packages.

> Would that just be packages that are currently in core? Because AUCTeX
> definitely fits that description. I have always been curious as why it is
> not in core currently.

No, it wouldn't be just packages currently in core, we could consider others
for inclusion as well. AucTeX might be an excellent candidate.

I like that ELPA could give us better flexibility about what goes in the
tarball, without having to make decisions that directly affect core.

John

RE: ELPA policy

[Drew Adams]

> We would like developers of future useful packages to want to put
> their packages into the GNU ELPA.
> 
> To recommend putting some packages -- whatever they might be -- into
> MELPA would work against these goals.  It would have been a bad
> approach.

FWIW -

I will continue to use MELPA for my libraries, and not GNU ELPA,
for one reason:

I post my code to Emacs Wiki (to locked pages).  MELPA pulls
the code from the wiki (daily).  I do not need to use GIT etc.

This is the way I prefer to work.  Call it old-fashioned or
bizarre; I don't care.

So you might recommend that packages not be put in MELPA, but
some will continue to be put there, including perhaps some that
you might someday want to include in Emacs.

>   > Which raises the question; what is the rationale for moving from
>   > MELPA to Gnu ELPA?
> 
> If a package is useful to recommend, and we CAN get papers for it,
> we definitely want to move it to GNU ELPA.

This was refused by Stefan for my libraries (for which FSF has papers),
based on the fact that I do not use GIT, so any updates I make to them
would not be done directly in the repository.  It was not acceptable to
update elsewhere (e.g. the wiki) and then have someone or a program
pull from there to the repository when appropriate.  So be it.

Re: ELPA policy

[John Wiegley]

>>>>> Drew Adams <drew.adams@oracle.com> writes:

> I will continue to use MELPA for my libraries, and not GNU ELPA, for one
> reason:
>
> I post my code to Emacs Wiki (to locked pages). MELPA pulls the code from
> the wiki (daily). I do not need to use GIT etc.
>
> This is the way I prefer to work. Call it old-fashioned or bizarre; I don't
> care.

This is a legitimate complaint, Drew. One of the things I want to avoid with
ELPA is making it harder for people to contribute to than other alternatives.

Since pulling in data from the Web indiscriminately is not possible for this
project (we have to make sure we're not unwittingly including code without a
proper copyright assignment), perhaps we need as an "integrator": someone
willing to guide the update of ELPA from various sources on the Web, keeping
an eye out for changes that might affect copyright.

It wouldn't be too hard to setup a process that updates a local clone of ELPA
with all your latest versions from GitHub and EmacsWiki, and then to check
over a diff of those changes before pushing them to ELPA. One might even
suggest that you could do this yourself, as part of your own release work.
Would that be something you'd find worth doing?

John

RE: ELPA policy

[Drew Adams]

> This is a legitimate complaint, Drew. One of the things I want to avoid
> with ELPA is making it harder for people to contribute to than other
> alternatives.
> 
> Since pulling in data from the Web indiscriminately is not possible for
> this project (we have to make sure we're not unwittingly including code without
> a proper copyright assignment), perhaps we need as an "integrator": someone
> willing to guide the update of ELPA from various sources on the Web,
> keeping an eye out for changes that might affect copyright.

I'm glad to hear that; all of it.  And thanks for replying.

> It wouldn't be too hard to setup a process that updates a local clone of
> ELPA with all your latest versions from GitHub and EmacsWiki, and then to check
> over a diff of those changes before pushing them to ELPA. One might even
> suggest that you could do this yourself, as part of your own release work.
> Would that be something you'd find worth doing?

I don't want to develop anything like that; sorry.  I don't want to fiddle
with a local GIT clone.  I would be willing to identify versions of my
libraries to be pulled at specific times ("releases") from the wiki (or
from MELPA) to be put (updated) in GNU ELPA.

I doubt that my case is typical, however.  I'm not saying that Emacs Dev
needs to cater to such a case.  I'm just explaining that I update my
libraries locally, without GIT etc., and I upload updates to the wiki.
They are automatically pulled to MELPA.

What do I need to know, currently?  Only how to write some Lisp (a
library) and how to copy and paste it into a Web page.  Simple.  Maybe
too simple to accommodate by Emacs Dev.

Re: ELPA policy

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Since pulling in data from the Web indiscriminately is not
  > possible for this project (we have to make sure we're not
  > unwittingly including code without a proper copyright assignment),
  > perhaps we need as an "integrator": someone willing to guide the
  > update of ELPA from various sources on the Web, keeping an eye out
  > for changes that might affect copyright.

What we need to solve Drew's problem is to merge changes into a few
specific files from a specific place, and commit them using git.
Isn't that easy enough to do?

There could be one page where he provides the commit log info.
Whenever that page changes (we could check every 5 minutes), our demon
could merge in all the files which have changed.  They it would email
the diffs to Drew so he could confirm what he installed.

Thus, when Drew wants to install a new version, he would change the
other pages first, then write the change log text into the log page.
Then wait 5 minutes and it's done.

This is assuming there are no other obstacles aside from the
mechanics of installing changes into our repository.



-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: ELPA policy

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 787 bytes --]

On 11 Nov 2015 11:27 pm, "Richard Stallman" <rms@gnu.org> wrote:
>
> There could be one page where he provides the commit log info.
> Whenever that page changes (we could check every 5 minutes), our demon
> could merge in all the files which have changed.  They it would email
> the diffs to Drew so he could confirm what he installed.

Richard, do think such as system would be viable copyright-wise?
Of course, we would only do this when the author assigns copyright, but
there's a very real possibility that a careless maintainer might add
someone else's code to this page. And then this (non-copyright-assigned)
code would be automatically pulled and committed to Elpa.

I ask this because if we are OK with this risk then I'm perfectly willing
to write such a script (eager, even).

[-- Attachment #2: Type: text/html, Size: 949 bytes --]

Re: ELPA policy

[Dmitry Gutov]

On 11/12/2015 02:35 AM, Artur Malabarba wrote:

> Richard, do think such as system would be viable copyright-wise?
> Of course, we would only do this when the author assigns copyright, but
> there's a very real possibility that a careless maintainer might add
> someone else's code to this page. And then this (non-copyright-assigned)
> code would be automatically pulled and committed to Elpa.

We should also consider whether this is sound from the security 
standpoint. I mean, pushing the code to ELPA without any review from the 
core developers.

Re: ELPA policy

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Richard, do think such as system would be viable copyright-wise?

In terms of copyright, there is no difference between the system I
proposed (installing what Drew puts in certain web pages) and the
method Drew doesn't like, which is for him to install the same changes
via Git.

Whichever method he uses, he technically could install code written by
others whom we don't have papers for.  If we permit him to use Git to
install changes, that entails trusting him to handle this
responsibility.  If we can trust him to do it with one method, we can
trust him to do it with the other method.

  > We should also consider whether this is sound from the security 
  > standpoint. I mean, pushing the code to ELPA without any review from the 
  > core developers.

The same would occur if we invite him to install changes in his packages
through Git.

If we can check enough one way, we can check enough the other way.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: ELPA policy

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 937 bytes --]

On 12 Nov 2015 10:34 pm, "Richard Stallman" <rms@gnu.org> wrote:
>   > Richard, do think such as system would be viable copyright-wise?
>
> In terms of copyright, there is no difference between the system I
> proposed (installing what Drew puts in certain web pages) and the
> method Drew doesn't like, which is for him to install the same changes
> via Git.
>
> Whichever method he uses, he technically could install code written by
> others whom we don't have papers for.  If we permit him to use Git to
> install changes, that entails trusting him to handle this
> responsibility.  If we can trust him to do it with one method, we can
> trust him to do it with the other method.

I'm trying to determine whether this method can be employed as a general
elpa mechanism. Of course we trust Drew, but would it be acceptable to use
such a script with other people who submit packages to Elpa? (assuming the
author has assigned copyright)

[-- Attachment #2: Type: text/html, Size: 1135 bytes --]

Re: ELPA policy

[Stephen Leake]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > Since pulling in data from the Web indiscriminately is not
>   > possible for this project (we have to make sure we're not
>   > unwittingly including code without a proper copyright assignment),
>   > perhaps we need as an "integrator": someone willing to guide the
>   > update of ELPA from various sources on the Web, keeping an eye out
>   > for changes that might affect copyright.
>
> What we need to solve Drew's problem is to merge changes into a few
> specific files from a specific place, and commit them using git.
> Isn't that easy enough to do?
>
> There could be one page where he provides the commit log info.
> Whenever that page changes (we could check every 5 minutes), our demon
> could merge in all the files which have changed.  They it would email
> the diffs to Drew so he could confirm what he installed.
>
> Thus, when Drew wants to install a new version, he would change the
> other pages first, then write the change log text into the log page.
> Then wait 5 minutes and it's done.
>
> This is assuming there are no other obstacles aside from the
> mechanics of installing changes into our repository.

Any malicious hacker can drop completely different code in that web
page, and thus get it into Gnu ELPA.

We will have replaced the security of private machines with whatever web
login that web page requires; that's a huge step backwards.

-- 
-- Stephe

RE: ELPA policy

[Drew Adams]

> Any malicious hacker can drop completely different code in that web
> page, and thus get it into Gnu ELPA.  We will have replaced the
> security of private machines with whatever web login that web page
> requires; that's a huge step backwards.

Again: "I post my code to Emacs Wiki (to locked pages)."

But of course that doesn't mean it is uncrackable.

Re: ELPA policy

[Alex Schröder]

Drew Adams <drew.adams@oracle.com> writes:

> Again: "I post my code to Emacs Wiki (to locked pages)."
>
> But of course that doesn't mean it is uncrackable.

Given that Emacs Wiki currently does not use HTTPS, we should consider
Emacs Wiki insecure. Even then, the admin passwords are shared and there
are no other efforts made to identify editors once they present the
admin password. The current setup only protects against the simplest of
threat models, e.g. spammers, vandals, well-meaning contributors that
want to edit wiki pages instead of informing maintainers, and so on. I
think that's the threat model wiki pages need to defend against, but I
suspect that's not good enough for an inclusion into Emacs Core.

-- 
Public Key Fingerprint = DF94 46EB 7B78 4638 7CCC  018B C78C A29B ACEC FEAE

Re: ELPA policy

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Any malicious hacker can drop completely different code in that web
  > page, and thus get it into Gnu ELPA.

Drew said the pages were locked.
Doesn't that mean that only he has access to change them?

  > We will have replaced the security of private machines with whatever web
  > login that web page requires; that's a huge step backwards.

I think you are concerned that someone might break the security on that other
server and then install changes on it using Drew's account.

In general, someone who breaks the security on a machine used by
an Emacs contributor might be able to insert changes in Emacs
by pretending to be that contributor.  I don't think this is
fundamentally different.  But maybe the web site's security is
not quite as good.

We can make the security tighter.  Drew, are you willing to GPG-sign
your new versions?

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: ELPA policy

[Stephen Leake]

Richard Stallman <rms@gnu.org> writes:

>   > We will have replaced the security of private machines with whatever web
>   > login that web page requires; that's a huge step backwards.
>
> I think you are concerned that someone might break the security on that other
> server and then install changes on it using Drew's account.
>
> In general, someone who breaks the security on a machine used by
> an Emacs contributor might be able to insert changes in Emacs
> by pretending to be that contributor.  I don't think this is
> fundamentally different.  But maybe the web site's security is
> not quite as good.

Yes, my concern is about the strength of the web security, as opposed to
the strength of Drew's private machine.

I suppose there's no a priori reason to assume one is more secure than
the other; maybe I'm just reacting to all the news reports about hacking.

-- 
-- Stephe

Re: ELPA policy

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I suppose there's no a priori reason to assume one is more secure than
  > the other; maybe I'm just reacting to all the news reports about hacking.

Would you please call that "cracking"?  Hacking is what we do as Emacs
developers.  See http://stallman.org/articles/on-hacking.html.

Anyway, now I understand your concern, and I think it's a valid one.
That's why I've proposed another solution.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

RE: ELPA policy

[Drew Adams]

>   > Any malicious hacker can drop completely different code in that web
>   > page, and thus get it into Gnu ELPA.
> 
> Drew said the pages were locked.
> Doesn't that mean that only he has access to change them?

No, anyone with admin privileges for the wiki has access to do so.
There are a few people in this category.  And see Alex Schroeder's
clarification of what this means.  This is not watertight security,
by any means.

Perhaps one way to look at it is similar to submitting something
by email (which would be another possibility, for me).

>   > We will have replaced the security of private machines with whatever
>   > web login that web page requires; that's a huge step backwards.
> 
> I think you are concerned that someone might break the security on that
> other server and then install changes on it using Drew's account.

See above.

> In general, someone who breaks the security on a machine used by
> an Emacs contributor might be able to insert changes in Emacs
> by pretending to be that contributor.  I don't think this is
> fundamentally different.  But maybe the web site's security is
> not quite as good.
> 
> We can make the security tighter.  Drew, are you willing to GPG-sign
> your new versions?

I don't really know what that entails.

Dunno whether you really want to discuss my case in particular in
detail here.  Again, I doubt that it is typical.  The reason for
my initial message about this was to suggest that some people do
use MELPA, and that perhaps some way to accommodate them could be
devised.  But maybe not.

To repeat the summary of my original point:

  So you might recommend that packages not be put in MELPA, but
  some will continue to be put there, including perhaps some that
  you might someday want to include in Emacs.

Regarding my own case, this was the point:

  I do not use GIT, so any updates I make to them
  would not be done directly in the repository.  It was not
  acceptable to update elsewhere (e.g. the wiki) and then have
  someone or a program pull from there to the repository when
  appropriate.

In sum, some people will post code to MELPA, including some
that you might someday want in Emacs.  And some input to MELPA
comes from the wiki, not from GIT - but this is probably a small
portion of what is in MELPA.

Re: ELPA policy

[Eli Zaretskii]

> Date: Thu, 12 Nov 2015 15:05:50 -0800 (PST)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: jwiegley@gmail.com, drew.adams@oracle.com, emacs-devel@gnu.org
> 
> Regarding my own case, this was the point:
> 
>   I do not use GIT, so any updates I make to them
>   would not be done directly in the repository.  It was not
>   acceptable to update elsewhere (e.g. the wiki) and then have
>   someone or a program pull from there to the repository when
>   appropriate.
> 
> In sum, some people will post code to MELPA, including some
> that you might someday want in Emacs.  And some input to MELPA
> comes from the wiki, not from GIT - but this is probably a small
> portion of what is in MELPA.

FWIW, I very much doubt that many people nowadays would share Drew's
issue with using Git.  Using Git is a no-brainer nowadays.  I actually
think this is a very unique situation, and anyone else who insists on
using MELPA doesn't do that for such simple technical reasons.  I
think the legal paperwork is the main issue, and also perhaps not
everyone would like to be accountable for their code practices before
the Emacs maintainers and their peculiar requirements and
expectations.

Re: ELPA policy

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > We can make the security tighter.  Drew, are you willing to GPG-sign
  > > your new versions?

  > I don't really know what that entails.

It means you would make diffs to show the change to install,
plus a log entry, sign that whole thing with GnuPG, and mail
it to a certain address.  The signature would demonstrate it came
from you.

Is that ok for you?

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

RE: ELPA policy

[Drew Adams]

>   > > We can make the security tighter.  Drew, are you willing
>   > > to GPG-sign your new versions?
> 
>   > I don't really know what that entails.
> 
> It means you would make diffs to show the change to install,
> plus a log entry, sign that whole thing with GnuPG, and mail
> it to a certain address.  The signature would demonstrate it
> came from you.
> 
> Is that ok for you?

I have no problem creating diffs & log entries, and emailing
them in.  I've mailed in plenty of patches that way.

Dunno what "sign that whole thing with GnuPG" entails.  What
is needed for that (e.g., on MS Windows)?

Looking at the doc at https://www.gnupg.org/documentation/howtos.html
and http://www.dewinter.com/gnupg_howto/english/GPGMiniHowto-2.html,
it seems I would need to install GnuPG from source code,
using a C compiler etc.  I don't plan to install and use tools
such as a C compiler on my laptop, so I guess the answer would
be no; sorry.

If there were an easy-to-use Windows binary for such signing,
then I would probably do as you propose.

(I don't do software development anymore.  I use MS Windows
binaries of Emacs, and I write some Emacs-Lisp code.  I do
have (an old version of) Cygwin installed, so that I can use
a few simple commands from within Emacs.  That's about it.)

I don't mean to make a big deal about this.  I'm not saying
no definitively.  But AFAIK there is not necessarily even
any interest in adding any of the code I've written to Emacs.
I don't feel like going through a lot of preparation for
nothing.

My libraries are there, and have been for quite a while.
If someone is really interested in something, let me know
and maybe we can work something out.  Otherwise, we are
likely wasting each other's time here.  I don't mean to
waste your time.  (But I do appreciate your having
considered this.)

Again, my original point was about MELPA and updates to
it from the wiki.  And again, my use of MELPA is likely
not typical.  Perhaps I should have said nothing at all
about it.

Re: ELPA policy

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Dunno what "sign that whole thing with GnuPG" entails.  What
  > is needed for that (e.g., on MS Windows)?

GnuPG works on Windows.  I expect we can find someone to help
you install it.  Then we need you to send us your public key
so we can verify your signatures.  We can find a way to do that.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: ELPA policy

[Michael Heerdegen]

Richard Stallman <rms@gnu.org> writes:

> GnuPG works on Windows.  I expect we can find someone to help
> you install it.  Then we need you to send us your public key
> so we can verify your signatures.  We can find a way to do that.

I think that's not much easier for Drew than to use git.

I know Drew doesn't want to use git in his workflow (which he doesn't
have to), he does not dislike git for political reasons or so AFAIR.

Once he has set up things correctly (we can help him with that), he just
has to change the file in his repo clone, pull, add and push.  That's not
more complicated than what he does now when uploading to the wiki.  He
doesn't have to use git for anything else, or while developing.

Does that sound acceptable to you, Drew?  I can help you with any
problems you might encounter.  After everything has been set up
properly, uploading to Gnu Elpa is trivial, you don't have to learn git
for that.


Regards,

Michael.

RE: ELPA policy

[Drew Adams]

> Once he has set up things correctly (we can help him with that), he
> just has to change the file in his repo clone, pull, add and push.  
> That's not more complicated than what he does now when uploading to
> the wiki.  He doesn't have to use git for anything else, or while 
> developing.
> 
> Does that sound acceptable to you, Drew?  I can help you with any
> problems you might encounter.  After everything has been set up
> properly, uploading to Gnu Elpa is trivial, you don't have to
> learn git for that.

I will take a look.  Please follow up with me off-list.

(I've tried to make clear that my reply to this thread was
not about getting my libraries into Emacs, but was about
"MELPA and updates to it from the wiki."  And as I said,
"AFAIK there is not necessarily even any interest in adding
any of the code I've written to Emacs."  This has largely
been much ado about nothing, I think.  I'm sorry for the
resulting waste of people's time.  But I'm glad that the
question of MELPA and wiki code was at least considered
to some extent.)

Re: ELPA policy

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I don't think this is "much ado about nothing".
The reasons that affect one person may affect others.

We want people to arrange to contribute their Emacs add-ons
to Emacs, and that suggests that whenever someone says a reason
why he didn't want to put his code in GNU ELPA, we should try
to correct it.



-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: ELPA policy

[John Wiegley]

>>>>> Richard Stallman <rms@gnu.org> writes:

> We want people to arrange to contribute their Emacs add-ons to Emacs, and
> that suggests that whenever someone says a reason why he didn't want to put
> his code in GNU ELPA, we should try to correct it.

Agreed. ELPA should become a more positive experience, both for developers,
contributors and users.

John

Re: ELPA policy

[Artur Malabarba]

2015-11-17 23:17 GMT+00:00 John Wiegley <jwiegley@gmail.com>:
>>>>>> Richard Stallman <rms@gnu.org> writes:
>
>> We want people to arrange to contribute their Emacs add-ons to Emacs, and
>> that suggests that whenever someone says a reason why he didn't want to put
>> his code in GNU ELPA, we should try to correct it.
>
> Agreed. ELPA should become a more positive experience, both for developers,
> contributors and users.

I'm deeply in favor of making it easier for people to contribute code
to Elpa, and (like I said), I'm willing to help write the code for
that.
But I think the hardest part of the problem is not being discussed here.

Whatever this method is, it needs to meet two criteria simultaneously:
1. Not commit any code to Elpa until someone with push access has OK'd
it. This is just to ensure nothing malicious is being done.
2. Be fairly automated (not completely), so that the Elpa maintainers
don't have to manually commit+push all code that gets sent to them.
This is to preserve the sanity of the maintainers. The current model
where everything is commited by us just isn't scalable.

While those two points are not mutually exclusive, they are quite
difficult to reconcile. It needs to show the maintainers a diff of the
changes and then only proceed with the commit after receiving some
very minimal interaction (a reply to an email, or the click of a
button).

Re: ELPA policy

[David Kastrup]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> 2015-11-17 23:17 GMT+00:00 John Wiegley <jwiegley@gmail.com>:
>>>>>>> Richard Stallman <rms@gnu.org> writes:
>>
>>> We want people to arrange to contribute their Emacs add-ons to Emacs, and
>>> that suggests that whenever someone says a reason why he didn't want to put
>>> his code in GNU ELPA, we should try to correct it.
>>
>> Agreed. ELPA should become a more positive experience, both for developers,
>> contributors and users.
>
> I'm deeply in favor of making it easier for people to contribute code
> to Elpa, and (like I said), I'm willing to help write the code for
> that.
> But I think the hardest part of the problem is not being discussed here.
>
> Whatever this method is, it needs to meet two criteria simultaneously:
> 1. Not commit any code to Elpa until someone with push access has OK'd
> it. This is just to ensure nothing malicious is being done.
> 2. Be fairly automated (not completely), so that the Elpa maintainers
> don't have to manually commit+push all code that gets sent to them.
> This is to preserve the sanity of the maintainers. The current model
> where everything is commited by us just isn't scalable.
>
> While those two points are not mutually exclusive, they are quite
> difficult to reconcile. It needs to show the maintainers a diff of the
> changes and then only proceed with the commit after receiving some
> very minimal interaction (a reply to an email, or the click of a
> button).

With the GNU ELPA requiring copyright assignments, I doubt that we are
going to hit scalability issues of the kind you fear.  But of course
there is nothing wrong with trying to create a contributing process that
is minimal effort for the sake of such archives where this is not the
case.

-- 
David Kastrup

Re: ELPA policy

[Achim Gratz]

Drew Adams writes:
> Dunno what "sign that whole thing with GnuPG" entails.  What
> is needed for that (e.g., on MS Windows)?

http://www.gpg4win.org/

> (I don't do software development anymore.  I use MS Windows
> binaries of Emacs, and I write some Emacs-Lisp code.  I do
> have (an old version of) Cygwin installed, so that I can use
> a few simple commands from within Emacs.  That's about it.)

You'd really be off easier if you updated Cygwin so that you could use
Git and SSH and GPG from that.  Perhaps you'll find that using Emacs
from Cygwin is a lot easier now, too -- you can use emacs-w32 if you
don't care about running X11.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf microQ V2.22R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

RE: ELPA policy

[Drew Adams]

>   > Since pulling in data from the Web indiscriminately is not
>   > possible for this project (we have to make sure we're not
>   > unwittingly including code without a proper copyright assignment),
>   > perhaps we need as an "integrator": someone willing to guide the
>   > update of ELPA from various sources on the Web, keeping an eye out
>   > for changes that might affect copyright.
> 
> What we need to solve Drew's problem is to merge changes into a few
> specific files from a specific place, and commit them using git.
> Isn't that easy enough to do?
> 
> There could be one page where he provides the commit log info.
> Whenever that page changes (we could check every 5 minutes), our demon
> could merge in all the files which have changed.  They it would email
> the diffs to Drew so he could confirm what he installed.
> 
> Thus, when Drew wants to install a new version, he would change the
> other pages first, then write the change log text into the log page.
> Then wait 5 minutes and it's done.
> 
> This is assuming there are no other obstacles aside from the
> mechanics of installing changes into our repository.

I will again say that I don't expect that my situation is typical.
There may be no one else out there who ever prefers something like it.

But FWIW I think that what you describe would work for me.

Re: ELPA policy

[Stephen Leake]

David Engster <deng@randomsample.de> writes:

> John Wiegley writes:
>>>>>>> David Engster <deng@randomsample.de> writes:
>>
>>> This is not about reaching a consensus. This is about you giving proper
>>> reasons for such a big change.
>>
>> To be clear, I would also put Eshell (though not pcomplete) into the category
>> of "big things that should be in ELPA" -- albeit, the subset of ELPA that will
>> be in the release tarball.
>>
>> It's hard to pin down why in a manner that fits in an e-mail. If Eshell were
>> in ELPA today, and we were talking about moving it into core, I would have
>> just as much trouble justifying that move too. Perhaps this simply underscores
>> the fact that we don't have an agreed upon ELPA policy we can all refer to.
>
> In my opinion, the main question is whether something provides
> infrastructure for other packages to use. 

I think it is a reasonable goal to allow ELPA packages to serve in that
role, as long as the "other packages" are also normal or tarball ELPA
packages.

> This is precisely what CEDET tries to do.

Yes.

However, now the shoe is on the other foot: why must infrastructure
packages be in Emacs core?

I think the trigger is "some other _core_ code wants to depend on it".
That would trigger a move to Emacs git, as a core ELPA package (the
package could still have intermediate released via the Gnu ELPA server).

> I wouldn't have much trouble with putting parts of CEDET in ELPA,
> namely those parts that do not directly provide infrastructure, like
> support for certain languages, project types, indexing tools, etc.

Good, but let's not try to do that for Emacs 25; since we are trying to
get to feature freeze, it's too much.

> It is still not clear to me what exactly is gained by moving core
> packages to ELPA. 

One gain is making it clear that other core code is not allowed to depend
on it. This is in turn to ensure that it doesn't creep into the dumped
code. But I'm not sure that's an important reason.

-- 
-- Stephe

Re: ELPA policy

[Stephen Leake]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> David Engster <deng@randomsample.de> writes:
>
>> This is not about reaching a consensus. This is about you giving proper
>> reasons for such a big change.
>
> To be clear, I would also put Eshell (though not pcomplete) into the category
> of "big things that should be in ELPA" -- albeit, the subset of ELPA that will
> be in the release tarball.
>
> It's hard to pin down why in a manner that fits in an e-mail. If Eshell were
> in ELPA today, and we were talking about moving it into core, I would have
> just as much trouble justifying that move too. Perhaps this simply underscores
> the fact that we don't have an agreed upon ELPA policy we can all
> refer to.

Yes.

> OK, David, I won't decide this by fiat. Let us decide what our ELPA policy
> will be, until it becomes clear, based on that document, what should go where,
> and why. We'll defer discussions of package movement until we have that in
> place.

Thank you.


-- 
-- Stephe

Re: ELPA policy

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Cc: Stephen Leake <stephen_leake@stephe-leake.org>,  aaronecay@gmail.com,  Eli Zaretskii <eliz@gnu.org>,  Stromeko@nexgo.de,  Dmitry Gutov <dgutov@yandex.ru>,  emacs-devel@gnu.org
> Date: Tue, 10 Nov 2015 10:52:41 -0800
> 
> > Why?
> 
> There will never be 100% agreement on whether they should be in ELPA, or be in
> Core, so I'm making the decision that they belong in ELPA.

IMO, it's a mistake to move CEDET.

Re: ELPA policy

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: John Wiegley <jwiegley@gmail.com>
>> Cc: Stephen Leake <stephen_leake@stephe-leake.org>,
>> aaronecay@gmail.com, Eli Zaretskii <eliz@gnu.org>,
>> Stromeko@nexgo.de, Dmitry Gutov <dgutov@yandex.ru>,
>> emacs-devel@gnu.org
>> Date: Tue, 10 Nov 2015 10:52:41 -0800
>> 
>> > Why?
>> 
>> There will never be 100% agreement on whether they should be in ELPA, or be in
>> Core, so I'm making the decision that they belong in ELPA.
>
> IMO, it's a mistake to move CEDET.

Bald statements of preference, without rationale, do not help move this
discussion forward.

We are trying to establish a general ELPA policy, not just vote on one
particular package.

What is it about CEDET that makes it a poor candidate?

How would your workflow suffer if it was moved?

How would users suffer?

-- 
-- Stephe

Re: ELPA policy

[Stephen Leake]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> David Engster <deng@randomsample.de> writes:
>
>>> Large packages like CEDET should move outside of Emacs.git and into
>>> Elpa.git.
>
>> Why?
>
> There will never be 100% agreement on whether they should be in ELPA, or be in
> Core, so I'm making the decision that they belong in ELPA. 

I'm happy to have someone making decisions, but there needs to be some
rationale given, so we understand the tradeoffs.

I believe the rationale here is:

- CEDET is not used by other core Emacs code (xref.el will be fixed)

- code that is not used by other core Emacs code clutters the workspace,
  producing extra hits in greps, etc.

- CEDET can still be included in the Emacs tarball, so users do
  not have to use list-packages to install it (this requires a to be
  implemented mechanism).

>> From my experience, git submodules are never seamless to work with.
>
> Even still, they work perfectly fine for this purpose.

I would like to see an actual demonstration, please; there are lots of
devils in the details.

Which means:

Please post a clear recipe for setting up an Emacs + ELPA developer
workspace, starting from git clone, so I can try this on my machine.


The test is then that some core Emacs code depend on some ELPA package;
we'd need branches in both to make that happen, since it doesn't now.

-- 
-- Stephe

Re: ELPA policy

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  aaronecay@gmail.com,  Stromeko@nexgo.de,  emacs-devel@gnu.org,  Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 10 Nov 2015 10:28:26 -0800
> 
> Large packages like CEDET should move outside of Emacs.git and into Elpa.git.

"Should" based on what? just the fact that it's large?  I think we
should decide this stuff on a case by case basis.  For example:

> If xref.el depends on CEDET, it would move to Elpa.git as well.

IMO, the exact opposite: if there are core features that we want to be
in Emacs no matter what, and those features depend on a package which
could be a candidate to move to ELPA, that package should NOT move to
ELPA.

Re: ELPA policy

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

>> Large packages like CEDET should move outside of Emacs.git and into
>> Elpa.git.
> "Should" based on what? just the fact that it's large? I think we should
> decide this stuff on a case by case basis. For example:

I'm surprised you say this, Eli, because in another thread you agree that
packages like this should be in Elpa, didn't you?

>> If xref.el depends on CEDET, it would move to Elpa.git as well.

> IMO, the exact opposite: if there are core features that we want to be in
> Emacs no matter what, and those features depend on a package which could be
> a candidate to move to ELPA, that package should NOT move to ELPA.

Core should provide functionality along the lines of a "standard library" and
a "standard environment", where having them in core is as much a statement
about consistency of interface, as it is about universal availability of the
functionality.

Since xref.el does not need to depend on CEDET, I don't see a reason why it
should, causing CEDET to remain in core.

John

Re: ELPA policy

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Cc: stephen_leake@stephe-leake.org,  aaronecay@gmail.com,  Stromeko@nexgo.de,  emacs-devel@gnu.org,  dgutov@yandex.ru
> Date: Tue, 10 Nov 2015 10:54:51 -0800
> 
> >>>>> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Large packages like CEDET should move outside of Emacs.git and into
> >> Elpa.git.
> > "Should" based on what? just the fact that it's large? I think we should
> > decide this stuff on a case by case basis. For example:
> 
> I'm surprised you say this, Eli, because in another thread you agree that
> packages like this should be in Elpa, didn't you?

I said I _could_agree.  But not unconditionally.  We should discuss
each case.

> >> If xref.el depends on CEDET, it would move to Elpa.git as well.
> 
> > IMO, the exact opposite: if there are core features that we want to be in
> > Emacs no matter what, and those features depend on a package which could be
> > a candidate to move to ELPA, that package should NOT move to ELPA.
> 
> Core should provide functionality along the lines of a "standard library" and
> a "standard environment", where having them in core is as much a statement
> about consistency of interface, as it is about universal availability of the
> functionality.

Sorry, I don't understand what this means in practice.

> Since xref.el does not need to depend on CEDET, I don't see a reason why it
> should, causing CEDET to remain in core.

xref is as useful as its backends.  If you take away backends, it
becomes less useful, or supports less programming languages, or both.

Re: ELPA policy

[Eli Zaretskii]

How about if we leave this issue alone for a while?  Let's concentrate
on releasing Emacs 25.1 first, and return to this issue later.  Rome
wasn't built in one day, we don't have to rush.

WDYT?

Re: ELPA policy

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> How about if we leave this issue alone for a while? Let's concentrate on
> releasing Emacs 25.1 first, and return to this issue later. Rome wasn't
> built in one day, we don't have to rush.
> 
> WDYT?

Agreed. Let's table for now and revisit. If the suggested moves really trouble
core developers, we'll talk about it again before making a final decision.

John

Re: ELPA policy

[Dmitry Gutov]

On 11/10/2015 09:21 PM, Eli Zaretskii wrote:

> xref is as useful as its backends.  If you take away backends, it
> becomes less useful, or supports less programming languages, or both.

None of the xref backends use Semantic. There are still only two of 
them: elisp and etags.

When CEDET grows a backend, it can just as well add it when 
semantic-mode is enabled. No need to have it in the core for that.

Re: ELPA policy

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> Since xref.el does not need to depend on CEDET, I don't see a reason why it
>> should, causing CEDET to remain in core.
>
> xref is as useful as its backends.  If you take away backends, it
> becomes less useful, or supports less programming languages, or both.

Yes.

That's an argument for making the CEDET xref backend available
somewhere; I've committed to ensuring that.

But this discussion is about ELPA vs Emacs core:

Does it matter whether that backend is in ELPA or Emacs core?

-- 
-- Stephe

Re: ELPA policy

[Stephen Leake]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> Large packages like CEDET should move outside of Emacs.git and into
>>> Elpa.git.
>> "Should" based on what? just the fact that it's large? I think we should
>> decide this stuff on a case by case basis. For example:
>
> I'm surprised you say this, Eli, because in another thread you agree that
> packages like this should be in Elpa, didn't you?
>
>>> If xref.el depends on CEDET, it would move to Elpa.git as well.
>
>> IMO, the exact opposite: if there are core features that we want to be in
>> Emacs no matter what, and those features depend on a package which could be
>> a candidate to move to ELPA, that package should NOT move to ELPA.
>
> Core should provide functionality along the lines of a "standard library" and
> a "standard environment", where having them in core is as much a statement
> about consistency of interface, as it is about universal availability of the
> functionality.

Right. Since CEDET provides lots of interesting infrastrucure, I assumed
it was part of the Emacs standard library.

But I also see no reason not to have two levels of "standard
infrastructure"; one in core Emacs, on in ELPA.

That said, moving CEDET out of core to ELPA can be seen as a demotion.
For example, I was assuming that the EDE package system should be
prefered over any package system that is in ELPA, precisely because it
is in Emacs git. So I was spending my time improving EDE, rather than
some other ELPA package.

That aspect should be considered.

In other emails, I've said "CEDET is a tarball package; in
Emacs git solely to be included in the tarball".

Is it fair to add "also because it is approved as part of the standard
Emacs library"?

> Since xref.el does not need to depend on CEDET, I don't see a reason why it
> should, causing CEDET to remain in core.

I would object to moving CEDET to ELPA git if it were not easy to
refactor the current dependency to still be provided by an ELPA package.

-- 
-- Stephe

Re: ELPA policy

[Dmitry Gutov]

On 11/10/2015 08:49 PM, Eli Zaretskii wrote:

> IMO, the exact opposite: if there are core features that we want to be
> in Emacs no matter what, and those features depend on a package which
> could be a candidate to move to ELPA, that package should NOT move to
> ELPA.

xref only depends on a minor feature of CEDET (the code indexing tools 
integration), and I've done that in a quick-and-dirty way, as a proof of 
concept.

It would be better to have functionality like that in the core, 
independent of CEDET.

Re: ELPA policy

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: John Wiegley <jwiegley@gmail.com>
>> Cc: Eli Zaretskii <eliz@gnu.org>, aaronecay@gmail.com,
>> Stromeko@nexgo.de, emacs-devel@gnu.org, Dmitry Gutov
>> <dgutov@yandex.ru>
>> Date: Tue, 10 Nov 2015 10:28:26 -0800
>> 
>> Large packages like CEDET should move outside of Emacs.git and into Elpa.git.
>
> "Should" based on what? just the fact that it's large?  I think we
> should decide this stuff on a case by case basis.  For example:
>
>> If xref.el depends on CEDET, it would move to Elpa.git as well.
>
> IMO, the exact opposite: if there are core features that we want to be
> in Emacs no matter what, and those features depend on a package which
> could be a candidate to move to ELPA, that package should NOT move to
> ELPA.

In general, I agree.

However, there are two _independent_ aspects of "move to ELPA":

1) Move to ELPA git.

2) Be distributed by the ELPA package server.


If other core code depends on a package, it should not move to ELPA git
(unless the dependency is easily refactored, as in the CEDET/xgit case;
there will always be corner cases).

However, if there is a desire to do package releases between Emacs
releases, making the package available via the ELPA package server makes
sense.

I think this discussion is mainly focused on "move to ELPA git", but it
would help to be clear.

-- 
-- Stephe

Re: ELPA policy

[Stephen Leake]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> Stephen Leake <stephen_leake@stephe-leake.org> writes:
>
>> ELPA packages that other core code depends on (like CEDET; xref uses it -
>> called "core ELPA packages" hereinafter) have to be in every developer's
>> build environment everyday; the other core code has to see the current
>> version of the package, and it has to be the same for every developer.
>> 
>> If core ELPA packages are in the ELPA git repository, you would copy some
>> subtrees of the ELPA git workspace into the Emacs git workspace. People have
>> looked into doing this, but it gets messy and confusing. "git fetch" is
>> dealing with two upstreams for one workspace. There has been talk about git
>> submodules, etc, but nothing concrete.
>
> Elpa.git should be a submodule referenced from within Emacs.git (under
> "elpa").

That's overkill; there are many packages in Gnu ELPA that core code
should _not_ depend on.

We could try to have a list of approved packages in a file somewhere,
but I don't think we can enforce that; better to use git to enforce it.

We need a way to explicitly label and enforce which ELPA packages core
code can depend on. The only demonstrated way to do that is to keep them
in Emacs git.

> This allows us to expressly state which "version" of Elpa is expected
> to work with the current Emacs.git.

That's only required for the ELPA packages that core code can depend on.
For the other ELPA packages, we explicitly don't want to know if they
work with Emacs master; that's up to the individual package maintainers.
That's one of the primary purposes of ELPA.

> Large packages like CEDET should move outside of Emacs.git and into
> Elpa.git.

If this said "large packages that the rest of Emacs core does not depend
on ...", I could agree.

I just checked; the only other core code that depends on CEDET is
xref.el, which can be fixed. So I agree, CEDET should move to elpa git.

However, I assume the reason CEDET is in Emacs git is because there is a
desire to distribute it in the tarball; a sufficient number of users
expect it to be there without going thru package install. So the
mechanism to put ELPA packages in the tarball (but _not_ in the emacs
development workspace) must also be implemented. That should not be too
hard; no git submodules, just directory tree copy.

So this introduces a third kind of ELPA package: "distributed in Emacs
tarball". A "tarball package" for short?

Except that Eric maintains that there is more to "CEDET" in Emacs core
than just lisp/cedet/*. So to be clear, we are proposing to move only
lisp/cedet/* to elpa git.

> If xref.el depends on CEDET, it would move to Elpa.git as well.

As Dmitry points out, better to remove the dependency from the core
xref.el. Then we can add it back either in the ELPA CEDET package or in
a new xref-cedet ELPA package.

>> If we use the approach of keeping core ELPA packages in the Emacs git
>> repository, there is _zero_ friction for the current core Emacs developers;
>> the only change is in the ELPA config scripts.
>
> I do not want this code replication. 

How is it replication?

There are two styles of ELPA packages currently:

1) The only repository for the code is Gnu ELPA git

2) There is an external repository that the package maintainer uses;
   releases are pushed to Gnu ELPA git

The choice is up to the ELPA package maintainer.

The proposal is to change both to say:

1a) The only repository for the code is Gnu ELPA git or Gnu Emacs git.

2a) There is an external repository that the package maintainer uses;
    releases are pushed to Gnu ELPA git or Gnu Emacs git.

The ELPA packages in Emacs git are "core ELPA packages"; core Emacs code
may rely on them. 

There is no additional replication by keeping the core ELPA packages in
Emacs git.

The only reason the core Emacs packages are in the ELPA server is to
allow package releases in addition to Emacs releases. The package
maintainers have to accept maintaining backward compatibility, so a core
package in an Emacs 25.1 binary install can work with a more recent
version of the core ELPA package.

In summary, I'm proposing that there are three kinds of ELPA packages:

1) "normal ELPA package". 

    - Stored in Gnu ELPA git, and possibly in the package maintainer's
      separate repository.

    - Releases only occur via the ELPA package server, whenever the
      version number increments in the ELPA git main package file.
    
    - Core Emacs code may not depend on these.

2) "tarball ELPA package". 

    - Stored in Gnu ELPA git, and possibly in the package maintainer's
      separate repository.

    - Each Emacs release includes a released version of the package in
      the Emacs tarball, via copy from an ELPA workspace at Emacs
      tarball build time. Other releases also occur, via the ELPA
      package server, whenever the version number increments in the ELPA
      git main package file.
    
    - Core Emacs code may not depend on these.

3) "core ELPA package".  

    - Stored in Gnu Emacs git, and possibly in the package maintainer's
      separate repository.

    - Each Emacs release includes a version of the package in the
      tarball, because it is in the Emacs git workspace. Other releases
      also occur, via the ELPA package server, whenever the version
      number increments in the Emacs git main package file.
    
    - Core Emacs code may depend on these.

-- 
-- Stephe

Re: ELPA policy

[John Wiegley]

>>>>> Stephen Leake <stephen_leake@stephe-leake.org> writes:

>> Elpa.git should be a submodule referenced from within Emacs.git (under
>> "elpa").

> That's overkill; there are many packages in Gnu ELPA that core code should
> _not_ depend on.

I believe that no core code should depend on any ELPA packages. Such a
dependency would be a reason to move that package into core, no? If that's
really the case, no submodule reference is necessary after all.

> If this said "large packages that the rest of Emacs core does not depend on
> ...", I could agree.

Yes, I meant that.

> So this introduces a third kind of ELPA package: "distributed in Emacs
> tarball". A "tarball package" for short?

Yes, for ELPA packages, there would be two flavors: in the tarball, and not.
Some have suggested just putting everything in the tarball for now. So we
could defer this question until the size of the tarball mattered, or if a
particular package was so dynamic it didn't want a version frozen into the
tarball.

Note that going down this road starts to make the line between "core" and
"ELPA" very thin, with easy migration between the two when desired. For
developers it allows Git to focus better, but for users, the difference is
largely invisible.

> Except that Eric maintains that there is more to "CEDET" in Emacs core than
> just lisp/cedet/*. So to be clear, we are proposing to move only
> lisp/cedet/* to elpa git.

Yes.

> In summary, I'm proposing that there are three kinds of ELPA packages:

I think we have a lot of agreement here, but I'd still like a clearer policy
than just "if core depends on it".

John

RE: ELPA policy

[Drew Adams]

Here's a naive question regarding where you place Lisp code
that is distributed with Emacs (part of the tarball): Will
anything change for users, depending on where you happen to
manage the code wrt ELPA etc.?

I'm guessing (and hoping) not, so that the distributed
directory structure will remain the same.  Is that correct?

Re: ELPA policy

[Stephen Leake]

Drew Adams <drew.adams@oracle.com> writes:

> Here's a naive question regarding where you place Lisp code
> that is distributed with Emacs (part of the tarball): Will
> anything change for users, depending on where you happen to
> manage the code wrt ELPA etc.?
>
> I'm guessing (and hoping) not, so that the distributed
> directory structure will remain the same.  Is that correct?

That's a good point. But I'm not clear what previous state you are
comparing to.

There are two cases;

1) A package that is currently in Emacs git moves to Gnu ELPA git but
   remains in the Emacs tarball.

2) A package that is currently in Gnu ELPA git stays there, and is added
   to the Emacs tarball.

I would argue that tarball ELPA packages should be installed in some
system level elpa directory (similar to the user's .emacs.d/elpa), not
in the main emacs/lisp install directory.

I don't think the history of a package should determine where it is
installed. So in case 1), I expect the package to be in a different
installation directory from the previous Emacs release. But it's still
in the default load-path, so this change is transparent to the user.

But it's not too important; the important distinction is that tarball
ELPA packages are _not_ in the developer Emacs workspace, so core
code cannot accidently depend on them.

-- 
-- Stephe

RE: ELPA policy

[Drew Adams]

> > Here's a naive question regarding where you place Lisp code
> > that is distributed with Emacs (part of the tarball): Will
> > anything change for users, depending on where you happen to
> > manage the code wrt ELPA etc.?
> >
> > I'm guessing (and hoping) not, so that the distributed
> > directory structure will remain the same.  Is that correct?
> 
> That's a good point. But I'm not clear what previous state you are
> comparing to.

Previous delivery of Emacs (back to ~Day One), including its
source code, especially Lisp, under directory lisp/.  What's
unclear about that?

> There are two cases;
> 1) A package that is currently in Emacs git moves to Gnu ELPA git but
>    remains in the Emacs tarball.
> 2) A package that is currently in Gnu ELPA git stays there, and is added
>    to the Emacs tarball.
> 
> I would argue that tarball ELPA packages should be installed in some
> system level elpa directory (similar to the user's .emacs.d/elpa), not
> in the main emacs/lisp install directory.

You say you would argue that, but where's the argument?  How about
a reason?

As one user, I would prefer that all distributed Lisp code be under
lisp/.  Makes things simpler for users (who might even have existing
scripts etc. that expect this).

Is there a good reason not to keep this behavior?

I don't really care where you stick code, or how you access it, for
development purposes.  Where an Emacs distribution puts it for
_users_ should remain what it has been, IMO, if possible - somewhere
under lisp/, at least.

Why should changes in where you keep code for development affect
where users of Emacs find it?

> I don't think the history of a package should determine where it is
> installed. So in case 1), I expect the package to be in a different
> installation directory from the previous Emacs release. But it's still
> in the default load-path, so this change is transparent to the user.

`load-path' is not everything, so no, it is _not_ transparent to
users, depending on what they do and how they use Emacs (including
its source code).

Emacs should be able to present its source code to users in the
same way as in the past, I would think.  They certainly should
not be _required_ to use the package system or git or ... to
access it.

`package.el' should be a nice-to-have.  It should not become an
obligatory hoop for users to jump through.  Most users will find
it handy, just as some users might find a menu handy.  But it
should not become a necessary way to access Emacs code.

> But it's not too important; the important distinction is that tarball
> ELPA packages are _not_ in the developer Emacs workspace, so core
> code cannot accidently depend on them.

That might be important to developers, but it is not the issue I
asked about.

Re: ELPA policy

[Achim Gratz]

Eli Zaretskii writes:
> The suggestion was to move _all_ of them, except the few that are
> needed for bootstrap, out of the Emacs repository.

You keep arguing about things that realistically aren't even on the
table yet (I don't want to prevent that discussion, mind you).  I said
it would be possible to do this, given some not-yet-existing
infrastructure and the only reason I even mentioned it was because
that's the other end of the spectrum from moving everything that looks
halfway useful into core.  Even if that infrastructure develops (which I
have serious doubts about given how this discussion goes), nobody in
their right mind would move all packages big-bang style to ELPA.
There'd certainly be a transition period with some pilot packages, I'd
probably pick Org, CEDET and Tramp based on the list traffic.  You'd
then see how that works and make some fixes that will certainly be
necessary before you get to the point to move more packages over to
ELPA.  This being Emacs, for better or worse, I'd expect that at this
point we'd be looking at _years_ before the extreme end you seem to be
so worried about is even within reach.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: ELPA policy

[Eli Zaretskii]

> From: Achim Gratz <Stromeko@nexgo.de>
> Date: Mon, 09 Nov 2015 19:22:50 +0100
> 
> Eli Zaretskii writes:
> > The suggestion was to move _all_ of them, except the few that are
> > needed for bootstrap, out of the Emacs repository.
> 
> You keep arguing about things that realistically aren't even on the
> table yet (I don't want to prevent that discussion, mind you).

If that's not on the table, then please tell what in your opinion _is_
on the table.  You did want to propose something practical that could
be done soon, right?  Because this discussion was about what we want
to do from now on with ELPA, not what we might want considering in
some distant future that might never come.

So please, let's be practical and consider alternatives that
realistically can be taken soon.  Like tomorrow or the next month.
Okay?

> There'd certainly be a transition period with some pilot packages, I'd
> probably pick Org, CEDET and Tramp based on the list traffic.

Moving out a few large packages that are developed outside Emacs
anyway is a no-brainer, provided that their developers agree.  I
already said I'll probably agree, and no one else objected.  The only
question is what is the opinion of the respective developers: we
cannot move the packages if they disagree.

Is there some other alternative on the table?  If not, we can finally
agree about something, I think.

Thanks.

Re: ELPA policy

[David Engster]

Eli Zaretskii writes:
>> From: Achim Gratz <Stromeko@nexgo.de>
>> Date: Mon, 09 Nov 2015 19:22:50 +0100

>> There'd certainly be a transition period with some pilot packages, I'd
>> probably pick Org, CEDET and Tramp based on the list traffic.
>
> Moving out a few large packages that are developed outside Emacs
> anyway is a no-brainer, provided that their developers agree.  I
> already said I'll probably agree, and no one else objected.  The only
> question is what is the opinion of the respective developers: we
> cannot move the packages if they disagree.

As long as Emacs core cannot depend on packages that are bundled from
ELPA, I would definitely not agree to move CEDET to ELPA.

-David

Re: ELPA policy

[Eli Zaretskii]

> From: David Engster <deng@randomsample.de>
> Date: Mon, 09 Nov 2015 19:58:29 +0100
> Cc: Achim Gratz <Stromeko@nexgo.de>, emacs-devel@gnu.org
> 
> As long as Emacs core cannot depend on packages that are bundled from
> ELPA, I would definitely not agree to move CEDET to ELPA.

Of course.  I don't think we should move anything before such a
dependency is possible.

Re: ELPA policy

[David Engster]

Eli Zaretskii writes:
>> From: David Engster <deng@randomsample.de>
>> Date: Mon, 09 Nov 2015 19:58:29 +0100
>> Cc: Achim Gratz <Stromeko@nexgo.de>, emacs-devel@gnu.org
>> 
>> As long as Emacs core cannot depend on packages that are bundled from
>> ELPA, I would definitely not agree to move CEDET to ELPA.
>
> Of course.  I don't think we should move anything before such a
> dependency is possible.

But if you have such dependencies, what's the point? IIRC, Stefan
decidedly did not want such dependencies, since the whole point was to
add those bundles only when creating releases.

-David

Re: ELPA policy

[Eli Zaretskii]

> From: David Engster <deng@randomsample.de>
> Cc: Stromeko@nexgo.de,  emacs-devel@gnu.org
> Date: Mon, 09 Nov 2015 20:35:04 +0100
> 
> Eli Zaretskii writes:
> >> From: David Engster <deng@randomsample.de>
> >> Date: Mon, 09 Nov 2015 19:58:29 +0100
> >> Cc: Achim Gratz <Stromeko@nexgo.de>, emacs-devel@gnu.org
> >> 
> >> As long as Emacs core cannot depend on packages that are bundled from
> >> ELPA, I would definitely not agree to move CEDET to ELPA.
> >
> > Of course.  I don't think we should move anything before such a
> > dependency is possible.
> 
> But if you have such dependencies, what's the point? IIRC, Stefan
> decidedly did not want such dependencies, since the whole point was to
> add those bundles only when creating releases.

I'm not pushing for the move, I just said that if the technical
problems are successfully resolved, I won't object.

Re: ELPA policy

[Rasmus]

Eli Zaretskii <eliz@gnu.org> writes:

> Moving out a few large packages that are developed outside Emacs
> anyway is a no-brainer, provided that their developers agree.  I
> already said I'll probably agree, and no one else objected.  The only
> question is what is the opinion of the respective developers: we
> cannot move the packages if they disagree.

I'm pretty sure Org would be happy to be distributed this way.  In fact,
we have got the 8.3 release now.  It really *must* be in 25.1, but I think
it would be fine to try out Fabian’s system to allow it to be upgraded
later.  Otherwise, I will hopefully be able to merge it soon the
"copy-paste" way.

Rasmus


-- 
However beautiful the theory, you should occasionally look at the evidence

Re: ELPA policy

[Achim Gratz]

Eli Zaretskii writes:
> If that's not on the table, then please tell what in your opinion _is_
> on the table.  You did want to propose something practical that could
> be done soon, right?  Because this discussion was about what we want
> to do from now on with ELPA, not what we might want considering in
> some distant future that might never come.

I did want to propose something that could become a vision for some time
in the foreseeable, yet somewhat distant future that we can then take
steps towards.  ELPA is foremost for _users_ and any decisions taken
should put the users first and developers second.  As a user I don't
have the luxury of chosing which Emacs version I get to use and what
packages are already installed on the system (indeed in my day job that
changes when I switch between project workspaces).  Wouldn't it be
splendid if ELPA helped me making those differences fade into the
background, if not disappear entirely?

> So please, let's be practical and consider alternatives that
> realistically can be taken soon.  Like tomorrow or the next month.
> Okay?

No, not okay.  There's still no consensus of even roughly where we want
to end up, so starting a random walk to find out who's feeling well or
not with whichever direction is certainly not my idea of progress.

> Moving out a few large packages that are developed outside Emacs
> anyway is a no-brainer, provided that their developers agree.

Again, it's not.  It simply doesn't work at this moment, not for Emacs,
not for the package developers and not for the users.  That doesn't have
anything to do with the size of those packages best I can tell.  To
change that (without moving the sources anywhere, just to not go down
that rabbit hole again) we'd need to devise a way to have an Emacs core
package whose autoload definitions can be separated and that can be
overridden (maybe only with a newer version) on the installation level
or by the user.  That would be the first step before any others can
follow, provided you even want to go that direction.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

DIY Stuff:
http://Synth.Stromeko.net/DIY.html

ELPA policy

[Julien Danjou]

Hi,

This topic has been recently brought by Lars, Ted and me on the gnus
mailing list. I've been asked to move this discussion here for
clarification.

I am currently worried by the Emacs ELPA archive and what will be that
policy. As Lars pointed out, until now, packages have been added over
the years into Emacs trunk, maintained by a extended set of skilled
hands, assuring good quality in (almost :-)) every place of every
packages furnished with Emacs.

Now, with the introduction of ELPA I'm worried about a lot of things.

- Who will be able to upload to ELPA?
- Who will fix the packages' bugs?
- Who will assure there's no regression for Emacs 24.1 user when people
  will starts uploading packages using 24.2 only function?
- Who will assure there's no regression at all?
- Who will assure there's no really bad things uploaded?
- Will all new packages go to ELPA, or with some still go to Emacs
  trunk? And if some can go to the trunk, upon which rules?
- Will all/most of the current packages be moved to ELPA?

There's probably a lot more question outstanding. I've the feeling ELPA
is adding a second class citizenship for packages, and I really do not
like that, at least in its current form.

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

Re: ELPA policy

[Chong Yidong]

Julien Danjou <julien@danjou.info> writes:

> I am currently worried by the Emacs ELPA archive and what will be that
> policy. As Lars pointed out, until now, packages have been added over
> the years into Emacs trunk, maintained by a extended set of skilled
> hands, assuring good quality in (almost :-)) every place of every
> packages furnished with Emacs.

Emacs contains many packages that are maintained "externally".  While
Emacs developers might make small changes to the in-tree version, most
development is done upstream and periodically merged in.  Those upstream
maintainers handle bugfixes and ensure compatibility among Emacs
versions.

One good reason to put a package in elpa.gnu.org, rather than in the
Emacs tarball, is if it is likely to benefit only a small segment of
Emacs users (even if it's of tremendous usefulness to that segment).
Especially, but not necessarily, if a package is large and complex, like
Auctex and Muse.

There are other good reasons too, e.g. packages that we want to merge
into Emacs core in the future, but not yet (for whatever reason).

(Incidentally, there are also some packages in Emacs that might be
usefully moved out into elpa.gnu.org, e.g. since they are so rarely
used.  We could also move some of the files in obsolete/ into a
subrepository, e.g. elpa.gnu.org/packages/obsolete)

(Also, as stated before, the FSF's policy is that for a package to be
listed on elpa.gnu.org its copyright must be assigned, in the exact same
way as if it's included in Emacs core.)

> - Who will be able to upload to ELPA?
> - Who will assure there's no really bad things uploaded?

The way it's currently set up is that only a couple of people can upload
to ELPA; package maintainers, when they release a new version, should
email me (or Ted) to get the package uploaded.  The system is still a
work in progress; we might set up a more elaborate "staging area" system
in the future.  (Such a system would still involve a human component, of
course, to reduce the possibility of malicious uploads.)  I will add a
page to the elpa.gnu.org webpage explaining this.

Re: ELPA policy

[Lars Magne Ingebrigtsen]

Chong Yidong <cyd@stupidchicken.com> writes:

> One good reason to put a package in elpa.gnu.org, rather than in the
> Emacs tarball, is if it is likely to benefit only a small segment of
> Emacs users (even if it's of tremendous usefulness to that segment).
> Especially, but not necessarily, if a package is large and complex, like
> Auctex and Muse.
>
> There are other good reasons too, e.g. packages that we want to merge
> into Emacs core in the future, but not yet (for whatever reason).

The reason that I raised the question now was that I was trying to
figure out how to do reasonable colour distance calculations for
rendering foreground colours in shr.el.  rainbow-mode (for colourising
colour specs in, for instance, CSS files) was brought up, that had
similar needs, and its calculations in the same area might possibly be
reusable by shr.el.

But apparently rainbow-mode went to ELPA instead of into Emacs, even
though it's small, it's clearly written, and it seems to be generally
useful for anybody editing stuff like CSS or .Xdefaults files or
anything.  So I wondered why that was...

> The way it's currently set up is that only a couple of people can upload
> to ELPA; package maintainers, when they release a new version, should
> email me (or Ted) to get the package uploaded.

If ELPA is supposed to be a repository of really special-needs code, I
think it's a good idea.  It would be nice to have an automatic way to
download, say, one of the Emacs-based mp3 players.  But if it's going to
carry stuff that people do really want to have in the Emacs
distribution, then I think it's counter-productive.

Stuff that is in Emacs gets loving care.  When url.el needs tweaking
(for instance), people step up to the plate and makes sure that it
works.  Things that live in a package repository bit-rot at an alarming
rate.  One of the great things about Emacs is that when you "apt-get
install emacs", you get a fully-featured system where you can be
reasonably sure that things work.  Over-modularisation can be costly in
the long run.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: ELPA policy

[Chong Yidong]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> figure out how to do reasonable colour distance calculations for
> rendering foreground colours in shr.el.  rainbow-mode (for colourising
> colour specs in, for instance, CSS files) was brought up, that had
> similar needs, and its calculations in the same area might possibly be
> reusable by shr.el.
>
> But apparently rainbow-mode went to ELPA instead of into Emacs, even
> though it's small, it's clearly written, and it seems to be generally
> useful for anybody editing stuff like CSS or .Xdefaults files or
> anything.  So I wondered why that was...

Rainbow-mode, as presented, is a tool for colorizing strings in Emacs
buffers, the kind of neat hack that doesn't really need to be in Emacs.
From your description, it sounds like parts of it can be usefully
refactored into a general color-manipulation library, in which case we
could include that part in Emacs (preferably, using a better prefix than
`rainbow').

> If ELPA is supposed to be a repository of really special-needs code, I
> think it's a good idea.  It would be nice to have an automatic way to
> download, say, one of the Emacs-based mp3 players.  But if it's going
> to carry stuff that people do really want to have in the Emacs
> distribution, then I think it's counter-productive.  Stuff that is in
> Emacs gets loving care.  Things that live in a package repository
> bit-rot at an alarming rate.

Well, obviously it's going to be a judgement call, analogous to the
debates about what goes into distribution DVDs (with programs like Emacs
nowadays frequently not making the cut :-P).  The level of bit-rot is
dependent on the upstream package maintainers; I don't think you can
argue that packages like Auctex (or SLIME or ESS, which are not
packaged) have bit-rotted.

Some packages integrated into Emacs are no longer active upstream but
are still tinkered with by Emacs developers, but what's really happening
here is that we have effectively forked the project.  If a package on
elpa.gnu.org ends up being abandoned, there's little difference in how
we could solve the problem, except the fork would be more explicit than
implicit.

Re: ELPA policy

[Edward O'Connor]

> The reason that I raised the question now was that I was trying to
> figure out how to do reasonable colour distance calculations for
> rendering foreground colours in shr.el.  rainbow-mode (for colourising
> colour specs in, for instance, CSS files) was brought up, that had
> similar needs, and its calculations in the same area might possibly be
> reusable by shr.el.

Maybe generalizing `tty-color-approximate' is the right thing to do?


Ted

Re: ELPA policy

[Glenn Morris]

Chong Yidong wrote:

> (Incidentally, there are also some packages in Emacs that might be
> usefully moved out into elpa.gnu.org, e.g. since they are so rarely
> used.  We could also move some of the files in obsolete/ into a
> subrepository, e.g. elpa.gnu.org/packages/obsolete)

I don't see how that would work. If, as soon as something becomes
obsolete, you move it to elpa, you've defeated the entire point of
obsolescence. Emacs goes from "you can assume we have function foo" to
"you can't rely on us having function foo" (unless your users have
added an optional package).

If on the other hand you move it to elpa at the point it which it
would normally be deleted altogether, then you're just prolonging its
life, when the whole point of making things obsolete is to migrate
people away from them.

And to move it at some intermediate stage seems a little pointless.

> (Also, as stated before, the FSF's policy is that for a package to be
> listed on elpa.gnu.org its copyright must be assigned, in the exact same
> way as if it's included in Emacs core.)

I was surprised to see AUCTeX there, since I thought its copyright was
notoriously hard to assign. At a quick glance, tex-fptex.el, tex-jp.el
have non-FSF copyrights in the headers, and there is no information
for the images/ directory. The texinfo source of the manuals and pdfs
also appears to be missing (maybe this is by design).

Re: ELPA policy

[Tom Tromey]

>>>>> "Julien" == Julien Danjou <julien@danjou.info> writes:

Julien> - Who will assure there's no regression for Emacs 24.1 user when people
Julien>   will starts uploading packages using 24.2 only function?

You could try making it so uploads require a warning-free byte-compile
on various Emacs versions first.

Packages can require a minimum Emacs version, if the author knows that
this is needed.

Julien> - Who will assure there's no regression at all?

You can't catch all the possible bugs.
I've had ok results on my site just forwarding reports to the package
maintainer for fixing.

Julien> There's probably a lot more question outstanding. I've the feeling ELPA
Julien> is adding a second class citizenship for packages, and I really do not
Julien> like that, at least in its current form.

My view leans more towards package inclusion maximalism -- put
everything in Emacs, but also do more frequent releases of some included
packages via ELPA.

Tom

Re: ELPA policy

[Glenn Morris]

Tom Tromey wrote:

> You could try making it so uploads require a warning-free byte-compile
> on various Emacs versions first.

Hah, we can't even get this for Emacs proper...

Re: ELPA policy

[Stefan Monnier]

> This topic has been recently brought by Lars, Ted and me on the gnus
> mailing list. I've been asked to move this discussion here for
> clarification.

As mentioned Chong, it's still "work in progress" from this point
of view.  But since your questions use the future tense, I'll try to
reply based on what I think should happen in the future.

> - Who will be able to upload to ELPA?

Depends: new packages or updates to existing packages?
For new packages, I'd expect only a few people to have such rights, but
for updates, I'd expect something like "anybody with access to the Bzr
repository".  After all, if they can screw with the main Emacs codebase,
why not with the ELPA packages.

> - Who will fix the packages' bugs?

The upstream maintainers, I'd expect.  That's one of the reasons to have
packages outside of Emacs's tree.

> - Who will assure there's no regression for Emacs 24.1 user when people
>   will starts uploading packages using 24.2 only function?

The upstream maintainer, for the same reason.

> - Who will assure there's no regression at all?

In which sense?  I'd expect the upstream maintainer to take some role
here, but admittedly, since those packages are easily available in
a central place, it's more likely that Emacs maintainers will pay
attention to them when introducing changes to the core.

> - Who will assure there's no really bad things uploaded?

The same people who currently assure that there's no really bad things
installed in the Emacs trunk.

> - Will all new packages go to ELPA, or with some still go to Emacs
>   trunk? And if some can go to the trunk, upon which rules?

We don't have rules for it, right now.  Here are some considerations, on
top of what Chong mentioned:
- in order to keep Emacs maintainable, we'd rather have packages in
  ELPA, all things being equal.
- packages of general usefulness (e.g. libraries) are good candidates
  for inclusion in Emacs, to reduce dependencies among ELPA packages.
  
> - Will all/most of the current packages be moved to ELPA?

Definitely not (in the foreseeable future).

> There's probably a lot more question outstanding. I've the feeling ELPA
> is adding a second class citizenship for packages, and I really do not
> like that, at least in its current form.

We currently have two kinds of packages: bundled and unbundled.  So ELPA
is about adding a third kind in-between, with properties such as
"smooth integration", "ease of installation", ... to be halfway between
the other two.


        Stefan

Re: ELPA policy

[Chong Yidong]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> For new packages, I'd expect only a few people to have such rights,
> but for updates, I'd expect something like "anybody with access to the
> Bzr repository".  After all, if they can screw with the main Emacs
> codebase, why not with the ELPA packages.

One difference, though, is that screwing with the main Emacs codebase
affects only those using the development version of Emacs, and we have
mechanisms like the diffs mailing list for problems to be easily
spotted.  After Emacs 24 is released, by screwing with elpa.gnu.org you
can immediately affect users of deployed stable Emacs versions.

So, we need to be a bit more paranoid for elpa.gnu.org than for our main
repository.

I agree, though, that it would be nice for Emacs developers to easily
edit packages in elpa.gnu.org without going through an onerous package
upload process.  I'm not sure how to set this up, though maybe the way
the Org daily builds are handled can be used as a starting point.

Re: ELPA policy

[Ted Zlatanov]

On Mon, 15 Nov 2010 15:12:15 -0500 Chong Yidong <cyd@stupidchicken.com> wrote: 

CY> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> For new packages, I'd expect only a few people to have such rights,
>> but for updates, I'd expect something like "anybody with access to the
>> Bzr repository".  After all, if they can screw with the main Emacs
>> codebase, why not with the ELPA packages.

CY> One difference, though, is that screwing with the main Emacs codebase
CY> affects only those using the development version of Emacs, and we have
CY> mechanisms like the diffs mailing list for problems to be easily
CY> spotted.  After Emacs 24 is released, by screwing with elpa.gnu.org you
CY> can immediately affect users of deployed stable Emacs versions.

CY> So, we need to be a bit more paranoid for elpa.gnu.org than for our main
CY> repository.

This is one good reason to keep a separate repository per major Emacs
version, at least.  Then you can fix packages without so much fear.

CY> I agree, though, that it would be nice for Emacs developers to easily
CY> edit packages in elpa.gnu.org without going through an onerous package
CY> upload process.  I'm not sure how to set this up, though maybe the way
CY> the Org daily builds are handled can be used as a starting point.

I hope we make it easy to submit changes but harder to publish them.
See below for my staging suggestion.

CY> Maintaining a separate repository for every Emacs release sounds like a
CY> lot of work.  Currently, it's up to maintainers of upstream/external
CY> packages ensure backward compatibility with older Emacs versions,
CY> e.g. by introducing compatibility functions.  Obviously this system
CY> doesn't work perfectly, but it doesn't seem like we have the manpower
CY> for anything more elaborate.

Setting it up is really not hard, even if we don't use it.  Just add
"packages-MAJOR" and "packages-MAJOR.MINOR" to the default "packages"
repository on startup.  Those can be empty but when we need something in
them (and we will, I promise you), they'll Just Work.

The dev checkout version should also have "packages-dev" or some such
that's only enabled if you make a dev build.

All this is not hard if we use VCS branches.  It's a 1-to-1 mapping to
the Emacs repo's branches.  In fact the elpa.gnu.org repo could mirror a
subdirectory of the Emacs repo, which would solve the staging problem
(developers would just stage to the Emacs repo and the elpa.gnu.org
maintainers would publish from there).

I think the alternative approach, keeping it really simple now, will
require more manpower and drain more time long-term.

Ted

Re: ELPA policy

[Richard Stallman]

    - Who will assure there's no really bad things uploaded?

We should treat inclusion in ELPA like inclusion in Emacs.

-- 
Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org, www.gnu.org