Bundling GNU ELPA packages

Bundling GNU ELPA packages

[Stefan Monnier]

In Emacs-25.1, I'd like to start bundling some GNU ELPA packages
into Emacs.  The most obvious such package is Org (whose Git code
should move out of emacs.git and into elpa.git), but others will be
added, I'm sure (probably Company, for example).

Who'd like to help?


        Stefan

Re: Bundling GNU ELPA packages

[Kelvin White]

> Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> In Emacs-25.1, I'd like to start bundling some GNU ELPA packages
> into Emacs.  The most obvious such package is Org (whose Git code
> should move out of emacs.git and into elpa.git), but others will be
> added, I'm sure (probably Company, for example).
>
> Who'd like to help?

I would be glad to help

Re: Bundling GNU ELPA packages

[Stefan Monnier]

What is needed, I think it something like the following:

- a file that lists which packages we want to bundle.
- a makefile target that fetches those packages and places them where we
  want them.
- this target should be used work when creating the release tarballs
  and pretests.
- this target should also work when building from the emacs.git
  repository, but we wouldn't want to update every time we do "make".
- when building from the trunk, it should fetch the latest code
  from elpa.git.
- when making the tarballs or building from the release branch, it
  should fetch particular revisions (the file that lists the packages
  would also list the revision we want for each package).
- it should work both for "subtree" and "external" packages.


        Stefan

Re: Bundling GNU ELPA packages

[Dmitry Gutov]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> The most obvious such package is Org (whose Git code
> should move out of emacs.git and into elpa.git)

CEDET also comes to mind as a great candidate, although it might be
harder to move.

> Who'd like to help?

This sounds like an infrastructure task to me: standardize in which dir
the ELPA checkout should be relative to the core, and then update the
"build release" script to copy files from certain dirs when the time
comes.

This isn't something I'd be particularly good at, but... what's the main
difficulty?

Re: Bundling GNU ELPA packages

[David Engster]

Dmitry Gutov writes:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>> The most obvious such package is Org (whose Git code
>> should move out of emacs.git and into elpa.git)
>
> CEDET also comes to mind as a great candidate, although it might be
> harder to move.

I think the Big Three are good candidates: Org, Gnus, and CEDET. They
all live in their separate directories underneath lisp/, so moving them
shouldn't be too hard. I had envisioned that after the move to Git,
CEDET could be imported from a stable upstream branch as a subtree
lisp/cedet, but I guess using GNU ELPA would be even better.

-David

Re: Bundling GNU ELPA packages

[Achim Gratz]

David Engster writes:
> I think the Big Three are good candidates: Org, Gnus, and CEDET.

+1


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Q+, Q and microQ:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: Bundling GNU ELPA packages

[Tassilo Horn]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> In Emacs-25.1, I'd like to start bundling some GNU ELPA packages into
> Emacs.

Now I'm curious.  What's the purpose of having a package system and then
bundling packages?

Bye,
Tassilo

Re: Bundling GNU ELPA packages

[Nicolas Richard]

Tassilo Horn <tsdh@gnu.org> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>> In Emacs-25.1, I'd like to start bundling some GNU ELPA packages into
>> Emacs.
>
> Now I'm curious.  What's the purpose of having a package system and then
> bundling packages?

I have one reason in mind : if org is bundled, opening an .org file will
automagically work (like it does now).

-- 
Nicolas Richard

Re: Bundling GNU ELPA packages

[Stefan Monnier]

>> In Emacs-25.1, I'd like to start bundling some GNU ELPA packages into
>> Emacs.
> Now I'm curious.  What's the purpose of having a package system and then
> bundling packages?

I'm sure the XEmacs guys could tell you ;-)

Having a package in ELPA means that it can be updated independently
from Emacs.

Having packages in elpa.git instead of emacs.git makes their release
schedules independent.

Having bundled packages in both emacs.git and in elpa.git means
2 branches to keep in sync.


        Stefan

Re: Bundling GNU ELPA packages

[David Engster]

Stefan Monnier writes:
>>> In Emacs-25.1, I'd like to start bundling some GNU ELPA packages into
>>> Emacs.
>> Now I'm curious.  What's the purpose of having a package system and then
>> bundling packages?
>
> I'm sure the XEmacs guys could tell you ;-)
>
> Having a package in ELPA means that it can be updated independently
> from Emacs.
>
> Having packages in elpa.git instead of emacs.git makes their release
> schedules independent.

Actually, you forgot the most important thing: ELPA packages don't have
to provide Changelogs... That alone would be reason enough for me to
move CEDET to ELPA.

Another advantage could be that you'd be able to provide security fixes
through ELPA for those packages without having to do a new Emacs release
(see also: Emacs 23.4), although this would involve more work to
"somehow" make this automatic.

-David

Re: Bundling GNU ELPA packages

[Glenn Morris]

David Engster wrote:

> to provide Changelogs... That alone would be reason enough for me to
> move CEDET to ELPA.

Also, no need to rename everything to work on MS-DOS!
I imagine this would have saved a lot of work, had it been an option
when CEDET was first added to Emacs. :(

Re: Bundling GNU ELPA packages

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Date: Thu, 06 Nov 2014 12:19:10 -0500
> 
> David Engster wrote:
> 
> > to provide Changelogs... That alone would be reason enough for me to
> > move CEDET to ELPA.
> 
> Also, no need to rename everything to work on MS-DOS!

Never miss an opportunity to kick old Eli in the butt!

I challenge you to recall when was someone required to rename a file
due to that reason.  (Yes, we have quite a few 8+3 conflicts in the
tree.)

> I imagine this would have saved a lot of work, had it been an option
> when CEDET was first added to Emacs. :(

That was back when I thought I had more friends here.  I'm much better
now, thank you.

Re: Bundling GNU ELPA packages

[Glenn Morris]

It seems impossible for us to have a non-emotive discussion about this,
so I'll simply say that friends can disagree on issues and still remain
friends; and I'll say no more on this subject.

Re: Bundling GNU ELPA packages

[Tassilo Horn]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> In Emacs-25.1, I'd like to start bundling some GNU ELPA packages
>>> into Emacs.
>> Now I'm curious.  What's the purpose of having a package system and
>> then bundling packages?
>
> I'm sure the XEmacs guys could tell you ;-)

So Stephen, if you read that, feel free to elaborate.

> Having a package in ELPA means that it can be updated independently
> from Emacs.
>
> Having packages in elpa.git instead of emacs.git makes their release
> schedules independent.
>
> Having bundled packages in both emacs.git and in elpa.git means
> 2 branches to keep in sync.

Yes, yes, and yes.  I understand all those points.  What I don't
understand is why we don't move org, gnus, and other built-in packages
which aren't "super-core" (i.e., not everybody needs them) from
emacs.git to elpa.git?  Then all points above still apply, and emacs
releases are a bit more lightweight.  I mean, for fast-evolving packages
like org and company, if emacs 25.1 bundles version X, the next day
version X+1 is available from ELPA anyway.

The only downside I can see is that users upgrading from Emacs 24 to 25
might get startup errors because formerly built-in packages aren't
anymore.  But that can be documented easily:

  If you used the built-in org-mode version in Emacs < 25, do

    1. emacs -Q
    2. M-x package-install RET org RET
    3. Now you can restart emacs without -Q

Or even better, there could be some hack that when emacs 25.1 is started
the first time puts the user in a package manager tutorial that guides
him thru the process of installing packages with an emphasis on formerly
built-in packages.

Bye,
Tassilo

Re: Bundling GNU ELPA packages

[Jonathan Leech-Pepin]

[-- Attachment #1: Type: text/plain, Size: 2149 bytes --]

On 6 November 2014 14:30, Tassilo Horn <tsdh@gnu.org> wrote:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> [...]
>
> Having a package in ELPA means that it can be updated independently
> > from Emacs.
> >
> > Having packages in elpa.git instead of emacs.git makes their release
> > schedules independent.
> >
> > Having bundled packages in both emacs.git and in elpa.git means
> > 2 branches to keep in sync.
>
> Yes, yes, and yes.  I understand all those points.  What I don't
> understand is why we don't move org, gnus, and other built-in packages
> which aren't "super-core" (i.e., not everybody needs them) from
> emacs.git to elpa.git?  Then all points above still apply, and emacs
> releases are a bit more lightweight.  I mean, for fast-evolving packages
> like org and company, if emacs 25.1 bundles version X, the next day
> version X+1 is available from ELPA anyway.
>
> The only downside I can see is that users upgrading from Emacs 24 to 25
> might get startup errors because formerly built-in packages aren't
> anymore.  But that can be documented easily:
>
>   If you used the built-in org-mode version in Emacs < 25, do
>
>     1. emacs -Q
>     2. M-x package-install RET org RET
>     3. Now you can restart emacs without -Q
>
> Or even better, there could be some hack that when emacs 25.1 is started
> the first time puts the user in a package manager tutorial that guides
> him thru the process of installing packages with an emphasis on formerly
> built-in packages.
>

Or even (assuming this is feasible):

1. Define a set of 'partial-core' packages (e.g Gnus, Org, CEDET) that have
autoloads* defined.
2. When the functions are called, advise that the package is not yet
installed and prompt the user to install it
3. Prompt user to restart Emacs (Assuming it is required if the package was
not previously present in that version

* Autoloads is perhaps the wrong term.  More something akin to
disabled-commands where it attempts to require the package, if the package
is found (installed to load path from site-lisp or locally) then proceed,
otherwise prompt the user.

Regards,
Jonathan
---


> Bye,
> Tassilo
>
>

[-- Attachment #2: Type: text/html, Size: 3067 bytes --]

Re: Bundling GNU ELPA packages

[Eli Zaretskii]

> From: Tassilo Horn <tsdh@gnu.org>
> Date: Thu, 06 Nov 2014 20:30:08 +0100
> Cc: emacs-devel@gnu.org
> 
> What I don't understand is why we don't move org, gnus, and other
> built-in packages which aren't "super-core" (i.e., not everybody
> needs them) from emacs.git to elpa.git?  Then all points above still
> apply, and emacs releases are a bit more lightweight.

There's no direct relation between moving packages between
repositories and excluding them from the release tarballs.  We can
have one, but not the other.

What is the advantage of having a more lightweight tarball?  Disk
space is no longer at premium, and Emacs is a relatively small package
by modern standards.

> I mean, for fast-evolving packages like org and company, if emacs
> 25.1 bundles version X, the next day version X+1 is available from
> ELPA anyway.

Yes, but then installing a tarball gives me Org and Gnus etc., even if
they are slightly outdated.  If we go your way, I don't have them at
all, and need a live, reliable, and uncensored network connection to
get them; until I do, my Emacs is crippled or might not even start at
all.  That's a net loss.

When I install XEmacs, I always want the "sumo" package, for that very
reason.

> The only downside I can see is that users upgrading from Emacs 24 to 25
> might get startup errors because formerly built-in packages aren't
> anymore.  But that can be documented easily:
> 
>   If you used the built-in org-mode version in Emacs < 25, do
> 
>     1. emacs -Q
>     2. M-x package-install RET org RET
>     3. Now you can restart emacs without -Q

There are only disadvantages here.  You add conditions that, if they
are not satisfied, will interfere with the upgrade.  It's a nuisance
for no good reason.

Re: Bundling GNU ELPA packages

[joakim]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Tassilo Horn <tsdh@gnu.org>
>> Date: Thu, 06 Nov 2014 20:30:08 +0100
>> Cc: emacs-devel@gnu.org
>> 
>> What I don't understand is why we don't move org, gnus, and other
>> built-in packages which aren't "super-core" (i.e., not everybody
>> needs them) from emacs.git to elpa.git?  Then all points above still
>> apply, and emacs releases are a bit more lightweight.
>
> There's no direct relation between moving packages between
> repositories and excluding them from the release tarballs.  We can
> have one, but not the other.
>
> What is the advantage of having a more lightweight tarball?  Disk
> space is no longer at premium, and Emacs is a relatively small package
> by modern standards.
>
>> I mean, for fast-evolving packages like org and company, if emacs
>> 25.1 bundles version X, the next day version X+1 is available from
>> ELPA anyway.
>
> Yes, but then installing a tarball gives me Org and Gnus etc., even if
> they are slightly outdated.  If we go your way, I don't have them at
> all, and need a live, reliable, and uncensored network connection to
> get them; until I do, my Emacs is crippled or might not even start at
> all.  That's a net loss.
>
> When I install XEmacs, I always want the "sumo" package, for that very
> reason.
>
>> The only downside I can see is that users upgrading from Emacs 24 to 25
>> might get startup errors because formerly built-in packages aren't
>> anymore.  But that can be documented easily:
>> 
>>   If you used the built-in org-mode version in Emacs < 25, do
>> 
>>     1. emacs -Q
>>     2. M-x package-install RET org RET
>>     3. Now you can restart emacs without -Q
>
> There are only disadvantages here.  You add conditions that, if they
> are not satisfied, will interfere with the upgrade.  It's a nuisance
> for no good reason.
>

+1 for Eli:s entire post from me! Theres no real point in a slimmed
release tarball, and if you still want one, you could still make one
separately from the main release tarball.

-- 
Joakim Verona

Re: Bundling GNU ELPA packages

[Nic Ferrier]

joakim@verona.se writes:

>>> The only downside I can see is that users upgrading from Emacs 24 to 25
>>> might get startup errors because formerly built-in packages aren't
>>> anymore.  But that can be documented easily:
>>> 
>>>   If you used the built-in org-mode version in Emacs < 25, do
>>> 
>>>     1. emacs -Q
>>>     2. M-x package-install RET org RET
>>>     3. Now you can restart emacs without -Q

I don't see why packages like this couldn't be delivered at
installation. The package could be bundled in the source repository at a
specific version and package-install-file'd into ELPA by the source
build.


I don't think the slimmed source build idea is a great idea because it's
already complicated. If we added that there would be 3 different package
styles.


It seems to me the package system is a bit confused, because people are
confused about packages. The confusion is greater in the wider Emacs
world.

One thing that causes that confusion is the ELPA repository. It's a
representation of packages, instead of packages.

I said before that I think ELPA should move to a package artifact store,
like marmalade-repo.

I think this because artifact stores are clearer. They just accept
package uploads and then give them out to users.

With ELPA there is a hidden build process. How many ELPA package authors
build their package and test it before committing? (this situation is
one of the reasons I don't like MELPA, the build is also hidden).

It's also not very scalable I think, eventually the git repo will get so
big it will be hard to move around. I'm sure we can spend time and energy
coming up with a solution to that but why?

GNU wants a package archive? make a package _archive_ not a source
repository. The source repositories for ELPA packages could be required
to exist on savannah, where projects belong, and they should be able to
have their own project management.

And they should deliver packages. Not some half thought out
representation of their package as a directory.

This might require package archive software, but I have one that is
GPLed (and even written in elisp) and it might require a build tool, but
I have one (written in elisp).

So I don't see why we shouldn't do packages _properly_


TL,DR - current ELPA combines release and build, hides build and merges
projects together. We should stop and go to the "standard" model.

Re: Bundling GNU ELPA packages

[Tassilo Horn]

Eli Zaretskii <eliz@gnu.org> writes:

>> The only downside I can see is that users upgrading from Emacs 24 to
>> 25 might get startup errors because formerly built-in packages aren't
>> anymore.  But that can be documented easily:
>> 
>>   If you used the built-in org-mode version in Emacs < 25, do
>> 
>>     1. emacs -Q
>>     2. M-x package-install RET org RET
>>     3. Now you can restart emacs without -Q
>
> There are only disadvantages here.  You add conditions that, if they
> are not satisfied, will interfere with the upgrade.  It's a nuisance
> for no good reason.

Ok, I can understand bundling org and gnus because they've previously
been built-in.  But when we now start bundling even more packages, those
are the next that can never ever be removed for that very same reason.
If their maintainers are hit by a truck, then you and the other Emacs
devs will have to take over.  Ditto if such a package is superseeded by
some better alternative.

Just saying: if a package is distributed with emacs, that's a kind of
guarantee that it'll still be there in the next decades to come.

Bye,
Tassilo

Re: Bundling GNU ELPA packages

[David Engster]

Tassilo Horn writes:
> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> The only downside I can see is that users upgrading from Emacs 24 to
>>> 25 might get startup errors because formerly built-in packages aren't
>>> anymore.  But that can be documented easily:
>>> 
>>>   If you used the built-in org-mode version in Emacs < 25, do
>>> 
>>>     1. emacs -Q
>>>     2. M-x package-install RET org RET
>>>     3. Now you can restart emacs without -Q
>>
>> There are only disadvantages here.  You add conditions that, if they
>> are not satisfied, will interfere with the upgrade.  It's a nuisance
>> for no good reason.
>
> Ok, I can understand bundling org and gnus because they've previously
> been built-in.  But when we now start bundling even more packages, those
> are the next that can never ever be removed for that very same reason.

We shouldn't bundle packages willy-nilly, obviously. You are right that
bundled packages become "core" packages, which will have to be
maintained for a long time. company-mode has been a contender for moving
into Emacs proper for quite some time now. Emacs is in dire need of an
"pop-up-alike" completion method.

> Just saying: if a package is distributed with emacs, that's a kind of
> guarantee that it'll still be there in the next decades to come.

True.

-David

Re: Bundling GNU ELPA packages

[Eli Zaretskii]

> From: Tassilo Horn <tsdh@gnu.org>
> Cc: monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Thu, 06 Nov 2014 21:40:50 +0100
> 
> Ok, I can understand bundling org and gnus because they've previously
> been built-in.  But when we now start bundling even more packages, those
> are the next that can never ever be removed for that very same reason.
> If their maintainers are hit by a truck, then you and the other Emacs
> devs will have to take over.  Ditto if such a package is superseeded by
> some better alternative.

This is true to some extent, but it is not a catastrophe, and is
nowhere a kind of Catholic marriage.  Package maintainers do
disappear, and we did take over some of the packages over the years.
Eventually, if a package ceases to be maintained, it is superseded and
retired into 'obsolete' (or maybe now it will actually be removed).

> Just saying: if a package is distributed with emacs, that's a kind of
> guarantee that it'll still be there in the next decades to come.

Which I think is a Good Thing.  We don't accept packages into the
bundle easily, which guarantees that they are generally useful, and
will therefore be popular with high probability.

Re: Bundling GNU ELPA packages

[Stephen Leake]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> In Emacs-25.1, I'd like to start bundling some GNU ELPA packages into
>>> Emacs.
>> Now I'm curious.  What's the purpose of having a package system and then
>> bundling packages?
>
> I'm sure the XEmacs guys could tell you ;-)
>
> Having a package in ELPA means that it can be updated independently
> from Emacs.
>
> Having packages in elpa.git instead of emacs.git makes their release
> schedules independent.
>
> Having bundled packages in both emacs.git and in elpa.git means
> 2 branches to keep in sync.

This says why we should have ELPA, and packages either in elpa.git or
emacs.git but not both.

The question was why we should then bundle some packages from ELPA in
the release tarball.

I'm wondering that myself.

-- 
-- Stephe

Re: Bundling GNU ELPA packages

[David Engster]

Stephen Leake writes:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>>> In Emacs-25.1, I'd like to start bundling some GNU ELPA packages into
>>>> Emacs.
>>> Now I'm curious.  What's the purpose of having a package system and then
>>> bundling packages?
>>
>> I'm sure the XEmacs guys could tell you ;-)
>>
>> Having a package in ELPA means that it can be updated independently
>> from Emacs.
>>
>> Having packages in elpa.git instead of emacs.git makes their release
>> schedules independent.
>>
>> Having bundled packages in both emacs.git and in elpa.git means
>> 2 branches to keep in sync.
>
> This says why we should have ELPA, and packages either in elpa.git or
> emacs.git but not both.
>
> The question was why we should then bundle some packages from ELPA in
> the release tarball.

Because users expect that Emacs ships with Org, Gnus and CEDET by
default. Bundling ELPA packages gives us the best of both worlds.

-David

Re: Bundling GNU ELPA packages

[Stephen Leake]

David Engster <deng@randomsample.de> writes:

> Stephen Leake writes:
>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>
>>>>> In Emacs-25.1, I'd like to start bundling some GNU ELPA packages into
>>>>> Emacs.
>>>> Now I'm curious.  What's the purpose of having a package system and then
>>>> bundling packages?
>>>
>>> I'm sure the XEmacs guys could tell you ;-)
>>>
>>> Having a package in ELPA means that it can be updated independently
>>> from Emacs.
>>>
>>> Having packages in elpa.git instead of emacs.git makes their release
>>> schedules independent.
>>>
>>> Having bundled packages in both emacs.git and in elpa.git means
>>> 2 branches to keep in sync.
>>
>> This says why we should have ELPA, and packages either in elpa.git or
>> emacs.git but not both.
>>
>> The question was why we should then bundle some packages from ELPA in
>> the release tarball.
>
> Because users expect that Emacs ships with Org, Gnus and CEDET by
> default. 

Perhaps it's time to change that expectation.

For example, I don't use Org or CEDET. So your "users" is leaving out at
least me :).

Debian and Ubuntu must have similar issues about what to include in
their "standard installers"; anyone know enough about that to
contribute? 

-- 
-- Stephe

Re: Bundling GNU ELPA packages

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Date: Sat, 08 Nov 2014 13:57:05 -0600
> 
> > Because users expect that Emacs ships with Org, Gnus and CEDET by
> > default. 
> 
> Perhaps it's time to change that expectation.

Why?

> For example, I don't use Org or CEDET. So your "users" is leaving out at
> least me :).

For every Emacs user, there are many bundled packages that user
doesn't use.  So what?

Re: Bundling GNU ELPA packages

[Stephen J. Turnbull]

Stefan Monnier writes:
 > >> In Emacs-25.1, I'd like to start bundling some GNU ELPA packages into
 > >> Emacs.
 > > Now I'm curious.  What's the purpose of having a package system and then
 > > bundling packages?
 > 
 > I'm sure the XEmacs guys could tell you ;-)

Well, we have some ideas about that, but in practice we don't.

 > Having a package in ELPA means that it can be updated independently
 > from Emacs.

Correct.

 > Having packages in elpa.git instead of emacs.git makes their release
 > schedules independent.

This isn't actually true.  Although we haven't put it in practice, the
discussions about it came to the conclusion that package releases are
likely to be frequent in a window before a core release, when there's
more incentive to get things committed and pushed.

 > Having bundled packages in both emacs.git and in elpa.git means
 > 2 branches to keep in sync.

Sure, but I think the question is more "why bundle packages?  why not
have the user use the package manager to get the latest and greatest?"

There are three answers:

1.  Latest is not always greatest.  If you don't keep a stock of
    "bundlable versions" known to be stable, you're putting the users
    at risk.

2.  Convenience: not all users have network connections all the time.

3.  Some modules that users consider "core" functionality may be
    highly decoupled from the "rest of core", and therefore such
    modules benefit from decoupled development cycles, but should be
    bundled so they are immediately available (including to the build
    process!)

Re: Bundling GNU ELPA packages

[Achim Gratz]

Stefan Monnier writes:
> In Emacs-25.1, I'd like to start bundling some GNU ELPA packages
> into Emacs.  The most obvious such package is Org (whose Git code
> should move out of emacs.git and into elpa.git), but others will be
> added, I'm sure (probably Company, for example).
>
> Who'd like to help?

I do.  Depending on the exact timing of the change I may have more or
less time to spend on this topic, though.

Here's one issue that I've not seen discussed bradly so far: currently
builtin packages aren't packages at all and are indeed available
site-wide for this particular Emacs version.  ELPA packages on the other
hand are a per-user thing only.  With ELPA packages bundled into Emacs,
I suggest that there should be a possibility for site-wide configuration
of which packages are available and enabled in the same way that
site-lisp currently works for non-packaged libraries.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Waldorf MIDI Implementation & additional documentation:
http://Synth.Stromeko.net/Downloads.html#WaldorfDocs

Re: Bundling GNU ELPA packages

[Stefan Monnier]

> I do.  Depending on the exact timing of the change I may have more or
> less time to spend on this topic, though.

Any time now would be good.

> Here's one issue that I've not seen discussed bradly so far: currently
> builtin packages aren't packages at all and are indeed available
> site-wide for this particular Emacs version.  ELPA packages on the other
> hand are a per-user thing only.  With ELPA packages bundled into Emacs,
> I suggest that there should be a possibility for site-wide configuration
> of which packages are available and enabled in the same way that
> site-lisp currently works for non-packaged libraries.

While there is no UI for it, package.el can definitely handle
site-wide packages: just add the corresponding directory to
package-directory-list.  And /usr/local/share/emacs/site-lisp/elpa is
included in there by default.

I'm not sure exactly what kind of "configuration of which packages are
available" you're thinking of, but I don't plan to provide a way to
"disable" bundled packages, just like we currently don't offer a way to
disable the things "activated" in lisp/loaddefs.el.

Elsewhere some other people talked about:
> > CEDET also comes to mind as a great candidate, although it might be
> > harder to move.
> I think the Big Three are good candidates: Org, Gnus, and CEDET.

Once the infrastructure is in place, we will have the opportunity to
look at those things.  Note that Gnus is probably not an easy option
since several parts of it are also used by non-Gnus code
(e.g. message.el for bug reports).


        Stefan

Re: Bundling GNU ELPA packages

[Achim Gratz]

Stefan Monnier writes:
> While there is no UI for it, package.el can definitely handle
> site-wide packages: just add the corresponding directory to
> package-directory-list.  And /usr/local/share/emacs/site-lisp/elpa is
> included in there by default.

That might take care of adding a package into site-lisp, but unless I'm
mistaken there is no obvious way for the user to "delete" such a package
(unless he's got write access to site-lisp) or even just chose a
different version.  Yes you can fiddle with the data structures, but
that is too error-prone, I'd think.

> I'm not sure exactly what kind of "configuration of which packages are
> available" you're thinking of, but I don't plan to provide a way to
> "disable" bundled packages, just like we currently don't offer a way to
> disable the things "activated" in lisp/loaddefs.el.

I'm thinking of a site administrator who wants to have a customized
selection of packages available, perhaps for multiple versions of Emacs;
without foisting that default on any user who might want or need
different packages.  So there needs to be a way to override the
selection of packages that came with Emacs on a site-wide basis and then
again on a per-user basis.  A user needs to be able to update the
packages she added, while the site administrator needs to be able to do
the same for the site collection, independently of each other.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Waldorf MIDI Implementation & additional documentation:
http://Synth.Stromeko.net/Downloads.html#WaldorfDocs

Re: Bundling GNU ELPA packages

[Stefan Monnier]

>> While there is no UI for it, package.el can definitely handle
>> site-wide packages: just add the corresponding directory to
>> package-directory-list.  And /usr/local/share/emacs/site-lisp/elpa is
>> included in there by default.
> That might take care of adding a package into site-lisp, but unless I'm
> mistaken there is no obvious way for the user to "delete" such a package
> (unless he's got write access to site-lisp)

Indeed, just like she can't "delete" a package that's bundled with Emacs.

The assumption is that installing a package is "harmless" and if that's
not the case, it's a bug in the package rather than a failure of the
package manager.

Tho, I think you can also prevent activation of a package via
package-pinned-packages (i.e. pinning to version that doesn't exist).

> or even just chose a different version.  Yes you can fiddle with the
> data structures, but that is too error-prone, I'd think.

If multiple versions of the same package are installed (that can also
happen in the user's own ~/.emacs.d/elpa), package.el should only
activate the latest version, and if that's not the right one, the user
can pick the one she wants with package-pinned-packages.

At least that's the theory.  This part of the code hasn't really been
tested, AFAIK.

> A user needs to be able to update the packages she added, while the
> site administrator needs to be able to do the same for the site
> collection, independently of each other.

Indeed.  And AFAIK the current code should behave correctly in
this respect.


        Stefan

Re: Bundling GNU ELPA packages

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> In Emacs-25.1, I'd like to start bundling some GNU ELPA packages
> into Emacs.  The most obvious such package is Org (whose Git code
> should move out of emacs.git and into elpa.git), but others will be
> added, I'm sure (probably Company, for example).

(A very confusing long thread followed.)

Some people seem to interpret this as you saying that you want to remove
some packages from Emacs and put them into Elpa.  But what you're saying
here is the opposite -- you want to bundle (some) Elpa packages into
Emacs.

I think you're not quite communicating what the purpose here is.  Do you
want to move some bits of Emacs into a place where it can be updated
faster than other bits of Emacs?

Or do you wish to slim down "Emacs" by moving bits of it into Elpa, but
then bundle Elpa into Emacs anyway?

But then there's this:

> Once the infrastructure is in place, we will have the opportunity to
> look at those things.  Note that Gnus is probably not an easy option
> since several parts of it are also used by non-Gnus code
> (e.g. message.el for bug reports).

So if you're bundling Elpa into Emacs, why would this matter anyway?
Won't it always be available anyway?

To put my query here more succinctly:

"Huh?"

Please clarify.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Bundling GNU ELPA packages

[Stefan Monnier]

> I think you're not quite communicating what the purpose here is.

Currently the purpose is to move Org to elpa.git and to add company to
Emacs without moving it to emacs.git.

Once that is supported, maybe some other packages will want to do like
Org, but it's a separate discussion and there's no hurry for that one.


        Stefan

Re: Bundling GNU ELPA packages

[David Engster]

Stefan Monnier writes:
>> I think you're not quite communicating what the purpose here is.
>
> Currently the purpose is to move Org to elpa.git and to add company to
> Emacs without moving it to emacs.git.
>
> Once that is supported, maybe some other packages will want to do like
> Org, but it's a separate discussion and there's no hurry for that one.

I second Lars' confusion here, and you didn't really answer his
question: if we "bundle a package from ELPA", it is *always* available,
isn't it? So how's that a problem if a package is also used by other
code in Emacs?

-David

Re: Bundling GNU ELPA packages

[David Engster]

David Engster writes:
> Stefan Monnier writes:
>>> I think you're not quite communicating what the purpose here is.
>>
>> Currently the purpose is to move Org to elpa.git and to add company to
>> Emacs without moving it to emacs.git.
>>
>> Once that is supported, maybe some other packages will want to do like
>> Org, but it's a separate discussion and there's no hurry for that one.
>
> I second Lars' confusion here, and you didn't really answer his
> question: if we "bundle a package from ELPA", it is *always* available,
> isn't it? So how's that a problem if a package is also used by other
> code in Emacs?

And to add to your second paragraph: I'm afraid there's a bit of a hurry
w.r.t. to CEDET. Once Emacs has moved to git, all my merging setup will
be tears in rain. CEDET will switch to git as well, but I'm not eager to
first setup a new patch-by-patch merging script and then move to yet
another solution. As I've said, my idea was to import CEDET as a
git-subtree in Emacs, but if we go the ELPA-bundle route, that should be
done rather sooner than later before the two repositories diverge too
much.

-David

Re: Bundling GNU ELPA packages

[Stefan Monnier]

> I second Lars' confusion here, and you didn't really answer his
> question: if we "bundle a package from ELPA", it is *always* available,
> isn't it?

No, it's obviously not present after "git clone".
Technically, we could *require* people to fetch the bundled elpa.git
packages before proceeding with the build, but I think it's preferable
if the bundled packages kept in elpa.git stay "loosely coupled".


        Stefan

Re: Bundling GNU ELPA packages

[James Cloos]

>>>>> "SM" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

SM> In Emacs-25.1, I'd like to start bundling some GNU ELPA packages
SM> into Emacs.  The most obvious such package is Org (whose Git code
SM> should move out of emacs.git and into elpa.git), but others will be
SM> added, I'm sure (probably Company, for example).

Does that mean building from git will require elpa as a submodule?

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 0x997A9F17ED7DAEA6