Adding ELPA to Emacs core

Adding ELPA to Emacs core

[Phillip Lord]

I've had a bit of a play at allowing ELPA packages to be directly
distributed with a core emacs installation. I tried this before where
files from ELPA were adding into the core as package.el packages. This
version does something simpler -- the raw files are just dumping into
the Emacs core, and then loaded like normal.

It currently uses some heuristics to achieve this, but it supports
installing the package, any tests into EMACS/test/lisp and texinfo
files. I've added an extensibility point though -- packages could
control their own deployment where something complicated needs to
happen.

Currently, it's make file driven (i.e. you edit the make file to add or
update a new package), but it would probably make sense to put the data
in a different file.

It's on branch elparized-core. Comments welcome.

Phil

Re: Adding ELPA to Emacs core

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Date: Sat, 10 Mar 2018 12:13:01 +0000
> 
> 
> I've had a bit of a play at allowing ELPA packages to be directly
> distributed with a core emacs installation. I tried this before where
> files from ELPA were adding into the core as package.el packages. This
> version does something simpler -- the raw files are just dumping into
> the Emacs core, and then loaded like normal.
> 
> It currently uses some heuristics to achieve this, but it supports
> installing the package, any tests into EMACS/test/lisp and texinfo
> files. I've added an extensibility point though -- packages could
> control their own deployment where something complicated needs to
> happen.
> 
> Currently, it's make file driven (i.e. you edit the make file to add or
> update a new package), but it would probably make sense to put the data
> in a different file.

Thanks.

Is this supposed to be used while building a development version as
well?  Because if so, its use of rsync might be a portability issue.
Can't we do the same using some Git magic instead?

Re: Adding ELPA to Emacs core

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> Currently, it's make file driven (i.e. you edit the make file to add or
>> update a new package), but it would probably make sense to put the data
>> in a different file.
>
> Thanks.
>
> Is this supposed to be used while building a development version as
> well?  Because if so, its use of rsync might be a portability issue.
> Can't we do the same using some Git magic instead?

It would *only* be used during a development build. During a "normal"
from source tarball, all the files would already been in place.

I've blitzed rsync. I was just using instead of cp.

Phil

Re: Adding ELPA to Emacs core

[Stefan Monnier]

> It would *only* be used during a development build. During a "normal"
> from source tarball, all the files would already been in place.

Is there some way to "pin" a package to particular revision (on release
branches, we'll want to do that for all packages)?


        Stefan

Re: Adding ELPA to Emacs core

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> It would *only* be used during a development build. During a "normal"
>> from source tarball, all the files would already been in place.
>
> Is there some way to "pin" a package to particular revision (on release
> branches, we'll want to do that for all packages)?


At the moment, it's the opposite. There isn't any way to unpin. Packages
are pulled out by a specific SHA individual for each package. This is
necessary because ELPA is a mixed economy: packages on branches will
have a different SHA from HEAD on master; and on trunk, package releases
are not co-ordinated, so each package needs to define it's own SHA.

Personally, I think that this is correct and not just for release
branches. Otherwise, the build would be far from repeatable, depending
on the state of ELPA. Having a "ignore the make file and take HEAD"
would be a nice option to have for running on a CI system.

It would also be possible to support other repos beyond ELPA
(i.e. org-mode has it's own git and we could take from there
directly). Whether this is a good idea or not, I leave as an open
question.

One thing that does worry me is that it means Emacs source as references
to ELPA git. If Emacs (or ELPA) ever needs to move VCS, it's going to
require some significant hackery.

Phil

Re: Adding ELPA to Emacs core

[Achim Gratz]

Phillip Lord writes:
> One thing that does worry me is that it means Emacs source as references
> to ELPA git. If Emacs (or ELPA) ever needs to move VCS, it's going to
> require some significant hackery.

Well, if you replace those SHA you were previously talking about with
tags, then the required hackery gets significantly reduced.  And no, I
don't think that EMACS needs to be able to reference any SHA directly
since ELPA is well enough under control to be able to place tags where
you need them, maybe even of the form emacs-x.y.z or similar so it
becomes clear which of these are in use where.  Naming the tags
explicitly to indicate Emacs' use of them would certainly help any
maintainer who would want to backport important patches to a branch that
older Emacsen can use.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Samples for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra

Re: Adding ELPA to Emacs core

[Phillip Lord]

Achim Gratz <Stromeko@nexgo.de> writes:

> Phillip Lord writes:
>> One thing that does worry me is that it means Emacs source as references
>> to ELPA git. If Emacs (or ELPA) ever needs to move VCS, it's going to
>> require some significant hackery.
>
> Well, if you replace those SHA you were previously talking about with
> tags, then the required hackery gets significantly reduced.  And no, I
> don't think that EMACS needs to be able to reference any SHA directly
> since ELPA is well enough under control to be able to place tags where
> you need them, maybe even of the form emacs-x.y.z or similar so it
> becomes clear which of these are in use where.  Naming the tags
> explicitly to indicate Emacs' use of them would certainly help any
> maintainer who would want to backport important patches to a branch that
> older Emacsen can use.


Yeah, I thought about that. But tags are just aliases for SHAs so I am
unconvinced this reduces the hackery that would come if we were to move
VCS. So, for example, we move to nugit (the new VCS I am not working
on). git tags would have to be translated to nugit tags. But, if we
could do this, we could just put git SHAs onto nugit as a nugit tag; or
we could edit Emacs git history so that git SHAs become nugit
identifiers.

Of course, my current solution does allow use of tags instead of SHAs,
since you can use anywhere a SHA is valid in git, you can use a tag. But
I don't share your confidence that tags could be controlled on
ELPA. There are multiple branches, and even on master the "current
version" maps to multiple commits. So, you need "A-emacs-x.y.z" and
"B-emacs.x.y.z" where A and B are package names. Putting this much
semantics in a string goes against my better judgement, I think.

Phil

Re: Adding ELPA to Emacs core

[Stefan Monnier]

> Of course, my current solution does allow use of tags instead of SHAs,
> since you can use anywhere a SHA is valid in git, you can use a tag. But
> I don't share your confidence that tags could be controlled on
> ELPA. There are multiple branches, and even on master the "current
> version" maps to multiple commits. So, you need "A-emacs-x.y.z" and
> "B-emacs.x.y.z" where A and B are package names. Putting this much
> semantics in a string goes against my better judgement, I think.

FWIW, I tend to agree that SHAs will work at least as well as tags.
Also if you want reproducibility, you need SHAs rather than tags.


        Stefan

Re: Adding ELPA to Emacs core

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Of course, my current solution does allow use of tags instead of SHAs,
>> since you can use anywhere a SHA is valid in git, you can use a tag. But
>> I don't share your confidence that tags could be controlled on
>> ELPA. There are multiple branches, and even on master the "current
>> version" maps to multiple commits. So, you need "A-emacs-x.y.z" and
>> "B-emacs.x.y.z" where A and B are package names. Putting this much
>> semantics in a string goes against my better judgement, I think.
>
> FWIW, I tend to agree that SHAs will work at least as well as tags.
> Also if you want reproducibility, you need SHAs rather than tags.
>


One issue with my branch so far is that the packages that I have copied
into the build become built-in, and package.el doesn't see versions on
ELPA as upgrades.

I guess this brings us to a related and perhaps bigger problem. ELPA
(the web end of the archive, rather than git) doesn't support multiple
versions of packages.

Say Emacs-27 bundles bob-3.1.el which is also in ELPA. Now bob-4.1.el is
released which is an upgrade. So surely Emacs-27 should offer to update?
But, then bob-5.1.el comes out along with Emacs-28 and bob-5.1.el does
not support Emacs-27. Now Emacs-27 should no longer install
bob-5.1.el. But surely, it should allow installation of bob-4.1.el.

AFAICT, neither package.el, nor ELPA supports multiple versions with
different Emacs dependencies.

Phil

Re: Adding ELPA to Emacs core

[Stefan Monnier]

> One issue with my branch so far is that the packages that I have copied
> into the build become built-in, and package.el doesn't see versions on
> ELPA as upgrades.

This is normally done via package--builtin-versions, set in loaddefs.el,
generated from lisp/emacs-lisp/autoload.el.

> Say Emacs-27 bundles bob-3.1.el which is also in ELPA. Now bob-4.1.el is
> released which is an upgrade. So surely Emacs-27 should offer to update?
> But, then bob-5.1.el comes out along with Emacs-28 and bob-5.1.el does
> not support Emacs-27. Now Emacs-27 should no longer install
> bob-5.1.el. But surely, it should allow installation of bob-4.1.el.

This problem is not specific to "Emacs with ELPA packages", tho.

But yes, it's a problem that would be good to fix.


        Stefan