Keeping an ELPA checkout

Keeping an ELPA checkout

[Stefan Monnier]

I've been trying to use a checkout of ELPA with my "always run in-place"
development of Emacs, and I see some difficulties:

I currently basically add a symlink from emacs/lisp/packages to
elpa/packages.  A few tweaks are needed to convince Emacs to follow that
link when building Elisp files.
But this is too crude:
1- some files fail to compile (e.g. because they require ome package we
   don't distribute).
2- the automatic decision of what to include in load-path leads to
   shadows such as "auctex-11.86/style/url.el" vs "url/url.el".

I think we need to come up with a way to do the above but more robustly.


        Stefan

Re: Keeping an ELPA checkout

[Reiner Steib]

On Thu, Mar 24 2011, Stefan Monnier wrote:

> 2- the automatic decision of what to include in load-path leads to
>    shadows such as "auctex-11.86/style/url.el" vs "url/url.el".

AUCTeX has style/.nosearch since ages.  Why isn't it respected in your
scenario?

$ cat style/.nosearch 
;; AUCTeX style/ and auto/ directories should not appear in load path.

IIRC, this file is also copied to the auto/ directory on installation.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: Keeping an ELPA checkout

[Stefan Monnier]

>> 2- the automatic decision of what to include in load-path leads to
>> shadows such as "auctex-11.86/style/url.el" vs "url/url.el".
> AUCTeX has style/.nosearch since ages.  Why isn't it respected in your
> scenario?

Because emacs/lisp/Makefile.el doesn't pay attention to it.


        Stefan

Re: Keeping an ELPA checkout

[Ted Zlatanov]

On Thu, 24 Mar 2011 18:21:59 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

SM> I've been trying to use a checkout of ELPA with my "always run in-place"
SM> development of Emacs, and I see some difficulties:

Is that supported by ELPA/package.el?  I didn't think so...  

Are you trying to *use* the packages or *compile* them?  I can't figure
it out from your post.

SM> I currently basically add a symlink from emacs/lisp/packages to
SM> elpa/packages.
SM> But this is too crude:
SM> 1- some files fail to compile (e.g. because they require ome package we
SM>    don't distribute).
SM> 2- the automatic decision of what to include in load-path leads to
SM>    shadows such as "auctex-11.86/style/url.el" vs "url/url.el".
SM> I think we need to come up with a way to do the above but more robustly.

Shouldn't package.el, which knows how to manage the load path and the
dependencies, be used for this?  In the Commentary it mentions this
phase:

;; * Byte compile.  Currently this phase is done during install,
;;   but we may change this.

but there are many TODO items regarding byte-compilation.

Ted

Re: Keeping an ELPA checkout

[Stefan Monnier]

SM> I've been trying to use a checkout of ELPA with my "always run in-place"
SM> development of Emacs, and I see some difficulties:
> Is that supported by ELPA/package.el?  I didn't think so...

`package.el' is not involved.

> Are you trying to *use* the packages or *compile* them?  I can't figure
> it out from your post.

Both.  I want to just do "bzr update; make" and have all the ELPA
packages be updated, byte-compiled, ready for use, and ready for edit
(i.e. C-h f gives me a link to the file under Bzr).  As is the case for
all the files in emacs/lisp.


        Stefan

Re: Keeping an ELPA checkout

[Ted Zlatanov]

On Fri, 25 Mar 2011 15:52:02 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

SM> I've been trying to use a checkout of ELPA with my "always run in-place"
SM> development of Emacs, and I see some difficulties:

SM> I want to just do "bzr update; make" and have all the ELPA packages
SM> be updated, byte-compiled, ready for use, and ready for edit
SM> (i.e. C-h f gives me a link to the file under Bzr).  As is the case
SM> for all the files in emacs/lisp.

But ELPA packages are... packages!  Everything in emacs/lisp is source
code, meant to work as you describe.  Some ELPA packages happen to be
single-file libraries but you are trying to skip the installation and
activation steps which track dependencies, byte-compile, and adjust the
load path.  At least I don't think that's how it is supposed to work in
Tom Tromey's design.

I think these package.el features may be useful:

revno: 103553
committer: Chong Yidong <cyd@stupidchicken.com>
message:
  Allow specifying local ELPA mirrors in package-archives.

revno: 103229
author: Phil Hagelberg <phil@hagelb.org>
committer: Chong Yidong <cyd@stupidchicken.com>
message:
  Allow packages to be reinstalled.

Which I think are so people can install ELPA locally, edit things
locally, and reinstall the package when ready.  Does that help?  It can
probably be automated to be part of a "make local-activate-all" process.

Or do you just want to use the ELPA packages without the ELPA wrappings?

Ted

Re: Keeping an ELPA checkout

[Chong Yidong]

Ted Zlatanov <tzz@lifelogs.com> writes:

> But ELPA packages are... packages!  Everything in emacs/lisp is source
> code, meant to work as you describe.  Some ELPA packages happen to be
> single-file libraries but you are trying to skip the installation and
> activation steps which track dependencies, byte-compile, and adjust the
> load path.

Currently, we only have a script for deploying the elpa branch to an
archive, i.e. to a place from which the packages can be installed.

With a bit more work, we could add another script (or a Makefile rule)
for *installing* all the packages in the archive to a given location (or
even in place).  By pointing `package-directory-list' to that directory,
one would have all the packages in the archive installed in one step.
This is probably what Stefan has in mind.

Re: Keeping an ELPA checkout

[Ted Zlatanov]

On Fri, 25 Mar 2011 17:06:38 -0400 Chong Yidong <cyd@stupidchicken.com> wrote: 

CY> Ted Zlatanov <tzz@lifelogs.com> writes:
>> But ELPA packages are... packages!  Everything in emacs/lisp is source
>> code, meant to work as you describe.  Some ELPA packages happen to be
>> single-file libraries but you are trying to skip the installation and
>> activation steps which track dependencies, byte-compile, and adjust the
>> load path.

CY> Currently, we only have a script for deploying the elpa branch to an
CY> archive, i.e. to a place from which the packages can be installed.

CY> With a bit more work, we could add another script (or a Makefile rule)
CY> for *installing* all the packages in the archive to a given location (or
CY> even in place).  By pointing `package-directory-list' to that directory,
CY> one would have all the packages in the archive installed in one step.
CY> This is probably what Stefan has in mind.

That seems like a useful package.el feature that can be driven from a
Makefile and it can be used to easily run a full build test against the
GNU ELPA in particular.

But is it just for the GNU ELPA or for ELPA in general?  If the latter,
Phil Hagelberg and Tom Tromey will probably have feedback to make it
easier to implement and I imagine they've done something similar.

Ted

Re: Keeping an ELPA checkout

[Tom Tromey]

>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com> writes:

CY> With a bit more work, we could add another script (or a Makefile rule)
CY> for *installing* all the packages in the archive to a given location (or
CY> even in place).  By pointing `package-directory-list' to that directory,
CY> one would have all the packages in the archive installed in one step.
CY> This is probably what Stefan has in mind.

Ted> But is it just for the GNU ELPA or for ELPA in general?  If the latter,
Ted> Phil Hagelberg and Tom Tromey will probably have feedback to make it
Ted> easier to implement and I imagine they've done something similar.

I haven't.  When I've tested things, I've done it by having a local
repository.

Tom

Re: Keeping an ELPA checkout

[Stefan Monnier]

> But ELPA packages are... packages!  Everything in emacs/lisp is source
> code, meant to work as you describe.  Some ELPA packages happen to be
> single-file libraries but you are trying to skip the installation and
> activation steps which track dependencies, byte-compile, and adjust the
> load path.  At least I don't think that's how it is supposed to work in
> Tom Tromey's design.

I know, but I don't think there's any fundamental good reason why we
can't tweak emacs/lisp/Makefile.in so that it gets things to work.

> Which I think are so people can install ELPA locally, edit things
> locally, and reinstall the package when ready.  Does that help?  It can
> probably be automated to be part of a "make local-activate-all" process.

An important part of my use case is that files be used directly from the
Bzr checkout.  Also I don't want to copy any of those files anywhere
near my ~/ or ~/.emacs.d directory: everything should stay cleanly
within the Bzr checkout.


        Stefan

Re: Keeping an ELPA checkout

[Ted Zlatanov]

On Fri, 25 Mar 2011 17:12:57 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> But ELPA packages are... packages!  Everything in emacs/lisp is source
>> code, meant to work as you describe.  Some ELPA packages happen to be
>> single-file libraries but you are trying to skip the installation and
>> activation steps which track dependencies, byte-compile, and adjust the
>> load path.  At least I don't think that's how it is supposed to work in
>> Tom Tromey's design.

SM> I know, but I don't think there's any fundamental good reason why we
SM> can't tweak emacs/lisp/Makefile.in so that it gets things to work.

They are packages, that's a great reason :)  Yeah, you can do it, but
it's going to break sooner or later.  I like Chong's approach better
than tweaking things.

Also, isn't it better to put the rules inside the GNU ELPA (thus the
"elpa" branch) than inside Emacs' lisp/Makefile.in, as Chong suggested?

>> Which I think are so people can install ELPA locally, edit things
>> locally, and reinstall the package when ready.  Does that help?  It can
>> probably be automated to be part of a "make local-activate-all" process.

SM> An important part of my use case is that files be used directly from the
SM> Bzr checkout.  Also I don't want to copy any of those files anywhere
SM> near my ~/ or ~/.emacs.d directory: everything should stay cleanly
SM> within the Bzr checkout.

Sure, I think that's what Chong had in mind too.  I'm pretty sure
package.el can do that (maybe a special load file will be needed though).

Ted

Re: Keeping an ELPA checkout

[Stefan Monnier]

SM> I know, but I don't think there's any fundamental good reason why we
SM> can't tweak emacs/lisp/Makefile.in so that it gets things to work.

> They are packages, that's a great reason :)  Yeah, you can do it, but
> it's going to break sooner or later.  I like Chong's approach better
> than tweaking things.

> Also, isn't it better to put the rules inside the GNU ELPA (thus the
> "elpa" branch) than inside Emacs' lisp/Makefile.in, as Chong suggested?

Where they go doesn't matter much, yes..  The important part is to be
able to use them in-place and have them all activated from a single
place with "bzr up; make".


        Stefan

Re: Keeping an ELPA checkout

[Ted Zlatanov]

On Fri, 25 Mar 2011 19:20:37 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

SM> I know, but I don't think there's any fundamental good reason why we
SM> can't tweak emacs/lisp/Makefile.in so that it gets things to work.

>> They are packages, that's a great reason :)  Yeah, you can do it, but
>> it's going to break sooner or later.  I like Chong's approach better
>> than tweaking things.

>> Also, isn't it better to put the rules inside the GNU ELPA (thus the
>> "elpa" branch) than inside Emacs' lisp/Makefile.in, as Chong suggested?

SM> Where they go doesn't matter much, yes..  The important part is to be
SM> able to use them in-place and have them all activated from a single
SM> place with "bzr up; make".

OK.  So who's going to do the work?  And can you summarize what you're
asking for after this discussion?

Ted

Re: Keeping an ELPA checkout

[Stefan Monnier]

SM> Where they go doesn't matter much, yes..  The important part is to be
SM> able to use them in-place and have them all activated from a single
SM> place with "bzr up; make".

> OK.  So who's going to do the work?  And can you summarize what you're
> asking for after this discussion?

What I wrote above is the summary, AFAIK.


        Stefan

Re: Keeping an ELPA checkout

[Ted Zlatanov]

On Mon, 28 Mar 2011 21:16:08 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

SM> Where they go doesn't matter much, yes..  The important part is to be
SM> able to use them in-place and have them all activated from a single
SM> place with "bzr up; make".

>> OK.  So who's going to do the work?  And can you summarize what you're
>> asking for after this discussion?

SM> What I wrote above is the summary, AFAIK.

OK, I'll try to summarize because your requirements are not clear to
me.  Tell me if I understand what you're looking for.

1) check out or update Emacs trunk to $trunk and build it (should you
need to install?)

2) check out or update the ELPA branch to $elpa

3) cd $elpa; make local-install EMACS=$trunk/src/emacs (this runs the
trunk-built Emacs to `package-install' each ELPA package in
$elpa/packages and then adds "$elpa/packages" to
`package-directory-list' or maybe gives you a snippet to put in your
.emacs).  The trunk-built Emacs may need to have been installed
system-wide.

4) Now all the ELPA packages should be available as if you had installed
them through `package-install' into ~/.emacs.d because
`package-directory-list' has been augmented

5) any package build and installation artifacts in $elpa need to be
ignored by Bazaar

Is this correct?

Thanks
Ted

Re: Keeping an ELPA checkout

[Stefan Monnier]

>>> OK.  So who's going to do the work?  And can you summarize what you're
>>> asking for after this discussion?
SM> What I wrote above is the summary, AFAIK.
> OK, I'll try to summarize because your requirements are not clear to
> me.  Tell me if I understand what you're looking for.

You describe a possible solution rather than a requirement ;-)

> 2) check out or update the ELPA branch to $elpa

OK.

> 3) cd $elpa; make local-install EMACS=$trunk/src/emacs (this runs the
> trunk-built Emacs to `package-install' each ELPA package in
> $elpa/packages and then adds "$elpa/packages" to
> `package-directory-list' or maybe gives you a snippet to put in your
> .emacs).

The name "install" is confusing here: I don't want the .el files to
be copied.  I just want to compile them, extract their autoloads into
a file somewhere, and things like that.

> The trunk-built Emacs may need to have been installed system-wide.

No, that's undesirable.

> 4) Now all the ELPA packages should be available as if you had installed
> them through `package-install' into ~/.emacs.d because
> `package-directory-list' has been augmented

If you say so.

> 5) any package build and installation artifacts in $elpa need to be
> ignored by Bazaar

Right.

> Is this correct?

I'm not sure, but your point 5 makes me think it is.


        Stefan

Re: Keeping an ELPA checkout

[Ted Zlatanov]

On Tue, 29 Mar 2011 17:16:39 -0400 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>>>> OK.  So who's going to do the work?  And can you summarize what you're
>>>> asking for after this discussion?
SM> What I wrote above is the summary, AFAIK.
>> OK, I'll try to summarize because your requirements are not clear to
>> me.  Tell me if I understand what you're looking for.

SM> You describe a possible solution rather than a requirement ;-)

Yes, I tend to do that :)

>> 2) check out or update the ELPA branch to $elpa

SM> OK.

>> 3) cd $elpa; make local-install EMACS=$trunk/src/emacs (this runs the
>> trunk-built Emacs to `package-install' each ELPA package in
>> $elpa/packages and then adds "$elpa/packages" to
>> `package-directory-list' or maybe gives you a snippet to put in your
>> .emacs).

SM> The name "install" is confusing here: I don't want the .el files to
SM> be copied.  I just want to compile them, extract their autoloads into
SM> a file somewhere, and things like that.

I'm pretty sure this will require a `package-install-in-place' function,
after looking at the source.  Ugh.

Would "make inplace" be a better name?

>> The trunk-built Emacs may need to have been installed system-wide.

SM> No, that's undesirable.

...so are we then installing in the locally-compiled Emacs (modifying
files under $trunk) or for the user (modifying init/custom files)?

(All this would be much easier if the elpa branch was a subdirectory
under trunk!)

Ted

Re: Keeping an ELPA checkout

[Stefan Monnier]

>>> The trunk-built Emacs may need to have been installed system-wide.
SM> No, that's undesirable.
> ...so are we then installing in the locally-compiled Emacs (modifying
> files under $trunk) or for the user (modifying init/custom files)?

No, I'd rather we only modify files under $elpa.
Then the user is in charge of adding a (load "$elpa/foo-bar.el") to his
.emacs (or site-start, ...).

Remember, this is a setup intended for maintainers, that you do once and
then you forget about it (because you just "bzr update; make").


        Stefan