package.el, auto-installation, and auto-removal

package.el, auto-installation, and auto-removal

[Stefan Monnier]

`package.el' is great to install packages more easily, but it falls short
when you want to keep the same set of installed packages on various
machines, or when it's time to remove package you don't need because it
doesn't distinguish between the packages you want and those required
by those packages.

So I'm looking for some kind soul who could extend package.el so as to
keep track of the packages that the user installed explicitly.
This should likely be kept in a custom variable.

Then we could add a command to install all the packages listed in
that variable.  And we could also write a function that deletes all
"orphan" packages.


        Stefan

Re: package.el, auto-installation, and auto-removal

[Ted Zlatanov]

On Fri, 07 Nov 2014 08:45:51 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

SM> `package.el' is great to install packages more easily, but it falls short
SM> when you want to keep the same set of installed packages on various
SM> machines, or when it's time to remove package you don't need because it
SM> doesn't distinguish between the packages you want and those required
SM> by those packages.

SM> So I'm looking for some kind soul who could extend package.el so as to
SM> keep track of the packages that the user installed explicitly.
SM> This should likely be kept in a custom variable.

SM> Then we could add a command to install all the packages listed in
SM> that variable.  And we could also write a function that deletes all
SM> "orphan" packages.

(CC to Robert Dallas Gray, Pallet's author)

Pallet does this: https://github.com/rdallasgray/pallet

I am not crazy about Cask because it requires Python, but the simplicity
of Pallet itself is appealing.  It uses advice and can be easily
improved to integrate better with package.el with a few hooks.

Pallet is GPL-ed and I don't know if the author would be interested in
assigning the copyright to Emacs and modifying Pallet to support a
non-Cask package maintenance backend. If that was possible, the problem
would be reduced to writing that backend and the package.el hooks.
Otherwise I think at least looking at pallet.el is worthwhile.

Ted

Re: package.el, auto-installation, and auto-removal

[joakim]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Fri, 07 Nov 2014 08:45:51 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 
>
> SM> `package.el' is great to install packages more easily, but it falls short
> SM> when you want to keep the same set of installed packages on various
> SM> machines, or when it's time to remove package you don't need because it
> SM> doesn't distinguish between the packages you want and those required
> SM> by those packages.
>
> SM> So I'm looking for some kind soul who could extend package.el so as to
> SM> keep track of the packages that the user installed explicitly.
> SM> This should likely be kept in a custom variable.
>
> SM> Then we could add a command to install all the packages listed in
> SM> that variable.  And we could also write a function that deletes all
> SM> "orphan" packages.
>
> (CC to Robert Dallas Gray, Pallet's author)
>
> Pallet does this: https://github.com/rdallasgray/pallet
>
> I am not crazy about Cask because it requires Python, but the simplicity
> of Pallet itself is appealing.  It uses advice and can be easily
> improved to integrate better with package.el with a few hooks.
>
> Pallet is GPL-ed and I don't know if the author would be interested in
> assigning the copyright to Emacs and modifying Pallet to support a
> non-Cask package maintenance backend. If that was possible, the problem
> would be reduced to writing that backend and the package.el hooks.
> Otherwise I think at least looking at pallet.el is worthwhile.

Also there is "el-get". It has some kind of funky license though.

>
> Ted
>
>

-- 
Joakim Verona

Re: package.el, auto-installation, and auto-removal

[Nic Ferrier]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> `package.el' is great to install packages more easily, but it falls short
> when you want to keep the same set of installed packages on various
> machines, or when it's time to remove package you don't need because it
> doesn't distinguish between the packages you want and those required
> by those packages.
>
> So I'm looking for some kind soul who could extend package.el so as to
> keep track of the packages that the user installed explicitly.
> This should likely be kept in a custom variable.
>
> Then we could add a command to install all the packages listed in
> that variable.  And we could also write a function that deletes all
> "orphan" packages.

Elpakit pretty much does this already.


Nic

Re: package.el, auto-installation, and auto-removal

[Stefan Monnier]

> Also there is "el-get". It has some kind of funky license though.

I'd like this new facility to be well-integrated into package.el.
E.g. M-x package-install should update the list of packages explicitly
requested by the user.

I don't think el-get qualifies in this respect.


        Stefan

Re: package.el, auto-installation, and auto-removal

[Stefan Monnier]

> Elpakit pretty much does this already.

Could someone turn it into a patch that we could install into Emacs?


        Stefan

Re: package.el, auto-installation, and auto-removal

[Nic Ferrier]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Elpakit pretty much does this already.
>
> Could someone turn it into a patch that we could install into Emacs?

Possibly. Do you only wants bits of it?


Nic

Re: package.el, auto-installation, and auto-removal

[Stefan Monnier]

>>> Elpakit pretty much does this already.
>> Could someone turn it into a patch that we could install into Emacs?
> Possibly. Do you only wants bits of it?

As stated originally, I want to change the behavior of package.el so as
to keep track of which packages were installed explicitly.

I don't really know elpakit does, so I can't really tell you what "bits
of it" would mean and what the other implicit option (probably "the
whole elpakit package") would do instead.


        Stefan

Re: package.el, auto-installation, and auto-removal

[Phillip Lord]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Fri, 07 Nov 2014 08:45:51 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 
>
> SM> `package.el' is great to install packages more easily, but it falls short
> SM> when you want to keep the same set of installed packages on various
> SM> machines, or when it's time to remove package you don't need because it
> SM> doesn't distinguish between the packages you want and those required
> SM> by those packages.
>
> (CC to Robert Dallas Gray, Pallet's author)
>
> Pallet does this: https://github.com/rdallasgray/pallet
>
> I am not crazy about Cask because it requires Python, but the simplicity
> of Pallet itself is appealing.  It uses advice and can be easily
> improved to integrate better with package.el with a few hooks.
>

Cask does use python, but only to launch emacs. Mostly the python side
finds an executable, works out what version it is and so forth. If it
were distributed with Emacs, all of this would be unnecessary (since it
would all be known in advance). It should be possible to produce a
shell/bat file for most OSes.

One advantage of having a command line tool for updating Emacs is that
it would solve at least some of the current problems of updating
packages in a running Emacs.

> Pallet is GPL-ed and I don't know if the author would be interested in
> assigning the copyright to Emacs and modifying Pallet to support a
> non-Cask package maintenance backend. If that was possible, the problem
> would be reduced to writing that backend and the package.el hooks.
> Otherwise I think at least looking at pallet.el is worthwhile.


All pallet does is pimp up package.el so that it rewrites the Cask file.
And (from a package management POV) all cask does is run package.el.

Phil

Re: package.el, auto-installation, and auto-removal

[Ted Zlatanov]

On Mon, 10 Nov 2014 14:55:28 +0000 phillip.lord@newcastle.ac.uk (Phillip Lord) wrote: 

PL> Ted Zlatanov <tzz@lifelogs.com> writes:
>> On Fri, 07 Nov 2014 08:45:51 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 
>> 
SM> `package.el' is great to install packages more easily, but it falls short
SM> when you want to keep the same set of installed packages on various
SM> machines, or when it's time to remove package you don't need because it
SM> doesn't distinguish between the packages you want and those required
SM> by those packages.
>> 
>> (CC to Robert Dallas Gray, Pallet's author)
>> 
>> Pallet does this: https://github.com/rdallasgray/pallet
>> 
>> I am not crazy about Cask because it requires Python, but the simplicity
>> of Pallet itself is appealing.  It uses advice and can be easily
>> improved to integrate better with package.el with a few hooks.
>> 

PL> Cask does use python, but only to launch emacs. Mostly the python side
PL> finds an executable, works out what version it is and so forth. If it
PL> were distributed with Emacs, all of this would be unnecessary (since it
PL> would all be known in advance). It should be possible to produce a
PL> shell/bat file for most OSes.

Yup.  Pallet and Cask could be integrated together.

PL> One advantage of having a command line tool for updating Emacs is that
PL> it would solve at least some of the current problems of updating
PL> packages in a running Emacs.

Avoiding the problems doesn't solve them :)

>> Pallet is GPL-ed and I don't know if the author would be interested in
>> assigning the copyright to Emacs and modifying Pallet to support a
>> non-Cask package maintenance backend. If that was possible, the problem
>> would be reduced to writing that backend and the package.el hooks.
>> Otherwise I think at least looking at pallet.el is worthwhile.

PL> All pallet does is pimp up package.el so that it rewrites the Cask file.
PL> And (from a package management POV) all cask does is run package.el.

Yes. I think that's what Stefan was asking for. But Stefan seems to
prefer the elpakit approach, which is fine too (Nic has a nice
comparison section at https://github.com/nicferrier/elpakit).

(Stefan: would you and the other maintainers be open to Cask+Pallet
*and* elpakit in Emacs?  Or is that too much choice?)

Ted

Re: package.el, auto-installation, and auto-removal

[Nic Ferrier]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Yes. I think that's what Stefan was asking for. But Stefan seems to
> prefer the elpakit approach, which is fine too (Nic has a nice
> comparison section at https://github.com/nicferrier/elpakit).
>
> (Stefan: would you and the other maintainers be open to Cask+Pallet
> *and* elpakit in Emacs?  Or is that too much choice?)

I don't think that was Stefan's position. His position was "can we have
this feature?"

I don't think he cares who does it or how it's done.

I'll add the feature he wants via bits of elpakit. I'm not sure the rest
of elpakit is needed. Maybe it is.



I am still cross about the whole state of packaging. It's wrong. But no
one seems to be listening to me.


/me throws toys down, kicks trumpet, shoves hands in pockets and stands
in corner.


Nic

Re: package.el, auto-installation, and auto-removal

[Stefan Monnier]

> Yes. I think that's what Stefan was asking for. But Stefan seems to
> prefer the elpakit approach, which is fine too (Nic has a nice
> comparison section at https://github.com/nicferrier/elpakit).

That's a misunderstanding.  I don't prefer any of those options because
I don't actually know anything about them (or close enough).

> (Stefan: would you and the other maintainers be open to Cask+Pallet
> *and* elpakit in Emacs?  Or is that too much choice?)

If someone cared to explain what that would do and why that would imply
"choice", I could tell you.

Nic says:
> I don't think that was Stefan's position. His position was "can we have
> this feature?"
> I don't think he cares who does it or how it's done.

Exactly.


        Stefan

Re: package.el, auto-installation, and auto-removal

[Stefan Monnier]

> I am still cross about the whole state of packaging. It's wrong. But no
> one seems to be listening to me.

I'm not sure if you're referring to the way GNU ELPA works or about
package.el.  About GNU ELPA, it's not that I don't listen, but that
I disagree.  The point of GNU ELPA to not to just distribute other
people's packages.


        Stefan

Re: package.el, auto-installation, and auto-removal

[Ted Zlatanov]

On Mon, 10 Nov 2014 16:37:50 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> Yes. I think that's what Stefan was asking for. But Stefan seems to
>> prefer the elpakit approach, which is fine too (Nic has a nice
>> comparison section at https://github.com/nicferrier/elpakit).

SM> That's a misunderstanding.  I don't prefer any of those options because
SM> I don't actually know anything about them (or close enough).

>> (Stefan: would you and the other maintainers be open to Cask+Pallet
>> *and* elpakit in Emacs?  Or is that too much choice?)

SM> If someone cared to explain what that would do and why that would imply
SM> "choice", I could tell you.

I provided URLs to Pallet's repo on Github and to Nic's comparison of
Cask (which Pallet uses underneath) and elpakit (the comparison is on
the elpakit repo home page, also on Github). I described how Pallet
currently works with advice and how it could be improved to use hooks.

Sorry if I didn't explain things better.  What's unclear?

In terms of popularity, Cask seems really popular so it's worth at least
a look.

Ted

Re: package.el, auto-installation, and auto-removal

[Stefan Monnier]

> I provided URLs to Pallet's repo on Github and to Nic's comparison of
> Cask (which Pallet uses underneath) and elpakit (the comparison is on
> the elpakit repo home page, also on Github).  I described how Pallet
> currently works with advice and how it could be improved to use hooks.
> Sorry if I didn't explain things better.  What's unclear?

I guess I just don't understand what those things do, nor how they do
it, and your explanations didn't really help.

E.g. does one of those things make it so that M-x package-install will
add the package to some list of "explicitly installed packages"?
If so, which tool, and how does it do it?  AFAICT it can only do that
via advice currently.  Of course, it can be changed to use hooks, but
why bother, when we can just modify package-install directly?


        Stefan

Re: package.el, auto-installation, and auto-removal

[Ted Zlatanov]

On Mon, 10 Nov 2014 21:26:14 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> I provided URLs to Pallet's repo on Github and to Nic's comparison of
>> Cask (which Pallet uses underneath) and elpakit (the comparison is on
>> the elpakit repo home page, also on Github).  I described how Pallet
>> currently works with advice and how it could be improved to use hooks.
>> Sorry if I didn't explain things better.  What's unclear?

SM> I guess I just don't understand what those things do, nor how they do
SM> it, and your explanations didn't really help.

Sorry.

SM> E.g. does one of those things make it so that M-x package-install will
SM> add the package to some list of "explicitly installed packages"?
SM> If so, which tool, and how does it do it?  AFAICT it can only do that
SM> via advice currently.  Of course, it can be changed to use hooks, but
SM> why bother, when we can just modify package-install directly?

Pallet hooks (through advice) into package installations, plus it can
discover the starting list of packages. It then maintains a Cask file.
So it does what you asked for at the beginning of this thread, but
relays the actual package management to Cask.

I agree that the integration of (Pallet|elpakit) can be as tight as we
want, no need for hooks.

Cask manages a list of packages in an external file. It's aware of
dependencies, repos, etc. and is quite popular for its ease of use. As
Phillip Lord said, bundling Cask would remove its Python requirement and
perhaps improve the user experience with package.el and the various
ELPAs.

I would love to hear from people that actively use Pallet/Cask or
elpakit. I've used el-get and use-package over the last few years, and
tried Pallet/Cask only recently as I heard about them from friends.

Ted

Re: package.el, auto-installation, and auto-removal

[Stefan Monnier]

> Pallet hooks (through advice) into package installations, plus it can
> discover the starting list of packages. It then maintains a Cask file.

OK, next question: what's a Cask file?

> So it does what you asked for at the beginning of this thread, but
> relays the actual package management to Cask.

You mean package.el isn't actually used?

> Cask manages a list of packages in an external file. It's aware of
> dependencies, repos, etc. and is quite popular for its ease of use.

"Aware of" isn't sufficient for me to understand what it does with that info.

> As Phillip Lord said, bundling Cask would remove its Python
> requirement and perhaps improve the user experience with package.el
> and the various ELPAs.

What functionality does it provide for the user (other than
auto-removal of the packages that were not explicitly installed, and
the ability to install the same set of packages, obviously)?


        Stefan

Re: package.el, auto-installation, and auto-removal

[Nic Ferrier]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I provided URLs to Pallet's repo on Github and to Nic's comparison of
>> Cask (which Pallet uses underneath) and elpakit (the comparison is on
>> the elpakit repo home page, also on Github).  I described how Pallet
>> currently works with advice and how it could be improved to use hooks.
>> Sorry if I didn't explain things better.  What's unclear?
>
> I guess I just don't understand what those things do, nor how they do
> it, and your explanations didn't really help.
>
> E.g. does one of those things make it so that M-x package-install will
> add the package to some list of "explicitly installed packages"?
> If so, which tool, and how does it do it?  AFAICT it can only do that
> via advice currently.  Of course, it can be changed to use hooks, but
> why bother, when we can just modify package-install directly?

Having thought about it that seems like the best thing. Just add another
property to the package somehow that says "the user installed this
directly". Maybe keep that along side the archive as well as in the
archive so that it can be mixed in when you refresh the indexes.

It would be nice to keep it as a directory structure because they are
easier to edit:

.emacs.d/elpa/
.emacs.d/elpa/archives/
.emacs.d/elpa/archives/marmalade/
.emacs.d/elpa/archives/gnu/
.emacs.d/elpa/archives/user-installed/
.emacs.d/elpa/archives/user-installed/elnode
.emacs.d/elpa/archives/user-installed/rcirc


Nic

Re: package.el, auto-installation, and auto-removal

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Pallet hooks (through advice) into package installations, plus it can
>> discover the starting list of packages. It then maintains a Cask file.
>
> OK, next question: what's a Cask file?
>
>> So it does what you asked for at the beginning of this thread, but
>> relays the actual package management to Cask.
>
> You mean package.el isn't actually used?
>
>> Cask manages a list of packages in an external file. It's aware of
>> dependencies, repos, etc. and is quite popular for its ease of use.
>
> "Aware of" isn't sufficient for me to understand what it does with that info.
>
>> As Phillip Lord said, bundling Cask would remove its Python
>> requirement and perhaps improve the user experience with package.el
>> and the various ELPAs.
>
> What functionality does it provide for the user (other than
> auto-removal of the packages that were not explicitly installed, and
> the ability to install the same set of packages, obviously)?


So, a brief outline of the various mechanisms for managing packages out of band.

Cask is a package (with a small python launcher) that uses a "Cask"
file. This is the cask file from cask itself.


(source gnu)
(source melpa)
(package-file "cask.el")
;; (files "*.el" "bin" "templates")
(development
(depends-on "f")
(depends-on "s"))

It looks like Lisp but actually isn't; you can't put conditional
statements in it for instance. Emacs reads it, and installs the
relevant files, using package.el to do so. Cask can be used to
configure your "daily" emacs. You can synchronise between several
machines by just syncing your Cask file, and then running "cask
install" which will pull down all the relevant packages. I do not use
Cask for this purpose though.

It also lets you run Emacs in batch, with a "sandbox" load-path; I use
this to run unit tests both locally and on a continuous integration
server. For this use, cask also integrates nicely with evm, which lets
you tests for several versions of Emacs. I do use cask for this, and
find it works well.

Pallet is a package which advices the interactive features of
package.el. When you install a package, it also writes the Cask file
for you. I don't use pallet myself. So, it provides a GUI mechanism
for managing your Cask file.

In both cases, there is not specific support for deleting unused
packages, although the data is there. In practice, you can just delete
your elpa directory, and then run "cask install" to pull the stuff you
need again.

For myself, I John Wigely's use-package -- it provides a nice
convenient way to configure and customise packages (something that
Cask does not). So, you can do something like this...

(use-package ace-jump-mode
  :bind ("C-." . ace-jump-mode))


It also has an ":ensure" keyword. So 

(use-package ace-jump-mode
  :ensure t
  :bind ("C-." . ace-jump-mode))

Will ensure that the package is installed and do so if not. I like
this because I can just sync my .emacs file between (which I am going
to do anyway). When I want to use a new package, I add a use-package
statement to my .emacs, and the package installs on my machines next
time I start Emacs. This works nicely, although I would quite like a
command line tool for managing packages, it doesn't seem worth the
effort of moving to Cask at the moment.

At the moment, I have not tried elpakit. Sorry, Nic, I probably will
give it a go now.

I think that any of the solutions would work. What is interesting is
that both Elpakit and Cask have additional stuff added. I've been
really glad to get Cask working and have it running my test cases for
me, outside of my daily emacs. But it's not strictly necessary for
persistantly managing packages.

Re: package.el, auto-installation, and auto-removal

[Nic Ferrier]

phillip.lord@newcastle.ac.uk (Phillip Lord) writes:

> At the moment, I have not tried elpakit. Sorry, Nic, I probably will
> give it a go now.

Well, you don't need to.

And I'm in the middle of moving it to a version 2.0.


> I think that any of the solutions would work. What is interesting is
> that both Elpakit and Cask have additional stuff added. I've been
> really glad to get Cask working and have it running my test cases for
> me, outside of my daily emacs. But it's not strictly necessary for
> persistantly managing packages.

The point about these things is that they're the only way you've got of
testing really. There's an enormous amount of grunt work in testing
stuff:

- make your package
- start up a test emacs
- install your package
- run the tests
- see the results

that's horrid boring work that both cask and elpakit try and
automate. For me it's not the out-of-bandness so much... though
repeatability clearly guarantees that.


Nic

Re: package.el, auto-installation, and auto-removal

[Stefan Monnier]

> It would be nice to keep it as a directory structure because they are
> easier to edit:
> .emacs.d/elpa/
> .emacs.d/elpa/archives/
> .emacs.d/elpa/archives/marmalade/
> .emacs.d/elpa/archives/gnu/
> .emacs.d/elpa/archives/user-installed/
> .emacs.d/elpa/archives/user-installed/elnode
> .emacs.d/elpa/archives/user-installed/rcirc

I was thinking of keeping it in a Custom var, since that's easy to edit.
So the equivalent of a Cask file could look like

   % cat CaskNG.el
   (setq package-archives ((gnu ...)
                           (melpa ...)))
   (setq packages-installed '(dash foo bar))
   ...
   %

and you could run your tests with something like

   emacs --batch -l CaskNG.el -f package-install-packages ...

So, my request is for someone to help set up the core of this, i.e. add
the `packages-installed' var (automatically updated by
package-install/delete and friends (I notice now that package-delete is
not a command, surprisingly)), and add the package-install-packages
command (so users can install their packages on a new system simply by
setting packages-installed and then running package-install-packages).
Bonus points for writing `package-gc' which deletes packages not
required by packages-installed.


        Stefan

Re: package.el, auto-installation, and auto-removal

[Nic Ferrier]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> It would be nice to keep it as a directory structure because they are
>> easier to edit:
>> .emacs.d/elpa/
>> .emacs.d/elpa/archives/
>> .emacs.d/elpa/archives/marmalade/
>> .emacs.d/elpa/archives/gnu/
>> .emacs.d/elpa/archives/user-installed/
>> .emacs.d/elpa/archives/user-installed/elnode
>> .emacs.d/elpa/archives/user-installed/rcirc
>
> I was thinking of keeping it in a Custom var, since that's easy to edit.
> So the equivalent of a Cask file could look like
>
>    % cat CaskNG.el
>    (setq package-archives ((gnu ...)
>                            (melpa ...)))
>    (setq packages-installed '(dash foo bar))
>    ...
>    %
>
> and you could run your tests with something like
>
>    emacs --batch -l CaskNG.el -f package-install-packages ...

I don't think a custom var is as good as something persistent in the
file system. It would be completely orthogonal to the existing data
storage which is all in the elpa dir (even the package-archives variable
is just an index of what is actually stored nicely in the elpa
directory, you can't recreate it wholly from there, but it's good
enough).

Great that you used a cask example. Because you'd actually need python
to run that. Hurray!


> So, my request is for someone to help set up the core of this, i.e. add
> the `packages-installed' var (automatically updated by
> package-install/delete and friends (I notice now that package-delete is
> not a command, surprisingly)), and add the package-install-packages
> command (so users can install their packages on a new system simply by
> setting packages-installed and then running package-install-packages).
> Bonus points for writing `package-gc' which deletes packages not
> required by packages-installed.

Yeah, as I say elpakit already has this. I could extract the code.

One thing about elpakit's solution is that elpakit does this by making a
package with the right dependencies.

Elpakit's solution is in fact just a package depend builder. Choose X, Y
and Z packages and then make a new package depending on those.

Maybe the current package installed by user state could be constantly
regenerated as a package.

The reason elpakit does that was in response to teams of clojure users
who wanted to homogenize their emacs config. So we made a system where
they could generate a base package to pull in all standard packages. And
then they could all iterate on top of that.

So making a new package all the time would be kinda useful because you
could easily share it with other people.

What do you think of that idea?


Nic

Re: package.el, auto-installation, and auto-removal

[Achim Gratz]

Nic Ferrier writes:
> One thing about elpakit's solution is that elpakit does this by making a
> package with the right dependencies.
>
> Elpakit's solution is in fact just a package depend builder. Choose X, Y
> and Z packages and then make a new package depending on those.

So I'd have an (empty) package named "user-packages" and it depends on
exactly those packages I named, possibly with their version specified?
And I could maybe have a number of additional packages to create
something of a hierarchy of such things?  Like a "system-packages"
package that my "user-packages" package can depend on (or not if I don't
want to)?

> Maybe the current package installed by user state could be constantly
> regenerated as a package.

Or maybe just skip the actual packaging step and really only deal with
the dependencies.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Terratec KOMPLEXER:
http://Synth.Stromeko.net/Downloads.html#KomplexerWaves

Re: package.el, auto-installation, and auto-removal

[Nic Ferrier]

Achim Gratz <Stromeko@nexgo.de> writes:

> Nic Ferrier writes:
>> One thing about elpakit's solution is that elpakit does this by making a
>> package with the right dependencies.
>>
>> Elpakit's solution is in fact just a package depend builder. Choose X, Y
>> and Z packages and then make a new package depending on those.
>
> So I'd have an (empty) package named "user-packages" and it depends on
> exactly those packages I named, possibly with their version specified?
> And I could maybe have a number of additional packages to create
> something of a hierarchy of such things?  Like a "system-packages"
> package that my "user-packages" package can depend on (or not if I don't
> want to)?

Yes.

Elpakit let's you do that. Stefan is asking for something simpler than
that. And I agree, something simpler than that is a good thing.


>> Maybe the current package installed by user state could be constantly
>> regenerated as a package.
>
> Or maybe just skip the actual packaging step and really only deal with
> the dependencies.

Sure. But packages are a useful encapsulation of dependencies.


Nic

Re: package.el, auto-installation, and auto-removal

[Robert Dallas Gray]

As an addendum to the points that have been raised on this topic, and as the author of Pallet, I thought it worth chipping in a few tuppences. 

There are several solutions to the problem Stefan outlines, with varying degrees of complexity and flexibility. To my mind, Cask is the least complex and most flexible. 

Cask only requires Python when used from the command line. There is a version available as a package.el package; this is what Pallet uses as its backend.

Pallet, as has been stated, is a dirt-simple layer over Cask, which rewrites the Cask manifest when the user installs or deletes packages using the package.el subsystem.

Pallet doesn't yet support version-controlled packages (outside of the package.el ecosystem), but Cask does. Pallet will also, when I have time.

Cask uses a tried-and-tested text-file manifest strategy, which has worked well for, e.g., Ruby's bundler and Javascript's npm. The syntax is a subset of Emacs Lisp.

Cask's author, Johan Andersson (https://github.com/rejeep) has also written epl (https://github.com/cask/epl), a layer over package.el which abstracts the package.el api, and servant (https://github.com/cask/servant), a mock package.el server. He knows the vagaries of package.el very well indeed.

I don't really have a dog in this fight, as I respect the authors and codebases of elpakit, Cask and use-package equally (I don't know enough about el-get to have an opinion), and as Pallet is so functionally trivial. 

But from the point of view of testability, at least, I've had success with using the Cask/epl/servant ecosystem to develop a package, and a set of integration tests which work across several Emacs versions (not so trivial). So I'll throw my hat in the ring for something like Cask and something on top of it like Pallet, even if it's neither of those actual things.

I think a consensus on something simple and flexible would be a great thing for Emacs.

Robert Dallas Gray

Re: package.el, auto-installation, and auto-removal

[Nic Ferrier]

Robert Dallas Gray <mail@robertdallasgray.com> writes:

> But from the point of view of testability, at least, I've had success
> with using the Cask/epl/servant ecosystem to develop a package, and a
> set of integration tests which work across several Emacs versions (not
> so trivial). So I'll throw my hat in the ring for something like Cask
> and something on top of it like Pallet, even if it's neither of those
> actual things.

This is something elpakit doesn't try and do, facilitate testing across
emacs-versions.

Cask will let you install various versions of emacs and test on them.


Nic

Re: package.el, auto-installation, and auto-removal

[Stefan Monnier]

> I don't think a custom var is as good as something persistent in the
> file system.

OTOH, a Custom var is the standard way Emacs stores persistent
user choices (and the set of installed packages is a user choice).

>> So I'd have an (empty) package named "user-packages" and it depends on
>> exactly those packages I named, possibly with their version specified?
>> And I could maybe have a number of additional packages to create
>> something of a hierarchy of such things?  Like a "system-packages"
>> package that my "user-packages" package can depend on (or not if I don't
>> want to)?
> Elpakit let's you do that.

Ah, that's where you're going.  I kind of like the idea, but I don't
like the idea of editing the content of packages in ~/.emacs.d/elpa
(even if only for some "special" packages like "user-package").


        Stefan

Re: package.el, auto-installation, and auto-removal

[Nic Ferrier]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Ah, that's where you're going.  I kind of like the idea, but I don't
> like the idea of editing the content of packages in ~/.emacs.d/elpa
> (even if only for some "special" packages like "user-package").

You don't edit that. You edit the source somewhere else and then
automatically install it.

I'd suggest that the package could be constructed automatically every
time. If the necessary data is kept in a customize variable anyway it
doesn't matter about the package source.

It only matters that a package is a good way to load other packages.


Nic

Re: package.el, auto-installation, and auto-removal

[Stefan Monnier]

>> Ah, that's where you're going.  I kind of like the idea, but I don't
>> like the idea of editing the content of packages in ~/.emacs.d/elpa
>> (even if only for some "special" packages like "user-package").
> You don't edit that. You edit the source somewhere else and then
> automatically install it.
> I'd suggest that the package could be constructed automatically every
> time.  If the necessary data is kept in a customize variable anyway it
> doesn't matter about the package source.
> It only matters that a package is a good way to load other packages.

If you want your "user-packages" package to be versioned, or if other
packages start refer to such meta-packages, then I can agree that it
could make sense, but otherwise I get the sensation that
- de-install the previous user-packages package.
- make up a dummy .el file with the Package-Requires: header.
- install this dummy .el package.
is kind of contrived.

Within package.el we could handle it is a special kind of package
instead, but I'd rather try to avoid reifying it into an actual
directory under ~/.emacs.d/elpa.


        Stefan

Re: package.el, auto-installation, and auto-removal

[Nic Ferrier]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> Ah, that's where you're going.  I kind of like the idea, but I don't
>>> like the idea of editing the content of packages in ~/.emacs.d/elpa
>>> (even if only for some "special" packages like "user-package").
>> You don't edit that. You edit the source somewhere else and then
>> automatically install it.
>> I'd suggest that the package could be constructed automatically every
>> time.  If the necessary data is kept in a customize variable anyway it
>> doesn't matter about the package source.
>> It only matters that a package is a good way to load other packages.
>
> If you want your "user-packages" package to be versioned, or if other
> packages start refer to such meta-packages, then I can agree that it
> could make sense, but otherwise I get the sensation that
> - de-install the previous user-packages package.
> - make up a dummy .el file with the Package-Requires: header.
> - install this dummy .el package.
> is kind of contrived.

Unless you have such a "fake" package you cannot easily share your
packages with anyone else.

In fact, I don't think that a package representing just my package state
is particularly useful. I use this sort of package as a way of passing
round a curated list of packages to other users, in a team at work for
example.

With what you're proposing though I can see a "starter kit" like thing
happening.


"new users should install technomancy's user-packages because they're a
great starting point" people will say.


> Within package.el we could handle it is a special kind of package
> instead, but I'd rather try to avoid reifying it into an actual
> directory under ~/.emacs.d/elpa.

I don't see the point otherwise. It's the simplest thing to do. Users
don't spend a lot of time looking at this stuff.

Nic

Re: package.el, auto-installation, and auto-removal

[Stefan Monnier]

> Unless you have such a "fake" package you cannot easily share your
> packages with anyone else.

It can be shared the same way other Custom settings can be shared.

> In fact, I don't think that a package representing just my package state
> is particularly useful.  I use this sort of package as a way of passing
> round a curated list of packages to other users, in a team at work for
> example.

But, IIUC, the current package.el supports this use case already without
any problem, right?

Anyway, I'll just drop this discussion because it's taking much more
time than writing the code.  All I'm after is a way to get things like
auto-removal of unrequired packages, and to have a more declarative way
to specify which packages you want to install and see which packages
you installed.

> "new users should install technomancy's user-packages because they're a
> great starting point" people will say.

But people can already do that, so it's a different issue from the one
I'm after.


        Stefan

Re: package.el, auto-installation, and auto-removal

[Phillip Lord]

Nic Ferrier <nferrier@ferrier.me.uk> writes:

> Robert Dallas Gray <mail@robertdallasgray.com> writes:
>
>> But from the point of view of testability, at least, I've had success
>> with using the Cask/epl/servant ecosystem to develop a package, and a
>> set of integration tests which work across several Emacs versions (not
>> so trivial). So I'll throw my hat in the ring for something like Cask
>> and something on top of it like Pallet, even if it's neither of those
>> actual things.
>
> This is something elpakit doesn't try and do, facilitate testing across
> emacs-versions.
>
> Cask will let you install various versions of emacs and test on them.


It's evm which does this, rather than cask. Probably you could get it to
work with elpakit also.

Phil