package.el, auto-installation, and auto-removal

package.el, auto-installation, and auto-removal

[Stefan Monnier]

`package.el' is great to install packages more easily, but it falls short
when you want to keep the same set of installed packages on various
machines, or when it's time to remove package you don't need because it
doesn't distinguish between the packages you want and those required
by those packages.

So I'm looking for some kind soul who could extend package.el so as to
keep track of the packages that the user installed explicitly.
This should likely be kept in a custom variable.

Then we could add a command to install all the packages listed in
that variable.  And we could also write a function that deletes all
"orphan" packages.


        Stefan

Re: package.el, auto-installation, and auto-removal

[Ted Zlatanov]

On Fri, 07 Nov 2014 08:45:51 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

SM> `package.el' is great to install packages more easily, but it falls short
SM> when you want to keep the same set of installed packages on various
SM> machines, or when it's time to remove package you don't need because it
SM> doesn't distinguish between the packages you want and those required
SM> by those packages.

SM> So I'm looking for some kind soul who could extend package.el so as to
SM> keep track of the packages that the user installed explicitly.
SM> This should likely be kept in a custom variable.

SM> Then we could add a command to install all the packages listed in
SM> that variable.  And we could also write a function that deletes all
SM> "orphan" packages.

(CC to Robert Dallas Gray, Pallet's author)

Pallet does this: https://github.com/rdallasgray/pallet

I am not crazy about Cask because it requires Python, but the simplicity
of Pallet itself is appealing.  It uses advice and can be easily
improved to integrate better with package.el with a few hooks.

Pallet is GPL-ed and I don't know if the author would be interested in
assigning the copyright to Emacs and modifying Pallet to support a
non-Cask package maintenance backend. If that was possible, the problem
would be reduced to writing that backend and the package.el hooks.
Otherwise I think at least looking at pallet.el is worthwhile.

Ted

Re: package.el, auto-installation, and auto-removal

[joakim]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Fri, 07 Nov 2014 08:45:51 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 
>
> SM> `package.el' is great to install packages more easily, but it falls short
> SM> when you want to keep the same set of installed packages on various
> SM> machines, or when it's time to remove package you don't need because it
> SM> doesn't distinguish between the packages you want and those required
> SM> by those packages.
>
> SM> So I'm looking for some kind soul who could extend package.el so as to
> SM> keep track of the packages that the user installed explicitly.
> SM> This should likely be kept in a custom variable.
>
> SM> Then we could add a command to install all the packages listed in
> SM> that variable.  And we could also write a function that deletes all
> SM> "orphan" packages.
>
> (CC to Robert Dallas Gray, Pallet's author)
>
> Pallet does this: https://github.com/rdallasgray/pallet
>
> I am not crazy about Cask because it requires Python, but the simplicity
> of Pallet itself is appealing.  It uses advice and can be easily
> improved to integrate better with package.el with a few hooks.
>
> Pallet is GPL-ed and I don't know if the author would be interested in
> assigning the copyright to Emacs and modifying Pallet to support a
> non-Cask package maintenance backend. If that was possible, the problem
> would be reduced to writing that backend and the package.el hooks.
> Otherwise I think at least looking at pallet.el is worthwhile.

Also there is "el-get". It has some kind of funky license though.

>
> Ted
>
>

-- 
Joakim Verona

Re: package.el, auto-installation, and auto-removal

[Stefan Monnier]

> Also there is "el-get". It has some kind of funky license though.

I'd like this new facility to be well-integrated into package.el.
E.g. M-x package-install should update the list of packages explicitly
requested by the user.

I don't think el-get qualifies in this respect.


        Stefan

Re: package.el, auto-installation, and auto-removal

[Phillip Lord]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Fri, 07 Nov 2014 08:45:51 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 
>
> SM> `package.el' is great to install packages more easily, but it falls short
> SM> when you want to keep the same set of installed packages on various
> SM> machines, or when it's time to remove package you don't need because it
> SM> doesn't distinguish between the packages you want and those required
> SM> by those packages.
>
> (CC to Robert Dallas Gray, Pallet's author)
>
> Pallet does this: https://github.com/rdallasgray/pallet
>
> I am not crazy about Cask because it requires Python, but the simplicity
> of Pallet itself is appealing.  It uses advice and can be easily
> improved to integrate better with package.el with a few hooks.
>

Cask does use python, but only to launch emacs. Mostly the python side
finds an executable, works out what version it is and so forth. If it
were distributed with Emacs, all of this would be unnecessary (since it
would all be known in advance). It should be possible to produce a
shell/bat file for most OSes.

One advantage of having a command line tool for updating Emacs is that
it would solve at least some of the current problems of updating
packages in a running Emacs.

> Pallet is GPL-ed and I don't know if the author would be interested in
> assigning the copyright to Emacs and modifying Pallet to support a
> non-Cask package maintenance backend. If that was possible, the problem
> would be reduced to writing that backend and the package.el hooks.
> Otherwise I think at least looking at pallet.el is worthwhile.


All pallet does is pimp up package.el so that it rewrites the Cask file.
And (from a package management POV) all cask does is run package.el.

Phil

Re: package.el, auto-installation, and auto-removal

[Ted Zlatanov]

On Mon, 10 Nov 2014 14:55:28 +0000 phillip.lord@newcastle.ac.uk (Phillip Lord) wrote: 

PL> Ted Zlatanov <tzz@lifelogs.com> writes:
>> On Fri, 07 Nov 2014 08:45:51 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 
>> 
SM> `package.el' is great to install packages more easily, but it falls short
SM> when you want to keep the same set of installed packages on various
SM> machines, or when it's time to remove package you don't need because it
SM> doesn't distinguish between the packages you want and those required
SM> by those packages.
>> 
>> (CC to Robert Dallas Gray, Pallet's author)
>> 
>> Pallet does this: https://github.com/rdallasgray/pallet
>> 
>> I am not crazy about Cask because it requires Python, but the simplicity
>> of Pallet itself is appealing.  It uses advice and can be easily
>> improved to integrate better with package.el with a few hooks.
>> 

PL> Cask does use python, but only to launch emacs. Mostly the python side
PL> finds an executable, works out what version it is and so forth. If it
PL> were distributed with Emacs, all of this would be unnecessary (since it
PL> would all be known in advance). It should be possible to produce a
PL> shell/bat file for most OSes.

Yup.  Pallet and Cask could be integrated together.

PL> One advantage of having a command line tool for updating Emacs is that
PL> it would solve at least some of the current problems of updating
PL> packages in a running Emacs.

Avoiding the problems doesn't solve them :)

>> Pallet is GPL-ed and I don't know if the author would be interested in
>> assigning the copyright to Emacs and modifying Pallet to support a
>> non-Cask package maintenance backend. If that was possible, the problem
>> would be reduced to writing that backend and the package.el hooks.
>> Otherwise I think at least looking at pallet.el is worthwhile.

PL> All pallet does is pimp up package.el so that it rewrites the Cask file.
PL> And (from a package management POV) all cask does is run package.el.

Yes. I think that's what Stefan was asking for. But Stefan seems to
prefer the elpakit approach, which is fine too (Nic has a nice
comparison section at https://github.com/nicferrier/elpakit).

(Stefan: would you and the other maintainers be open to Cask+Pallet
*and* elpakit in Emacs?  Or is that too much choice?)

Ted

Re: package.el, auto-installation, and auto-removal

[Nic Ferrier]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Yes. I think that's what Stefan was asking for. But Stefan seems to
> prefer the elpakit approach, which is fine too (Nic has a nice
> comparison section at https://github.com/nicferrier/elpakit).
>
> (Stefan: would you and the other maintainers be open to Cask+Pallet
> *and* elpakit in Emacs?  Or is that too much choice?)

I don't think that was Stefan's position. His position was "can we have
this feature?"

I don't think he cares who does it or how it's done.

I'll add the feature he wants via bits of elpakit. I'm not sure the rest
of elpakit is needed. Maybe it is.



I am still cross about the whole state of packaging. It's wrong. But no
one seems to be listening to me.


/me throws toys down, kicks trumpet, shoves hands in pockets and stands
in corner.


Nic

Re: package.el, auto-installation, and auto-removal

[Stefan Monnier]

> I am still cross about the whole state of packaging. It's wrong. But no
> one seems to be listening to me.

I'm not sure if you're referring to the way GNU ELPA works or about
package.el.  About GNU ELPA, it's not that I don't listen, but that
I disagree.  The point of GNU ELPA to not to just distribute other
people's packages.


        Stefan

package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Nic Ferrier]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I am still cross about the whole state of packaging. It's wrong. But no
>> one seems to be listening to me.
>
> I'm not sure if you're referring to the way GNU ELPA works or about
> package.el.  About GNU ELPA, it's not that I don't listen, but that
> I disagree.  The point of GNU ELPA to not to just distribute other
> people's packages.

I am cross about the way we are building this system, ELPA.

Not about package.el, or about what ELPA is. But about how ELPA (and
MELPA) work, and to some extent, the provisions we have made for package
creation.

I don't have any complaints about single file packages. My complaints
are all about multi file packages.


The multi-packages users load in their emacs are tars. But the packages
that are checked in to ELPA are directories of files.

So package authors are not checking in what gets delivered to the
user. So there is a magic build step somewhere.

This discourages testing of packages before they are distributed.

And I am really starting to think we need better testing. 24.4 looked
like a slog to release and it still has many bugs.

And yes, I promise to help more. But that's not really the issue. This
isn't scaling. We need to have better QA tools. And packages having
hidden builds is going the other way.


Nic

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Stefan Monnier]

> I am cross about the way we are building this system, ELPA.

Just to avoid confusion in the discussion: ELPA is the name of the
general package distribution system, IOW the protocol between package.el
and the web servers.

There are several servers which follow this protocol, such as Marmalade,
MELPA, and GNU ELPA.

> Not about package.el, or about what ELPA is.  But about how ELPA (and
> MELPA) work, and to some extent, the provisions we have made for package
> creation.

IIUC by "ELPA" above I guess you mean GNU ELPA (since "ELPA and MELPA"
would otherwise combine with "and" two things of different nature).

> The multi-packages users load in their emacs are tars.  But the packages
> that are checked in to ELPA are directories of files.
           ^^^^^^^^^^^^^^^^^^
So I think here you mean elpa.git (part of the greater GNU ELPA).

> So package authors are not checking in what gets delivered to the
> user.  So there is a magic build step somewhere.

Indeed.  Note that for all packages (even single-file packages) there
are yet more build steps elsewhere:
- the creation of the <foo>-autoloads.el.
- the creation of the *.elc files.

> This discourages testing of packages before they are distributed.

The "magic step" that turns a directory in elpa.git into a tarball is
fairly limited currently.  It is basically limited to "create
<foo>-pkg.el then tar".  Given the desire to limit the number of external
tools required (which is why we distribute *.tar and not *.tar.gz, for
example), I think that "tar+untar" is pretty close to the minimum way to
transfer a directory of files from the developer's machine to the user
who installs the package.

Note that there is a fair bit a pressure to *add* rather than remove
magic steps (the first candidate in the list is to build the *.info
files from the *.texi files).

The way to "test" for elpa.git is "cd .../elpa; make" which should
create the *-autoloads.el and *-pkg.el and *.elc files in the "same" way
as what will happen on the user's machine.

> And I am really starting to think we need better testing.  24.4 looked
> like a slog to release and it still has many bugs.

FWIW, I haven't heard of any case where the issues you're pointing out
have resulted in actual problems.  The only real reliability problem
I've heard about linked to ELPA packages is the issue of
compiling/installing a package when an older version is
already installed.


        Stefan

RE: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Drew Adams]

> Just to avoid confusion in the discussion: ELPA is the name of
> the general package distribution system, IOW the protocol
> between package.el and the web servers.

Really?  I thought that ELPA stands for "Emacs Lisp Package
Archive", and it thus means a repository of Emacs-Lisp code.

I think of `package.el', not ELPA, as something that provides
(the basis for) the "package system".

Perhaps Tom T. has something to offer about the terminology?

> There are several servers which follow this protocol, such as
> Marmalade, MELPA, and GNU ELPA.

Several ELPAs, several Emacs-Lisp repositories.  And yes, a
"repository" can include the ways of accessing it, including,
for example, a web server.

> IIUC by "ELPA" above I guess you mean GNU ELPA (since "ELPA
> and MELPA" would otherwise combine with "and" two things
> of different nature).

That's my thought too - GNU ELPA and MELPA are two ELPAs.
Marmalade is also an ELPA.  There can be any number of ELPAs.
Presumably all will support the package system offered by
(supported by) `package.el', though in different ways.

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Nic Ferrier]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> IIUC by "ELPA" above I guess you mean GNU ELPA (since "ELPA and MELPA"
> would otherwise combine with "and" two things of different nature).
>
>> The multi-packages users load in their emacs are tars.  But the packages
>> that are checked in to ELPA are directories of files.
>            ^^^^^^^^^^^^^^^^^^
> So I think here you mean elpa.git (part of the greater GNU ELPA).

Excellent clarification. That's what I mean in this instance.


> Indeed.  Note that for all packages (even single-file packages) there
> are yet more build steps elsewhere:
> - the creation of the <foo>-autoloads.el.
> - the creation of the *.elc files.

No. Those are not build steps. Those are installation steps. The
package artifacts (the things delivered to end users) do NOT have
auto-loads files or elc files in them. If they do they shouldn't because
that's what Emacs package.el does when it installs them.

It should go:

  build -> release -> install

elpa.git and MELPA are promoting:

  release -> (build) -> install

I don't think it's bad that MELPA promotes that. That's their
business. But I do think it matters that elpa.git works that way.


> Note that there is a fair bit a pressure to *add* rather than remove
> magic steps (the first candidate in the list is to build the *.info
> files from the *.texi files).

I agree. If the build was clearly separated this pressure would move
away into the build tools.


>> And I am really starting to think we need better testing.  24.4 looked
>> like a slog to release and it still has many bugs.
>
> FWIW, I haven't heard of any case where the issues you're pointing out
> have resulted in actual problems.  The only real reliability problem
> I've heard about linked to ELPA packages is the issue of
> compiling/installing a package when an older version is
> already installed.

Ok... I'm not sure I've come across an elpa.git package that has bugs
because it wasn't tested. But then I don't use many right now.

But my point is we agree (don't we?) that moving stuff into elpa.git
from core is broadly right and I want to encourage a testing culture.

Why do we want to move things into elpa.git? because we can fix bugs and
add features quicker.


Nic

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Eli Zaretskii]

> From: Nic Ferrier <nferrier@ferrier.me.uk>
> Date: Tue, 11 Nov 2014 11:41:16 +0000
> Cc: emacs-devel@gnu.org
> 
> > Note that there is a fair bit a pressure to *add* rather than remove
> > magic steps (the first candidate in the list is to build the *.info
> > files from the *.texi files).
> 
> I agree. If the build was clearly separated this pressure would move
> away into the build tools.
> [...]
> Why do we want to move things into elpa.git? because we can fix bugs and
> add features quicker.

Please let's not forget that most users of ELPA are not developers,
they are users.  For a user, it's an annoyance when fetching a package
requires to "build" it, especially if that "build" stage requires
extra tools (beyond Emacs itself) and other dependencies, let alone
dealing with possible error messages, failure to build, etc.

Sacrificing user convenience of installing a package to make our lives
as developers easier sounds backwards to me.  So, from my POV, the
ideal should be "no build stage", and the only files to be generated
during the installation should be those that Emacs can do itself, as
part of the fetching and installation.

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Nic Ferrier]

Eli Zaretskii <eliz@gnu.org> writes:

> Please let's not forget that most users of ELPA are not developers,
> they are users.  For a user, it's an annoyance when fetching a package
> requires to "build" it, especially if that "build" stage requires
> extra tools (beyond Emacs itself) and other dependencies, let alone
> dealing with possible error messages, failure to build, etc.

No one is arguing for that. I am NOT arguing for that.

Build is something package authors SHOULD do.

Build is something package authors for elpa.git do not do currently and
that inconveniences users in the end because authors won't test and the
packages will be broken. This happens often now with MELPA.


> Sacrificing user convenience of installing a package to make our lives
> as developers easier sounds backwards to me.  So, from my POV, the
> ideal should be "no build stage", and the only files to be generated
> during the installation should be those that Emacs can do itself, as
> part of the fetching and installation.

No one is saying that.

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Stefan Monnier]

> Sacrificing user convenience of installing a package to make our lives
> as developers easier sounds backwards to me.  So, from my POV, the
> ideal should be "no build stage", and the only files to be generated
> during the installation should be those that Emacs can do itself, as
> part of the fetching and installation.

There are 2 build stages: one when turning an elpa.git directory and
turning it into a tarball (done by elpa.gnu.org, not by users) and one
done by Emacs on the user's machine to compile the .el and generate
the autoloads.

So, the build normally done on the user's machine currently doesn't
require any external tools, and we aim to keep it this way
whenever possible.


        Stefan

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Nic Ferrier <nferrier@ferrier.me.uk>,  emacs-devel@gnu.org
> Date: Tue, 11 Nov 2014 12:20:23 -0500
> 
> So, the build normally done on the user's machine currently doesn't
> require any external tools, and we aim to keep it this way
> whenever possible.

That's all I care about.  Apologies if I misunderstood this
discussion.

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Stefan Monnier]

> It should go:
>   build -> release -> install
> elpa.git and MELPA are promoting:
>   release -> (build) -> install
> I don't think it's bad that MELPA promotes that. That's their
> business. But I do think it matters that elpa.git works that way.

Can you detail which part of GNU ELPA's "(build)" you don't like and why?


        Stefan

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Nic Ferrier]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> It should go:
>>   build -> release -> install
>> elpa.git and MELPA are promoting:
>>   release -> (build) -> install
>> I don't think it's bad that MELPA promotes that. That's their
>> business. But I do think it matters that elpa.git works that way.
>
> Can you detail which part of GNU ELPA's "(build)" you don't like and why?

the part where it happens without the user doing it.


I am saying I think the only way to encourage a testing culture is to
have an artifact store against which continuous integration can be done.


Nic

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Mon, 10 Nov 2014 18:24:21 -0500
> Cc: emacs-devel@gnu.org
> 
> Note that there is a fair bit a pressure to *add* rather than remove
> magic steps (the first candidate in the list is to build the *.info
> files from the *.texi files).

I hope we will be able to resist that pressure.

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Stefan Monnier]

>> Note that there is a fair bit a pressure to *add* rather than remove
>> magic steps (the first candidate in the list is to build the *.info
>> files from the *.texi files).
> I hope we will be able to resist that pressure.

Why?  Having to keep the .info files in the Git is not good.


        Stefan

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: nferrier@ferrier.me.uk,  emacs-devel@gnu.org
> Date: Tue, 11 Nov 2014 12:18:18 -0500
> 
> >> Note that there is a fair bit a pressure to *add* rather than remove
> >> magic steps (the first candidate in the list is to build the *.info
> >> files from the *.texi files).
> > I hope we will be able to resist that pressure.
> 
> Why?  Having to keep the .info files in the Git is not good.

Not good for whom?  For users it _is_ good, because it saves them from
the annoyance to have Texinfo of the "right" version installed on
their systems.

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Cc: nferrier@ferrier.me.uk,  emacs-devel@gnu.org
>> Date: Tue, 11 Nov 2014 12:18:18 -0500
>> 
>> >> Note that there is a fair bit a pressure to *add* rather than remove
>> >> magic steps (the first candidate in the list is to build the *.info
>> >> files from the *.texi files).
>> > I hope we will be able to resist that pressure.
>> 
>> Why?  Having to keep the .info files in the Git is not good.
>
> Not good for whom?  For users it _is_ good, because it saves them from
> the annoyance to have Texinfo of the "right" version installed on
> their systems.

Git is not "for users".  It is for developers.  Make does not work
reliably when generated files are checked into Git because Git does not
tamper with the file creation dates: those are the dates of the
checkout.

For "users" it is important that the _tarballs_ contain some of the
generated files.

-- 
David Kastrup

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Stephen J. Turnbull]

David Kastrup writes:

 > Git is not "for users".  It is for developers.

That, unfortunately, is a distinction that doesn't exist with git.  I
have many students with no detectable neural activity as far as
software development goes keeping their theses in a group-accessible
git repo.  That means I don't have to fish their content out of some
broken mail message, and it keeps a record of when they are actually
working on their documents.  (If I could only get them to use Emacsen
and TeX so git diff would be useful!  But that's another story.)  Some
of them have actually discovered that the log is useful to them too!

True, from the developer's POV, git is not a great way to distribute
products-with-source because it doesn't keep Make relationships up to
date.  However, that doesn't mean it doesn't make a good way to
distribute product to users.

Perhaps this tension could be addressed by having a "developer" branch
(which could be trunk = "master", of course) which assumes a complete
suite of tools for developers, and a "distribute" branch containing
products that require tools other than Emacs itself for users.
Thinking out loud, using the current "maintenance" branch as a
"distribute" branch might even work.  With appropriate tools (to be
written AFAIK, unfortunately), you can recover (to a close
approximation) the necessary timestamp information to avoid
unnecessary rebuilding by Make, too.

And of course you could resolve the tension by declaring Emacs's
and/or ELPA's git repos to be developer-only.  I'm pointing out
possibilities, not imperatives.

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> David Kastrup writes:
>
>  > Git is not "for users".  It is for developers.
>
> That, unfortunately, is a distinction that doesn't exist with git.

But it exists with the Git repository of Emacs.

> I have many students with no detectable neural activity as far as
> software development goes keeping their theses in a group-accessible
> git repo.  That means I don't have to fish their content out of some
> broken mail message, and it keeps a record of when they are actually
> working on their documents.  (If I could only get them to use Emacsen
> and TeX so git diff would be useful!  But that's another story.)  Some
> of them have actually discovered that the log is useful to them too!

So they are using it as a developer repo.  They are not using it as an
archival system for generated files, and indeed Git very explicitly is
not an archival system.  The discussion to preserve file modification
dates was there in its beginnings, and Git does not preserve them (and
does not offer ways to preserve them) on purpose.

> True, from the developer's POV, git is not a great way to distribute
> products-with-source because it doesn't keep Make relationships up to
> date.  However, that doesn't mean it doesn't make a good way to
> distribute product to users.

Emacs' tarball is not a finished product.  It needs to have some
relations intact to maintain pregenerated info files and stuff.

> Perhaps this tension could be addressed by having a "developer" branch
> (which could be trunk = "master", of course) which assumes a complete
> suite of tools for developers, and a "distribute" branch containing
> products that require tools other than Emacs itself for users.

Doesn't work since Git does not preserve file dates on checkout.

> Thinking out loud, using the current "maintenance" branch as a
> "distribute" branch might even work.  With appropriate tools (to be
> written AFAIK, unfortunately), you can recover (to a close
> approximation) the necessary timestamp information to avoid
> unnecessary rebuilding by Make, too.

The tool is written already, of course.  Make has some flag for touching
everything instead of building it.  But that would include non-version
controlled binaries in the tree that have not been touched on the last
checkout.

-- 
David Kastrup

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Nic Ferrier]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Date: Mon, 10 Nov 2014 18:24:21 -0500
>> Cc: emacs-devel@gnu.org
>> 
>> Note that there is a fair bit a pressure to *add* rather than remove
>> magic steps (the first candidate in the list is to build the *.info
>> files from the *.texi files).
>
> I hope we will be able to resist that pressure.

You're mistaking the build step. The build step is where you take
sources and generate an artifact. Our artifacts are called packages.

The user's install packages/artifacts.

So any build complication is all on the author. As it should be.


Nic

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Achim Gratz]

Stefan Monnier writes:
> The "magic step" that turns a directory in elpa.git into a tarball is
> fairly limited currently.  It is basically limited to "create
> <foo>-pkg.el then tar".  Given the desire to limit the number of external
> tools required (which is why we distribute *.tar and not *.tar.gz, for
> example), I think that "tar+untar" is pretty close to the minimum way to
> transfer a directory of files from the developer's machine to the user
> who installs the package.

Yes, but for more complex packages this is too limiting.  What gets
packaged is only ever accidentally exactly the same as the source for
that package.  So the first question to decide would be if elpa.git is a
source archive or a package archive.

> Note that there is a fair bit a pressure to *add* rather than remove
> magic steps (the first candidate in the list is to build the *.info
> files from the *.texi files).

...if it is meant to be a source archive, this is actually quite
sensible.  Why check build results into a source archive?  On the other
hand, if it's just a package archive, then the implicit requirement is
that the sources live someplace else.

To give you an idea of what can be involved in moving from source to
package, this is how Org creates its package archives:

http://orgmode.org/cgit.cgi/org-mode.git/plain/mk/server.mk

In words, the package and version file, a separate autoload file that
Org needs (in addition to the autoload file that later gets created on
the users' machine) and the documentation all need to be created before
packaging things up.  These steps actually happen in different
directories (lisp/ and doc/) that get stripped directly by tar rather
than copying the files around.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Stefan Monnier]

> that package.  So the first question to decide would be if elpa.git is a
> source archive or a package archive.

I want it to be a source archive.

> In words, the package and version file, a separate autoload file that
> Org needs (in addition to the autoload file that later gets created on
> the users' machine)

The org-pkg.el file is nowadays built by the GNU ELPA scripts (there's
even a special exception in this script specifically for Org, which is
the only package where this file is not auto-built yet).

What is the "version" file?

Can you give further details on this extra autoload file?

> and the documentation all need to be created before packaging
> things up.

Right, the doc-generation is a step that's clearly lacking in the
current GNU ELPA scripts.


        Stefan

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Stephen Leake]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> that package.  So the first question to decide would be if elpa.git is a
>> source archive or a package archive.
>
> I want it to be a source archive.
>
>
>> In words, the package and version file, a separate autoload file that
>> Org needs (in addition to the autoload file that later gets created on
>> the users' machine)
>
> The org-pkg.el file is nowadays built by the GNU ELPA scripts (there's
> even a special exception in this script specifically for Org, which is
> the only package where this file is not auto-built yet).
>
> What is the "version" file?
>
> Can you give further details on this extra autoload file?
>
>> and the documentation all need to be created before packaging
>> things up.
>
> Right, the doc-generation is a step that's clearly lacking in the
> current GNU ELPA scripts.

Is the intent to build the docs on the GNU ELPA server, and include them
in the tarball?

Or to build them on the user's machine?

I'm guessing the former, to avoid burdening users with tools. That
requires standardizing the tools that the Gnu ELPA server provides.

What will we do when packages start including source code for Emacs
dynamic modules (ie curl, ada-mode parser)? That introduces
user-specific parameters like OS, machine architecture, compiler
version, etc.

I've suggested one solution; use the existing (non-Emacs) OS packaging
systems for those binaries; that avoids the issue for GNU ELPA.

-- 
-- Stephe

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Nic Ferrier]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> that package.  So the first question to decide would be if elpa.git is a
>> source archive or a package archive.
>
> I want it to be a source archive.


You're wrong to want that for several reasons:

- you constrain packages to have source that is managed only by the
  elpa.git makefile

- you hide the build system from most authors

- eventually it won't scale, massive branching in a git repo just to
  manage totally different packages is not what git was made for.


I don't understand why you don't do the obvious thing that nearly every
packaging system does and accept the need for specifically made
artifacts.


Nic

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Stefan Monnier]

>>> that package.  So the first question to decide would be if elpa.git is a
>>> source archive or a package archive.
>> I want it to be a source archive.
> You're wrong to want that for several reasons:

I don't think I can be "wrong" there, tho you may disagree with the
importance of some of my wishes: there has to be a source somewhere, and
I wish it to be available at a single place (which is a currently single
Git repository but could be a tree of repositories, as long as I can
"make pull" and get them all, and as long as they're on gnu.org property
to remind authors that they shouldn't be pushing stuff there that
doesn't have proper copyright).

> - you constrain packages to have source that is managed only by the
>   elpa.git makefile

I could live with a secondary archive of "pre-built" thingies so that
the build can be done externally.

> I don't understand why you don't do the obvious thing that nearly every
> packaging system does and accept the need for specifically made
> artifacts.

I don't know what you mean,


        Stefan

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Nic Ferrier]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I don't think I can be "wrong" there, tho you may disagree with the
> importance of some of my wishes: there has to be a source somewhere, and
> I wish it to be available at a single place (which is a currently single
> Git repository but could be a tree of repositories, as long as I can
> "make pull" and get them all, and as long as they're on gnu.org property
> to remind authors that they shouldn't be pushing stuff there that
> doesn't have proper copyright).

And if they were savannah gnu projects that would be achieved without
the over centralization you have now.

The centralization isn't really a problem right now you think. But I bet
it is. You're making people work inside a source tree that doesn't
belong to them and you're constraining the technical content they put
there.

You're also inviting people to break the Makefile because they want
their own build.

You're also inviting people to check in non-working code.

You might say "these things have not happened yet". But that's because
there are very few ELPA authors so far. Maybe one of the reasons there
are so few ELPA authors is that it's weird.


>> - you constrain packages to have source that is managed only by the
>>   elpa.git makefile
>
> I could live with a secondary archive of "pre-built" thingies so that
> the build can be done externally.

I think that's my main point. There should be a package archive where
authors send their "done" packages.

But I also think you are wrong about requiring everyone to use one
source archive. Just because you want one place where the source
is. I've heard no good justification for that.


>> I don't understand why you don't do the obvious thing that nearly every
>> packaging system does and accept the need for specifically made
>> artifacts.
>
> I don't know what you mean,

I mean: You're doing something very weird. Why?

Weird can be good if you've got a good reason. If you haven't it's just
weird.


Nic

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Stefan Monnier]

> And if they were savannah gnu projects that would be achieved without
> the over centralization you have now.

No: many (most?) savannah gnu projects don't require copyright assignments.
Also, when maintainers disappear, it's rather problematic to get bugs fixed.

> The centralization isn't really a problem right now you think.  But I bet
> it is.  You're making people work inside a source tree that doesn't
> belong to them and you're constraining the technical content they put
> there.

That depends on what you compare it with.  You're comparing it to having
your package on some random Git server somewhere, but if you compare it
to having your package in Emacs itself, then it's much more "your"
source tree, and it has fewer constraints.

> You're also inviting people to break the Makefile because they want
> their own build.

I don't know what you mean by that.

> You're also inviting people to check in non-working code.

How?  Are you referring to "distributing tarballs without testing them"?

> I think that's my main point.  There should be a package archive where
> authors send their "done" packages.

I'd accept patches to the GNU ELPA scripts which lets authors do that.
Note that I've heard comments from other authors who find the "just bump
the version number" way of making a release to be really handy, so
I wouldn't want to force people to make their own archive.

> But I also think you are wrong about requiring everyone to use one
> source archive.  Just because you want one place where the source
> is.  I've heard no good justification for that.

As I said, it could be a tree of repositories.  I have limited time to
spend writing those GNU ELPA scripts, so I did it in the way that seemed
the easiest for me.

As long as I can get a Git checkout of all of them in one "make"
command, and that any Emacs maintainer can commit to them from there
without having to go and chase the "upstream repository/maintainer" du
jour, another organization would probably work just as well.

> I mean: You're doing something very weird. Why?

I guess I just don't know what's weird about it.


        Stefan

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Nic Ferrier]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> No: many (most?) savannah gnu projects don't require copyright
> assignments.  Also, when maintainers disappear, it's rather
> problematic to get bugs fixed.

A GNU project is a GNU project, is my understanding. savannah can host
non-gnu or GNU. GNU projects all need (C) assignment. You can't be GNU
without that.

Maybe that changed? But that's the way I recall it from when savannah
was setup and I was involved with the Java stuff.


I don't see why it's any more easy to get bugs fixed. If a maintainer
for a GNU project disappears there's a regular course of action to chase
them up or hand off control. Isn't there? There used to be.


> That depends on what you compare it with.  You're comparing it to having
> your package on some random Git server somewhere, but if you compare it
> to having your package in Emacs itself, then it's much more "your"
> source tree, and it has fewer constraints.

But my comparison is what most authors will experience.

Unless you're going to only talk to authors who already contribute to
gnu emacs.


>> You're also inviting people to break the Makefile because they want
>> their own build.
>
> I don't know what you mean by that.

I mean that people who want to have an odd build will attempt to make
the Makefile do it and then break it.


>> You're also inviting people to check in non-working code.
>
> How?  Are you referring to "distributing tarballs without testing
> them"?

Yes. But also, my repo is mine. We have to have discipline around the
emacs source tree and I think everyone undestands that's shared. But the
expectation surely would be that my branch of the elpa.git is mine.


>> I think that's my main point.  There should be a package archive where
>> authors send their "done" packages.
>
> I'd accept patches to the GNU ELPA scripts which lets authors do that.
> Note that I've heard comments from other authors who find the "just bump
> the version number" way of making a release to be really handy, so
> I wouldn't want to force people to make their own archive.

Really handy vs safe is something I think should err on the side of
safe.


>> I mean: You're doing something very weird. Why?
>
> I guess I just don't know what's weird about it.

Maybe you don't know enough about software ecosystems?


Nic

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Stefan Monnier]

>> No: many (most?) savannah gnu projects don't require copyright
>> assignments.  Also, when maintainers disappear, it's rather
>> problematic to get bugs fixed.
> A GNU project is a GNU project, is my understanding. savannah can host
> non-gnu or GNU.  GNU projects all need (C) assignment.  You can't be GNU
> without that.

Of course you can.  Bazaar is/was a GNU project, and its copyright
belongs very clearly to Canonical (who used a similar copyright
assignment principle, except without the same guarantees that they
wouldn't misuse that copyright ;-).

See https://www.gnu.org/help/evaluation.html for details of what it
means to be "a GNU package".

> Maybe that changed?

It's been that way since before Savannah existed, AFAIK.

> I don't see why it's any more easy to get bugs fixed.  If a maintainer
> for a GNU project disappears there's a regular course of action to chase
> them up or hand off control.  Isn't there?  There used to be.

It takes months, going through various intermediaries to get access
rights, ...  Whereas with the current setup I can install a bug fix
as soon as git.sv.gnu.org is back online (oh wait, it is back online!),
without even having to think about it.

> But my comparison is what most authors will experience.

Then they won't contribute to GNU ELPA.  We're no worse, since they
wouldn't contribute to Emacs either.

I don't see the point of making GNU ELPA into a copy of MELPA/Marmalade:
those already exist and they work well, AFAICT.

> I mean that people who want to have an odd build will attempt to make
> the Makefile do it and then break it.

Which Makefile?  elpa/Makefile?  This file is not used directly by
elpa.gnu.org, there's a manual step to update the file used by
elpa.gnu.org, so the risk is very small (as long as someone monitors
the elpa-diffs commits, of course).

> Yes.  But also, my repo is mine.  We have to have discipline around the
> Emacs source tree and I think everyone undestands that's shared.  But the
> expectation surely would be that my branch of the elpa.git is mine.

If you don't want Emacs maintainers to contribute directly to your code,
then elpa.git might indeed not be the best choice for you.

> Really handy vs safe is something I think should err on the side of safe.

Given the general quality of Elisp packages, I'm not sure it's worth the
trouble.  Also, in case there's a packaging bug, you can fix it
trivially with a single commit bumping the version one more time.

> Maybe you don't know enough about software ecosystems?

Maybe.


        Stefan

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Stephen J. Turnbull]

Nic Ferrier writes:

 > A GNU project is a GNU project, is my understanding. savannah can host
 > non-gnu or GNU. GNU projects all need (C) assignment. You can't be GNU
 > without that.

No, GNU has never tried to monopolize copyright.  A few "strategic"
projects do require assignment or equivalent (mostly those RMS was
directly involved with creating, such as Emacs, of course, and GCC),
and it's recommended for all projects based on experience with license
migrations.  But it's a project by project decision, and always has
been.

 > I don't see why it's any more easy to get bugs fixed. If a
 > maintainer for a GNU project disappears there's a regular course of
 > action to chase them up or hand off control. Isn't there? There
 > used to be.

Sure, but somebody has recognize that the maintainer has gone AWOL and
then go do the chasing.  This way the bugs reports are shared, to some
extent channels are shared, and it's relatively easy to recognize when
a maintainer has become unresponsive.

Consider the (lack of) success of the "regular course of action"
vis-a-vis Bazaar.  Somebody raised his hand, became the official
maintainer of GNU Bazaar, and has never been heard of again.  The
people who have explicitly disclaimed responsibility for that project
have been far more active!

 > > That depends on what you compare it with.  You're comparing it to having
 > > your package on some random Git server somewhere, but if you compare it
 > > to having your package in Emacs itself, then it's much more "your"
 > > source tree, and it has fewer constraints.
 > 
 > But my comparison is what most authors will experience.

So?  GNU ELPA is part of Emacs, it's not a collection of random packages.

 > Unless you're going to only talk to authors who already contribute to
 > gnu emacs.

s/already/want to/ and you have it!

 > I mean that people who want to have an odd build will attempt to make
 > the Makefile do it and then break it.

I don't see a difference.

 > >> You're also inviting people to check in non-working code.

Again, I don't see a difference.  Between Emacs core and "some random
Git server somewhere", yes.  Not between ELPA and a separate git
project.  People who check in non-working code in random projects in a
common repository can be slapped hard (I've removed commit privileges
after two warnings to revert, Stefan might or might not be more polite
:-).  Ask Micheal Albinus or Alan Mackenzie how often random people
have broken TRAMP or CC-mode in the XEmacs package repo.  It just
doesn't need to happen, because core and the ELPA projects are
different.  The people who maintain Emacs core are grownups, they
don't think that because they're "core" that gives them rights to mess
with projects they're only loosely associated with.

 > > I'd accept patches to the GNU ELPA scripts which lets authors do that.
 > > Note that I've heard comments from other authors who find the "just bump
 > > the version number" way of making a release to be really handy, so
 > > I wouldn't want to force people to make their own archive.
 > 
 > Really handy vs safe is something I think should err on the side of
 > safe.

"Bump the version number" has worked for XEmacs for a decade.  I don't
recall any packaging bugs of the "somebody broke make bootstrap, just
needs a trivial patch" level.

 > >> I mean: You're doing something very weird. Why?
 > >
 > > I guess I just don't know what's weird about it.
 > 
 > Maybe you don't know enough about software ecosystems?

Maybe you don't know enough about Emacs and GNU? :-)  Specifically, the
term "software ecosystem" is deprecated here.  You'd have to read the
FSF FAQ or ask RMS for the party line, but my take on it is that the
FSF and GNU have taken the role of guardian (legal and developer,
repectively) of software freedom, and decisions are taken explicitly
about project and Project organization as well as specific development
practices, and even features, in the interest of free software.
Decisions in these matters are not based on "natural ecological"
relationships among participants.

Note well: In practice, although I'd quibble about details, I prefer
to run things basically in the way you suggest.  But that is not the
way things are done in Emacs or in the free software movement
(vs. open source community) in general.

Regards,

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > A GNU project is a GNU project, is my understanding. savannah can host
  > non-gnu or GNU. GNU projects all need (C) assignment. You can't be GNU
  > without that.

Some GNU packages are FSF-copyrighted, and some are not.
The package's developers decide whether to assign copyright.

Please see the GNU maintainers guide.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Nic Ferrier]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > A GNU project is a GNU project, is my understanding. savannah can host
>   > non-gnu or GNU. GNU projects all need (C) assignment. You can't be GNU
>   > without that.
>
> Some GNU packages are FSF-copyrighted, and some are not.
> The package's developers decide whether to assign copyright.
>
> Please see the GNU maintainers guide.

I stand corrected, thanks rms.

Apologies Stefan.


Nic

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Stephen Leake]

Nic Ferrier <nferrier@ferrier.me.uk> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>> I mean: You're doing something very weird. Why?
>>
>> I guess I just don't know what's weird about it.
>
> Maybe you don't know enough about software ecosystems?

Nic; instead of casting vague accusations/assumptions, please explain
more clearly what your favorite alternative to the current GNU ELPA
system is.

-- 
-- Stephe

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Nic Ferrier]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> Nic Ferrier <nferrier@ferrier.me.uk> writes:
>
>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>
>>>> I mean: You're doing something very weird. Why?
>>>
>>> I guess I just don't know what's weird about it.
>>
>> Maybe you don't know enough about software ecosystems?
>
> Nic; instead of casting vague accusations/assumptions, please explain
> more clearly what your favorite alternative to the current GNU ELPA
> system is.

No? That's not the point. I've provided a critique of what the problems
are with the current approach.

Apparently people don't believe me or think I am exaggerating.

So we should probably just wait to see.



Nic

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Stefan Monnier]

> Apparently people don't believe me or think I am exaggerating.

Sounds like a misunderstanding.  Among the things you seem to want
(IIUC), there are 2 different kinds:
- some that are incompatible with some of the things I want.
- some that are perfectly feasible, but noone has done them yet.
Among the second point, there's for instance "improve the build scripts
of GNU ELPA such that it's easier to run them locally", or "improve the
build scripts such that some or all of the build can be done manually by
the maintainer".


        Stefan

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 1537 bytes --]

() Stefan Monnier <monnier@iro.umontreal.ca>
() Wed, 12 Nov 2014 17:40:34 -0500

   I guess I just don't know what's weird about it.

Well, non-trivial packages require "make dist" to actually
DTRT.  That step presupposes "make" (and for conscientious
authors, "make check").  These are visisble steps and their
machinery is transparent to the authors.  For the user, the
steps are "configure" and "make install".  (All this you
probably know, i expect.)

For trivial packages, it's possible to conflate many of these
explicit steps, and that's what the GNU ELPA system design
does.  The weirdness is that this over-specialization is a
step backward from the Generally Accepted Packaging Practice.
It gives up the benefits of the "separate compilation" model
and forces the "interpreter" (phase-less) model on everyone.

I was initially enthusiastic about "bump version to release"
but now have come to the conclusion that i prefer GAPP.  I
feel more comfortable being responsible for "make check" and
"make dist", and delivering a fully-elaborated package, thus
owning (and learning from) my errors and not pointing fingers.
Call me a control freak, i won't deny it.

Besides, philosophically, when "make dist" is on the server,
then GNU ELPA is effectively SaaSS.  'Nuff said...

-- 
Thien-Thi Nguyen
   GPG key: 4C807502
   (if you're human and you know it)
      read my lisp: (responsep (questions 'technical)
                               (not (via 'mailing-list)))
                     => nil

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Phillip Lord]

Nic Ferrier <nferrier@ferrier.me.uk> writes:
> The centralization isn't really a problem right now you think. But I bet
> it is. You're making people work inside a source tree that doesn't
> belong to them and you're constraining the technical content they put
> there.
>
> You're also inviting people to break the Makefile because they want
> their own build.
>
> You're also inviting people to check in non-working code.
>
> You might say "these things have not happened yet". But that's because
> there are very few ELPA authors so far. Maybe one of the reasons there
> are so few ELPA authors is that it's weird.


There is some truth in this. I feel rather more nervous commting into
ELPA because it contains so many pieces of work from others. With my own
repo's, that's fine. If I screw things up, then it's my problem.

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Nic Ferrier]

phillip.lord@newcastle.ac.uk (Phillip Lord) writes:

> Nic Ferrier <nferrier@ferrier.me.uk> writes:
>> The centralization isn't really a problem right now you think. But I bet
>> it is. You're making people work inside a source tree that doesn't
>> belong to them and you're constraining the technical content they put
>> there.
>>
>> You're also inviting people to break the Makefile because they want
>> their own build.
>>
>> You're also inviting people to check in non-working code.
>>
>> You might say "these things have not happened yet". But that's because
>> there are very few ELPA authors so far. Maybe one of the reasons there
>> are so few ELPA authors is that it's weird.
>
>
> There is some truth in this. I feel rather more nervous commting into
> ELPA because it contains so many pieces of work from others. With my own
> repo's, that's fine. If I screw things up, then it's my problem.

Kinda implies there isn't any truth in the rest of it :-(


Nic

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Phillip Lord]

Nic Ferrier <nferrier@ferrier.me.uk> writes:

> phillip.lord@newcastle.ac.uk (Phillip Lord) writes:
>
>> Nic Ferrier <nferrier@ferrier.me.uk> writes:
>>> The centralization isn't really a problem right now you think. But I bet
>>> it is. You're making people work inside a source tree that doesn't
>>> belong to them and you're constraining the technical content they put
>>> there.
>>>
>>> You're also inviting people to break the Makefile because they want
>>> their own build.
>>>
>>> You're also inviting people to check in non-working code.
>>>
>>> You might say "these things have not happened yet". But that's because
>>> there are very few ELPA authors so far. Maybe one of the reasons there
>>> are so few ELPA authors is that it's weird.
>>
>>
>> There is some truth in this. I feel rather more nervous commting into
>> ELPA because it contains so many pieces of work from others. With my own
>> repo's, that's fine. If I screw things up, then it's my problem.
>
> Kinda implies there isn't any truth in the rest of it :-(

But only kinda implies, not actually implies.

For the record, I only sort of agree with you. First, I think you are
overplaying things somewhat. I can see the issues that you are talking
about, but I am not certain how common they are. If it's just org-mode
and maybe semantic that a complicated build, then perhaps special case
treatment is the best way forward.

Second, I do not think that the problem is that elpa.git is a source
archive, or that the artifacts are build away from the developer.
It is a worry that it may not be so easy to reproduce. So, for example,
with MELPA, I can pull down the whole thing (small at 17Mb, because it
only contains recipies). Then

make recipies/pabbrev

builds my package. The point is that the build is *replicable* from the
source; I can do it locally, even though normally, I do not.

Finally, I think your complaints about elpa.git are also reasonable; I
would prefer to be using my own repo for my packages. Getting everything
set up on an ELPA branch has not been trivial (and is still not entirely
working for reasons I cannot figure). However, I can appreciate Stefan's
position. He has already made some changes to my packages and
improved them. For me, the cost of raising my own activation energy in
contributing to ELPA is probably worth the benefit of lowering his.

Phil

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Nic Ferrier]

phillip.lord@newcastle.ac.uk (Phillip Lord) writes:

> For the record, I only sort of agree with you. First, I think you are
> overplaying things somewhat.

Am not.

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Achim Gratz]

Stefan Monnier writes:
> The org-pkg.el file is nowadays built by the GNU ELPA scripts (there's
> even a special exception in this script specifically for Org, which is
> the only package where this file is not auto-built yet).
>
> What is the "version" file?

I meant org-version.el, which makes each release traceable for bug
reports.

> Can you give further details on this extra autoload file?

Org has split the autoload into the "normal" autoloads that are roughly
for commands that the user should be able to use without loading Org
first and a separate one that provides autoloads for the entry points
into more rarely used parts of Org that don't need to be loaded up-front
(these only get loaded when Org is actually used).  That setup was
gleaned from calc IIRC, with some nudging from Glenn.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Stefan Monnier]

>> Can you give further details on this extra autoload file?
> Org has split the autoload into the "normal" autoloads that are roughly
> for commands that the user should be able to use without loading Org
> first and a separate one that provides autoloads for the entry points
> into more rarely used parts of Org that don't need to be loaded up-front
> (these only get loaded when Org is actually used).  That setup was
> gleaned from calc IIRC, with some nudging from Glenn.

If there is no file that has both types of autoloads, then you don't
need a separate run of generate-autoloads, you can just use

   ;; Local variables:
   ;; generated-autoload-file: "org-secondary-autoloads.el"
   ;; End:


-- Stefan

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Achim Gratz]

Stefan Monnier writes:
> If there is no file that has both types of autoloads, then you don't
> need a separate run of generate-autoloads, you can just use

But Org does have those.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf rackAttack V1.04R1:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Stephen Leake]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> The way to "test" for elpa.git is "cd .../elpa; make" which should
> create the *-autoloads.el and *-pkg.el and *.elc files in the "same" way
> as what will happen on the user's machine.

This is not a good test. For example, if the developer forgets to add
a file to elpa.git, this test will pass, but a true install will fail
(happened to me recently).

I use these make targets to build a package-archive that package.el can
use and install from it:

build-elpa : force
	rm -rf ../../elpa/archive-tmp
	mkdir -p ../../elpa/archive-tmp/packages
	cp -a ../../elpa/packages/ada-mode ../../elpa/archive-tmp/packages
	cp -a ../../elpa/packages/ada-ref-man ../../elpa/archive-tmp/packages
	cp -a ../../elpa/packages/wisi     ../../elpa/archive-tmp/packages
	make -C ../../elpa/ process-archive

install-elpa :
	emacs -Q --eval '(progn (load-file "install-elpa.el")(kill-emacs))'

That is a test of the complete process, assuming there are no steps on
the GNU ELPA server that are not in the elpa.git makefile.

-- 
-- Stephe

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Phillip Lord]

Nic Ferrier <nferrier@ferrier.me.uk> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>> I am still cross about the whole state of packaging. It's wrong. But no
>>> one seems to be listening to me.
>>
>> I'm not sure if you're referring to the way GNU ELPA works or about
>> package.el.  About GNU ELPA, it's not that I don't listen, but that
>> I disagree.  The point of GNU ELPA to not to just distribute other
>> people's packages.
>
> The multi-packages users load in their emacs are tars. But the packages
> that are checked in to ELPA are directories of files.
>
> So package authors are not checking in what gets delivered to the
> user. So there is a magic build step somewhere.


It is certainly the case that I most "test" my packages by using them.
And I do not install my own packages through ELPA but live from their
VC'd repository. So, there is a worry here. This is even true during my
continuous integration tests (the dependencies are loaded from an ELPA
repo, but nothing else).

Having said that, I generally use an automatic tool to move from the
source repository format to the package format; or, in the case of
MELPA, someone else does it for me. Once this is set up, I am struggling
to think of bugs that have come at package time.


> This discourages testing of packages before they are distributed.
>
> And I am really starting to think we need better testing. 24.4 looked
> like a slog to release and it still has many bugs.
>
> And yes, I promise to help more. But that's not really the issue. This
> isn't scaling. We need to have better QA tools. And packages having
> hidden builds is going the other way.


Not going to argue here. Slowing moving my elisp to a testable state has
made me happier.

Phil

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Nic Ferrier]

phillip.lord@newcastle.ac.uk (Phillip Lord) writes:

> It is certainly the case that I most "test" my packages by using them.
> And I do not install my own packages through ELPA but live from their
> VC'd repository. So, there is a worry here. This is even true during my
> continuous integration tests (the dependencies are loaded from an ELPA
> repo, but nothing else).
>
> Having said that, I generally use an automatic tool to move from the
> source repository format to the package format; or, in the case of
> MELPA, someone else does it for me. Once this is set up, I am struggling
> to think of bugs that have come at package time.

I do this too.

Obvious things that break because of no package testing:

- autoloads missing (only really fixed by package-install-ing)
- expected depend missing (can be fixed by requiring the source in fresh
  emacs)



Nic

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Stefan Monnier]

> - autoloads missing (only really fixed by package-install-ing)
> - expected depend missing (can be fixed by requiring the source in fresh
>   emacs)

Neither of this is affected by the "(build)" step of GNU ELPA, AFAICT.


        Stefan

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Nic Ferrier]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> - autoloads missing (only really fixed by package-install-ing)
>> - expected depend missing (can be fixed by requiring the source in fresh
>>   emacs)
>
> Neither of this is affected by the "(build)" step of GNU ELPA, AFAICT.

That's right. But because the build isn't exposed people don't test
their packages and this bit is never tested.

As Philip said, when he's building software he'll just edit and eval in
his emacs. In the normal way (me too btw).

But this isn't smalltalk. When you're done you can't just drop what you
did into elpa.git and expect it to work. You have to test the
package-installed version in a new emacs session with no config because
that's the only time you have byte compilation, autoload generation and
fresh dependency analysis.


Nic

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Stefan Monnier]

> That's right.  But because the build isn't exposed people don't test
> their packages and this bit is never tested.

How would you ideally like to see it exposed?


        Stefan

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Stephen Leake]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> That's right.  But because the build isn't exposed people don't test
>> their packages and this bit is never tested.
>
> How would you ideally like to see it exposed?

It should be in the elpa.git Makefile and/or associated scripts, so
developers can run it on their machine for testing.

Is there part of the current build that is not in elpa.git?

-- 
-- Stephe

Re: package and testing rant (was Re: package.el, auto-installation, and auto-removal)

[Stefan Monnier]

> Is there part of the current build that is not in elpa.git?

The only part that's not in there is the archive signing code.


        Stefan

Re: package.el, auto-installation, and auto-removal

[Stefan Monnier]

> Yes. I think that's what Stefan was asking for. But Stefan seems to
> prefer the elpakit approach, which is fine too (Nic has a nice
> comparison section at https://github.com/nicferrier/elpakit).

That's a misunderstanding.  I don't prefer any of those options because
I don't actually know anything about them (or close enough).

> (Stefan: would you and the other maintainers be open to Cask+Pallet
> *and* elpakit in Emacs?  Or is that too much choice?)

If someone cared to explain what that would do and why that would imply
"choice", I could tell you.

Nic says:
> I don't think that was Stefan's position. His position was "can we have
> this feature?"
> I don't think he cares who does it or how it's done.

Exactly.


        Stefan

Re: package.el, auto-installation, and auto-removal

[Ted Zlatanov]

On Mon, 10 Nov 2014 16:37:50 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> Yes. I think that's what Stefan was asking for. But Stefan seems to
>> prefer the elpakit approach, which is fine too (Nic has a nice
>> comparison section at https://github.com/nicferrier/elpakit).

SM> That's a misunderstanding.  I don't prefer any of those options because
SM> I don't actually know anything about them (or close enough).

>> (Stefan: would you and the other maintainers be open to Cask+Pallet
>> *and* elpakit in Emacs?  Or is that too much choice?)

SM> If someone cared to explain what that would do and why that would imply
SM> "choice", I could tell you.

I provided URLs to Pallet's repo on Github and to Nic's comparison of
Cask (which Pallet uses underneath) and elpakit (the comparison is on
the elpakit repo home page, also on Github). I described how Pallet
currently works with advice and how it could be improved to use hooks.

Sorry if I didn't explain things better.  What's unclear?

In terms of popularity, Cask seems really popular so it's worth at least
a look.

Ted

Re: package.el, auto-installation, and auto-removal

[Stefan Monnier]

> I provided URLs to Pallet's repo on Github and to Nic's comparison of
> Cask (which Pallet uses underneath) and elpakit (the comparison is on
> the elpakit repo home page, also on Github).  I described how Pallet
> currently works with advice and how it could be improved to use hooks.
> Sorry if I didn't explain things better.  What's unclear?

I guess I just don't understand what those things do, nor how they do
it, and your explanations didn't really help.

E.g. does one of those things make it so that M-x package-install will
add the package to some list of "explicitly installed packages"?
If so, which tool, and how does it do it?  AFAICT it can only do that
via advice currently.  Of course, it can be changed to use hooks, but
why bother, when we can just modify package-install directly?


        Stefan

Re: package.el, auto-installation, and auto-removal

[Ted Zlatanov]

On Mon, 10 Nov 2014 21:26:14 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> I provided URLs to Pallet's repo on Github and to Nic's comparison of
>> Cask (which Pallet uses underneath) and elpakit (the comparison is on
>> the elpakit repo home page, also on Github).  I described how Pallet
>> currently works with advice and how it could be improved to use hooks.
>> Sorry if I didn't explain things better.  What's unclear?

SM> I guess I just don't understand what those things do, nor how they do
SM> it, and your explanations didn't really help.

Sorry.

SM> E.g. does one of those things make it so that M-x package-install will
SM> add the package to some list of "explicitly installed packages"?
SM> If so, which tool, and how does it do it?  AFAICT it can only do that
SM> via advice currently.  Of course, it can be changed to use hooks, but
SM> why bother, when we can just modify package-install directly?

Pallet hooks (through advice) into package installations, plus it can
discover the starting list of packages. It then maintains a Cask file.
So it does what you asked for at the beginning of this thread, but
relays the actual package management to Cask.

I agree that the integration of (Pallet|elpakit) can be as tight as we
want, no need for hooks.

Cask manages a list of packages in an external file. It's aware of
dependencies, repos, etc. and is quite popular for its ease of use. As
Phillip Lord said, bundling Cask would remove its Python requirement and
perhaps improve the user experience with package.el and the various
ELPAs.

I would love to hear from people that actively use Pallet/Cask or
elpakit. I've used el-get and use-package over the last few years, and
tried Pallet/Cask only recently as I heard about them from friends.

Ted

Re: package.el, auto-installation, and auto-removal

[Stefan Monnier]

> Pallet hooks (through advice) into package installations, plus it can
> discover the starting list of packages. It then maintains a Cask file.

OK, next question: what's a Cask file?

> So it does what you asked for at the beginning of this thread, but
> relays the actual package management to Cask.

You mean package.el isn't actually used?

> Cask manages a list of packages in an external file. It's aware of
> dependencies, repos, etc. and is quite popular for its ease of use.

"Aware of" isn't sufficient for me to understand what it does with that info.

> As Phillip Lord said, bundling Cask would remove its Python
> requirement and perhaps improve the user experience with package.el
> and the various ELPAs.

What functionality does it provide for the user (other than
auto-removal of the packages that were not explicitly installed, and
the ability to install the same set of packages, obviously)?


        Stefan

Re: package.el, auto-installation, and auto-removal

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Pallet hooks (through advice) into package installations, plus it can
>> discover the starting list of packages. It then maintains a Cask file.
>
> OK, next question: what's a Cask file?
>
>> So it does what you asked for at the beginning of this thread, but
>> relays the actual package management to Cask.
>
> You mean package.el isn't actually used?
>
>> Cask manages a list of packages in an external file. It's aware of
>> dependencies, repos, etc. and is quite popular for its ease of use.
>
> "Aware of" isn't sufficient for me to understand what it does with that info.
>
>> As Phillip Lord said, bundling Cask would remove its Python
>> requirement and perhaps improve the user experience with package.el
>> and the various ELPAs.
>
> What functionality does it provide for the user (other than
> auto-removal of the packages that were not explicitly installed, and
> the ability to install the same set of packages, obviously)?


So, a brief outline of the various mechanisms for managing packages out of band.

Cask is a package (with a small python launcher) that uses a "Cask"
file. This is the cask file from cask itself.


(source gnu)
(source melpa)
(package-file "cask.el")
;; (files "*.el" "bin" "templates")
(development
(depends-on "f")
(depends-on "s"))

It looks like Lisp but actually isn't; you can't put conditional
statements in it for instance. Emacs reads it, and installs the
relevant files, using package.el to do so. Cask can be used to
configure your "daily" emacs. You can synchronise between several
machines by just syncing your Cask file, and then running "cask
install" which will pull down all the relevant packages. I do not use
Cask for this purpose though.

It also lets you run Emacs in batch, with a "sandbox" load-path; I use
this to run unit tests both locally and on a continuous integration
server. For this use, cask also integrates nicely with evm, which lets
you tests for several versions of Emacs. I do use cask for this, and
find it works well.

Pallet is a package which advices the interactive features of
package.el. When you install a package, it also writes the Cask file
for you. I don't use pallet myself. So, it provides a GUI mechanism
for managing your Cask file.

In both cases, there is not specific support for deleting unused
packages, although the data is there. In practice, you can just delete
your elpa directory, and then run "cask install" to pull the stuff you
need again.

For myself, I John Wigely's use-package -- it provides a nice
convenient way to configure and customise packages (something that
Cask does not). So, you can do something like this...

(use-package ace-jump-mode
  :bind ("C-." . ace-jump-mode))


It also has an ":ensure" keyword. So 

(use-package ace-jump-mode
  :ensure t
  :bind ("C-." . ace-jump-mode))

Will ensure that the package is installed and do so if not. I like
this because I can just sync my .emacs file between (which I am going
to do anyway). When I want to use a new package, I add a use-package
statement to my .emacs, and the package installs on my machines next
time I start Emacs. This works nicely, although I would quite like a
command line tool for managing packages, it doesn't seem worth the
effort of moving to Cask at the moment.

At the moment, I have not tried elpakit. Sorry, Nic, I probably will
give it a go now.

I think that any of the solutions would work. What is interesting is
that both Elpakit and Cask have additional stuff added. I've been
really glad to get Cask working and have it running my test cases for
me, outside of my daily emacs. But it's not strictly necessary for
persistantly managing packages.

Re: package.el, auto-installation, and auto-removal

[Nic Ferrier]

phillip.lord@newcastle.ac.uk (Phillip Lord) writes:

> At the moment, I have not tried elpakit. Sorry, Nic, I probably will
> give it a go now.

Well, you don't need to.

And I'm in the middle of moving it to a version 2.0.


> I think that any of the solutions would work. What is interesting is
> that both Elpakit and Cask have additional stuff added. I've been
> really glad to get Cask working and have it running my test cases for
> me, outside of my daily emacs. But it's not strictly necessary for
> persistantly managing packages.

The point about these things is that they're the only way you've got of
testing really. There's an enormous amount of grunt work in testing
stuff:

- make your package
- start up a test emacs
- install your package
- run the tests
- see the results

that's horrid boring work that both cask and elpakit try and
automate. For me it's not the out-of-bandness so much... though
repeatability clearly guarantees that.


Nic

Re: package.el, auto-installation, and auto-removal

[Nic Ferrier]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I provided URLs to Pallet's repo on Github and to Nic's comparison of
>> Cask (which Pallet uses underneath) and elpakit (the comparison is on
>> the elpakit repo home page, also on Github).  I described how Pallet
>> currently works with advice and how it could be improved to use hooks.
>> Sorry if I didn't explain things better.  What's unclear?
>
> I guess I just don't understand what those things do, nor how they do
> it, and your explanations didn't really help.
>
> E.g. does one of those things make it so that M-x package-install will
> add the package to some list of "explicitly installed packages"?
> If so, which tool, and how does it do it?  AFAICT it can only do that
> via advice currently.  Of course, it can be changed to use hooks, but
> why bother, when we can just modify package-install directly?

Having thought about it that seems like the best thing. Just add another
property to the package somehow that says "the user installed this
directly". Maybe keep that along side the archive as well as in the
archive so that it can be mixed in when you refresh the indexes.

It would be nice to keep it as a directory structure because they are
easier to edit:

.emacs.d/elpa/
.emacs.d/elpa/archives/
.emacs.d/elpa/archives/marmalade/
.emacs.d/elpa/archives/gnu/
.emacs.d/elpa/archives/user-installed/
.emacs.d/elpa/archives/user-installed/elnode
.emacs.d/elpa/archives/user-installed/rcirc


Nic

Re: package.el, auto-installation, and auto-removal

[Stefan Monnier]

> It would be nice to keep it as a directory structure because they are
> easier to edit:
> .emacs.d/elpa/
> .emacs.d/elpa/archives/
> .emacs.d/elpa/archives/marmalade/
> .emacs.d/elpa/archives/gnu/
> .emacs.d/elpa/archives/user-installed/
> .emacs.d/elpa/archives/user-installed/elnode
> .emacs.d/elpa/archives/user-installed/rcirc

I was thinking of keeping it in a Custom var, since that's easy to edit.
So the equivalent of a Cask file could look like

   % cat CaskNG.el
   (setq package-archives ((gnu ...)
                           (melpa ...)))
   (setq packages-installed '(dash foo bar))
   ...
   %

and you could run your tests with something like

   emacs --batch -l CaskNG.el -f package-install-packages ...

So, my request is for someone to help set up the core of this, i.e. add
the `packages-installed' var (automatically updated by
package-install/delete and friends (I notice now that package-delete is
not a command, surprisingly)), and add the package-install-packages
command (so users can install their packages on a new system simply by
setting packages-installed and then running package-install-packages).
Bonus points for writing `package-gc' which deletes packages not
required by packages-installed.


        Stefan

Re: package.el, auto-installation, and auto-removal

[Nic Ferrier]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> It would be nice to keep it as a directory structure because they are
>> easier to edit:
>> .emacs.d/elpa/
>> .emacs.d/elpa/archives/
>> .emacs.d/elpa/archives/marmalade/
>> .emacs.d/elpa/archives/gnu/
>> .emacs.d/elpa/archives/user-installed/
>> .emacs.d/elpa/archives/user-installed/elnode
>> .emacs.d/elpa/archives/user-installed/rcirc
>
> I was thinking of keeping it in a Custom var, since that's easy to edit.
> So the equivalent of a Cask file could look like
>
>    % cat CaskNG.el
>    (setq package-archives ((gnu ...)
>                            (melpa ...)))
>    (setq packages-installed '(dash foo bar))
>    ...
>    %
>
> and you could run your tests with something like
>
>    emacs --batch -l CaskNG.el -f package-install-packages ...

I don't think a custom var is as good as something persistent in the
file system. It would be completely orthogonal to the existing data
storage which is all in the elpa dir (even the package-archives variable
is just an index of what is actually stored nicely in the elpa
directory, you can't recreate it wholly from there, but it's good
enough).

Great that you used a cask example. Because you'd actually need python
to run that. Hurray!


> So, my request is for someone to help set up the core of this, i.e. add
> the `packages-installed' var (automatically updated by
> package-install/delete and friends (I notice now that package-delete is
> not a command, surprisingly)), and add the package-install-packages
> command (so users can install their packages on a new system simply by
> setting packages-installed and then running package-install-packages).
> Bonus points for writing `package-gc' which deletes packages not
> required by packages-installed.

Yeah, as I say elpakit already has this. I could extract the code.

One thing about elpakit's solution is that elpakit does this by making a
package with the right dependencies.

Elpakit's solution is in fact just a package depend builder. Choose X, Y
and Z packages and then make a new package depending on those.

Maybe the current package installed by user state could be constantly
regenerated as a package.

The reason elpakit does that was in response to teams of clojure users
who wanted to homogenize their emacs config. So we made a system where
they could generate a base package to pull in all standard packages. And
then they could all iterate on top of that.

So making a new package all the time would be kinda useful because you
could easily share it with other people.

What do you think of that idea?


Nic

Re: package.el, auto-installation, and auto-removal

[Achim Gratz]

Nic Ferrier writes:
> One thing about elpakit's solution is that elpakit does this by making a
> package with the right dependencies.
>
> Elpakit's solution is in fact just a package depend builder. Choose X, Y
> and Z packages and then make a new package depending on those.

So I'd have an (empty) package named "user-packages" and it depends on
exactly those packages I named, possibly with their version specified?
And I could maybe have a number of additional packages to create
something of a hierarchy of such things?  Like a "system-packages"
package that my "user-packages" package can depend on (or not if I don't
want to)?

> Maybe the current package installed by user state could be constantly
> regenerated as a package.

Or maybe just skip the actual packaging step and really only deal with
the dependencies.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Terratec KOMPLEXER:
http://Synth.Stromeko.net/Downloads.html#KomplexerWaves

Re: package.el, auto-installation, and auto-removal

[Nic Ferrier]

Achim Gratz <Stromeko@nexgo.de> writes:

> Nic Ferrier writes:
>> One thing about elpakit's solution is that elpakit does this by making a
>> package with the right dependencies.
>>
>> Elpakit's solution is in fact just a package depend builder. Choose X, Y
>> and Z packages and then make a new package depending on those.
>
> So I'd have an (empty) package named "user-packages" and it depends on
> exactly those packages I named, possibly with their version specified?
> And I could maybe have a number of additional packages to create
> something of a hierarchy of such things?  Like a "system-packages"
> package that my "user-packages" package can depend on (or not if I don't
> want to)?

Yes.

Elpakit let's you do that. Stefan is asking for something simpler than
that. And I agree, something simpler than that is a good thing.


>> Maybe the current package installed by user state could be constantly
>> regenerated as a package.
>
> Or maybe just skip the actual packaging step and really only deal with
> the dependencies.

Sure. But packages are a useful encapsulation of dependencies.


Nic

Re: package.el, auto-installation, and auto-removal

[Stefan Monnier]

> I don't think a custom var is as good as something persistent in the
> file system.

OTOH, a Custom var is the standard way Emacs stores persistent
user choices (and the set of installed packages is a user choice).

>> So I'd have an (empty) package named "user-packages" and it depends on
>> exactly those packages I named, possibly with their version specified?
>> And I could maybe have a number of additional packages to create
>> something of a hierarchy of such things?  Like a "system-packages"
>> package that my "user-packages" package can depend on (or not if I don't
>> want to)?
> Elpakit let's you do that.

Ah, that's where you're going.  I kind of like the idea, but I don't
like the idea of editing the content of packages in ~/.emacs.d/elpa
(even if only for some "special" packages like "user-package").


        Stefan

Re: package.el, auto-installation, and auto-removal

[Nic Ferrier]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Ah, that's where you're going.  I kind of like the idea, but I don't
> like the idea of editing the content of packages in ~/.emacs.d/elpa
> (even if only for some "special" packages like "user-package").

You don't edit that. You edit the source somewhere else and then
automatically install it.

I'd suggest that the package could be constructed automatically every
time. If the necessary data is kept in a customize variable anyway it
doesn't matter about the package source.

It only matters that a package is a good way to load other packages.


Nic

Re: package.el, auto-installation, and auto-removal

[Stefan Monnier]

>> Ah, that's where you're going.  I kind of like the idea, but I don't
>> like the idea of editing the content of packages in ~/.emacs.d/elpa
>> (even if only for some "special" packages like "user-package").
> You don't edit that. You edit the source somewhere else and then
> automatically install it.
> I'd suggest that the package could be constructed automatically every
> time.  If the necessary data is kept in a customize variable anyway it
> doesn't matter about the package source.
> It only matters that a package is a good way to load other packages.

If you want your "user-packages" package to be versioned, or if other
packages start refer to such meta-packages, then I can agree that it
could make sense, but otherwise I get the sensation that
- de-install the previous user-packages package.
- make up a dummy .el file with the Package-Requires: header.
- install this dummy .el package.
is kind of contrived.

Within package.el we could handle it is a special kind of package
instead, but I'd rather try to avoid reifying it into an actual
directory under ~/.emacs.d/elpa.


        Stefan

Re: package.el, auto-installation, and auto-removal

[Nic Ferrier]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> Ah, that's where you're going.  I kind of like the idea, but I don't
>>> like the idea of editing the content of packages in ~/.emacs.d/elpa
>>> (even if only for some "special" packages like "user-package").
>> You don't edit that. You edit the source somewhere else and then
>> automatically install it.
>> I'd suggest that the package could be constructed automatically every
>> time.  If the necessary data is kept in a customize variable anyway it
>> doesn't matter about the package source.
>> It only matters that a package is a good way to load other packages.
>
> If you want your "user-packages" package to be versioned, or if other
> packages start refer to such meta-packages, then I can agree that it
> could make sense, but otherwise I get the sensation that
> - de-install the previous user-packages package.
> - make up a dummy .el file with the Package-Requires: header.
> - install this dummy .el package.
> is kind of contrived.

Unless you have such a "fake" package you cannot easily share your
packages with anyone else.

In fact, I don't think that a package representing just my package state
is particularly useful. I use this sort of package as a way of passing
round a curated list of packages to other users, in a team at work for
example.

With what you're proposing though I can see a "starter kit" like thing
happening.


"new users should install technomancy's user-packages because they're a
great starting point" people will say.


> Within package.el we could handle it is a special kind of package
> instead, but I'd rather try to avoid reifying it into an actual
> directory under ~/.emacs.d/elpa.

I don't see the point otherwise. It's the simplest thing to do. Users
don't spend a lot of time looking at this stuff.

Nic

Re: package.el, auto-installation, and auto-removal

[Stefan Monnier]

> Unless you have such a "fake" package you cannot easily share your
> packages with anyone else.

It can be shared the same way other Custom settings can be shared.

> In fact, I don't think that a package representing just my package state
> is particularly useful.  I use this sort of package as a way of passing
> round a curated list of packages to other users, in a team at work for
> example.

But, IIUC, the current package.el supports this use case already without
any problem, right?

Anyway, I'll just drop this discussion because it's taking much more
time than writing the code.  All I'm after is a way to get things like
auto-removal of unrequired packages, and to have a more declarative way
to specify which packages you want to install and see which packages
you installed.

> "new users should install technomancy's user-packages because they're a
> great starting point" people will say.

But people can already do that, so it's a different issue from the one
I'm after.


        Stefan

Re: package.el, auto-installation, and auto-removal

[Nic Ferrier]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> `package.el' is great to install packages more easily, but it falls short
> when you want to keep the same set of installed packages on various
> machines, or when it's time to remove package you don't need because it
> doesn't distinguish between the packages you want and those required
> by those packages.
>
> So I'm looking for some kind soul who could extend package.el so as to
> keep track of the packages that the user installed explicitly.
> This should likely be kept in a custom variable.
>
> Then we could add a command to install all the packages listed in
> that variable.  And we could also write a function that deletes all
> "orphan" packages.

Elpakit pretty much does this already.


Nic

Re: package.el, auto-installation, and auto-removal

[Stefan Monnier]

> Elpakit pretty much does this already.

Could someone turn it into a patch that we could install into Emacs?


        Stefan

Re: package.el, auto-installation, and auto-removal

[Nic Ferrier]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Elpakit pretty much does this already.
>
> Could someone turn it into a patch that we could install into Emacs?

Possibly. Do you only wants bits of it?


Nic

Re: package.el, auto-installation, and auto-removal

[Stefan Monnier]

>>> Elpakit pretty much does this already.
>> Could someone turn it into a patch that we could install into Emacs?
> Possibly. Do you only wants bits of it?

As stated originally, I want to change the behavior of package.el so as
to keep track of which packages were installed explicitly.

I don't really know elpakit does, so I can't really tell you what "bits
of it" would mean and what the other implicit option (probably "the
whole elpakit package") would do instead.


        Stefan

Re: package.el, auto-installation, and auto-removal

[Robert Dallas Gray]

As an addendum to the points that have been raised on this topic, and as the author of Pallet, I thought it worth chipping in a few tuppences. 

There are several solutions to the problem Stefan outlines, with varying degrees of complexity and flexibility. To my mind, Cask is the least complex and most flexible. 

Cask only requires Python when used from the command line. There is a version available as a package.el package; this is what Pallet uses as its backend.

Pallet, as has been stated, is a dirt-simple layer over Cask, which rewrites the Cask manifest when the user installs or deletes packages using the package.el subsystem.

Pallet doesn't yet support version-controlled packages (outside of the package.el ecosystem), but Cask does. Pallet will also, when I have time.

Cask uses a tried-and-tested text-file manifest strategy, which has worked well for, e.g., Ruby's bundler and Javascript's npm. The syntax is a subset of Emacs Lisp.

Cask's author, Johan Andersson (https://github.com/rejeep) has also written epl (https://github.com/cask/epl), a layer over package.el which abstracts the package.el api, and servant (https://github.com/cask/servant), a mock package.el server. He knows the vagaries of package.el very well indeed.

I don't really have a dog in this fight, as I respect the authors and codebases of elpakit, Cask and use-package equally (I don't know enough about el-get to have an opinion), and as Pallet is so functionally trivial. 

But from the point of view of testability, at least, I've had success with using the Cask/epl/servant ecosystem to develop a package, and a set of integration tests which work across several Emacs versions (not so trivial). So I'll throw my hat in the ring for something like Cask and something on top of it like Pallet, even if it's neither of those actual things.

I think a consensus on something simple and flexible would be a great thing for Emacs.

Robert Dallas Gray

Re: package.el, auto-installation, and auto-removal

[Nic Ferrier]

Robert Dallas Gray <mail@robertdallasgray.com> writes:

> But from the point of view of testability, at least, I've had success
> with using the Cask/epl/servant ecosystem to develop a package, and a
> set of integration tests which work across several Emacs versions (not
> so trivial). So I'll throw my hat in the ring for something like Cask
> and something on top of it like Pallet, even if it's neither of those
> actual things.

This is something elpakit doesn't try and do, facilitate testing across
emacs-versions.

Cask will let you install various versions of emacs and test on them.


Nic

Re: package.el, auto-installation, and auto-removal

[Phillip Lord]

Nic Ferrier <nferrier@ferrier.me.uk> writes:

> Robert Dallas Gray <mail@robertdallasgray.com> writes:
>
>> But from the point of view of testability, at least, I've had success
>> with using the Cask/epl/servant ecosystem to develop a package, and a
>> set of integration tests which work across several Emacs versions (not
>> so trivial). So I'll throw my hat in the ring for something like Cask
>> and something on top of it like Pallet, even if it's neither of those
>> actual things.
>
> This is something elpakit doesn't try and do, facilitate testing across
> emacs-versions.
>
> Cask will let you install various versions of emacs and test on them.


It's evm which does this, rather than cask. Probably you could get it to
work with elpakit also.

Phil