cl-defstruct-based package.el, now with ert tests!

cl-defstruct-based package.el, now with ert tests!

[Daniel Hackney]

I've mentioned this project on the list before, and I want to update
everyone now that I have more thorough testing.

My aim has been to clean up package.el, primarily by using
`cl-defstruct' to create a `package-desc' structure which is used
throughout the library. This replaces the current situation, in which
package information floats through the library in several different
forms. Sometimes a package is represented as a `cons' of the package
name and a vector of information, sometimes (as in "foo-pkg.el" files),
it is represented more like a list, and sometimes as a name and version
as separate arguments. Moving to a single, uniform representation
simplifies extensibility.

The changes touch most of the code, but the overall structure of the
code has not changed significantly. Some parts of the code which were
questionable in the first place (such as using `tar-mode' to extract tar
archives) have been replaced with simpler alternatives. In the case of
tar, the file is simply downloaded to a temporary directory and
extracted using "tar xaf foo-1.2.3.tar".

This reorganization will make it easier to add features, most notably
the use of compressed archives and signatures, since the code is better
organized and easier to modify.

An important part of the work is that it is fully compatible with the
externally-facing "API" of the current package.el. It produces and
consumes exactly the same "archive-contents" and "*-pkg.el" files as the
current package.el, so none of the existing infrastructure needs to
change.

Finally, I have written a set of tests using ERT to ensure correct
behavior. The file is in "test/automated" and tests:

- single- and multi-file package installation
- installing packages with dependencies
- refreshing archive contents
- displaying the "*Packages*" buffer
- Upgrading packages using `package-menu-mark-upgrades'

All of the tests pass for me.

It is rebased off of the current trunk. My rewrite is here:

https://github.com/haxney/emacs/tree/package-rewrite

Hope you enjoy!

--
Daniel Hackney

Re: cl-defstruct-based package.el, now with ert tests!

[Tom Tromey]

Daniel> My aim has been to clean up package.el, primarily by using
Daniel> `cl-defstruct' to create a `package-desc' structure which is
Daniel> used throughout the library.

This sounds good.

Daniel> The changes touch most of the code, but the overall structure of the
Daniel> code has not changed significantly. Some parts of the code which were
Daniel> questionable in the first place (such as using `tar-mode' to extract tar
Daniel> archives) have been replaced with simpler alternatives. In the case of
Daniel> tar, the file is simply downloaded to a temporary directory and
Daniel> extracted using "tar xaf foo-1.2.3.tar".

This sounds worse to me, since it means that now package installation
has an external dependency.  Previously this was not the case --
intentionally.

Tom

Re: cl-defstruct-based package.el, now with ert tests!

[Dmitry Gutov]

Tom Tromey <tromey@redhat.com> writes:
> Daniel> The changes touch most of the code, but the overall structure of the
> Daniel> code has not changed significantly. Some parts of the code which were
> Daniel> questionable in the first place (such as using `tar-mode' to extract tar
> Daniel> archives) have been replaced with simpler alternatives. In the case of
> Daniel> tar, the file is simply downloaded to a temporary directory and
> Daniel> extracted using "tar xaf foo-1.2.3.tar".
>
> This sounds worse to me, since it means that now package installation
> has an external dependency.  Previously this was not the case --
> intentionally.

Indeed, if that means that it won't work on vanilla Windows
installations, then this decision should be reversed.

The change looks good otherwise, I especially like the presence of tests.

Re: cl-defstruct-based package.el, now with ert tests!

[Daniel Hackney]

Tom Tromey <tromey@redhat.com> wrote:
> Daniel> The changes touch most of the code, but the overall structure of the
> Daniel> code has not changed significantly. Some parts of the code which were
> Daniel> questionable in the first place (such as using `tar-mode' to extract tar
> Daniel> archives) have been replaced with simpler alternatives. In the case of
> Daniel> tar, the file is simply downloaded to a temporary directory and
> Daniel> extracted using "tar xaf foo-1.2.3.tar".
>
> This sounds worse to me, since it means that now package installation
> has an external dependency.  Previously this was not the case --
> intentionally.

The current package.el does use tar, in `package-tar-file-info', which is called
from `package-install-file'. It is used to extract the "foo-pkg.el" file from
the archive. People may not have run into this if they didn't use
`package-install-file'. I can revert this change and use Emacs's `tar-mode' if
people like.

On a related note, is there a way to un-gzip buffers without needing the
external gzip program? I'd like to be able to use gzipped tarballs for package
distribution; it seems strange to be sending around uncompressed tar files.

Are there any other suggestions before it can be committed to the trunk?

--
Daniel Hackney

Re: cl-defstruct-based package.el, now with ert tests!

[Ted Zlatanov]

On Tue, 12 Mar 2013 16:03:05 +0400 Dmitry Gutov <dgutov@yandex.ru> wrote: 

DG> Tom Tromey <tromey@redhat.com> writes:
Daniel> The changes touch most of the code, but the overall structure of the
Daniel> code has not changed significantly. Some parts of the code which were
Daniel> questionable in the first place (such as using `tar-mode' to extract tar
Daniel> archives) have been replaced with simpler alternatives. In the case of
Daniel> tar, the file is simply downloaded to a temporary directory and
Daniel> extracted using "tar xaf foo-1.2.3.tar".
>> 
>> This sounds worse to me, since it means that now package installation
>> has an external dependency.  Previously this was not the case --
>> intentionally.

DG> Indeed, if that means that it won't work on vanilla Windows
DG> installations, then this decision should be reversed.

Agreed, maybe Daniel can explain why using `tar-mode' was questionable.  Performance?

DG> The change looks good otherwise, I especially like the presence of tests.

Same here.  I like the standardization of the internals.

Ted

Re: cl-defstruct-based package.el, now with ert tests!

[Daniel Hackney]

Ted Zlatanov <tzz <at> lifelogs.com> writes:
> Agreed, maybe Daniel can explain why using `tar-mode' was questionable.
> Performance?

The answer is "it seemed yucky." `tar-mode' was clearly not intended to act as
an API for other programs looking to read tar files, so using it as such was
kind of hacky. Using the external "tar" program seems like the cleaner
alternative. Since package.el already uses the external tar program for
`package-tar-file-info', I figured depending on tar for more substantial stuff
was fine. If people really don't want to depend on the external program, I'll
remove it and just use `tar-mode'.

Is there any way to gunzip the data in a buffer without using an external
program? It would be nice to be able to use tar.gz files.

Re: cl-defstruct-based package.el, now with ert tests!

[Stephen J. Turnbull]

Daniel Hackney writes:
 > Ted Zlatanov <tzz <at> lifelogs.com> writes:
 > > Agreed, maybe Daniel can explain why using `tar-mode' was questionable.
 > > Performance?
 > 
 > The answer is "it seemed yucky." `tar-mode' was clearly not
 > intended to act as an API for other programs looking to read tar
 > files, so using it as such was kind of hacky. Using the external
 > "tar" program seems like the cleaner alternative.

FWIW, nobody has ever complained about XEmacs's assumption that users
have tar and gzip available to install packages.  (In fact, use of an
external ftp program to perform FTP for distribution has caused a lot
of problems over the years due to firewalls and issues with
configuration of passive FTP.  But no problems with tar and gzip, and
not with ftp the program that I can recall.)

 > Since package.el already uses the external tar program for
 > `package-tar-file-info', I figured depending on tar for more
 > substantial stuff was fine. If people really don't want to depend
 > on the external program, I'll remove it and just use `tar-mode'.
 > 
 > Is there any way to gunzip the data in a buffer without using an external
 > program? It would be nice to be able to use tar.gz files.

Is that the right question to ask?  It would not be hard to link in
zlib to do this, but it creates an additional *dependency*.  So I
think the question is "without external dependencies".  I suspect an
elisp implementation of gunzip would be pretty slow and possibly hard
to get right.

Re: cl-defstruct-based package.el, now with ert tests!

[Ted Zlatanov]

On Thu, 14 Mar 2013 04:36:36 +0000 (UTC) Daniel Hackney <dan@haxney.org> wrote: 

DH> Ted Zlatanov <tzz <at> lifelogs.com> writes:
>> Agreed, maybe Daniel can explain why using `tar-mode' was questionable.
>> Performance?

DH> The answer is "it seemed yucky." `tar-mode' was clearly not intended to act as
DH> an API for other programs looking to read tar files, so using it as such was
DH> kind of hacky. Using the external "tar" program seems like the cleaner
DH> alternative. Since package.el already uses the external tar program for
DH> `package-tar-file-info', I figured depending on tar for more substantial stuff
DH> was fine. If people really don't want to depend on the external program, I'll
DH> remove it and just use `tar-mode'.

I'm OK with the change if it doesn't break any existing W32 support.
Otherwise we're leaving many Emacs users unable to use package.el.

`tar-mode' seems pretty OK as far as the data extraction.  So maybe we
just need to pull the data functions out into a utility library?

Either way, I'll wait for this to happen before working on the ELPA
security (signing packages).  So if I can help, let me know.

DH> Is there any way to gunzip the data in a buffer without using an external
DH> program? It would be nice to be able to use tar.gz files.

Sorry, I don't know, but `ldd emacs' says libz is already available at
least on GNU/Linux.  So the C functions should be there.

Ted

Re: cl-defstruct-based package.el, now with ert tests!

[Stefan Monnier]

> was fine. If people really don't want to depend on the external program, I'll
> remove it and just use `tar-mode'.

On GNU/Linux, Mac OS X, Cygwin, ..., there's very little risk in
assuming that `tar' is available.
But IIUC under Windows the picture is completely different.

> Is there any way to gunzip the data in a buffer without using an external
> program? It would be nice to be able to use tar.gz files.

Providing direct access to libz would be nice.


        Stefan

Re: cl-defstruct-based package.el, now with ert tests!

[chad]

On 14 Mar 2013, at 06:49, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

>> was fine. If people really don't want to depend on the external program, I'll
>> remove it and just use `tar-mode'.
> 
> On GNU/Linux, Mac OS X, Cygwin, ..., there's very little risk in
> assuming that `tar' is available.
> But IIUC under Windows the picture is completely different.

Does the Windows-side picture get any better for zip instead of tar
or tar+gunzip? When I last had to care, the variations in unzip on
Windows were pretty annoying, but in those days Windows 95 was
common and NT 3.51 was `a good sign'. Hopefully, the situation
has improved.

~Chad

Re: cl-defstruct-based package.el, now with ert tests!

[Daniel Hackney]

Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> > was fine. If people really don't want to depend on the external program,
> > I'll
> > remove it and just use `tar-mode'.
>
> On GNU/Linux, Mac OS X, Cygwin, ..., there's very little risk in
> assuming that `tar' is available.
> But IIUC under Windows the picture is completely different.

So it looks like I'll have to go back to `tar-mode'. I should be able
to get some work done on that front over the weekend, and I'll resubmit
a `tar-mode'-based package.el.

> > Is there any way to gunzip the data in a buffer without using an
> > external
> > program? It would be nice to be able to use tar.gz files.
>
> Providing direct access to libz would be nice.

Agreed, especially since it appears to be linked in already. However,
I'd like my new package.el to work out-of-the-box on Emacs 24.3, so I
wouldn't be able to rely on libz support (should it appear in Emacs 24.4
or 25.1).

--
Daniel Hackney

Re: cl-defstruct-based package.el, now with ert tests!

[Stefan Monnier]

> Agreed, especially since it appears to be linked in already. However,
> I'd like my new package.el to work out-of-the-box on Emacs 24.3, so I
> wouldn't be able to rely on libz support (should it appear in Emacs 24.4
> or 25.1).

Of course, another option is to be evil:
- if there's gunzip, use it.
- if we're on Windows, download a prebuilt gunzip executable from GNU
  ELPA (presumably we do have network access since in all likelyhood we
  just downloaded the targz).
- if we're not on Windows and there's no gunzip, give up in disgust.


        Stefan ;-)

Re: cl-defstruct-based package.el, now with ert tests!

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 15 Mar 2013 08:56:44 -0400
> Cc: emacs-devel@gnu.org
> 
> Of course, another option is to be evil:
> - if there's gunzip, use it.
> - if we're on Windows, download a prebuilt gunzip executable from GNU
>   ELPA (presumably we do have network access since in all likelyhood we
>   just downloaded the targz).
> - if we're not on Windows and there's no gunzip, give up in disgust.

Before the disgust, and actually even before fetching gunzip from
ELPA, I would suggest to probe for bsdtar executable, which can unpack
any archive known to humanity without invoking any external programs.
It is part of the libarchive package.  A Windows port is available (I
use it all the time), so this is a good alternative for Windows as
well.

Re: cl-defstruct-based package.el, now with ert tests!

[Ted Zlatanov]

On Fri, 15 Mar 2013 00:00:03 -0400 Daniel Hackney <dan@haxney.org> wrote: 

DH> So it looks like I'll have to go back to `tar-mode'. I should be able
DH> to get some work done on that front over the weekend, and I'll resubmit
DH> a `tar-mode'-based package.el.

This is exciting, thank you!  I look forward to it.

Ted