plist-based package.el (was Re: cl-defstruct-based package.el, now with ert tests and no external tar!)

plist-based package.el (was Re: cl-defstruct-based package.el, now with ert tests and no external tar!)

[Daniel Hackney]

Looking at Bug#13291 ("The package description buffer needs an URL
button"), I'm having a crisis of confidence in my plan to use
`cl-defstruct' to represent `package-desc' structures. The big problem
with `cl-defstruct' in this case is its lack of extensibility. We are
going to want to add additional slots to `package-desc' structures over
time, but doing so would require redefining `package-desc' each time.
`cl-defstruct' requires that structures be of the exact length given in
the definition of the structure; if, for example, `package-desc-name'
gets a vector with an unexpected length, it will signal an error:

    (error "package-desc-name accessing a non-package-desc")

This is desirable for structures which don't change, but we are going
to want all sorts of extra slots in `package-desc' structures.

As a result, I'm going to refactor my refactor: I'll make
`package-desc' structures into plists, so that future additions don't
break existing code. I don't think this should be that invasive; all
accesses to `package-desc' structures occur through `package-desc-*'
functions anyway, so not much will change from my current iteration.

What a time to change my mind; I had just finished all of the coding
changes.

Daniel Hackney <dan@haxney.org> wrote:
> Dmitry Gutov <dgutov@yandex.ru> wrote:
>> Any news on this?
>
> I was finishing up on exams, so this had to be put on hold. I'm
> starting up on it again. I think it's close to a mergable state, so
> I'll finish up the NEWS entry and submit the patch for merging.
>
>> I've been holding off on Bug#13291 both because it would complicate
>> the rebase of this changeset, and also because I'd like to use the
>> test harness you have here.
>
> Sounds good. I'll try to get things on my end totally finished so
> development can resume on the new version.
>
>> Daniel Hackney <dan@haxney.org> writes:
>>> I'll do a better job of explaining the whole patch. Should I include
>>> that in some sort of file in the repo or just on the mailing list?
>>> Would a detailed-ish explanation of the changes and rationale be
>>> appropriate for the ChangeLog or NEWS files?
>>
>> You can start with ChangeLog files. When writing a changelog entry,
>> one usually briefly describes and sometimes justifies the mechanical
>> transformations being performed on the code, and it helps other people
>> understand the changes.
>
> Since it is a complete refactoring, what should I say? Should I include
> a line for each new or changed function, or simply refer people to the
> NEWS file?
>
>> I see you've also already started on NEWS entries.
>
> How does it look so far? Any suggestions on how to improve it?
>
>>> About package-x.el, is the HTML and RSS updating functionality actually
>>> used? Currently, the only way to access the functionality is calling
>>> `package-maint-add-news-item' or the non-interactive
>>> `package-upload-buffer-internal' directly. GNU ELPA clearly doesn't use
>>> the version in package-x.el, as the HTML generated is not what
>>> package-x produces.
>>
>> Probably not. Melpa and Elpakit don't seem to be using it either.
>>
>>> Can I consider it unused and delete it? If not, I'll refactor it with
>>> the rest of the code.
>>
>> Personally, I'd delete them, but that's not my call. Maybe decorate them
>> with FIXMEs, leave them unfixed and see if anyone complains up until the
>> pretest?
>
> I did some porting and they should continue to work, although they don't
> do much of anything useful, just like before.
>
> One other change I made, which probably deserves some discussion, is
> that I created a folder "test/automated/data" which is intended to hold
> test-related data. I created a subfolder "package" (so it is
> "test/automated/data/package") which contains a test "archive-contents"
> file, as well as test elisp files. My hope is that such a directory
> could be useful for other tests which need example data.
>
> I also added a rule in the "Makefile.in" to skip byte-compilation of the
> files in "data".
>
> --
> Daniel Hackney



--
Daniel Hackney

Re: plist-based package.el (was Re: cl-defstruct-based package.el, now with ert tests and no external tar!)

[Dmitry Gutov]

On 05.06.2013 2:44, Daniel Hackney wrote:
> Looking at Bug#13291 ("The package description buffer needs an URL
> button"), I'm having a crisis of confidence in my plan to use
> `cl-defstruct' to represent `package-desc' structures. The big problem
> with `cl-defstruct' in this case is its lack of extensibility. We are
> going to want to add additional slots to `package-desc' structures over
> time, but doing so would require redefining `package-desc' each time.
> `cl-defstruct' requires that structures be of the exact length given in
> the definition of the structure; if, for example, `package-desc-name'
> gets a vector with an unexpected length, it will signal an error:
>
>      (error "package-desc-name accessing a non-package-desc")
>
> This is desirable for structures which don't change, but we are going
> to want all sorts of extra slots in `package-desc' structures.

I haven't had the time to read your patch properly yet, but I don't 
think that's necessarily true. Just like I did in my current patch for 
#13291, one of the fields in the struct can contain an alist with all 
extra properties.

Setting their values would be a tad less convenient, I suppose, but for 
getters, for example, `package-desc-homepage' can be a plain function, 
delegating to (cdr (assoc :homepage (package-desc-extras desc))).

Or with a more dynamic approach: (package-desc-extras-get :homepage). 
The implementation would also be simple.

Just an option to consider.

Re: plist-based package.el (was Re: cl-defstruct-based package.el, now with ert tests and no external tar!)

[Ted Zlatanov]

On Wed, 05 Jun 2013 03:01:47 +0400 Dmitry Gutov <dgutov@yandex.ru> wrote: 

DG> On 05.06.2013 2:44, Daniel Hackney wrote:
>> The big problem with `cl-defstruct' in this case is its lack of
>> extensibility. We are going to want to add additional slots to
>> `package-desc' structures over time, but doing so would require
>> redefining `package-desc' each time.  `cl-defstruct' requires that
>> structures be of the exact length given in the definition of the
>> structure

DG> I haven't had the time to read your patch properly yet, but I don't
DG> think that's necessarily true. Just like I did in my current patch for
DG> #13291, one of the fields in the struct can contain an alist with all
DG> extra properties.

I agree with Dmitry.  Keep the top-level structure static with
`cl-defstruct' and just add an option field that holds a plist.  That
provides a good data API, while a pure-plist approach is inherently less
stable as an API.

I worked with plist-only data structures in auth-source.el and looking
back, wish I had used the approach suggested by Dmitry and kept plists
to just one corner of the data structure.  Emacs Lisp doesn't have
(AFAIK) good plist support and self-quoting symbols have some issues
too, so you may end up converting 'X to :X and back occasionally.  That
was just my experience and I may have simply used plists badly.

To address all this I used some `loop' tricks... it seemed like a good
idea at the time :)

Ted

Re: plist-based package.el (was Re: cl-defstruct-based package.el, now with ert tests and no external tar!)

[Stefan Monnier]

> provides a good data API, while a pure-plist approach is inherently less
> stable as an API.

plists are OK as a "UI" elements (e.g. like the CL style :key arguments
in macro and function calls).  But as data-structures, they suck rocks.
The only other data structure that comes to mind as being even worse are
strings.


        Stefan

RE: plist-based package.el (was Re: cl-defstruct-based package.el, now with ert tests and no external tar!)

[Drew Adams]

> plists are OK as a "UI" elements (e.g. like the CL style :key arguments
> in macro and function calls).  But as data-structures, they suck rocks.
> The only other data structure that comes to mind as being even worse are
> strings.

Hm.  http://steve-yegge.blogspot.com/2008/10/universal-design-pattern.html.
And then there's JSON...  And "big data" generally...

Maybe you meant something specific about Emacs-Lisp plists?
Or were you really speaking generally about key+value pairs?

Re: plist-based package.el (was Re: cl-defstruct-based package.el, now with ert tests and no external tar!)

[Ted Zlatanov]

On Wed, 5 Jun 2013 10:56:54 -0700 (PDT) Drew Adams <drew.adams@oracle.com> wrote: 

>> plists are OK as a "UI" elements (e.g. like the CL style :key arguments
>> in macro and function calls).  But as data-structures, they suck rocks.
>> The only other data structure that comes to mind as being even worse are
>> strings.

DA> Hm.  http://steve-yegge.blogspot.com/2008/10/universal-design-pattern.html.
DA> And then there's JSON...  And "big data" generally...

DA> Maybe you meant something specific about Emacs-Lisp plists?
DA> Or were you really speaking generally about key+value pairs?

plists as a sequence of odd=key and even=value don't actually associate
the key with the value, unlike hashtables and alists.

Ted

Re: plist-based package.el (was Re: cl-defstruct-based package.el, now with ert tests and no external tar!)

[Stephen J. Turnbull]

Ted Zlatanov writes:

 > plists as a sequence of odd=key and even=value don't actually associate
 > the key with the value, unlike hashtables and alists.

They do if you use plist-put and plist-get.  If you use list
primitives, you get what you deserve, which may not always be what you
want....

Re: plist-based package.el (was Re: cl-defstruct-based package.el, now with ert tests and no external tar!)

[Ted Zlatanov]

On Thu, 06 Jun 2013 18:30:11 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote: 

SJT> Ted Zlatanov writes:
>> plists as a sequence of odd=key and even=value don't actually associate
>> the key with the value, unlike hashtables and alists.

SJT> They do if you use plist-put and plist-get.  If you use list
SJT> primitives, you get what you deserve, which may not always be what you
SJT> want....

The association is positional.  All you have to do is insert an element
in the middle of the list and the plist is broken.  Also there's no
iteration primitive, so you have to use something like

#+begin_src lisp
         (setq keys (loop for i below (length spec) by 2
                     collect (nth i spec)))
#+end_src

By contrast, an alist is an actual list of association cells that works
with `mapcar', and a hashtable is opaque so you don't know or care how
the association is maintained, but you can't screw it up.

Ted

Re: plist-based package.el (was Re: cl-defstruct-based package.el, now with ert tests and no external tar!)

[Stephen J. Turnbull]

Ted Zlatanov writes:
 > On Thu, 06 Jun 2013 18:30:11 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote: 
 > 
 > SJT> Ted Zlatanov writes:
 > >> plists as a sequence of odd=key and even=value don't actually associate
 > >> the key with the value, unlike hashtables and alists.
 > 
 > SJT> They do if you use plist-put and plist-get.  If you use list
 > SJT> primitives, you get what you deserve, which may not always be what you
 > SJT> want....
 > 
 > The association is positional.  All you have to do is insert an
 > element in the middle of the list and the plist is broken.

That's what I said, except more generally.  Really, one should pay
proper respect to venerable data structures.  They're often fragile.

 > Also there's no iteration primitive,

That's a bug in Emacs, not in plists as an ADT.  It would be easy
enough to add, and would be efficient (at least if written in C where
the obvious pointer-chasing algorithm is O(n), rather than in cl with
an O(n^2) algorithm!)

 > By contrast, an alist is an actual list of association cells

Actually, alists are just as bad, because neither keys nor values are
well-defined.  Key identity is ambiguous because of `assq', and value
identity is ambiguous because you don't know whether cdr or cadr (or
perhaps something more complex, such as (nth i item)) fetches the
value from the item.  If you make a mistake in your map function,
mapcar can become a weapon of mass destruction.  In a plist, on the
other hand, key and value are well-defined.

That, and the clean print representation, are why I strongly prefer
plists in APIs where the machine won't make mistakes -- once the bugs
are fixed -- but humans often have to write to that API.  Can't use a
hash table for that!  Well, you can, but interestingly enough, the
readable print representation of a hash table is built around a plist.

 > that works with `mapcar', and a hashtable is opaque so you don't
 > know or care how the association is maintained, but you can't screw
 > it up.

Aaahhhh, *hash* *tables*.  Now I like where you're headed!  An opaque
type, the modern programmer's heartthrob.  I have to agree with you
there.  "Opaque types are one honkin' great idea!  We should do more
of those."[1]



Footnotes: 
[1]  With apologies to Tim Peters.

opaque data types (was: plist-based package.el (was Re: cl-defstruct-based package.el, now with ert tests and no external tar!))

[Ted Zlatanov]

On Fri, 07 Jun 2013 12:07:44 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote: 

SJT> Aaahhhh, *hash* *tables*.  Now I like where you're headed!  An opaque
SJT> type, the modern programmer's heartthrob.  I have to agree with you
SJT> there.  "Opaque types are one honkin' great idea!  We should do more
SJT> of those."[1]

SJT> Footnotes: 
SJT> [1]  With apologies to Tim Peters.

I'd really like an opaque "password" type that doesn't print unless
retrieved with a special function, and is stored encrypted in memory.
Currently auth-source.el uses a closure for that, which is not ideal.
It's on my TODO list for some rainy day.

Generally, I love how Lisp balances readability, accessibility, and
power by limiting itself to data structures that are not opaque.  The
exceptions to that rule are the interesting edge cases :)

(By contrast, Java makes data structures and objects impenetrable by
default and requires nasty tricks to get introspection.  And on top of
that we have Guava, Scala, and Clojure... amazing!)

Ted

Re: opaque data types

[Christopher Schmidt]

Ted Zlatanov <tzz@lifelogs.com> writes:
> I'd really like an opaque "password" type that doesn't print unless
> retrieved with a special function, and is stored encrypted in memory.

Why do you want that?

        Christopher

Re: opaque data types

[Ted Zlatanov]

On Fri,  7 Jun 2013 14:27:12 +0100 (BST) Christopher Schmidt <christopher@ch.ristopher.com> wrote: 

CS> Ted Zlatanov <tzz@lifelogs.com> writes:
>> I'd really like an opaque "password" type that doesn't print unless
>> retrieved with a special function, and is stored encrypted in memory.

CS> Why do you want that?

To store passwords and other sensitive data in a way so they are not
trivial to extract.  We've had some previous emacs-devel discussions
about this.

Ted

Re: opaque data types

[Christopher Schmidt]

Ted Zlatanov <tzz@lifelogs.com> writes:
> To store passwords and other sensitive data in a way so they are not
> trivial to extract.  We've had some previous emacs-devel discussions
> about this.

Can you please point me to the discussion?

Considering we are talking about a regular userspace application with no
distributed components I do not see any advantage at all by encrypting
passwords in memory.  How does interposing a function to extract
passwords from a new inbuild type increase security at all?  Who's
your attacker anyway?

        Christopher

Re: opaque data types

[Ted Zlatanov]

On Fri,  7 Jun 2013 16:44:18 +0100 (BST) Christopher Schmidt <christopher@ch.ristopher.com> wrote: 

CS> Ted Zlatanov <tzz@lifelogs.com> writes:
>> To store passwords and other sensitive data in a way so they are not
>> trivial to extract.  We've had some previous emacs-devel discussions
>> about this.

CS> Can you please point me to the discussion?

At least some of it was covered in the thread around
http://lists.gnu.org/archive/html/emacs-devel/2012-01/msg00407.html

CS> Considering we are talking about a regular userspace application with no
CS> distributed components I do not see any advantage at all by encrypting
CS> passwords in memory.  How does interposing a function to extract
CS> passwords from a new inbuild type increase security at all?

By making it less trivial to extract them.  The opaque type makes it
possible to change the implementation if better ways are available on a
platform, e.g. the Mac OS X keychain or the Secrets API or the W32
keychain.  The fallback mechanism can at least make it a little harder
to get someone's passwords.

CS> Who's your attacker anyway?

Do we have to do risk assessments too?

Ted

Re: opaque data types

[Christopher Schmidt]

Ted Zlatanov <tzz@lifelogs.com> writes:
> CS> Considering we are talking about a regular userspace application
> CS> with no distributed components I do not see any advantage at all
> CS> by encrypting passwords in memory.  How does interposing a
> CS> function to extract passwords from a new inbuild type increase
> CS> security at all?
>
> By making it less trivial to extract them.

That is security through obscurity.

> The opaque type makes it possible to change the implementation if
> better ways are available on a platform, e.g. the Mac OS X keychain or
> the Secrets API or the W32 keychain.  The fallback mechanism can at
> least make it a little harder to get someone's passwords.

Storing passwords using different backends does not require in-memory
encryption or a new opaque type.

How is this new type in combination with custom hard back ends superior
to what auth-info.el is doing already?

> CS> Who's your attacker anyway?
>
> Do we have to do risk assessments too?

I do not understand that question.

I was asking for an informal threat model because I did not understand
the problem you are trying to solve.  I still do not understand the
problem and I do not see how that new type makes provides any advantage
whatsoever.

        Christopher

Re: opaque data types

[Ted Zlatanov]

On Sat,  8 Jun 2013 10:19:34 +0100 (BST) Christopher Schmidt <christopher@ch.ristopher.com> wrote: 

CS> Ted Zlatanov <tzz@lifelogs.com> writes:
CS> Considering we are talking about a regular userspace application
CS> with no distributed components I do not see any advantage at all
CS> by encrypting passwords in memory.  How does interposing a
CS> function to extract passwords from a new inbuild type increase
CS> security at all?
>> 
>> By making it less trivial to extract them.

CS> That is security through obscurity.

I disagree with that statement.  Perhaps we're talking about different
things.  My goal is not to obscure the data but to encrypt it so it's
not trivially extracted.

>> The opaque type makes it possible to change the implementation if
>> better ways are available on a platform, e.g. the Mac OS X keychain or
>> the Secrets API or the W32 keychain.  The fallback mechanism can at
>> least make it a little harder to get someone's passwords.

CS> Storing passwords using different backends does not require in-memory
CS> encryption or a new opaque type.

True.  But my experience with auth-source.el (which in fact provides
these backends) was that it's very hard to make it even a little bit
secure at the Lisp level alone.

CS> How is this new type in combination with custom hard back ends superior
CS> to what auth-info.el is doing already?

It's harder to corrupt, inspect, or intercept the underlying data
transactions from Lisp code if they are at the C level.

You don't have to depend on external utilities if you call the libraries
directly.  OTOH you have to worry about the API changing...

The data you pass back and forth is not visible to every Lisp function.

CS> Who's your attacker anyway?
>> 
>> Do we have to do risk assessments too?

CS> I do not understand that question.

I am saying I hope I don't have to invent an attacker to justify my
proposal.  Look at it this way: do you feel Emacs provides good enough
security?  I don't, based on what I know about Emacs Lisp and the
underlying C code.  When it's not possible to write 15 lines of Lisp to
grab my passwords out of memory, I'll be less worried.

Ted

Re: opaque data types

[Stefan Monnier]

> Currently auth-source.el uses a closure for that, which is not ideal.

It can use the standard "opaque-at-print-time" object, i.e. the symbol
(which is indeed what is used by lexical-let closures, but not by
lexical-binding closures, where the password would be displayed in the
printed representation).


        Stefan

Re: opaque data types

[Ted Zlatanov]

On Fri, 07 Jun 2013 12:17:11 -0400 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>> Currently auth-source.el uses a closure for that, which is not ideal.
SM> It can use the standard "opaque-at-print-time" object, i.e. the symbol
SM> (which is indeed what is used by lexical-let closures, but not by
SM> lexical-binding closures, where the password would be displayed in the
SM> printed representation).

Right.  I think we can do better than just hide the data at print time.

Ted

opaque data types

[Stephen J. Turnbull]

Ted Zlatanov writes:

 > Generally, I love how Lisp balances readability, accessibility, and
 > power by limiting itself to data structures that are not opaque.  The
 > exceptions to that rule are the interesting edge cases :)

You mean exceptions like plists?

 > (By contrast, Java

 > On Fri, 07 Jun 2013 12:07:44 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote: 
ou 

 > makes data structures and objects impenetrable by
 > default and requires nasty tricks to get introspection.

Introspection is a question of API design.  The Lisp that you love is
a trivial design.  There's one data structure: the cons.
Introspection is provided by the access functions and the repl.

On the other hand, you seem to like hash tables.  How do you
introspect the number of buckets and the hash function in Emacs?
Another example: XEmacs takes you half-way there.  It displays the
number of buckets in the un(read)able print representation.

opaque data types [revise and resend]

[Stephen J. Turnbull]

Oops, keyboard bounce; fix and resend.

Ted Zlatanov writes:

 > Generally, I love how Lisp balances readability, accessibility, and
 > power by limiting itself to data structures that are not opaque.  The
 > exceptions to that rule are the interesting edge cases :)

You mean exceptions like plists?

 > (By contrast, Java makes data structures and objects impenetrable
 > by default and requires nasty tricks to get introspection.

Introspection is a question of API design.  The Lisp that you love is
a trivial design.  There's one data structure: the cons.
Introspection is provided by the access functions and the repl.

On the other hand, you seem to like hash tables.  How do you
introspect the number of buckets and the hash function in Emacs?
Inquiring minds want to know!  Another example: XEmacs takes you
half-way there.  It displays the number of buckets in the un(read)able
print representation.  If bucket count were useful to programs, it
would be easy enough to provide an API.

So it's a question of how you balance the needs.  I'm certainly not
recommending use of Java over Lisp!  However, there are some common
structures like mappings that are well-served by opaque data types,
and some specialized ones like characters (!) and keymaps where Emacs
implementers are well-served.

Re: opaque data types [revise and resend]

[Ted Zlatanov]

On Sat, 08 Jun 2013 15:28:00 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote: 

SJT> So it's a question of how you balance the needs.  I'm certainly not
SJT> recommending use of Java over Lisp!  However, there are some common
SJT> structures like mappings that are well-served by opaque data types,
SJT> and some specialized ones like characters (!) and keymaps where Emacs
SJT> implementers are well-served.

I agree :) In fact my other post in this thread is (again) defending an
opaque data type for password data.

Ted

Re: plist-based package.el (was Re: cl-defstruct-based package.el, now with ert tests and no external tar!)

[Stefan Monnier]

> button"), I'm having a crisis of confidence in my plan to use
> `cl-defstruct' to represent `package-desc' structures. The big problem
> with `cl-defstruct' in this case is its lack of extensibility. We are
> going to want to add additional slots to `package-desc' structures over
> time, but doing so would require redefining `package-desc' each time.
> `cl-defstruct' requires that structures be of the exact length given in
> the definition of the structure; if, for example, `package-desc-name'
> gets a vector with an unexpected length, it will signal an error:

>     (error "package-desc-name accessing a non-package-desc")

I don't see the relevance: those defstruct structures should be internal
to package.el (i.e. never saved to a file or loaded from a file), so
there's no risk of using one with the functions of another.


        Stefan

Re: plist-based package.el (was Re: cl-defstruct-based package.el, now with ert tests and no external tar!)

[Daniel Hackney]

Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> button"), I'm having a crisis of confidence in my plan to use
>> `cl-defstruct' to represent `package-desc' structures. The big problem
>> with `cl-defstruct' in this case is its lack of extensibility. We are
>> going to want to add additional slots to `package-desc' structures over
>> time, but doing so would require redefining `package-desc' each time.
>> `cl-defstruct' requires that structures be of the exact length given in
>> the definition of the structure; if, for example, `package-desc-name'
>> gets a vector with an unexpected length, it will signal an error:
>
>>     (error "package-desc-name accessing a non-package-desc")
>
> I don't see the relevance: those defstruct structures should be internal
> to package.el (i.e. never saved to a file or loaded from a file), so
> there's no risk of using one with the functions of another.

The only exception to that is in "finder-inf.el", though I could change
that.

My thought was that package.el could be made extensible with things
like hooks and extended properties, so functionality could be added by
adding some functions to `package-install-functions' or somesuch. That
way, adding, for example, digital signature checking could be done by
creating a property :sig on the `package-desc' structure and adding
some checks to package installation hooks.

That may be getting ahead of myself, though.

I did finish the `cl-defstruct' -> `plist' transition. It was little
more than creating `defsubst' and `gv-define-setter' calls for each
property and regenerating "finder-inf.el".

--
Daniel Hackney

Re: plist-based package.el (was Re: cl-defstruct-based package.el, now with ert tests and no external tar!)

[Rand User]

> From: Daniel Hackney <dan@haxney.org>

> To: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Emacs development discussions <emacs-devel@gnu.org>; Dmitry Gutov <dgutov@yandex.ru>
> Sent: Wednesday, June 5, 2013 8:31 PM
> Subject: Re: plist-based package.el (was Re: cl-defstruct-based package.el,
	now with ert tests and no external tar!)
> 
> Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>>>  button"), I'm having a crisis of confidence in my plan to use
>>>  `cl-defstruct' to represent `package-desc' structures. The big 
> problem
>>>  with `cl-defstruct' in this case is its lack of extensibility. We 
> are
>>>  going to want to add additional slots to `package-desc' structures 
> over
>>>  time, but doing so would require redefining `package-desc' each 
> time.
>>>  `cl-defstruct' requires that structures be of the exact length 
> given in
>>>  the definition of the structure; if, for example, 
> `package-desc-name'
>>>  gets a vector with an unexpected length, it will signal an error:
>> 
>>>      (error "package-desc-name accessing a non-package-desc")
>> 
>>  I don't see the relevance: those defstruct structures should be 
> internal
>>  to package.el (i.e. never saved to a file or loaded from a file), so
>>  there's no risk of using one with the functions of another.

The structures saved/loaded from files are the ones returned from the package servers, right?  Aren't these the ones that need to be flexible for extensibility?

Right now, the structures returned from the servers  basically consists of an alist of package symbols consed to package-desc vectors.  e.g.

(lex .
      [(1 1)
       nil "Lexical analyser construction" tar])

My suggestion would be to change the format of the data returned from the servers to something that allows arbitrary key-data pairs.  define-package should then allow package authors to specify arbitrary values that are passed along by the server.

Cheers,
Rand