RFC: User-defined pseudovectors

RFC: User-defined pseudovectors

[Lars Brinkhoff]

Hello,

With the current FFI discussion, this may be a good time to ask for
input on a Lisp extension I have lying around.

Some FFI bindings may want to introduce new Lisp types.  My idea is
that this should be possible to do in Lisp, not just in C.  This might
also be useful in other situations, e.g.

- Future new pseudovector types can be defined in Lisp.

- Lisp code can potentially interoperate more easily with code in
other languages.  Existing languages for Emacs include JavaScript
(Ejacs), Python (Pymacs), Ruby (El4r), Perl (EPL), Smalltalk (Eoops),
Common Lisp (Emacs Common Lisp).

- Possibly cl-defstruct can be extended to make new types if so requested.

The gist of my patch is to add a new type of pseudovector which is
like a normal vector, except the first element holds a symbol which is
its Lisp type.  So type-of returns whatever is in the first slot.
This may sound slightly reckless, and probably is.  It's just a first
shot.

Sample session:

(let ((x (make-typed-pseudovector 3 'foo nil)))
  (aset x 1 1)
  (aset x 2 2)
  (aset x 3 3)
  (list (read-from-string (with-output-to-string (prin1 x)))
        (typed-pseudovector-p x)
        (type-of x)
        (aref x 0)
        (aref x 3)
        (length x)))

 => ((#%[foo 1 2 3] . 13) t foo foo 3 4)

Re: RFC: User-defined pseudovectors

[Lars Brinkhoff]

I wrote:
> With the current FFI discussion, this may be a good time to ask for
> input on a Lisp extension I have lying around.

Here is the patch in its current (unfinished) state.  Suggestions are
welcome.  Would something like this be useful for inclusion in Emacs?


diff a/src/alloc.c b/src/alloc.c
--- a/src/alloc.c
+++ b/src/alloc.c
@@ -3048,6 +3048,12 @@ allocate_hash_table (void)
   return ALLOCATE_PSEUDOVECTOR (struct Lisp_Hash_Table, count, PVEC_HASH_TABLE);
 }
 
+struct Lisp_Vector *
+allocate_typed_pseudovector (int count)
+{
+  return allocate_pseudovector (count, count, PVEC_TYPED_PSEUDOVECTOR);
+}
+
 struct window *
 allocate_window (void)
 {
@@ -3096,6 +3102,30 @@ allocate_process (void)
   return p;
 }
 
+DEFUN ("make-typed-pseudovector", Fmake_typed_pseudovector, Smake_typed_pseudovector, 3, 3, 0,
+       doc: /* Create a new vector-like object of type TYPE with SLOTS elements, each initialized to INIT.  */)
+  (register Lisp_Object slots, Lisp_Object type, Lisp_Object init)
+{
+  Lisp_Object vector;
+  register ptrdiff_t size;
+  register ptrdiff_t i;
+  register struct Lisp_Vector *p;
+
+  CHECK_NATNUM (slots);
+  if (!SYMBOLP(type))
+    signal_error ("Invalid type; must be symbol", type);
+
+  size = XFASTINT (slots) + 1;
+  p = allocate_typed_pseudovector (size);
+  p->u.contents[0] = type;
+  for (i = 1; i < size; i++)
+    p->u.contents[i] = init;
+
+  XSETVECTOR (vector, p);
+  return vector;
+}
+
+
 DEFUN ("make-vector", Fmake_vector, Smake_vector, 2, 2, 0,
        doc: /* Return a newly created vector of length LENGTH, with each element being INIT.
 See also the function `vector'.  */)
@@ -6755,6 +6785,7 @@ The time is in seconds as a floating point value.  */);
   defsubr (&Smake_byte_code);
   defsubr (&Smake_list);
   defsubr (&Smake_vector);
+  defsubr (&Smake_typed_pseudovector);
   defsubr (&Smake_string);
   defsubr (&Smake_bool_vector);
   defsubr (&Smake_symbol);
diff a/src/data.c b/src/data.c
--- a/src/data.c
+++ b/src/data.c
@@ -290,6 +290,9 @@ for example, (type-of 1) returns `integer'.  */)
 	return Qfont_entity;
       if (FONT_OBJECT_P (object))
 	return Qfont_object;
+      if (TYPED_PSEUDOVECTOR_P (object))
+	return AREF (object, 0);
+
       return Qvector;
 
     case Lisp_Float:
@@ -370,6 +373,15 @@ DEFUN ("vectorp", Fvectorp, Svectorp, 1, 1, 0,
   return Qnil;
 }
 
+DEFUN ("typed-pseudovector-p", Ftyped_pseudovector_p, Styped_pseudovector_p, 1, 1, 0,
+       doc: /* Return t if OBJECT is a typed pseudovector.  */)
+  (Lisp_Object object)
+{
+  if (TYPED_PSEUDOVECTOR_P (object))
+    return Qt;
+  return Qnil;
+}
+
 DEFUN ("stringp", Fstringp, Sstringp, 1, 1, 0,
        doc: /* Return t if OBJECT is a string.  */)
   (Lisp_Object object)
@@ -2146,7 +2158,7 @@ or a byte-code object.  IDX starts at 0.  */)
       ptrdiff_t size = 0;
       if (VECTORP (array))
 	size = ASIZE (array);
-      else if (COMPILEDP (array))
+      else if (COMPILEDP (array) || TYPED_PSEUDOVECTOR_P (array))
 	size = ASIZE (array) & PSEUDOVECTOR_SIZE_MASK;
       else
 	wrong_type_argument (Qarrayp, array);
@@ -2167,7 +2179,8 @@ bool-vector.  IDX starts at 0.  */)
 
   CHECK_NUMBER (idx);
   idxval = XINT (idx);
-  CHECK_ARRAY (array, Qarrayp);
+  if (! TYPED_PSEUDOVECTOR_P (array))
+    CHECK_ARRAY (array, Qarrayp);
   CHECK_IMPURE (array);
 
   if (VECTORP (array))
@@ -2196,7 +2209,14 @@ bool-vector.  IDX starts at 0.  */)
       CHECK_CHARACTER (idx);
       CHAR_TABLE_SET (array, idxval, newelt);
     }
-  else
+  else if (TYPED_PSEUDOVECTOR_P (array))
+    {
+      ptrdiff_t size = ASIZE (array) & PSEUDOVECTOR_SIZE_MASK;
+      if (idxval < 0 || idxval >= size)
+	args_out_of_range (array, idx);
+      ASET (array, idxval, newelt);
+    }
+  else /* STRINGP */
     {
       int c;
 
@@ -3506,6 +3526,7 @@ syms_of_data (void)
   defsubr (&Sstringp);
   defsubr (&Smultibyte_string_p);
   defsubr (&Svectorp);
+  defsubr (&Styped_pseudovector_p);
   defsubr (&Schar_table_p);
   defsubr (&Svector_or_char_table_p);
   defsubr (&Sbool_vector_p);
diff a/src/fns.c b/src/fns.c
--- a/src/fns.c
+++ b/src/fns.c
@@ -115,7 +115,7 @@ To get the number of bytes, use `string-bytes'.  */)
     XSETFASTINT (val, MAX_CHAR);
   else if (BOOL_VECTOR_P (sequence))
     XSETFASTINT (val, XBOOL_VECTOR (sequence)->size);
-  else if (COMPILEDP (sequence))
+  else if (COMPILEDP (sequence) || TYPED_PSEUDOVECTOR_P (sequence))
     XSETFASTINT (val, ASIZE (sequence) & PSEUDOVECTOR_SIZE_MASK);
   else if (CONSP (sequence))
     {
diff a/src/lisp.h b/src/lisp.h
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -538,6 +538,7 @@ enum pvec_type
   PVEC_TERMINAL,
   PVEC_WINDOW_CONFIGURATION,
   PVEC_SUBR,
+  PVEC_TYPED_PSEUDOVECTOR,
   PVEC_OTHER,
   /* These should be last, check internal_equal to see why.  */
   PVEC_COMPILED,
@@ -2328,6 +2329,12 @@ FRAMEP (Lisp_Object a)
   return PSEUDOVECTORP (a, PVEC_FRAME);
 }
 
+INLINE bool
+TYPED_PSEUDOVECTOR_P (Lisp_Object a)
+{
+  return PSEUDOVECTORP (a, PVEC_TYPED_PSEUDOVECTOR);
+}
+
 /* Test for image (image . spec)  */
 INLINE bool
 IMAGEP (Lisp_Object x)
diff a/src/lread.c b/src/lread.c
--- a/src/lread.c
+++ b/src/lread.c
@@ -2603,6 +2603,19 @@ read1 (Lisp_Object readcharfun, int *pch, bool first_in_list)
 	  make_byte_code (vec);
 	  return tmp;
 	}
+      if (c == '%')
+	{
+	  c = READCHAR;
+	  if (c == '[')
+	    {
+	      Lisp_Object tmp;
+	      tmp = read_vector (readcharfun, 1);
+	      XSETPVECTYPE (XVECTOR(tmp), PVEC_TYPED_PSEUDOVECTOR);
+	      return tmp;
+	    }
+	  UNREAD (c);
+	  invalid_syntax ("#");
+	}
       if (c == '(')
 	{
 	  Lisp_Object tmp;
diff a/src/print.c b/src/print.c
--- a/src/print.c
+++ b/src/print.c
@@ -1945,6 +1945,12 @@ print_object (Lisp_Object obj, Lisp_Object printcharfun, bool escapeflag)
 	      PRINTCHAR ('#');
 	      size &= PSEUDOVECTOR_SIZE_MASK;
 	    }
+	  if (TYPED_PSEUDOVECTOR_P (obj))
+	    {
+	      PRINTCHAR ('#');
+	      PRINTCHAR ('%');
+	      size &= PSEUDOVECTOR_SIZE_MASK;
+	    }
 	  if (CHAR_TABLE_P (obj) || SUB_CHAR_TABLE_P (obj))
 	    {
 	      /* We print a char-table as if it were a vector,

Re: RFC: User-defined pseudovectors

[Lars Brinkhoff]

Just a small demo of how it would be possible to change cl-defstruct.
With the previous patch applied, plus this one, we get this:

(cl-defstruct foo x y z)

 => foo

(let ((x (make-foo :y 1)))
  (list (type-of x)
        (foo-p x)
        (typed-pseudovector-p x)
        (foo-y x)
        x))

 => (foo t t 1 #%[foo nil 1 nil])


diff a/lisp/emacs-lisp/cl-macs.el b/lisp/emacs-lisp/cl-macs.el
--- a/lisp/emacs-lisp/cl-macs.el
+++ b/lisp/emacs-lisp/cl-macs.el
@@ -2312,6 +2312,12 @@ Like `cl-callf', but PLACE is the second argument of FUNC, not the first.
 
 ;;; Structures.
 
+(defun typed-pseudovector (type &rest elements)
+  (let ((result (make-typed-pseudovector (length elements) type nil))
+	(i 0))
+    (dolist (elt elements result)
+      (aset result (cl-incf i) elt))))
+
 ;;;###autoload
 (defmacro cl-defstruct (struct &rest descs)
   "Define a struct type.
@@ -2450,21 +2456,25 @@ non-nil value, that slot cannot be set via `setf'.
 	    (or (memq type '(vector list))
 		(error "Invalid :type specifier: %s" type))
 	    (if named (setq tag name)))
-	(setq type 'vector named 'true)))
+	(setq type 'typed-pseudovector named 'true tag name)))
     (or named (setq descs (delq (assq 'cl-tag-slot descs) descs)))
     (push `(defvar ,tag-symbol) forms)
     (setq pred-form (and named
 			 (let ((pos (- (length descs)
 				       (length (memq (assq 'cl-tag-slot descs)
 						     descs)))))
-			   (if (eq type 'vector)
-			       `(and (vectorp cl-x)
-				     (>= (length cl-x) ,(length descs))
-				     (memq (aref cl-x ,pos) ,tag-symbol))
+			   (cond
+			    ((eq type 'vector)
+			     `(and (vectorp cl-x)
+				   (>= (length cl-x) ,(length descs))
+				   (memq (aref cl-x ,pos) ,tag-symbol)))
+			    ((eq type 'list)
 			     (if (= pos 0)
 				 `(memq (car-safe cl-x) ,tag-symbol)
 			       `(and (consp cl-x)
-				     (memq (nth ,pos cl-x) ,tag-symbol))))))
+				     (memq (nth ,pos cl-x) ,tag-symbol))))
+			    (t
+			     `(memq (type-of cl-x) ,tag-symbol)))))
 	  pred-check (and pred-form (> safety 0)
 			  (if (and (eq (cl-caadr pred-form) 'vectorp)
 				   (= safety 1))
@@ -2488,9 +2498,10 @@ non-nil value, that slot cannot be set via `setf'.
 			      (list `(or ,pred-check
                                          (error "%s accessing a non-%s"
                                                 ',accessor ',name))))
-                       ,(if (eq type 'vector) `(aref cl-x ,pos)
-                          (if (= pos 0) '(car cl-x)
-                            `(nth ,pos cl-x)))) forms)
+                       ,(if (eq type 'list)
+			    (if (= pos 0) '(car cl-x)
+			      `(nth ,pos cl-x))
+			  `(aref cl-x ,pos))) forms)
 	      (push (cons accessor t) side-eff)
               (if (cadr (memq :read-only (cddr desc)))
                   (push `(gv-define-expander ,accessor

Re: RFC: User-defined pseudovectors

[Dmitry Antipov]

On 10/10/2013 03:22 PM, Lars Brinkhoff wrote:

> The gist of my patch is to add a new type of pseudovector which is
> like a normal vector, except the first element holds a symbol which is
> its Lisp type.  So type-of returns whatever is in the first slot.
> This may sound slightly reckless, and probably is.  It's just a first
> shot.

Why make-typed-pseudovector & the rest can't be done in elisp?

(Look at lisp/emacs-lisp/timer.el to see how some non-trivial
things can be implemented on top of plain vector).

Dmitry

Re: RFC: User-defined pseudovectors

[Dmitry Antipov]

On 10/10/2013 04:52 PM, Dmitry Antipov wrote:

>> The gist of my patch is to add a new type of pseudovector which is
>> like a normal vector, except the first element holds a symbol which is
>> its Lisp type.  So type-of returns whatever is in the first slot.
>> This may sound slightly reckless, and probably is.  It's just a first
>> shot.
>
> Why make-typed-pseudovector & the rest can't be done in elisp?
>
> (Look at lisp/emacs-lisp/timer.el to see how some non-trivial
> things can be implemented on top of plain vector).

Plus, if we really need the ability to add builtin Lisp types to
interface foreign DSOs, I'll vote to design this on top of Lisp_Misc.

Dmitry

Re: RFC: User-defined pseudovectors

[Stefan Monnier]

> - Possibly cl-defstruct can be extended to make new types if so requested.

I'm not sure I understand the difference between your proposal and
cl-defstruct.  The only difference I can see is that `vectorp' returns
nil on your new objects (well, type-of also changes, but that is so
rarely used that it's basically irrelevant).  Oh, and there's a new
print syntax for those objects.

What is the benefit?


        Stefan

Re: RFC: User-defined pseudovectors

[Lars Brinkhoff]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:
> I'm not sure I understand the difference between your proposal and
> cl-defstruct.  The only difference I can see is that `vectorp'
> returns nil on your new objects (well, type-of also changes, but
> that is so rarely used that it's basically irrelevant).

The difference in return value from functions like vectorp and type-of
is the entire point, actually.

> Oh, and there's a new print syntax for those objects.

If it simplifies anything, that can be ignored for the moment.  I just
hacked something up on a whim.

> What is the benefit?

The benefit is that the types of these "typed pseudovectors" should be
disjoint from all previously existing types.  So e.g. this would be
guaranteed to work, even if someone passes [foo] to frob.

  (cl-defstruct foo ...)

  (defun frob (x)
    (cond
     ((vectorp x)   (frob-vector x))
     ((foop x)      (frob-foo x))))

In the case of an FFI, a wrapper may want to be able to pass a vector
or a "struct" (something like a pseudovector) to Emacs, and have Lisp
code be able to tell the difference.

Or in my case, I have an implementation of Common Lisp written in
Emacs Lisp, and it would be very nice if the Emacs Lisp vector type
could also be the Common Lisp type vector.  Instead I have to overload
many Common Lisp types on top of the Emacs Lisp vector.

Re: RFC: User-defined pseudovectors

[Lars Brinkhoff]

Dmitry Antipov <dmantipov@yandex.ru> writes:
> > Why make-typed-pseudovector & the rest can't be done in elisp?

Because the entire point is to be able to create a new Lisp type, and
there currently is no way to do that in Lisp.

> Plus, if we really need the ability to add builtin Lisp types to
> interface foreign DSOs, I'll vote to design this on top of
> Lisp_Misc.

That may well be better, I don't know.


I just now stumbled over this in lisp.h:

> To define a new data type, add one more Lisp_Misc subtype or one
> more pseudovector subtype.  Pseudovectors are more suitable for
> objects with several slots that need to support fast random access,
> while Lisp_Misc types are for everything else. A pseudovector object
> provides one or more slots for Lisp objects, followed by struct
> members that are accessible only from C.  A Lisp_Misc object is a
> wrapper for a C struct that can contain anything you like.

For my primary use case (Common Lisp data types, structs, and class
instances), I'm inclined to believe that pseudovectors are better.
But it may well be that Lisp_Misc is better for FFI datatypes.

Re: RFC: User-defined pseudovectors

[Stefan Monnier]

> The difference in return value from functions like vectorp and type-of
> is the entire point, actually.

It seems like a small benefit.  I understand it might be handy for your
Common-Lisp implementation, but you can already define your own
`vectorp' to get the same result.  Implementing a language on top of
another will always come with such costs because of (sometimes
subtle) mismatches.
Maybe the Guile guys are more likely to be wiling to make changes to
their system in order to better support some other language (such as
Common-Lisp).


        Stefan

Re: RFC: User-defined pseudovectors

[Stefan Monnier]

>   (defun frob (x)
>     (cond
>      ((vectorp x)   (frob-vector x))
>      ((foop x)      (frob-foo x))))

> In the case of an FFI, a wrapper may want to be able to pass a vector
> or a "struct" (something like a pseudovector) to Emacs, and have Lisp
> code be able to tell the difference.

In Elisp, such type-overlaps are common.  E.g. you should first check
`functionp' before `consp' if you can accept both lists and functions.
So similarly the above code should check `foop' before `vectorp'.


        Stefan

Re: RFC: User-defined pseudovectors

[Lars Brinkhoff]

As long as you work inside Emacs, I agree there's no great need for
user-defined types, because Emacs Lisp already provides the data
structures that are commonly needed.  And you can pass around lists or
vectors which provides a 99% solution in most cases.

But with an FFI, I would argue that the situation changes.  When you
want to interface with code written in other languages, it seems to me
that the need increases.  Most other languages have some kind of
user-defined record types, and they are used quite liberally.

Stefan Monnier <monnier@iro.umontreal.ca> writes:
> I understand it might be handy for your Common-Lisp implementation,
> but you can already define your own `vectorp' to get the same result.

I don't see that I could define one which would accept any vector wich
any content, and be able to tell a "proper vector" apart from user-
defined types.

>>     (cond
>>      ((vectorp x)   (frob-vector x))
>>      ((foop x)      (frob-foo x))))
>
> So similarly the above code should check `foop' before `vectorp'.

That would still fail for the input [foo].

> Maybe the Guile guys are more likely to be wiling to make changes to
> their system in order to better support some other language.

No change is needed.  Guile already has user-defined record types,
which is exactly what I want.  Maybe I should just wait for
GuileEmacs, but as you wrote elsewhere its schedule is somewhat
undependable.

Re: RFC: User-defined pseudovectors

[Stefan Monnier]

> that the need increases.  Most other languages have some kind of
> user-defined record types, and they are used quite liberally.

I'm not necessarily opposed to user-defined record types, but if
cl-defstruct is not sufficient and C-level changes are required, then
I'd hope for the new code to provide more exciting features than just
"vectorp returns nil".

>>> (cond
>>> ((vectorp x)   (frob-vector x))
>>> ((foop x)      (frob-foo x))))
>> So similarly the above code should check `foop' before `vectorp'.
> That would still fail for the input [foo].

Not sure if it should be considered as a bug, tho (it depends on what
was the intention of the caller.  IOW: would she have passed #%[foo] or
[foo] in your version of Emacs?).  Especially for cl-defstruct objects
where you'd need to pass [cl-defstruct-foo].  But if that's a problem,
you can use a more unique object as tag, e.g. an uninterned symbol.


        Stefan

User-defined record types

[Lars Brinkhoff]

Stefan Monnier <monnier@iro.umontreal.ca> writes:
> I'm not necessarily opposed to user-defined record types, but if
> cl-defstruct is not sufficient and C-level changes are required, then
> I'd hope for the new code to provide more exciting features than just
> "vectorp returns nil".

Great, I'd like to get to work on that.  What exciting features do you
have in mind?

Re: User-defined record types

[Stefan Monnier]

>> I'm not necessarily opposed to user-defined record types, but if
>> cl-defstruct is not sufficient and C-level changes are required, then
>> I'd hope for the new code to provide more exciting features than just
>> "vectorp returns nil".
> Great, I'd like to get to work on that.  What exciting features do you
> have in mind?

I don't have any, which is why I haven't implemented anything yet ;-)


        Stefan

Re: User-defined record types

[Ted Zlatanov]

On Sat, 12 Oct 2013 14:58:15 -0400 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>>> I'm not necessarily opposed to user-defined record types, but if
>>> cl-defstruct is not sufficient and C-level changes are required, then
>>> I'd hope for the new code to provide more exciting features than just
>>> "vectorp returns nil".
>> Great, I'd like to get to work on that.  What exciting features do you
>> have in mind?

SM> I don't have any, which is why I haven't implemented anything yet ;-)

Something that represents JSON and YAML well would be nice.  Currently
we don't have an ELisp data structure that can preserve all JSON nuances
without acrobatics (e.g. preserving the difference between "null" and
"empty list" or the native JSON data types).

I know about json.el and it's very convenient but not enough.

A native XML data structure would also be nice.  We have what libxml
produces, dumped in an awkward tree, but nothing native.

Ted

Re: User-defined record types

[Stefan Monnier]

> Something that represents JSON and YAML well would be nice.  Currently
> we don't have an ELisp data structure that can preserve all JSON nuances
> without acrobatics (e.g. preserving the difference between "null" and
> "empty list" or the native JSON data types).

I don't understand what you mean here.  It's very easy to have special
values, e.g. (defconst json-null (make-symbol "json-null")).

IIUC The problem of json.el is that it wants to provide a bridge between
traditional Elisp data structures and JSON, and there is no reliable
two-way conversion between the two.
IIUC the current json.el focuses on making sure that converting to JSON
and back returns the same object.

But I don't see any technical difficulty writing a json2.el alternative
which focuses on "to Elisp and back", i.e. when it reads a JSON object
it returns an Elisp object that preserves all JSON nuances so that it
can be reliably converted back to the "exact same" JSON object.

> a native XML data structure would also be nice.  We have what libxml
> produces, dumped in an awkward tree, but nothing native.

Not sure what "native" can mean in this context: we were talking about
new Lisp-defined types.

Whereas you seem to be talking about getting access to objects defined
in other (non-Lisp) libraries, i.e. an FFI.

Or maybe you're just talking about a more efficient representation for
an XML/JSON tree.  If Lisp-defined data types could offer that, it would be
great, of course, but I personally don't see how that would work.


        Stefan

Re: User-defined record types

[Ted Zlatanov]

On Fri, 18 Oct 2013 11:28:17 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> Something that represents JSON and YAML well would be nice.  Currently
>> we don't have an ELisp data structure that can preserve all JSON nuances
>> without acrobatics (e.g. preserving the difference between "null" and
>> "empty list" or the native JSON data types).

SM> I don't understand what you mean here.  It's very easy to have special
SM> values, e.g. (defconst json-null (make-symbol "json-null")).

Yes, but it's not something you can communicate externally.  Compare
with pure JSON or BSON, which are intended for communication across
programs.

SM> IIUC The problem of json.el is that it wants to provide a bridge between
SM> traditional Elisp data structures and JSON, and there is no reliable
SM> two-way conversion between the two.
SM> IIUC the current json.el focuses on making sure that converting to JSON
SM> and back returns the same object.

SM> But I don't see any technical difficulty writing a json2.el alternative
SM> which focuses on "to Elisp and back", i.e. when it reads a JSON object
SM> it returns an Elisp object that preserves all JSON nuances so that it
SM> can be reliably converted back to the "exact same" JSON object.

Sure.  I'm saying a custom data structure would help here, not that it's
the only way to accomplish it, and trying to answer your earlier
question about custom record types.

>> a native XML data structure would also be nice.  We have what libxml
>> produces, dumped in an awkward tree, but nothing native.

SM> Not sure what "native" can mean in this context: we were talking about
SM> new Lisp-defined types.

Native to ELisp, but in a way that represents the original data
structure cleanly and transparently.

SM> Whereas you seem to be talking about getting access to objects defined
SM> in other (non-Lisp) libraries, i.e. an FFI.

SM> Or maybe you're just talking about a more efficient representation for
SM> an XML/JSON tree.  If Lisp-defined data types could offer that, it would be
SM> great, of course, but I personally don't see how that would work.

I'm talking about custom data types that can be efficiently and
transparently converted to what the external libraries and protocols
expect, and provide a good ELisp interface to their contents.  I think
the currently available XML and JSON representation in ELisp don't do
both.  Am I misunderstanding the question?

Ted

Re: User-defined record types

[Stefan Monnier]

>>> Something that represents JSON and YAML well would be nice.  Currently
>>> we don't have an ELisp data structure that can preserve all JSON nuances
>>> without acrobatics (e.g. preserving the difference between "null" and
>>> "empty list" or the native JSON data types).
SM> I don't understand what you mean here.  It's very easy to have special
SM> values, e.g. (defconst json-null (make-symbol "json-null")).
> Yes, but it's not something you can communicate externally.  Compare
> with pure JSON or BSON, which are intended for communication across
> programs.

You lost me: I though "null" and "empty list" were JSON thingies, so
I just offered you ways to represent them on the Elisp side.  Of course,
when turning those elements into JSON, you'd be careful to map them to
the appropriate JSON elements.

> Sure.  I'm saying a custom data structure would help here, not that it's
> the only way to accomplish it, and trying to answer your earlier
> question about custom record types.

I still don't understand in what way a custom data structure would help.

>>> a native XML data structure would also be nice.  We have what libxml
>>> produces, dumped in an awkward tree, but nothing native.
SM> Not sure what "native" can mean in this context: we were talking about
SM> new Lisp-defined types.
> Native to ELisp, but in a way that represents the original data
> structure cleanly and transparently.

I still don't see what that means.  In which way would it be cleaner or
more transparent?

> I'm talking about custom data types that can be efficiently and
> transparently converted to what the external libraries and protocols
> expect, and provide a good ELisp interface to their contents.  I think
> the currently available XML and JSON representation in ELisp don't do
> both.  Am I misunderstanding the question?

What alternative do you have in mind that would be more efficient and/or
more transparent?


        Stefan

RE: User-defined record types

[Drew Adams]

> I though "null" and "empty list" were JSON thingies

FWIW: `null', yes; `empty list', no.

(But empty array and empty object, yes.)

Re: User-defined record types

[Ted Zlatanov]

On Fri, 18 Oct 2013 22:09:39 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>>>> Something that represents JSON and YAML well would be nice.  Currently
>>>> we don't have an ELisp data structure that can preserve all JSON nuances
>>>> without acrobatics (e.g. preserving the difference between "null" and
>>>> "empty list" or the native JSON data types).
SM> I don't understand what you mean here.  It's very easy to have special
SM> values, e.g. (defconst json-null (make-symbol "json-null")).
>> Yes, but it's not something you can communicate externally.  Compare
>> with pure JSON or BSON, which are intended for communication across
>> programs.

SM> You lost me: I though "null" and "empty list" were JSON thingies, so
SM> I just offered you ways to represent them on the Elisp side.  Of course,
SM> when turning those elements into JSON, you'd be careful to map them to
SM> the appropriate JSON elements.

Right.  So instead of a special mapping operation, the underlying C
storage (record) should hold the JSON/YAML/XML types and values
directly.  The ELisp API to that C storage would map things to
ELisp-land, but the original data would be preserved.

>> Sure.  I'm saying a custom data structure would help here, not that it's
>> the only way to accomplish it, and trying to answer your earlier
>> question about custom record types.

SM> I still don't understand in what way a custom data structure would help.

By holding the original data in a way that today's Emacs internals can't.

>>>> a native XML data structure would also be nice.  We have what libxml
>>>> produces, dumped in an awkward tree, but nothing native.
SM> Not sure what "native" can mean in this context: we were talking about
SM> new Lisp-defined types.
>> Native to ELisp, but in a way that represents the original data
>> structure cleanly and transparently.

SM> I still don't see what that means.  In which way would it be cleaner or
SM> more transparent?

>> I'm talking about custom data types that can be efficiently and
>> transparently converted to what the external libraries and protocols
>> expect, and provide a good ELisp interface to their contents.  I think
>> the currently available XML and JSON representation in ELisp don't do
>> both.  Am I misunderstanding the question?

SM> What alternative do you have in mind that would be more efficient and/or
SM> more transparent?

Well, let's look at your original question:

On Sat, 12 Oct 2013 14:58:15 -0400 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>>> I'm not necessarily opposed to user-defined record types, but if
>>> cl-defstruct is not sufficient and C-level changes are required, then
>>> I'd hope for the new code to provide more exciting features than just
>>> "vectorp returns nil".

Look.  This is JSON (YAML is similar; XML is much more complex but has
similar conversion and implementation issues):

{ "x": 1, "y": [ null, [], true, false ] }

A C structure would represent that as follows:

- the number 1 would be stored as a JSON-spec integer, which is
  essentially a tagged string.  These are not ELisp ints or bignums
  although the implementation can choose to limit them to Javascript
  ints.  See http://www.json.org/ for the details on number formatting.

- the value under the "y" key has four distinct values that should be
  represented and serialized differently, yet provide the expected
  semantics for ELisp access ("null", "[]", and "false" would be exposed
  as `nil').

- stored efficiently

All of this can be done in ELisp with custom symbols and other tricks,
but I think a C implementation would be cleaner by being closer to the
original input data and removing ELisp baggage from a simple data
format.  The C implementation would also be more efficient than an ELisp
implementation if it serializes to and from JSON in C, but that's a
minor concern.  Finally, a C implementation would be able to make null,
false, and [] look like `nil' in boolean or iteration contexts, yet
preserve the original data.  I don't think an ELisp implementation can
do that unless all access goes through a forced API.

So the exciting feature is that you don't have to work around the
absence of these record types in ELisp.  You may consider that an
advantage (less C code = less maintenance) but the original question was
about "exciting features" not maintainability.

Ted

Re: User-defined record types

[Stefan Monnier]

SM> You lost me: I though "null" and "empty list" were JSON thingies, so
SM> I just offered you ways to represent them on the Elisp side.  Of course,
SM> when turning those elements into JSON, you'd be careful to map them to
SM> the appropriate JSON elements.
> Right.  So instead of a special mapping operation, the underlying C
> storage (record) should hold the JSON/YAML/XML types and values
> directly.

Huh... no, I'm thinking about json.el, with no C code involved at all.

SM> I still don't understand in what way a custom data structure would help.
> By holding the original data in a way that today's Emacs internals can't.

For example?

> - the value under the "y" key has four distinct values that should be
>   represented and serialized differently, yet provide the expected
>   semantics for ELisp access ("null", "[]", and "false" would be exposed
>   as `nil').

Let's call [[e]] the Elisp representation of JSON's e.
Then you're saying that [[null]] would be exposed as nil and
[[false]] as well?  Then would it be the case that (eq [[false]] [[null]])?

I really don't see how you expect to turn those into `nil', preserve
sane semantics (and Elisp performance), and be able to recover null, [],
and false when converting them back to JSON.  You can probably pick any
two of the three, but all three together sounds pretty much impossible.

> - stored efficiently

Saying so is not sufficient to make it so.

> format.  The C implementation would also be more efficient than an ELisp
> implementation if it serializes to and from JSON in C, but that's a
> minor concern.

We're back at the FFI discussion, which is a different issue.

> Finally, a C implementation would be able to make null, false, and []
> look like `nil' in boolean or iteration contexts, yet preserve the
> original data.

C is not magic.  Such a "feature" would require changes deep in the
elisp/bytecode interpreter, the NILP function/macro, ...


        Stefan

Re: User-defined record types

[Ted Zlatanov]

On Sat, 19 Oct 2013 10:37:48 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> - the value under the "y" key has four distinct values that should be
>> represented and serialized differently, yet provide the expected
>> semantics for ELisp access ("null", "[]", and "false" would be exposed
>> as `nil').

SM> Let's call [[e]] the Elisp representation of JSON's e.
SM> Then you're saying that [[null]] would be exposed as nil and
SM> [[false]] as well?  Then would it be the case that (eq [[false]] [[null]])?

I would leave that up to the creator of the user-defined record type.

SM> I really don't see how you expect to turn those into `nil', preserve
SM> sane semantics (and Elisp performance), and be able to recover null, [],
SM> and false when converting them back to JSON.  You can probably pick any
SM> two of the three, but all three together sounds pretty much
SM> impossible.

The job of this hypothetical user-defined record type would be:

- preserve the original data

- provide the facilities to do value folding based on the underlying data type

- transparent conversion back and forth between ELisp and the C implementation

- preserve ELisp semantics as far as deemed necessary by the type's creator

>> Finally, a C implementation would be able to make null, false, and []
>> look like `nil' in boolean or iteration contexts, yet preserve the
>> original data.

SM> C is not magic.  Such a "feature" would require changes deep in the
SM> elisp/bytecode interpreter, the NILP function/macro, ...

I understand that.  Again, you asked about exciting features that could
come from user-defined record types.  You didn't qualify it further as
"must be implemented with no major changes" etc.

So to restate things clearly, I think some facility that represents
structured data such as JSON, YAML, and XML well based on some
user-supplied mappings would be terrific.  The current ELisp
representations of those three structured data formats are pretty lousy.
I think this facility would be useful in many other situations by
essentially encapsulating external data and providing an API on top.

Whether that's a realistic priority for anyone and whether it's
interesting to the Emacs maintainers, I don't know.

Ted

Re: User-defined record types

[Stefan Monnier]

> user-supplied mappings would be terrific.  The current ELisp
> representations of those three structured data formats are pretty lousy.

M-x report-emacs-bug and give some "sample API" you'd like to use instead.


        Stefan

User-defined record types

[Lars Brinkhoff]

Stefan Monnier wrote:
>> Tom Tromey wrote:
>>> It's kind of hacky though.  I think it's probably better to just add
>>> funcallable instances directly, and real types of some kind at the
>>> same time.
>> I tried to submit a patch for user-defined record types some years
>> ago.  Instances were pseudovectors, with the first element being a
>> symbol naming its type.
> Yes, I think we're pretty much ready to accept such a patch

This is my old patch dusted off and rebased to current master.
It's just a raw material posted for review.

A test case would be:

  (let ((x (make-record 'foo 3 nil)))
    (aset x 1 1)
    (aset x 2 2)
    (aset x 3 3)
    (list (read-from-string (with-output-to-string (prin1 x)))
    (recordp x)
    (type-of x)
    (aref x 0)
    (aref x 3)
    (length x)))

This evalates to ((#%[foo 1 2 3] . 13) t foo foo 3 4).


diff --git a/src/alloc.c b/src/alloc.c
index ae3e151..de08276 100644
--- a/src/alloc.c
+++ b/src/alloc.c
@@ -3392,6 +3392,46 @@ struct buffer *
   return b;
 }
 
+static void
+check_record_type (Lisp_Object type)
+{
+  if (!SYMBOLP(type))
+    error ("Invalid type; must be symbol");
+}
+
+static struct Lisp_Vector *
+allocate_record (int count)
+{
+  if (count >= (1 << PSEUDOVECTOR_SIZE_BITS))
+    error ("Record too large");
+
+  struct Lisp_Vector *p = allocate_vector (count);
+  XSETPVECTYPE (p, PVEC_RECORD);
+  return p;
+}
+
+DEFUN ("make-record", Fmake_record, Smake_record, 3, 3, 0,
+       doc: /* Create a new record of type TYPE with SLOTS elements, each initialized to INIT.  */)
+  (Lisp_Object type, Lisp_Object slots, Lisp_Object init)
+{
+  Lisp_Object vector;
+  ptrdiff_t size, i;
+  struct Lisp_Vector *p;
+
+  CHECK_NATNUM (slots);
+  check_record_type (type);
+
+  size = XFASTINT (slots) + 1;
+  p = allocate_record (size);
+  p->contents[0] = type;
+  for (i = 1; i < size; i++)
+    p->contents[i] = init;
+
+  XSETVECTOR (vector, p);
+  return vector;
+}
+
+
 DEFUN ("make-vector", Fmake_vector, Smake_vector, 2, 2, 0,
        doc: /* Return a newly created vector of length LENGTH, with each element being INIT.
 See also the function `vector'.  */)
@@ -7465,6 +7505,7 @@ This means that certain objects should be allocated in shared (pure) space.
   defsubr (&Smake_byte_code);
   defsubr (&Smake_list);
   defsubr (&Smake_vector);
+  defsubr (&Smake_record);
   defsubr (&Smake_string);
   defsubr (&Smake_bool_vector);
   defsubr (&Smake_symbol);
diff --git a/src/data.c b/src/data.c
index ae8dd97..eceb752 100644
--- a/src/data.c
+++ b/src/data.c
@@ -267,6 +267,7 @@ static void swap_in_symval_forwarding (struct Lisp_Symbol *,
         case PVEC_MUTEX: return Qmutex;
         case PVEC_CONDVAR: return Qcondition_variable;
         case PVEC_TERMINAL: return Qterminal;
+        case PVEC_RECORD: return AREF (object, 0);
         /* "Impossible" cases.  */
         case PVEC_XWIDGET:
         case PVEC_OTHER:
@@ -359,6 +360,15 @@ static void swap_in_symval_forwarding (struct Lisp_Symbol *,
   return Qnil;
 }
 
+DEFUN ("recordp", Frecordp_p, Srecordp, 1, 1, 0,
+       doc: /* Return t if OBJECT is a record.  */)
+  (Lisp_Object object)
+{
+  if (RECORDP (object))
+    return Qt;
+  return Qnil;
+}
+
 DEFUN ("stringp", Fstringp, Sstringp, 1, 1, 0,
        doc: /* Return t if OBJECT is a string.  */
        attributes: const)
@@ -2287,7 +2297,7 @@ If the current binding is global (the default), the value is nil.  */)
       ptrdiff_t size = 0;
       if (VECTORP (array))
 	size = ASIZE (array);
-      else if (COMPILEDP (array))
+      else if (COMPILEDP (array) || RECORDP (array))
 	size = ASIZE (array) & PSEUDOVECTOR_SIZE_MASK;
       else
 	wrong_type_argument (Qarrayp, array);
@@ -2308,7 +2318,8 @@ If the current binding is global (the default), the value is nil.  */)
 
   CHECK_NUMBER (idx);
   idxval = XINT (idx);
-  CHECK_ARRAY (array, Qarrayp);
+  if (! RECORDP (array))
+    CHECK_ARRAY (array, Qarrayp);
 
   if (VECTORP (array))
     {
@@ -2328,7 +2339,14 @@ If the current binding is global (the default), the value is nil.  */)
       CHECK_CHARACTER (idx);
       CHAR_TABLE_SET (array, idxval, newelt);
     }
-  else
+  else if (RECORDP (array))
+    {
+      ptrdiff_t size = ASIZE (array) & PSEUDOVECTOR_SIZE_MASK;
+      if (idxval < 0 || idxval >= size)
+	args_out_of_range (array, idx);
+      ASET (array, idxval, newelt);
+    }
+  else /* STRINGP */
     {
       int c;
 
@@ -3714,6 +3732,7 @@ enum bool_vector_op { bool_vector_exclusive_or,
   DEFSYM (Qbuffer, "buffer");
   DEFSYM (Qframe, "frame");
   DEFSYM (Qvector, "vector");
+  DEFSYM (Qrecord, "record");
   DEFSYM (Qchar_table, "char-table");
   DEFSYM (Qbool_vector, "bool-vector");
   DEFSYM (Qhash_table, "hash-table");
@@ -3750,6 +3769,7 @@ enum bool_vector_op { bool_vector_exclusive_or,
   defsubr (&Sstringp);
   defsubr (&Smultibyte_string_p);
   defsubr (&Svectorp);
+  defsubr (&Srecordp);
   defsubr (&Schar_table_p);
   defsubr (&Svector_or_char_table_p);
   defsubr (&Sbool_vector_p);
diff --git a/src/fns.c b/src/fns.c
index 1065355..36bde20 100644
--- a/src/fns.c
+++ b/src/fns.c
@@ -104,7 +104,7 @@ static void sort_vector_copy (Lisp_Object, ptrdiff_t,
     XSETFASTINT (val, MAX_CHAR);
   else if (BOOL_VECTOR_P (sequence))
     XSETFASTINT (val, bool_vector_size (sequence));
-  else if (COMPILEDP (sequence))
+  else if (COMPILEDP (sequence) || RECORDP (sequence))
     XSETFASTINT (val, ASIZE (sequence) & PSEUDOVECTOR_SIZE_MASK);
   else if (CONSP (sequence))
     {
diff --git a/src/lisp.h b/src/lisp.h
index ab4db4c..fb5fed1 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -874,6 +874,7 @@ enum pvec_type
   PVEC_TERMINAL,
   PVEC_WINDOW_CONFIGURATION,
   PVEC_SUBR,
+  PVEC_RECORD,
   PVEC_OTHER,            /* Should never be visible to Elisp code.  */
   PVEC_XWIDGET,
   PVEC_XWIDGET_VIEW,
@@ -2728,6 +2729,12 @@ enum char_bits
   return PSEUDOVECTORP (a, PVEC_FRAME);
 }
 
+INLINE bool
+RECORDP (Lisp_Object a)
+{
+  return PSEUDOVECTORP (a, PVEC_RECORD);
+}
+
 /* Test for image (image . spec)  */
 INLINE bool
 IMAGEP (Lisp_Object x)
diff --git a/src/lread.c b/src/lread.c
index 5c6a7f9..1fcbc37 100644
--- a/src/lread.c
+++ b/src/lread.c
@@ -2762,6 +2762,19 @@ BUFFER is the buffer to evaluate (nil means use current buffer),
 	  make_byte_code (vec);
 	  return tmp;
 	}
+      if (c == '%')
+	{
+	  c = READCHAR;
+	  if (c == '[')
+	    {
+	      Lisp_Object tmp;
+	      tmp = read_vector (readcharfun, 1);
+	      XSETPVECTYPE (XVECTOR(tmp), PVEC_RECORD);
+	      return tmp;
+	    }
+	  UNREAD (c);
+	  invalid_syntax ("#");
+	}
       if (c == '(')
 	{
 	  Lisp_Object tmp;
diff --git a/src/print.c b/src/print.c
index e857761..f7ecd3c 100644
--- a/src/print.c
+++ b/src/print.c
@@ -1966,6 +1966,7 @@
       case PVEC_SUB_CHAR_TABLE:
       case PVEC_COMPILED:
       case PVEC_CHAR_TABLE:
+      case PVEC_RECORD:
       case PVEC_NORMAL_VECTOR: ;
 	{
 	  ptrdiff_t size = ASIZE (obj);
@@ -1974,6 +1975,12 @@
 	      printchar ('#', printcharfun);
 	      size &= PSEUDOVECTOR_SIZE_MASK;
 	    }
+	  if (RECORDP (obj))
+	    {
+	      printchar ('#', printcharfun);
+	      printchar ('%', printcharfun);
+	      size &= PSEUDOVECTOR_SIZE_MASK;
+	    }
 	  if (CHAR_TABLE_P (obj) || SUB_CHAR_TABLE_P (obj))
 	    {
 	      /* We print a char-table as if it were a vector,

Re: User-defined record types

[Lars Brinkhoff]

Lars Brinkhoff wrote:
> This is my old patch dusted off and rebased to current master.
> It's just a raw material posted for review.

This is how cl-defstruct could be modified to optionally make record
instances.  More work would probably be needed in cl-preloaded.el and
cl-generic.el.

Test case:

  (cl-defstruct (foo (:type record)) x y z)
  (let ((x (make-foo :y 1)))
    (list (type-of x)
          (foo-p x)
          (recordp x)
          (foo-y x)
          x))



diff --git a/lisp/emacs-lisp/cl-macs.el b/lisp/emacs-lisp/cl-macs.el
index 40342f3..dead86e 100644
--- a/lisp/emacs-lisp/cl-macs.el
+++ b/lisp/emacs-lisp/cl-macs.el
@@ -2544,6 +2544,12 @@ cl--sublis
       (cons (cl--sublis alist (car tree)) (cl--sublis alist (cdr tree))))
      (t tree))))
 
+(defun record (type &rest elements)
+  (let ((result (make-record type (length elements) nil))
+	(i 0))
+    (dolist (elt elements result)
+      (aset result (cl-incf i) elt))))
+
 ;;; Structures.
 
 (defmacro cl--find-class (type)
@@ -2656,6 +2662,8 @@ cl-defstruct
 				  descs)))
 	      (t
 	       (error "Structure option %s unrecognized" opt)))))
+    (if (eq type 'record)
+        (setq named t))
     (unless (or include-name type)
       (setq include-name cl--struct-default-parent))
     (when include-name (setq include (cl--struct-get-class include-name)))
@@ -2684,7 +2692,7 @@ cl-defstruct
 	  (if (cl--struct-class-named include) (setq tag name named t)))
       (if type
 	  (progn
-	    (or (memq type '(vector list))
+	    (or (memq type '(vector list record))
 		(error "Invalid :type specifier: %s" type))
 	    (if named (setq tag name)))
 	(setq named 'true)))
@@ -2700,6 +2708,9 @@ cl-defstruct
                              `(and (vectorp cl-x)
                                    (>= (length cl-x) ,(length descs))
                                    (memq (aref cl-x ,pos) ,tag-symbol)))
+                            ((eq type 'record)
+                             `(and (recordp cl-x)
+                                   (memq (type-of cl-x) ,tag-symbol)))
                             ((= pos 0) `(memq (car-safe cl-x) ,tag-symbol))
                             (t `(and (consp cl-x)
 				     (memq (nth ,pos cl-x) ,tag-symbol))))))
@@ -2740,7 +2751,7 @@ cl-defstruct
 			      (list `(or ,pred-check
                                          (signal 'wrong-type-argument
                                                  (list ',name cl-x)))))
-                       ,(if (memq type '(nil vector)) `(aref cl-x ,pos)
+                       ,(if (memq type '(nil vector record)) `(aref cl-x ,pos)
                           (if (= pos 0) '(car cl-x)
                             `(nth ,pos cl-x))))
                     forms)

Re: User-defined record types

[Lars Brinkhoff]

Stefan Monnier wrote:
>> diff --git a/src/data.c b/src/data.c
>> index ae8dd97..eceb752 100644
>> --- a/src/data.c
>> +++ b/src/data.c
>> @@ -267,6 +267,7 @@ static void swap_in_symval_forwarding (struct
>> Lisp_Symbol *,
>>          case PVEC_MUTEX: return Qmutex;
>>          case PVEC_CONDVAR: return Qcondition_variable;
>>          case PVEC_TERMINAL: return Qterminal;
>> +        case PVEC_RECORD: return AREF (object, 0);
>>          /* "Impossible" cases.  */
>>          case PVEC_XWIDGET:
>>          case PVEC_OTHER:
>
> `type-of` is expected to return a symbol.  So the above code implies
> that `record` objects should have a symbol their slot 0 and that this
> symbol should be the record's type name.

Right.  Fmake_record does check that slot 0 is initialized to a symbol.

> Currently EIEIO and `cl-defstruct` indeed puts a symbol in slot 0 but
> this symbol is not exactly the struct type, so returning AREF (object,
> 0) wouldn't return quite the right value.

I don't understand everything about cl-defstruct, and even less about
Emacs flavor of generic functions or EIEIO.  I have a patch which
demostrates how to make cl-defstruct create record instances if you
explicitly ask for it.  It does put the type name in slot 0.

If you pass (:named foo) to cl-defstruct, foo will be put in list or
vector slot 0, so I suppose that should work for records too.

If you don't use the :named option, it seems there will be no type
information in the instances.

Re: User-defined record types

[Lars Brinkhoff]

> If you pass (:named foo) to cl-defstruct, foo will be put in list or
> vector slot 0, so I suppose that should work for records too.
>
> If you don't use the :named option, it seems there will be no type
> information in the instances.

I see now this is wrong, and there are more cases to consider.
This is the situation as of now:

- If you don't pass the :type option to cl-defstruct, you get a
cl-struct-FOO type tag in slot 0.

- If you pass :type but not :named, you don't get any type tag.

- If you pass :type and :named, you get a type tag in slot 0, which is
the same symbol as the struct name.

Re: User-defined record types

[Stefan Monnier]

>> If you pass (:named foo) to cl-defstruct, foo will be put in list or
>> vector slot 0, so I suppose that should work for records too.
>> If you don't use the :named option, it seems there will be no type
>> information in the instances.
> I see now this is wrong, and there are more cases to consider.
> This is the situation as of now:
> - If you don't pass the :type option to cl-defstruct, you get a
> cl-struct-FOO type tag in slot 0.

That's right.  And this symbol's value is the class object.
The same holds for EIEIO.

These are the main candidates for use of the new `record` type.

> - If you pass :type but not :named, you don't get any type tag.

And those should not use `record` but `vector` or `list`
depending on the :type that was specified.

> - If you pass :type and :named, you get a type tag in slot 0, which is
> the same symbol as the struct name.

But again, these shouldn't use `record`.


        Stefan

Re: User-defined record types

[Lars Brinkhoff]

Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> - If you don't pass the :type option to cl-defstruct, you get a
>> cl-struct-FOO type tag in slot 0.
>
> That's right.  And this symbol's value is the class object.
> The same holds for EIEIO.
>
> These are the main candidates for use of the new `record` type.

Yes, at first I tried to make cl-deftype do this by default.  But I ran
into problems in cl-preloaded.el and cl-generic.el.

>> - If you pass :type but not :named, you don't get any type tag.
>
> And those should not use `record` but `vector` or `list`
> depending on the :type that was specified.
>
>> - If you pass :type and :named, you get a type tag in slot 0, which is
>> the same symbol as the struct name.
>
> But again, these shouldn't use `record`.

Right, I just wanted to list all possible variations to gain some
understanding.  For example, I don't know if it makes a difference if
the slot 0 type tag has the form cl-struct-FOO as is the default, or
just FOO as when you specify :type.  But now I know there might be.

Re: User-defined record types

[Stefan Monnier]

>>> - If you don't pass the :type option to cl-defstruct, you get a
>>> cl-struct-FOO type tag in slot 0.
>> That's right.  And this symbol's value is the class object.
>> The same holds for EIEIO.
>> These are the main candidates for use of the new `record` type.
> Yes, at first I tried to make cl-deftype do this by default.  But I ran
> into problems in cl-preloaded.el and cl-generic.el.

I can help with that.

> Right, I just wanted to list all possible variations to gain some
> understanding.  For example, I don't know if it makes a difference if
> the slot 0 type tag has the form cl-struct-FOO as is the default, or
> just FOO as when you specify :type.  But now I know there might be.

The cl-struct-FOO symbol was used instead of just FOO in order to try
and avoid false-positives (i.e. to try and avoid mistakenly recognizing
some random array as being a cl-struct object).

With `record` you don't need that trick any more, so records could have
the FOO symbol in their slot 0.

For EIEIO objects, it would slow things down a bit, because EIEIO
accessors needs to consult the class object, so the time it takes to get
from (aref obj 0) to the actual class object is important: if we put the
actual type symbol, that means that we'd need something like

    (get (aref obj 0) 'cl--class)

whereas the current code uses

    (symbol-value (aref obj 0))

The problem being that the identifier used for the type name might
already be used as a variable as well as as function, so we can't use
the `symbol-value` or `symbol-function` slot.

As mentioned, ideally, we'd want to store the class object directly in
the slot 0.  The downside is that prin1 would then dump the class object
as well, so when reading dumped objects we'd end up creating another
copy of the class object rather than reusing an existing class object.
And this will break cl-generic dispatch which compares class objects
with `eq`.


        Stefan

Re: User-defined record types

[Lars Brinkhoff]

Stefan Monnier wrote:
>> Yes, at first I tried to make cl-deftype do this by default.  But I
>> ran into problems in cl-preloaded.el and cl-generic.el.
> I can help with that.

Thanks.  I got past that and made it through to a dumped emacs which
uses record types for defstruct by default.

However, there is still EIEIO which may well need some expert guidance
to update.

> As mentioned, ideally, we'd want to store the class object directly in
> the slot 0.  The downside is that prin1 would then dump the class object
> as well, so when reading dumped objects we'd end up creating another
> copy of the class object rather than reusing an existing class object.
> And this will break cl-generic dispatch which compares class objects
> with `eq`.

Understood.  I'll look into storing the class object in slot 0.  I
imagine there will be some circular bootstrapping problem, e.g. creating
the first record object requires a class object which is a record
object.

Some suggestions:

- type-of looks into the class object and returns the symbol naming the
  class.

- class-of could be introduced to return a class object, like CLOS.

- The read/print syntax for records uses the symbol for the type slot.
  Reading a record would then maybe only work right if the class object
  has been defined first.  Not sure if that's ok.

Re: User-defined record types

[Stefan Monnier]

> Thanks.  I got past that and made it through to a dumped emacs which
> uses record types for defstruct by default.

Cool.

> However, there is still EIEIO which may well need some expert guidance
> to update.

I can take care of that.  Can you put your code on a branch like
scratch/record?

>> As mentioned, ideally, we'd want to store the class object directly in
>> the slot 0.  The downside is that prin1 would then dump the class object
>> as well, so when reading dumped objects we'd end up creating another
>> copy of the class object rather than reusing an existing class object.
>> And this will break cl-generic dispatch which compares class objects
>> with `eq`.
> Understood.  I'll look into storing the class object in slot 0.
> I imagine there will be some circular bootstrapping problem, e.g. creating
> the first record object requires a class object which is a record
> object.

Yes, cl-preloaded.el would likely need some extra care to get the system
to bootstrap, but I'm not worried about that.

> Some suggestions:
> - type-of looks into the class object and returns the symbol naming the
>   class.

Yes, that would make a lot of sense, but that means we have to impose
a particular shape on the contents of slot 0.  IOW it means that the
representation of classes is at least somewhat known/imposed by the
C code.  It's probably OK.

> - class-of could be introduced to return a class object, like CLOS.

Sure.

> - The read/print syntax for records uses the symbol for the type slot.
>   Reading a record would then maybe only work right if the class object
>   has been defined first.  Not sure if that's ok.

Not sure either.  Of course, another option is for prin1 to print the
whole class object in slot 0, and when we read it in, we handle it
specially by looking at the class name inside the class object and
reusing the corresponding class if it already exists.  The would likely
hard-code even more of the notion of class-objects into the C code.


        Stefan

Re: User-defined record types

[Lars Brinkhoff]

Stefan Monnier wrote:
>> However, there is still EIEIO which may well need some expert guidance
>> to update.
>
> I can take care of that.  Can you put your code on a branch like
> scratch/record?

Certainly.  Do I need to apply for project membership or write
permission or something like that?

Re: User-defined record types

[Stefan Monnier]

> Certainly.  Do I need to apply for project membership or write
> permission or something like that?

If you don't have write access yet, then yes.  Just go to your Savannah
account and request membership in the `emacs` group.


        Stefan

Re: User-defined record types

[Lars Brinkhoff]

Stefan Monnier wrote:
>> However, there is still EIEIO which may well need some expert guidance
>> to update.
> I can take care of that.  Can you put your code on a branch like
> scratch/record?

Done, scratch/record it is.

Note that this doesn't survive a full build due to EIEIO breakage.
I will usually build from master, then checkout the branch and rebuild
the changed files.

Re: User-defined record types

[Stefan Monnier]

> Note that this doesn't survive a full build due to EIEIO breakage.

I'll see about fixing this, thanks,


        Stefan

Re: User-defined record types

[Stefan Monnier]

>> However, there is still EIEIO which may well need some expert guidance
>> to update.
> I can take care of that.  Can you put your code on a branch like
> scratch/record?

OK, it's not working.
I found a problem with cl-defstruct objects, tho:

We're going to have trouble preserving backward compatibility with
pre-existing .elc files.  If you look at the macro expansion of current
cl-defstruct:

    (macroexpand '(cl-defstruct (sm-test) a b))
  =>
    (progn
      (defvar cl-struct-sm-test-tags)
      (define-inline sm-test-p (x) (declare (side-effect-free error-free))
        (inline-letevals (x)
                         (inline-quote
                          (and (vectorp #1=,x) (>= (length #1#) 3)
                               (memq (aref #1# 0) cl-struct-sm-test-tags)
                               t))))
      (put 'sm-test 'cl-deftype-satisfies 'sm-test-p)
      (define-inline sm-test-a #2=(x)
        "Access slot \"a\" of `(sm-test)' struct CL-X."
        #3=(declare (side-effect-free t))
        (inline-letevals #4=(x)
          (inline-quote
           (progn
             (or (memq (aref #5=,x 0) cl-struct-sm-test-tags)
                 (signal #6='wrong-type-argument (list 'sm-test . #7=(,x))))
             (aref #8=,x 1)))))
      (define-inline sm-test-b #2# "Access slot \"b\" of `(sm-test)' struct CL-X." #3#
        (inline-letevals #4#
          (inline-quote
           (progn
             (or (memq (aref #5# 0) cl-struct-sm-test-tags)
                 (signal #6# (list 'sm-test . #7#)))
             (aref #8# 2)))))
      (defalias 'copy-sm-test (function copy-sequence))
      (cl-defsubst make-sm-test
          (&cl-defs (nil . #9=((cl-tag-slot) (a) (b))) &key a b)
        "Constructor for objects of type `sm-test'."
        (declare (side-effect-free t))
        (vector 'cl-struct-sm-test a b))
      (eval-and-compile
        (cl-struct-define 'sm-test nil 'cl-structure-object 'nil nil '#9# 'cl-struct-sm-test-tags 'cl-struct-sm-test 't))
      'sm-test)

As you can see, the code says it inherits from `cl-structure-object'
(which now is of `record` type), but its constructor `make-sm-test`
creates a vector rather than a record, so (cl-typep 'cl-structure-object)
will fail if it only considers `type-of'.

Even if we could arrange for cl-struct-define to override the
definition of make-sm-test, that wouldn't help with all the places
where make-sm-test has been inlined already.

[ Of course, the reverse could happen as well: a new struct type (using
  `record` in its constructor) could inherit from a parent that's still
  using `vector`.  But I think we can arrange for that to happen only in
  cases we don't care to support.  ]


        Stefan

Re: User-defined record types

[Stefan Monnier]

>>> However, there is still EIEIO which may well need some expert guidance
>>> to update.
>> I can take care of that.  Can you put your code on a branch like
>> scratch/record?
> OK, it's not working.
           ^^^
           now
Duh!


        Stefan

Re: User-defined record types

[Lars Brinkhoff]

Stefan Monnier wrote:
> OK, it's [now] working.

A full bootstrap build still fails for me:

cedet/ede.el:46:1:Error: Wrong type argument: sequencep, #%[cl-slot-descriptor expanded nil boolean ((:documentation . "State of an object being expanded in speedbar."))]

> We're going to have trouble preserving backward compatibility with
> pre-existing .elc files.  As you can see, the code says it inherits
> from `cl-structure-object' (which now is of `record` type), but its
> constructor `make-sm-test` creates a vector rather than a record, so
> (cl-typep 'cl-structure-object) will fail if it only considers
> `type-of'.

What if we relax the type check to use (aref object 0) instead?  That
works for both old and new instances, as long as :initial-offset is 0.

Re: User-defined record types

[Stefan Monnier]

> A full bootstrap build still fails for me:
> cedet/ede.el:46:1:Error: Wrong type argument: sequencep,
> #%[cl-slot-descriptor expanded nil boolean ((:documentation . "State of an
> object being expanded in speedbar."))]

Looks like I missed a place where a copy-sequence needs to be turned
into a copy-record?  I know there's still this bug in cl-defstruct's
definition of the copy-<foo> function, but I think this is never used,
so it's probably somewhere else.

>> We're going to have trouble preserving backward compatibility with
>> pre-existing .elc files.  As you can see, the code says it inherits
>> from `cl-structure-object' (which now is of `record` type), but its
>> constructor `make-sm-test` creates a vector rather than a record, so
>> (cl-typep 'cl-structure-object) will fail if it only considers
>> `type-of'.
> What if we relax the type check to use (aref object 0) instead?

But we need an arrayp or recordp check before, so it'd be something like

   (and (or (arrayp x) (recordp x)) (aref x 0))

Problem is that now we're slower than we were before the introduction of
`record`.  One option could be to change `type-of' so as to know about
the old struct format, and to make this backward compatibility hack
dependent on a boolean var which defaults to nil but would be set to
t as soon as we detect the use of an old-style struct.

> That works for both old and new instances, as long as :initial-offset
> is 0.

:initial-offset is always 0 if :type is not `vector` or `list`, so we
don't have to worry about this issue.


        Stefan

Re: User-defined record types

[Lars Brinkhoff]

Stefan Monnier wrote:
> Looks like I missed a place where a copy-sequence needs to be turned
> into a copy-record?  I know there's still this bug in cl-defstruct's
> definition of the copy-<foo> function, but I think this is never used,
> so it's probably somewhere else.

I fixed that, but it was probably the one in EIEIO's clone method.

Full bootstrap works now.

Re: User-defined record types

[Lars Brinkhoff]

Stefan Monnier wrote:
>> What if we relax the type check to use (aref object 0) instead?
>
> One option could be to change `type-of' so as to know about the old
> struct format, and to make this backward compatibility hack dependent
> on a boolean var which defaults to nil but would be set to t as soon
> as we detect the use of an old-style struct.

I have now implemented that in the latest commit to the scratch/record
branch.

Re: User-defined record types

[Stefan Monnier]

We could auto-detect the use of old-style structs (and set the
compatibility var) when cl-struct-define is called with a nil value of
the `type` argument.

[ Of course, this means we need to use a different value for the new
  `record` format.  ]


        Stefan

Re: User-defined record types

[Stefan Monnier]

One more thing: what's up with "record2"?  Which branch should I pay
attention to?  If you don't intend to keep using scratch/record, then
it's better to just replace scratch/record with scratch/record2: you can
rebase/rewrite those `scratch` branches at will.


        Stefan


>>>>> "Stefan" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

> We could auto-detect the use of old-style structs (and set the
> compatibility var) when cl-struct-define is called with a nil value of
> the `type` argument.

> [ Of course, this means we need to use a different value for the new
>   `record` format.  ]


>         Stefan

Re: User-defined record types

[Lars Brinkhoff]

Stefan Monnier wrote:
> We could auto-detect the use of old-style structs (and set the
> compatibility var) when cl-struct-define is called with a nil value of
> the `type` argument.

Great, I'll add that.

> Of course, this means we need to use a different value for the new
> `record` format.

The symbol `record' springs to mind.

> One more thing: what's up with "record2"?  Which branch should I pay
> attention to?  If you don't intend to keep using scratch/record, then
> it's better to just replace scratch/record with scratch/record2: you
> can rebase/rewrite those `scratch` branches at will.

I'm happy to do that.  I was not sure what the policy is with replaced
branched.  But if that is ok, I'll mercilessly rewrite it until it's
readly for merging.  I have deleted record2 and updated scratch/record.

Re: User-defined record types

[Stefan Monnier]

>> Of course, this means we need to use a different value for the new
>> `record` format.
> The symbol `record' springs to mind.

I'd prefer using a symbol like `blue-sky`, to avoid confusion.

> I'm happy to do that.  I was not sure what the policy is with replaced
> branched.  But if that is ok, I'll mercilessly rewrite it until it's
> readly for merging.  I have deleted record2 and updated scratch/record.

Great, thanks,


        Stefan