RFC: User-defined pseudovectors

RFC: User-defined pseudovectors

[Lars Brinkhoff]

Hello,

With the current FFI discussion, this may be a good time to ask for
input on a Lisp extension I have lying around.

Some FFI bindings may want to introduce new Lisp types.  My idea is
that this should be possible to do in Lisp, not just in C.  This might
also be useful in other situations, e.g.

- Future new pseudovector types can be defined in Lisp.

- Lisp code can potentially interoperate more easily with code in
other languages.  Existing languages for Emacs include JavaScript
(Ejacs), Python (Pymacs), Ruby (El4r), Perl (EPL), Smalltalk (Eoops),
Common Lisp (Emacs Common Lisp).

- Possibly cl-defstruct can be extended to make new types if so requested.

The gist of my patch is to add a new type of pseudovector which is
like a normal vector, except the first element holds a symbol which is
its Lisp type.  So type-of returns whatever is in the first slot.
This may sound slightly reckless, and probably is.  It's just a first
shot.

Sample session:

(let ((x (make-typed-pseudovector 3 'foo nil)))
  (aset x 1 1)
  (aset x 2 2)
  (aset x 3 3)
  (list (read-from-string (with-output-to-string (prin1 x)))
        (typed-pseudovector-p x)
        (type-of x)
        (aref x 0)
        (aref x 3)
        (length x)))

 => ((#%[foo 1 2 3] . 13) t foo foo 3 4)

Re: RFC: User-defined pseudovectors

[Lars Brinkhoff]

I wrote:
> With the current FFI discussion, this may be a good time to ask for
> input on a Lisp extension I have lying around.

Here is the patch in its current (unfinished) state.  Suggestions are
welcome.  Would something like this be useful for inclusion in Emacs?


diff a/src/alloc.c b/src/alloc.c
--- a/src/alloc.c
+++ b/src/alloc.c
@@ -3048,6 +3048,12 @@ allocate_hash_table (void)
   return ALLOCATE_PSEUDOVECTOR (struct Lisp_Hash_Table, count, PVEC_HASH_TABLE);
 }
 
+struct Lisp_Vector *
+allocate_typed_pseudovector (int count)
+{
+  return allocate_pseudovector (count, count, PVEC_TYPED_PSEUDOVECTOR);
+}
+
 struct window *
 allocate_window (void)
 {
@@ -3096,6 +3102,30 @@ allocate_process (void)
   return p;
 }
 
+DEFUN ("make-typed-pseudovector", Fmake_typed_pseudovector, Smake_typed_pseudovector, 3, 3, 0,
+       doc: /* Create a new vector-like object of type TYPE with SLOTS elements, each initialized to INIT.  */)
+  (register Lisp_Object slots, Lisp_Object type, Lisp_Object init)
+{
+  Lisp_Object vector;
+  register ptrdiff_t size;
+  register ptrdiff_t i;
+  register struct Lisp_Vector *p;
+
+  CHECK_NATNUM (slots);
+  if (!SYMBOLP(type))
+    signal_error ("Invalid type; must be symbol", type);
+
+  size = XFASTINT (slots) + 1;
+  p = allocate_typed_pseudovector (size);
+  p->u.contents[0] = type;
+  for (i = 1; i < size; i++)
+    p->u.contents[i] = init;
+
+  XSETVECTOR (vector, p);
+  return vector;
+}
+
+
 DEFUN ("make-vector", Fmake_vector, Smake_vector, 2, 2, 0,
        doc: /* Return a newly created vector of length LENGTH, with each element being INIT.
 See also the function `vector'.  */)
@@ -6755,6 +6785,7 @@ The time is in seconds as a floating point value.  */);
   defsubr (&Smake_byte_code);
   defsubr (&Smake_list);
   defsubr (&Smake_vector);
+  defsubr (&Smake_typed_pseudovector);
   defsubr (&Smake_string);
   defsubr (&Smake_bool_vector);
   defsubr (&Smake_symbol);
diff a/src/data.c b/src/data.c
--- a/src/data.c
+++ b/src/data.c
@@ -290,6 +290,9 @@ for example, (type-of 1) returns `integer'.  */)
 	return Qfont_entity;
       if (FONT_OBJECT_P (object))
 	return Qfont_object;
+      if (TYPED_PSEUDOVECTOR_P (object))
+	return AREF (object, 0);
+
       return Qvector;
 
     case Lisp_Float:
@@ -370,6 +373,15 @@ DEFUN ("vectorp", Fvectorp, Svectorp, 1, 1, 0,
   return Qnil;
 }
 
+DEFUN ("typed-pseudovector-p", Ftyped_pseudovector_p, Styped_pseudovector_p, 1, 1, 0,
+       doc: /* Return t if OBJECT is a typed pseudovector.  */)
+  (Lisp_Object object)
+{
+  if (TYPED_PSEUDOVECTOR_P (object))
+    return Qt;
+  return Qnil;
+}
+
 DEFUN ("stringp", Fstringp, Sstringp, 1, 1, 0,
        doc: /* Return t if OBJECT is a string.  */)
   (Lisp_Object object)
@@ -2146,7 +2158,7 @@ or a byte-code object.  IDX starts at 0.  */)
       ptrdiff_t size = 0;
       if (VECTORP (array))
 	size = ASIZE (array);
-      else if (COMPILEDP (array))
+      else if (COMPILEDP (array) || TYPED_PSEUDOVECTOR_P (array))
 	size = ASIZE (array) & PSEUDOVECTOR_SIZE_MASK;
       else
 	wrong_type_argument (Qarrayp, array);
@@ -2167,7 +2179,8 @@ bool-vector.  IDX starts at 0.  */)
 
   CHECK_NUMBER (idx);
   idxval = XINT (idx);
-  CHECK_ARRAY (array, Qarrayp);
+  if (! TYPED_PSEUDOVECTOR_P (array))
+    CHECK_ARRAY (array, Qarrayp);
   CHECK_IMPURE (array);
 
   if (VECTORP (array))
@@ -2196,7 +2209,14 @@ bool-vector.  IDX starts at 0.  */)
       CHECK_CHARACTER (idx);
       CHAR_TABLE_SET (array, idxval, newelt);
     }
-  else
+  else if (TYPED_PSEUDOVECTOR_P (array))
+    {
+      ptrdiff_t size = ASIZE (array) & PSEUDOVECTOR_SIZE_MASK;
+      if (idxval < 0 || idxval >= size)
+	args_out_of_range (array, idx);
+      ASET (array, idxval, newelt);
+    }
+  else /* STRINGP */
     {
       int c;
 
@@ -3506,6 +3526,7 @@ syms_of_data (void)
   defsubr (&Sstringp);
   defsubr (&Smultibyte_string_p);
   defsubr (&Svectorp);
+  defsubr (&Styped_pseudovector_p);
   defsubr (&Schar_table_p);
   defsubr (&Svector_or_char_table_p);
   defsubr (&Sbool_vector_p);
diff a/src/fns.c b/src/fns.c
--- a/src/fns.c
+++ b/src/fns.c
@@ -115,7 +115,7 @@ To get the number of bytes, use `string-bytes'.  */)
     XSETFASTINT (val, MAX_CHAR);
   else if (BOOL_VECTOR_P (sequence))
     XSETFASTINT (val, XBOOL_VECTOR (sequence)->size);
-  else if (COMPILEDP (sequence))
+  else if (COMPILEDP (sequence) || TYPED_PSEUDOVECTOR_P (sequence))
     XSETFASTINT (val, ASIZE (sequence) & PSEUDOVECTOR_SIZE_MASK);
   else if (CONSP (sequence))
     {
diff a/src/lisp.h b/src/lisp.h
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -538,6 +538,7 @@ enum pvec_type
   PVEC_TERMINAL,
   PVEC_WINDOW_CONFIGURATION,
   PVEC_SUBR,
+  PVEC_TYPED_PSEUDOVECTOR,
   PVEC_OTHER,
   /* These should be last, check internal_equal to see why.  */
   PVEC_COMPILED,
@@ -2328,6 +2329,12 @@ FRAMEP (Lisp_Object a)
   return PSEUDOVECTORP (a, PVEC_FRAME);
 }
 
+INLINE bool
+TYPED_PSEUDOVECTOR_P (Lisp_Object a)
+{
+  return PSEUDOVECTORP (a, PVEC_TYPED_PSEUDOVECTOR);
+}
+
 /* Test for image (image . spec)  */
 INLINE bool
 IMAGEP (Lisp_Object x)
diff a/src/lread.c b/src/lread.c
--- a/src/lread.c
+++ b/src/lread.c
@@ -2603,6 +2603,19 @@ read1 (Lisp_Object readcharfun, int *pch, bool first_in_list)
 	  make_byte_code (vec);
 	  return tmp;
 	}
+      if (c == '%')
+	{
+	  c = READCHAR;
+	  if (c == '[')
+	    {
+	      Lisp_Object tmp;
+	      tmp = read_vector (readcharfun, 1);
+	      XSETPVECTYPE (XVECTOR(tmp), PVEC_TYPED_PSEUDOVECTOR);
+	      return tmp;
+	    }
+	  UNREAD (c);
+	  invalid_syntax ("#");
+	}
       if (c == '(')
 	{
 	  Lisp_Object tmp;
diff a/src/print.c b/src/print.c
--- a/src/print.c
+++ b/src/print.c
@@ -1945,6 +1945,12 @@ print_object (Lisp_Object obj, Lisp_Object printcharfun, bool escapeflag)
 	      PRINTCHAR ('#');
 	      size &= PSEUDOVECTOR_SIZE_MASK;
 	    }
+	  if (TYPED_PSEUDOVECTOR_P (obj))
+	    {
+	      PRINTCHAR ('#');
+	      PRINTCHAR ('%');
+	      size &= PSEUDOVECTOR_SIZE_MASK;
+	    }
 	  if (CHAR_TABLE_P (obj) || SUB_CHAR_TABLE_P (obj))
 	    {
 	      /* We print a char-table as if it were a vector,

Re: RFC: User-defined pseudovectors

[Lars Brinkhoff]

Just a small demo of how it would be possible to change cl-defstruct.
With the previous patch applied, plus this one, we get this:

(cl-defstruct foo x y z)

 => foo

(let ((x (make-foo :y 1)))
  (list (type-of x)
        (foo-p x)
        (typed-pseudovector-p x)
        (foo-y x)
        x))

 => (foo t t 1 #%[foo nil 1 nil])


diff a/lisp/emacs-lisp/cl-macs.el b/lisp/emacs-lisp/cl-macs.el
--- a/lisp/emacs-lisp/cl-macs.el
+++ b/lisp/emacs-lisp/cl-macs.el
@@ -2312,6 +2312,12 @@ Like `cl-callf', but PLACE is the second argument of FUNC, not the first.
 
 ;;; Structures.
 
+(defun typed-pseudovector (type &rest elements)
+  (let ((result (make-typed-pseudovector (length elements) type nil))
+	(i 0))
+    (dolist (elt elements result)
+      (aset result (cl-incf i) elt))))
+
 ;;;###autoload
 (defmacro cl-defstruct (struct &rest descs)
   "Define a struct type.
@@ -2450,21 +2456,25 @@ non-nil value, that slot cannot be set via `setf'.
 	    (or (memq type '(vector list))
 		(error "Invalid :type specifier: %s" type))
 	    (if named (setq tag name)))
-	(setq type 'vector named 'true)))
+	(setq type 'typed-pseudovector named 'true tag name)))
     (or named (setq descs (delq (assq 'cl-tag-slot descs) descs)))
     (push `(defvar ,tag-symbol) forms)
     (setq pred-form (and named
 			 (let ((pos (- (length descs)
 				       (length (memq (assq 'cl-tag-slot descs)
 						     descs)))))
-			   (if (eq type 'vector)
-			       `(and (vectorp cl-x)
-				     (>= (length cl-x) ,(length descs))
-				     (memq (aref cl-x ,pos) ,tag-symbol))
+			   (cond
+			    ((eq type 'vector)
+			     `(and (vectorp cl-x)
+				   (>= (length cl-x) ,(length descs))
+				   (memq (aref cl-x ,pos) ,tag-symbol)))
+			    ((eq type 'list)
 			     (if (= pos 0)
 				 `(memq (car-safe cl-x) ,tag-symbol)
 			       `(and (consp cl-x)
-				     (memq (nth ,pos cl-x) ,tag-symbol))))))
+				     (memq (nth ,pos cl-x) ,tag-symbol))))
+			    (t
+			     `(memq (type-of cl-x) ,tag-symbol)))))
 	  pred-check (and pred-form (> safety 0)
 			  (if (and (eq (cl-caadr pred-form) 'vectorp)
 				   (= safety 1))
@@ -2488,9 +2498,10 @@ non-nil value, that slot cannot be set via `setf'.
 			      (list `(or ,pred-check
                                          (error "%s accessing a non-%s"
                                                 ',accessor ',name))))
-                       ,(if (eq type 'vector) `(aref cl-x ,pos)
-                          (if (= pos 0) '(car cl-x)
-                            `(nth ,pos cl-x)))) forms)
+                       ,(if (eq type 'list)
+			    (if (= pos 0) '(car cl-x)
+			      `(nth ,pos cl-x))
+			  `(aref cl-x ,pos))) forms)
 	      (push (cons accessor t) side-eff)
               (if (cadr (memq :read-only (cddr desc)))
                   (push `(gv-define-expander ,accessor

Re: RFC: User-defined pseudovectors

[Dmitry Antipov]

On 10/10/2013 03:22 PM, Lars Brinkhoff wrote:

> The gist of my patch is to add a new type of pseudovector which is
> like a normal vector, except the first element holds a symbol which is
> its Lisp type.  So type-of returns whatever is in the first slot.
> This may sound slightly reckless, and probably is.  It's just a first
> shot.

Why make-typed-pseudovector & the rest can't be done in elisp?

(Look at lisp/emacs-lisp/timer.el to see how some non-trivial
things can be implemented on top of plain vector).

Dmitry

Re: RFC: User-defined pseudovectors

[Dmitry Antipov]

On 10/10/2013 04:52 PM, Dmitry Antipov wrote:

>> The gist of my patch is to add a new type of pseudovector which is
>> like a normal vector, except the first element holds a symbol which is
>> its Lisp type.  So type-of returns whatever is in the first slot.
>> This may sound slightly reckless, and probably is.  It's just a first
>> shot.
>
> Why make-typed-pseudovector & the rest can't be done in elisp?
>
> (Look at lisp/emacs-lisp/timer.el to see how some non-trivial
> things can be implemented on top of plain vector).

Plus, if we really need the ability to add builtin Lisp types to
interface foreign DSOs, I'll vote to design this on top of Lisp_Misc.

Dmitry

Re: RFC: User-defined pseudovectors

[Stefan Monnier]

> - Possibly cl-defstruct can be extended to make new types if so requested.

I'm not sure I understand the difference between your proposal and
cl-defstruct.  The only difference I can see is that `vectorp' returns
nil on your new objects (well, type-of also changes, but that is so
rarely used that it's basically irrelevant).  Oh, and there's a new
print syntax for those objects.

What is the benefit?


        Stefan

Re: RFC: User-defined pseudovectors

[Lars Brinkhoff]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:
> I'm not sure I understand the difference between your proposal and
> cl-defstruct.  The only difference I can see is that `vectorp'
> returns nil on your new objects (well, type-of also changes, but
> that is so rarely used that it's basically irrelevant).

The difference in return value from functions like vectorp and type-of
is the entire point, actually.

> Oh, and there's a new print syntax for those objects.

If it simplifies anything, that can be ignored for the moment.  I just
hacked something up on a whim.

> What is the benefit?

The benefit is that the types of these "typed pseudovectors" should be
disjoint from all previously existing types.  So e.g. this would be
guaranteed to work, even if someone passes [foo] to frob.

  (cl-defstruct foo ...)

  (defun frob (x)
    (cond
     ((vectorp x)   (frob-vector x))
     ((foop x)      (frob-foo x))))

In the case of an FFI, a wrapper may want to be able to pass a vector
or a "struct" (something like a pseudovector) to Emacs, and have Lisp
code be able to tell the difference.

Or in my case, I have an implementation of Common Lisp written in
Emacs Lisp, and it would be very nice if the Emacs Lisp vector type
could also be the Common Lisp type vector.  Instead I have to overload
many Common Lisp types on top of the Emacs Lisp vector.

Re: RFC: User-defined pseudovectors

[Lars Brinkhoff]

Dmitry Antipov <dmantipov@yandex.ru> writes:
> > Why make-typed-pseudovector & the rest can't be done in elisp?

Because the entire point is to be able to create a new Lisp type, and
there currently is no way to do that in Lisp.

> Plus, if we really need the ability to add builtin Lisp types to
> interface foreign DSOs, I'll vote to design this on top of
> Lisp_Misc.

That may well be better, I don't know.


I just now stumbled over this in lisp.h:

> To define a new data type, add one more Lisp_Misc subtype or one
> more pseudovector subtype.  Pseudovectors are more suitable for
> objects with several slots that need to support fast random access,
> while Lisp_Misc types are for everything else. A pseudovector object
> provides one or more slots for Lisp objects, followed by struct
> members that are accessible only from C.  A Lisp_Misc object is a
> wrapper for a C struct that can contain anything you like.

For my primary use case (Common Lisp data types, structs, and class
instances), I'm inclined to believe that pseudovectors are better.
But it may well be that Lisp_Misc is better for FFI datatypes.

Re: RFC: User-defined pseudovectors

[Stefan Monnier]

> The difference in return value from functions like vectorp and type-of
> is the entire point, actually.

It seems like a small benefit.  I understand it might be handy for your
Common-Lisp implementation, but you can already define your own
`vectorp' to get the same result.  Implementing a language on top of
another will always come with such costs because of (sometimes
subtle) mismatches.
Maybe the Guile guys are more likely to be wiling to make changes to
their system in order to better support some other language (such as
Common-Lisp).


        Stefan

Re: RFC: User-defined pseudovectors

[Stefan Monnier]

>   (defun frob (x)
>     (cond
>      ((vectorp x)   (frob-vector x))
>      ((foop x)      (frob-foo x))))

> In the case of an FFI, a wrapper may want to be able to pass a vector
> or a "struct" (something like a pseudovector) to Emacs, and have Lisp
> code be able to tell the difference.

In Elisp, such type-overlaps are common.  E.g. you should first check
`functionp' before `consp' if you can accept both lists and functions.
So similarly the above code should check `foop' before `vectorp'.


        Stefan

Re: RFC: User-defined pseudovectors

[Lars Brinkhoff]

As long as you work inside Emacs, I agree there's no great need for
user-defined types, because Emacs Lisp already provides the data
structures that are commonly needed.  And you can pass around lists or
vectors which provides a 99% solution in most cases.

But with an FFI, I would argue that the situation changes.  When you
want to interface with code written in other languages, it seems to me
that the need increases.  Most other languages have some kind of
user-defined record types, and they are used quite liberally.

Stefan Monnier <monnier@iro.umontreal.ca> writes:
> I understand it might be handy for your Common-Lisp implementation,
> but you can already define your own `vectorp' to get the same result.

I don't see that I could define one which would accept any vector wich
any content, and be able to tell a "proper vector" apart from user-
defined types.

>>     (cond
>>      ((vectorp x)   (frob-vector x))
>>      ((foop x)      (frob-foo x))))
>
> So similarly the above code should check `foop' before `vectorp'.

That would still fail for the input [foo].

> Maybe the Guile guys are more likely to be wiling to make changes to
> their system in order to better support some other language.

No change is needed.  Guile already has user-defined record types,
which is exactly what I want.  Maybe I should just wait for
GuileEmacs, but as you wrote elsewhere its schedule is somewhat
undependable.

Re: RFC: User-defined pseudovectors

[Stefan Monnier]

> that the need increases.  Most other languages have some kind of
> user-defined record types, and they are used quite liberally.

I'm not necessarily opposed to user-defined record types, but if
cl-defstruct is not sufficient and C-level changes are required, then
I'd hope for the new code to provide more exciting features than just
"vectorp returns nil".

>>> (cond
>>> ((vectorp x)   (frob-vector x))
>>> ((foop x)      (frob-foo x))))
>> So similarly the above code should check `foop' before `vectorp'.
> That would still fail for the input [foo].

Not sure if it should be considered as a bug, tho (it depends on what
was the intention of the caller.  IOW: would she have passed #%[foo] or
[foo] in your version of Emacs?).  Especially for cl-defstruct objects
where you'd need to pass [cl-defstruct-foo].  But if that's a problem,
you can use a more unique object as tag, e.g. an uninterned symbol.


        Stefan

User-defined record types

[Lars Brinkhoff]

Stefan Monnier <monnier@iro.umontreal.ca> writes:
> I'm not necessarily opposed to user-defined record types, but if
> cl-defstruct is not sufficient and C-level changes are required, then
> I'd hope for the new code to provide more exciting features than just
> "vectorp returns nil".

Great, I'd like to get to work on that.  What exciting features do you
have in mind?

Re: User-defined record types

[Stefan Monnier]

>> I'm not necessarily opposed to user-defined record types, but if
>> cl-defstruct is not sufficient and C-level changes are required, then
>> I'd hope for the new code to provide more exciting features than just
>> "vectorp returns nil".
> Great, I'd like to get to work on that.  What exciting features do you
> have in mind?

I don't have any, which is why I haven't implemented anything yet ;-)


        Stefan

Re: User-defined record types

[Ted Zlatanov]

On Sat, 12 Oct 2013 14:58:15 -0400 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>>> I'm not necessarily opposed to user-defined record types, but if
>>> cl-defstruct is not sufficient and C-level changes are required, then
>>> I'd hope for the new code to provide more exciting features than just
>>> "vectorp returns nil".
>> Great, I'd like to get to work on that.  What exciting features do you
>> have in mind?

SM> I don't have any, which is why I haven't implemented anything yet ;-)

Something that represents JSON and YAML well would be nice.  Currently
we don't have an ELisp data structure that can preserve all JSON nuances
without acrobatics (e.g. preserving the difference between "null" and
"empty list" or the native JSON data types).

I know about json.el and it's very convenient but not enough.

A native XML data structure would also be nice.  We have what libxml
produces, dumped in an awkward tree, but nothing native.

Ted

Re: User-defined record types

[Stefan Monnier]

> Something that represents JSON and YAML well would be nice.  Currently
> we don't have an ELisp data structure that can preserve all JSON nuances
> without acrobatics (e.g. preserving the difference between "null" and
> "empty list" or the native JSON data types).

I don't understand what you mean here.  It's very easy to have special
values, e.g. (defconst json-null (make-symbol "json-null")).

IIUC The problem of json.el is that it wants to provide a bridge between
traditional Elisp data structures and JSON, and there is no reliable
two-way conversion between the two.
IIUC the current json.el focuses on making sure that converting to JSON
and back returns the same object.

But I don't see any technical difficulty writing a json2.el alternative
which focuses on "to Elisp and back", i.e. when it reads a JSON object
it returns an Elisp object that preserves all JSON nuances so that it
can be reliably converted back to the "exact same" JSON object.

> a native XML data structure would also be nice.  We have what libxml
> produces, dumped in an awkward tree, but nothing native.

Not sure what "native" can mean in this context: we were talking about
new Lisp-defined types.

Whereas you seem to be talking about getting access to objects defined
in other (non-Lisp) libraries, i.e. an FFI.

Or maybe you're just talking about a more efficient representation for
an XML/JSON tree.  If Lisp-defined data types could offer that, it would be
great, of course, but I personally don't see how that would work.


        Stefan

Re: User-defined record types

[Ted Zlatanov]

On Fri, 18 Oct 2013 11:28:17 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> Something that represents JSON and YAML well would be nice.  Currently
>> we don't have an ELisp data structure that can preserve all JSON nuances
>> without acrobatics (e.g. preserving the difference between "null" and
>> "empty list" or the native JSON data types).

SM> I don't understand what you mean here.  It's very easy to have special
SM> values, e.g. (defconst json-null (make-symbol "json-null")).

Yes, but it's not something you can communicate externally.  Compare
with pure JSON or BSON, which are intended for communication across
programs.

SM> IIUC The problem of json.el is that it wants to provide a bridge between
SM> traditional Elisp data structures and JSON, and there is no reliable
SM> two-way conversion between the two.
SM> IIUC the current json.el focuses on making sure that converting to JSON
SM> and back returns the same object.

SM> But I don't see any technical difficulty writing a json2.el alternative
SM> which focuses on "to Elisp and back", i.e. when it reads a JSON object
SM> it returns an Elisp object that preserves all JSON nuances so that it
SM> can be reliably converted back to the "exact same" JSON object.

Sure.  I'm saying a custom data structure would help here, not that it's
the only way to accomplish it, and trying to answer your earlier
question about custom record types.

>> a native XML data structure would also be nice.  We have what libxml
>> produces, dumped in an awkward tree, but nothing native.

SM> Not sure what "native" can mean in this context: we were talking about
SM> new Lisp-defined types.

Native to ELisp, but in a way that represents the original data
structure cleanly and transparently.

SM> Whereas you seem to be talking about getting access to objects defined
SM> in other (non-Lisp) libraries, i.e. an FFI.

SM> Or maybe you're just talking about a more efficient representation for
SM> an XML/JSON tree.  If Lisp-defined data types could offer that, it would be
SM> great, of course, but I personally don't see how that would work.

I'm talking about custom data types that can be efficiently and
transparently converted to what the external libraries and protocols
expect, and provide a good ELisp interface to their contents.  I think
the currently available XML and JSON representation in ELisp don't do
both.  Am I misunderstanding the question?

Ted

Re: User-defined record types

[Stefan Monnier]

>>> Something that represents JSON and YAML well would be nice.  Currently
>>> we don't have an ELisp data structure that can preserve all JSON nuances
>>> without acrobatics (e.g. preserving the difference between "null" and
>>> "empty list" or the native JSON data types).
SM> I don't understand what you mean here.  It's very easy to have special
SM> values, e.g. (defconst json-null (make-symbol "json-null")).
> Yes, but it's not something you can communicate externally.  Compare
> with pure JSON or BSON, which are intended for communication across
> programs.

You lost me: I though "null" and "empty list" were JSON thingies, so
I just offered you ways to represent them on the Elisp side.  Of course,
when turning those elements into JSON, you'd be careful to map them to
the appropriate JSON elements.

> Sure.  I'm saying a custom data structure would help here, not that it's
> the only way to accomplish it, and trying to answer your earlier
> question about custom record types.

I still don't understand in what way a custom data structure would help.

>>> a native XML data structure would also be nice.  We have what libxml
>>> produces, dumped in an awkward tree, but nothing native.
SM> Not sure what "native" can mean in this context: we were talking about
SM> new Lisp-defined types.
> Native to ELisp, but in a way that represents the original data
> structure cleanly and transparently.

I still don't see what that means.  In which way would it be cleaner or
more transparent?

> I'm talking about custom data types that can be efficiently and
> transparently converted to what the external libraries and protocols
> expect, and provide a good ELisp interface to their contents.  I think
> the currently available XML and JSON representation in ELisp don't do
> both.  Am I misunderstanding the question?

What alternative do you have in mind that would be more efficient and/or
more transparent?


        Stefan

RE: User-defined record types

[Drew Adams]

> I though "null" and "empty list" were JSON thingies

FWIW: `null', yes; `empty list', no.

(But empty array and empty object, yes.)

Re: User-defined record types

[Ted Zlatanov]

On Fri, 18 Oct 2013 22:09:39 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>>>> Something that represents JSON and YAML well would be nice.  Currently
>>>> we don't have an ELisp data structure that can preserve all JSON nuances
>>>> without acrobatics (e.g. preserving the difference between "null" and
>>>> "empty list" or the native JSON data types).
SM> I don't understand what you mean here.  It's very easy to have special
SM> values, e.g. (defconst json-null (make-symbol "json-null")).
>> Yes, but it's not something you can communicate externally.  Compare
>> with pure JSON or BSON, which are intended for communication across
>> programs.

SM> You lost me: I though "null" and "empty list" were JSON thingies, so
SM> I just offered you ways to represent them on the Elisp side.  Of course,
SM> when turning those elements into JSON, you'd be careful to map them to
SM> the appropriate JSON elements.

Right.  So instead of a special mapping operation, the underlying C
storage (record) should hold the JSON/YAML/XML types and values
directly.  The ELisp API to that C storage would map things to
ELisp-land, but the original data would be preserved.

>> Sure.  I'm saying a custom data structure would help here, not that it's
>> the only way to accomplish it, and trying to answer your earlier
>> question about custom record types.

SM> I still don't understand in what way a custom data structure would help.

By holding the original data in a way that today's Emacs internals can't.

>>>> a native XML data structure would also be nice.  We have what libxml
>>>> produces, dumped in an awkward tree, but nothing native.
SM> Not sure what "native" can mean in this context: we were talking about
SM> new Lisp-defined types.
>> Native to ELisp, but in a way that represents the original data
>> structure cleanly and transparently.

SM> I still don't see what that means.  In which way would it be cleaner or
SM> more transparent?

>> I'm talking about custom data types that can be efficiently and
>> transparently converted to what the external libraries and protocols
>> expect, and provide a good ELisp interface to their contents.  I think
>> the currently available XML and JSON representation in ELisp don't do
>> both.  Am I misunderstanding the question?

SM> What alternative do you have in mind that would be more efficient and/or
SM> more transparent?

Well, let's look at your original question:

On Sat, 12 Oct 2013 14:58:15 -0400 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>>> I'm not necessarily opposed to user-defined record types, but if
>>> cl-defstruct is not sufficient and C-level changes are required, then
>>> I'd hope for the new code to provide more exciting features than just
>>> "vectorp returns nil".

Look.  This is JSON (YAML is similar; XML is much more complex but has
similar conversion and implementation issues):

{ "x": 1, "y": [ null, [], true, false ] }

A C structure would represent that as follows:

- the number 1 would be stored as a JSON-spec integer, which is
  essentially a tagged string.  These are not ELisp ints or bignums
  although the implementation can choose to limit them to Javascript
  ints.  See http://www.json.org/ for the details on number formatting.

- the value under the "y" key has four distinct values that should be
  represented and serialized differently, yet provide the expected
  semantics for ELisp access ("null", "[]", and "false" would be exposed
  as `nil').

- stored efficiently

All of this can be done in ELisp with custom symbols and other tricks,
but I think a C implementation would be cleaner by being closer to the
original input data and removing ELisp baggage from a simple data
format.  The C implementation would also be more efficient than an ELisp
implementation if it serializes to and from JSON in C, but that's a
minor concern.  Finally, a C implementation would be able to make null,
false, and [] look like `nil' in boolean or iteration contexts, yet
preserve the original data.  I don't think an ELisp implementation can
do that unless all access goes through a forced API.

So the exciting feature is that you don't have to work around the
absence of these record types in ELisp.  You may consider that an
advantage (less C code = less maintenance) but the original question was
about "exciting features" not maintainability.

Ted

Re: User-defined record types

[Stefan Monnier]

SM> You lost me: I though "null" and "empty list" were JSON thingies, so
SM> I just offered you ways to represent them on the Elisp side.  Of course,
SM> when turning those elements into JSON, you'd be careful to map them to
SM> the appropriate JSON elements.
> Right.  So instead of a special mapping operation, the underlying C
> storage (record) should hold the JSON/YAML/XML types and values
> directly.

Huh... no, I'm thinking about json.el, with no C code involved at all.

SM> I still don't understand in what way a custom data structure would help.
> By holding the original data in a way that today's Emacs internals can't.

For example?

> - the value under the "y" key has four distinct values that should be
>   represented and serialized differently, yet provide the expected
>   semantics for ELisp access ("null", "[]", and "false" would be exposed
>   as `nil').

Let's call [[e]] the Elisp representation of JSON's e.
Then you're saying that [[null]] would be exposed as nil and
[[false]] as well?  Then would it be the case that (eq [[false]] [[null]])?

I really don't see how you expect to turn those into `nil', preserve
sane semantics (and Elisp performance), and be able to recover null, [],
and false when converting them back to JSON.  You can probably pick any
two of the three, but all three together sounds pretty much impossible.

> - stored efficiently

Saying so is not sufficient to make it so.

> format.  The C implementation would also be more efficient than an ELisp
> implementation if it serializes to and from JSON in C, but that's a
> minor concern.

We're back at the FFI discussion, which is a different issue.

> Finally, a C implementation would be able to make null, false, and []
> look like `nil' in boolean or iteration contexts, yet preserve the
> original data.

C is not magic.  Such a "feature" would require changes deep in the
elisp/bytecode interpreter, the NILP function/macro, ...


        Stefan

Re: User-defined record types

[Ted Zlatanov]

On Sat, 19 Oct 2013 10:37:48 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> - the value under the "y" key has four distinct values that should be
>> represented and serialized differently, yet provide the expected
>> semantics for ELisp access ("null", "[]", and "false" would be exposed
>> as `nil').

SM> Let's call [[e]] the Elisp representation of JSON's e.
SM> Then you're saying that [[null]] would be exposed as nil and
SM> [[false]] as well?  Then would it be the case that (eq [[false]] [[null]])?

I would leave that up to the creator of the user-defined record type.

SM> I really don't see how you expect to turn those into `nil', preserve
SM> sane semantics (and Elisp performance), and be able to recover null, [],
SM> and false when converting them back to JSON.  You can probably pick any
SM> two of the three, but all three together sounds pretty much
SM> impossible.

The job of this hypothetical user-defined record type would be:

- preserve the original data

- provide the facilities to do value folding based on the underlying data type

- transparent conversion back and forth between ELisp and the C implementation

- preserve ELisp semantics as far as deemed necessary by the type's creator

>> Finally, a C implementation would be able to make null, false, and []
>> look like `nil' in boolean or iteration contexts, yet preserve the
>> original data.

SM> C is not magic.  Such a "feature" would require changes deep in the
SM> elisp/bytecode interpreter, the NILP function/macro, ...

I understand that.  Again, you asked about exciting features that could
come from user-defined record types.  You didn't qualify it further as
"must be implemented with no major changes" etc.

So to restate things clearly, I think some facility that represents
structured data such as JSON, YAML, and XML well based on some
user-supplied mappings would be terrific.  The current ELisp
representations of those three structured data formats are pretty lousy.
I think this facility would be useful in many other situations by
essentially encapsulating external data and providing an API on top.

Whether that's a realistic priority for anyone and whether it's
interesting to the Emacs maintainers, I don't know.

Ted

Re: User-defined record types

[Stefan Monnier]

> user-supplied mappings would be terrific.  The current ELisp
> representations of those three structured data formats are pretty lousy.

M-x report-emacs-bug and give some "sample API" you'd like to use instead.


        Stefan