bug#11198: problems reading data with a "read-hash-extend" registered reader

bug#11198: problems reading data with a "read-hash-extend" registered reader

[Klaus Stehle]

Hi,

;;;; an example script to describe the problem
;;;; writing and reading records:

(use-modules (srfi srfi-9))

;; define a record as example
(define-record-type my-record
  (make-my-record one two)
  my-record?
  (one my-one)
  (two my-two))

;; fix another guile-2.0 bug
(if (string>= (version) "2")
    (struct-set! my-record (+ 2 vtable-offset-user) make-my-record))

;; the record write function
(define (record-printer s p)
  (let* ((rtd (record-type-descriptor s))
         (lst (apply append
                     (map (lambda (f) (list (symbol->keyword f)
                                            ((record-accessor rtd f) s)))
                          (record-type-fields rtd)))))
    (format p "#R~S" (cons (record-type-name rtd) lst))))

(struct-set! my-record vtable-index-printer record-printer)

;; the record read function
(define (read-R chr port)
  (let ((rlst (read port)))
    (if (not (pair? rlst))
        #f
        (let* ((name (car rlst))
               (lst (cdr rlst))
               (rtd (primitive-eval name))
               (fields (record-type-fields rtd)))
          (apply (record-constructor rtd)
                 (map (lambda (f) (let ((kl (memq (symbol->keyword f) 
lst)))
                                    (if (and kl (pair? (cdr kl)))
                                        (cadr kl)
                                        #f))) fields))))))

(read-hash-extend #\R read-R)

;; test-1: writing and reading records is working well ...
(define rec (make-my-record "bla" "bobo"))

(let ((opo (open-output-file "rec.data")))
  (write rec opo)
  (close-output-port opo))

(let ((ipo (open-input-file "rec.data")))
  (let ((data (read ipo)))
    (format #t "--> ~S~%" rec)
    (close-input-port ipo)))

;; test-2: ... but this doesn't work properly
(define rec #R(my-record #:two "bobo" #:one "bla"))
(format #t "==> ~S~%" rec)

;; end of script


The behavour of test-2 is very strange. If the script is running
from a file, there is a compiler error message, but the record is built
and displayed.

If we start the script with the -l option and enter the line
afterwards manually into the REPL ...

(define rec #R(my-record #:two "bobo" #:one "bla"))

... there is another cryptic error message:

=> While compiling expression:
=> ERROR: build-constant-store: unrecognized object
   #R(my-record #:one "bla" #:two "bobo")

We can see our record built correctly in the error message! But the record
remains unreachable.

Using guile-1.8 all these things are running perfectly!


Cheers,
Klaus Stehle


----------------------------
guile --version
guile (GNU Guile) 2.0.5

uname -srm
Linux 2.6.32-5-amd64 x86_64

bug#11198: problems reading data with a "read-hash-extend" registered reader

[Ludovic Courtès]

Hi Klaus,

Klaus Stehle <klaus.stehle@uni-tuebingen.de> skribis:

> (read-hash-extend #\R read-R)

Unlike previous versions, Guile 2.0 has distinct compilation and
run-time phases.  Here you probably want the reader extension to become
effective at compile-time (when compiling), and at evaluation-time (when
interpreting the code):

  (eval-when (compile eval)
    (read-hash-extend #\R read-R))

There’s an example of this in the manual, in the context of
‘current-reader’ (info "(guile) Loading").

Does it work for you?

Thanks,
Ludo’.

bug#11198: problems reading data with a "read-hash-extend" registered reader

[Klaus Stehle]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1233 bytes --]



On Mon, 9 Apr 2012, Ludovic Courtès wrote:

> Date: Mon, 09 Apr 2012 23:10:14 +0200
> From: Ludovic Courtès <ludo@gnu.org>
> To: Klaus Stehle <klaus.stehle@uni-tuebingen.de>
> Cc: 11198@debbugs.gnu.org
> Subject: Re: bug#11198: problems reading data with a "read-hash-extend"
>     registered reader
> 
> Hi Klaus,
> 
> Klaus Stehle <klaus.stehle@uni-tuebingen.de> skribis:
> 
> > (read-hash-extend #\R read-R)
> 
> Unlike previous versions, Guile 2.0 has distinct compilation and
> run-time phases.  Here you probably want the reader extension to become
> effective at compile-time (when compiling), and at evaluation-time (when
> interpreting the code):
> 
>   (eval-when (compile eval)
>     (read-hash-extend #\R read-R))
> 
> There’s an example of this in the manual, in the context of
> ‘current-reader’ (info "(guile) Loading").
> 
> Does it work for you?
> 
> Thanks,
> Ludo’.
> 

Hallo Ludo’,

No it doesn't work. The same behavour: the script runs with a compiler 
error message but nevertheless the record is created and displayed
correctly.
The only way to run the script without error message is to run it with
the --no-auto-compile option.


Thanks,
Klaus

bug#11198: problems reading data with a "read-hash-extend" registered reader

[Mark H Weaver]

ludo@gnu.org (Ludovic Courtès) writes:
> Klaus Stehle <klaus.stehle@uni-tuebingen.de> skribis:
>
>> (read-hash-extend #\R read-R)
>
> Unlike previous versions, Guile 2.0 has distinct compilation and
> run-time phases.  Here you probably want the reader extension to become
> effective at compile-time (when compiling), and at evaluation-time (when
> interpreting the code):
>
>   (eval-when (compile eval)
>     (read-hash-extend #\R read-R))

I don't think this will be sufficient by itself, because 'read-R' will
not yet be bound at compile time.  You also need to define 'read-R'
within the 'eval-when', and everything else that's needed to run
'read-R'.

Alternatively, you could refrain from using the #\R syntax within the
same file where it is defined.

    Mark

bug#11198: problems reading data with a "read-hash-extend" registered reader

[Klaus Stehle]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1727 bytes --]

Hallo Mark,

On Wed, 11 Apr 2012, Mark H Weaver wrote:

> Date: Wed, 11 Apr 2012 15:33:32 -0400
> From: Mark H Weaver <mhw@netris.org>
> To: Ludovic Courtès <ludo@gnu.org>
> Cc: Klaus Stehle <klaus.stehle@uni-tuebingen.de>, 11198@debbugs.gnu.org
> Subject: Re: bug#11198: problems reading data with a "read-hash-extend"
>     registered reader
> 
> ludo@gnu.org (Ludovic Courtès) writes:
> > Klaus Stehle <klaus.stehle@uni-tuebingen.de> skribis:
> >
> >> (read-hash-extend #\R read-R)
> >
> > Unlike previous versions, Guile 2.0 has distinct compilation and
> > run-time phases.  Here you probably want the reader extension to become
> > effective at compile-time (when compiling), and at evaluation-time (when
> > interpreting the code):
> >
> >   (eval-when (compile eval)
> >     (read-hash-extend #\R read-R))
> 
> I don't think this will be sufficient by itself, because 'read-R' will
> not yet be bound at compile time.  You also need to define 'read-R'
> within the 'eval-when', and everything else that's needed to run
> 'read-R'.

You are right. But now the record type is unknown at compile time.
So I also wrap the define-record-type expression into an 'eval-when'.

Then we arrive at the same error message which is displayed when
typing the #R-expression interactively:

=> ERROR: build-constant-store: unrecognized object
=> #<my-record one: "bla" two: "bobo">

This error message is not very informative. And in spite of an error the
record is built! You can see it in the error message but you can't use it.

> Alternatively, you could refrain from using the #\R syntax within the
> same file where it is defined.

Not yet tested ...  (soon ...)


Thanks,
Klaus

bug#11198: problems reading data with a "read-hash-extend" registered reader

[Mark H Weaver]

Klaus Stehle <klaus.stehle@uni-tuebingen.de> writes:
> On Wed, 11 Apr 2012, Mark H Weaver wrote:
>> ludo@gnu.org (Ludovic Courtès) writes:
>> > Klaus Stehle <klaus.stehle@uni-tuebingen.de> skribis:
>> >
>> >> (read-hash-extend #\R read-R)
>> >
>> > Unlike previous versions, Guile 2.0 has distinct compilation and
>> > run-time phases.  Here you probably want the reader extension to become
>> > effective at compile-time (when compiling), and at evaluation-time (when
>> > interpreting the code):
>> >
>> >   (eval-when (compile eval)
>> >     (read-hash-extend #\R read-R))
>> 
>> I don't think this will be sufficient by itself, because 'read-R' will
>> not yet be bound at compile time.  You also need to define 'read-R'
>> within the 'eval-when', and everything else that's needed to run
>> 'read-R'.
>
> You are right. But now the record type is unknown at compile time.
> So I also wrap the define-record-type expression into an 'eval-when'.
>
> Then we arrive at the same error message which is displayed when
> typing the #R-expression interactively:
>
> => ERROR: build-constant-store: unrecognized object
> => #<my-record one: "bla" two: "bobo">
>
> This error message is not very informative. And in spite of an error the
> record is built! You can see it in the error message but you can't use it.

I see now that this is a limitation of our compiler.  It serializes all
literals, and does not know how to serialize records.  The relevant
procedure 'build-constant-store' is in (language glil compile-assembly).

It is not obvious how to fix this.  The tricky part is how to serialize
the identity of the struct-vtable.  The best approach that comes to mind
is to serialize the identifier (syntax-object) of the module variable
that holds the record type descriptor.  These identifiers are already
being serialized within the compiled macros.  However, this approach
can only work for records defined at top-level.

What do you think?

      Mark

bug#11198: problems reading data with a "read-hash-extend" registered reader

[Ludovic Courtès]

Hello!

I think the reader should only return valid Scheme objects that have a
read syntax (info "(r5rs) Lexical Structure"), and records are not among
them.

So a short-term solution would be to change ‘read-R’ to return the
expressions that builds the record, instead of the record itself–just
like macros return syntax objects, not arbitrary Scheme objects.

  (define (read-R chr port)
    (let ((rlst (read port)))
      (if (not (pair? rlst))
          #f
          (let* ((name (car rlst))
                 (lst (cdr rlst))
                 (rtd (primitive-eval name))
                 (fields (record-type-fields rtd)))
            `(apply (record-constructor name)
                    ,@(map (lambda (f)
                            ...)
                           fields))))))

We could imagine changing the compiler to be able to serialize records
in the future, but I think that’s a longer-term approach, and not
directly relevant to this report.

WDYT?

Ludo’.

bug#11198: problems reading data with a "read-hash-extend" registered reader

[Andy Wingo]

On Sun 22 Apr 2012 20:01, ludo@gnu.org (Ludovic Courtès) writes:

> I think the reader should only return valid Scheme objects that have a
> read syntax (info "(r5rs) Lexical Structure"), and records are not among
> them.

I agree, FWIW.

> We could imagine changing the compiler to be able to serialize records
> in the future, but I think that’s a longer-term approach, and not
> directly relevant to this report.

Agreed, and this is a tricky area AFAIU -- see Racket's documentation on
prefab structures:

  http://docs.racket-lang.org/reference/structures.html?q=record&q=structs&q=records#(tech._prefab)

Regards,

Andy
-- 
http://wingolog.org/

bug#11198: problems reading data with a "read-hash-extend" registered reader

[Noah Lavine]

Hello,

>> I think the reader should only return valid Scheme objects that have a
>> read syntax (info "(r5rs) Lexical Structure"), and records are not among
>> them.
>
> I agree, FWIW.

This seems like circular logic to me - extending the reader should
mean that new types can have read syntax. The problem here, I think,
is that the compiler also needs to know how to serialize those types.

Why don't we provide an interface to define a serializer as well as a
reader, and have compile-assembly use these serializers? As long as
each type has both a serializer and a reader, it should work fine.
That also feels symmetrical, which I take as a good sign. :-)

Noah

bug#11198: problems reading data with a "read-hash-extend" registered reader

[Noah Lavine]

And just a quick note - this problem is more general than record
types. If someone added a type through C and wanted it to have a read
syntax, the same issue would apply. I can imagine times when this
would be very useful - for instance, you have a C extension to Guile,
and you want to be able to send data between your different Guile
instances in a way that is easy to manually inspect. You need a reader
and a serializer for your C things.

Noah

On Tue, Apr 24, 2012 at 7:24 AM, Noah Lavine <noah.b.lavine@gmail.com> wrote:
> Hello,
>
>>> I think the reader should only return valid Scheme objects that have a
>>> read syntax (info "(r5rs) Lexical Structure"), and records are not among
>>> them.
>>
>> I agree, FWIW.
>
> This seems like circular logic to me - extending the reader should
> mean that new types can have read syntax. The problem here, I think,
> is that the compiler also needs to know how to serialize those types.
>
> Why don't we provide an interface to define a serializer as well as a
> reader, and have compile-assembly use these serializers? As long as
> each type has both a serializer and a reader, it should work fine.
> That also feels symmetrical, which I take as a good sign. :-)
>
> Noah

bug#11198: problems reading data with a "read-hash-extend" registered reader

[Ludovic Courtès]

Hi Noah,

Noah Lavine <noah.b.lavine@gmail.com> skribis:

> Why don't we provide an interface to define a serializer as well as a
> reader, and have compile-assembly use these serializers? As long as
> each type has both a serializer and a reader, it should work fine.
> That also feels symmetrical, which I take as a good sign. :-)

Yes, sounds like a good idea.

However, this discussion should be moved to guile-devel@gnu.org IMO, as
it’s not going to address the bug report at hand in a timely fashion.

Thanks,
Ludo’.

bug#11198: prefab structs in guile

[Andy Wingo]

Hi,

We should think about supporting prefab structs in Guile.  For more
details, see:

  http://docs.racket-lang.org/guide/define-struct.html?q=prefab#%28part._prefab-struct%29
  http://docs.racket-lang.org/reference/structures.html?q=record&q=structs&q=records#(tech._prefab)

The reason is that sometimes you want to allow structures to be
serialized (to Scheme or into object code) and read back in.  See bug
11198 for an example.

I don't have time for this currently, but here's one plan on how you
would do it.

  You would create a new Scheme module, (ice-9 prefab) or
something.  It would contain a map from (name, number of fields) ->
record type descriptor.  (We don't have #:mutable, #:auto, or
supertypes; these would be separate projects.)

  Then you would modify the reader to call out to (ice-9 prefab) with
the list after #s, e.g. the (foo ...) in #s(foo ...).  (ice-9 prefab)
would return the record, creating the RTD if needed.

  You also modify the printer.

  You also modify SRFI-9's define-record-type to recognize #:prefab, or
some other name.

  You also add code to the glil->assembly compiler to recognize prefab
structs, and you add an opcode to look up a prefab struct RTD.

I think that would be it.

Anyone interested?

Andy
-- 
http://wingolog.org/

bug#11198: prefab structs in guile

[Ludovic Courtès]

Hello,

Andy Wingo <wingo@pobox.com> skribis:

>   Then you would modify the reader to call out to (ice-9 prefab) with
> the list after #s, e.g. the (foo ...) in #s(foo ...).  (ice-9 prefab)
> would return the record, creating the RTD if needed.

The problem with this is that one could precisely forge instances of a
given record type, thereby breaking the type safety we currently have
(each instance of a record type is genuine, in the sense of Rees’ “A
Security kernel Based on the Lambda-Calculus”.)

Does Racket address this somehow?

Thanks,
Ludo’.

bug#11198: prefab structs in guile

[Andy Wingo]

On Thu 05 Jul 2012 22:57, ludo@gnu.org (Ludovic Courtès) writes:

> Andy Wingo <wingo@pobox.com> skribis:
>
>>   Then you would modify the reader to call out to (ice-9 prefab) with
>> the list after #s, e.g. the (foo ...) in #s(foo ...).  (ice-9 prefab)
>> would return the record, creating the RTD if needed.
>
> The problem with this is that one could precisely forge instances of a
> given record type, thereby breaking the type safety we currently have
> (each instance of a record type is genuine, in the sense of Rees’ “A
> Security kernel Based on the Lambda-Calculus”.)
>
> Does Racket address this somehow?

See:

  http://docs.racket-lang.org/guide/define-struct.html?q=record&q=structs&q=records#(part._prefab-struct)

Specifically:

  Every prefab structure type is transparent—but even less abstract than
  a transparent type, because instances can be created without any
  access to a particular structure-type declaration or existing
  examples. Overall, the different options for structure types offer a
  spectrum of possibilities from more abstract to more convenient:

    Opaque (the default) : Instances cannot be inspected or forged without
    access to the structure-type declaration. As discussed in the next
    section, constructor guards and properties can be attached to the
    structure type to further protect or to specialize the behavior of its
    instances.

    Transparent : Anyone can inspect or create an instance without access
    to the structure-type declaration, which means that the value printer
    can show the content of an instance. All instance creation passes
    through a constructor guard, however, so that the content of an
    instance can be controlled, and the behavior of instances can be
    specialized through properties. Since the structure type is generated
    by its definition, instances cannot be manufactured simply through the
    name of the structure type, and therefore cannot be generated
    automatically by the expression reader.

    Prefab : Anyone can inspect or create an instance at any time, without
    prior access to a structure-type declaration or an example
    instance. Consequently, the expression reader can manufacture
    instances directly. The instance cannot have a constructor guard or
    properties.

  Since the expression reader can generate prefab instances, they are
  useful when convenient serialization is more important than
  abstraction. Opaque and transparent structures also can be serialized,
  however, if they are defined with define-serializable-struct as
  described in Datatypes and Serialization.

Andy
-- 
http://wingolog.org/

bug#11198: prefab structs in guile

[Ludovic Courtès]

Hi,

Andy Wingo <wingo@pobox.com> skribis:

>   Since the expression reader can generate prefab instances, they are
>   useful when convenient serialization is more important than
>   abstraction. Opaque and transparent structures also can be serialized,
>   however, if they are defined with define-serializable-struct as
>   described in Datatypes and Serialization.

So I’d be in the ‘define-serializable-struct’ camp, so to speak.

Prefabs raise an number of interesting issues.  For instance, what’s the
meaning of #s(sprout bean #f 17) in a module where ‘sprout’ is unbound?
In a module where it’s bound to a given RTD vs. in a module where it’s
bound to a different RTD?  Does ‘read’ have to be current-module-aware?
Etc.

This example is a bit scary to me:

    > (define lunch '#s(sprout bean))
                                     
    > (struct sprout (kind) #:prefab)
                                     
    > (sprout? lunch)                
    #t                               

since it implies that types are compared by name.

Thanks,
Ludo’.

bug#11198: prefab structs in guile

[Andy Wingo]

On Thu 05 Jul 2012 23:55, ludo@gnu.org (Ludovic Courtès) writes:

> So I’d be in the ‘define-serializable-struct’ camp, so to speak.

That's a valid position to have in general.  I can also imagine cases in
which you would choose other things.  It's a spectrum.

> Prefabs raise an number of interesting issues.  For instance, what’s the
> meaning of #s(sprout bean #f 17) in a module where ‘sprout’ is unbound?

Prefab structs are not modular.  It is the same as in a module where
`sprout' is bound.  Reading #s(sprout bean #f 17) may create an RTD if
needed, but it does not create any bindings.

> types are compared by name.

As the documentation clearly indicates :), prefab structs are indeed
compared by name, though in a combination with other characteristics
(number of fields, and more characteristics for racket).

Andy
-- 
http://wingolog.org/

bug#11198: prefab structs in guile

[Andy Wingo]

On Thu 05 Jul 2012 23:55, ludo@gnu.org (Ludovic Courtès) writes:

> Andy Wingo <wingo@pobox.com> skribis:
>
>>   Since the expression reader can generate prefab instances, they are
>>   useful when convenient serialization is more important than
>>   abstraction. Opaque and transparent structures also can be serialized,
>>   however, if they are defined with define-serializable-struct as
>>   described in Datatypes and Serialization.
>
> So I’d be in the ‘define-serializable-struct’ camp, so to speak.

To be clear: in Racket, serializable structs are not readable.  That's
the whole point of prefab structs: a struct that is readable.

Andy
-- 
http://wingolog.org/

bug#11198: prefab structs in guile

[Ludovic Courtès]

Hi,

Andy Wingo <wingo@pobox.com> skribis:

> On Thu 05 Jul 2012 23:55, ludo@gnu.org (Ludovic Courtès) writes:
>
>> So I’d be in the ‘define-serializable-struct’ camp, so to speak.
>
> That's a valid position to have in general.  I can also imagine cases in
> which you would choose other things.  It's a spectrum.

Yes, sure.  It can be convenient to have something that makes it easy to
serialize structs.  But given that prefabs have to be “built in”, and
that they look can-of-wormey to my demanding eye ;-), I’d be in favor of
anything built atop structs.

And actually, there’s (oop goops save), which is extensible and
everything.  :-)

Side note: while writing readers/writers by hand may look tedious, the
advantage is that it makes it easy to leave room for future extensions,
like allowing the reader to suitably map an old version of a serialized
struct to the new data structure.

>> Prefabs raise an number of interesting issues.  For instance, what’s the
>> meaning of #s(sprout bean #f 17) in a module where ‘sprout’ is unbound?
>
> Prefab structs are not modular.  It is the same as in a module where
> `sprout' is bound.  Reading #s(sprout bean #f 17) may create an RTD if
> needed, but it does not create any bindings.

OK.  So there can only be one ‘sprout’ prefab RTD in the whole process,
I guess.

>> types are compared by name.
>
> As the documentation clearly indicates :), prefab structs are indeed
> compared by name, though in a combination with other characteristics
> (number of fields, and more characteristics for racket).

Right.  Argh.

Thanks,
Ludo’.

bug#11198: prefab structs in guile

[Ludovic Courtès]

retitle 11198 serializing structs (prefabs)
tags 11198 wishlist
thanks