to serialize/deserialize closures; and multithreading

to serialize/deserialize closures; and multithreading

[Nicholas Paul Johnson]

Hello all,

I am writing a program in C which will make heavy use of the guile 
scheme interpreter, and want to use the SCM_ interface as opposed to the 
GH_ interface.  two questions:

1. If I have a scheme value that is a closure, is there any way that I can
serialize this closure (from C code) into a form that it can be
deserialized back into a SCM closure variable (again, by C code)?  I can
assume for this question that both ends of the serial line are running the
same version of gnu guile and my software, but cannot assume that both
ends are running on the same computer architecture.  Would it instead
only be possible for me to transfer closures as their scheme source.

2. Is it possible to have disjoint instances of the scheme interpreter 
running in the same process but in different threads?  I.E. If my C 
program was multithreaded, could more than one of the threads create a 
guile/scheme interpretter?  If so, would I have to use the initialization 
functions more than once.

Also, does anyone know of a good documentation of the SCM_ API?  I've not 
been able to find one.

-- 
Nicholas Paul Johnson
nickjohnsonSPAM^H^H^H^H@virginia.edu
http://manjac.ath.cx/nick
 _
( ) ascii ribbon campaign - against html mail 
 X                        - against microsoft attachments
/ \ http://www.google.com/search?q=ascii+ribbon
--




_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: to serialize/deserialize closures; and multithreading

[Paul Jarc]

Nicholas Paul Johnson <nickjohnson@virginia.edu> wrote:
> 1. If I have a scheme value that is a closure, is there any way that I can
> serialize this closure (from C code) into a form that it can be
> deserialized back into a SCM closure variable (again, by C code)?

Well, there's (procedure-source foo), which gets you a list
(lambda ...) which can be passed to (write) and recovered with
(read).  But the closure may also have captured some lexical bindings,
which won't survive this translation.  (And of course, anything that
can be done from Scheme can also be done from C, though the details
may vary depending on the procedure/function in question.)

> Also, does anyone know of a good documentation of the SCM_ API?  I've not
> been able to find one.

Relative to the manual released with 1.6.4, there have been many
additions in CVS.  Some of the changes are due to code changes, but
I'd expect most of it would hold for 1.6 as well.


paul


_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: to serialize/deserialize closures; and multithreading

[Greg Troxel]

  1. If I have a scheme value that is a closure, is there any way that I can
  serialize this closure (from C code) into a form that it can be
  deserialized back into a SCM closure variable (again, by C code)?  I can
  assume for this question that both ends of the serial line are running the
  same version of gnu guile and my software, but cannot assume that both
  ends are running on the same computer architecture.  Would it instead
  only be possible for me to transfer closures as their scheme source.

I'm not even sure it is well-defined what it means to migrate a
closure.  Should a mutation on the new computer of a captured variable
affect the old computer?

-- 
        Greg Troxel <gdt@ir.bbn.com>


_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: to serialize/deserialize closures; and multithreading

[Lynn Winebarger]

It's an interesting question all right.  What's the purpose of the
"migration"?
    Technically speaking, a closure is just a pair of pointers to its code and
its environment.  You could create a new "foreign reference" type (tag) that
would cause these to be looked up on the original computer whenever you needed
them.  That would handle synchronization.  Of course, you could also make copies
of everything.  I think tags are constant over all architechtures (last I looked)
but there are run-time allocated types that you'd have to do some work for
(I think - I don't remember what they're called).
    Anyway, besides taking care of the tags/synchronizing run-time allocated types/
handling endianness issues, you'd need to munge pointers according to whereever
they would be newly allocated on the new machine.  It would probably be easiest
to just number copies of the cells from 0 to whatever for the transport over and the
allocator on the other end could relabel them to the actual values.
    Also, you'd need to check that any symbols referenced by the code or variables were
actually in existence in the new guile instance's symbol table.  And that global
variables referenced by the code got copied (as well as the lexical variables).
So will be recursively nasty.

Good Luck.

Lynn

Greg Troxel wrote:
>   1. If I have a scheme value that is a closure, is there any way that I can
>   serialize this closure (from C code) into a form that it can be
>   deserialized back into a SCM closure variable (again, by C code)?  I can
>   assume for this question that both ends of the serial line are running the
>   same version of gnu guile and my software, but cannot assume that both
>   ends are running on the same computer architecture.  Would it instead
>   only be possible for me to transfer closures as their scheme source.
> 
> I'm not even sure it is well-defined what it means to migrate a
> closure.  Should a mutation on the new computer of a captured variable
> affect the old computer?
> 




_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: to serialize/deserialize closures; and multithreading

[Lynn Winebarger]

On the other hand, you could just rely on guile's printer/reader
syntax and use that on the environment to export it and the
global variables.  I don't know if it prints everything in a
uniquely readable way, but it should handle standard scheme
that way.
    And environments in guile (last time I looked at the
evaluator) really are just binding lists. With deBruijn indices.
I can't remember exactly.  Use the source.

Lynn

Lynn Winebarger wrote:
> It's an interesting question all right.  What's the purpose of the
> "migration"?
>    Technically speaking, a closure is just a pair of pointers to its 
> code and
> its environment.  You could create a new "foreign reference" type (tag) 
> that
> would cause these to be looked up on the original computer whenever you 
> needed
> them.  That would handle synchronization.  Of course, you could also 
> make copies
> of everything.  I think tags are constant over all architechtures (last 
> I looked)
> but there are run-time allocated types that you'd have to do some work for
> (I think - I don't remember what they're called).
>    Anyway, besides taking care of the tags/synchronizing run-time 
> allocated types/
> handling endianness issues, you'd need to munge pointers according to 
> whereever
> they would be newly allocated on the new machine.  It would probably be 
> easiest
> to just number copies of the cells from 0 to whatever for the transport 
> over and the
> allocator on the other end could relabel them to the actual values.
>    Also, you'd need to check that any symbols referenced by the code or 
> variables were
> actually in existence in the new guile instance's symbol table.  And 
> that global
> variables referenced by the code got copied (as well as the lexical 
> variables).
> So will be recursively nasty.
> 
> Good Luck.
> 
> Lynn
> 
> Greg Troxel wrote:
> 
>>   1. If I have a scheme value that is a closure, is there any way that 
>> I can
>>   serialize this closure (from C code) into a form that it can be
>>   deserialized back into a SCM closure variable (again, by C code)?  I 
>> can
>>   assume for this question that both ends of the serial line are 
>> running the
>>   same version of gnu guile and my software, but cannot assume that both
>>   ends are running on the same computer architecture.  Would it instead
>>   only be possible for me to transfer closures as their scheme source.
>>
>> I'm not even sure it is well-defined what it means to migrate a
>> closure.  Should a mutation on the new computer of a captured variable
>> affect the old computer?
>>
> 
> 
> 
> 
> _______________________________________________
> Guile-user mailing list
> Guile-user@gnu.org
> http://mail.gnu.org/mailman/listinfo/guile-user
> 
> 
> 




_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: to serialize/deserialize closures; and multithreading

[Paul Jarc]

Lynn Winebarger <owinebar@indiana.edu> wrote:
> there are run-time allocated types that you'd have to do some work for
> (I think - I don't remember what they're called).

Smobs are like that.  Maybe GOOPS classes too; I don't know.


paul


_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: to serialize/deserialize closures; and multithreading

[Faraz Shahbazker]

[-- Attachment #1.1: Type: text/html, Size: 2421 bytes --]

[-- Attachment #1.2: Type: text/plain, Size: 1857 bytes --]

nick,

I am curious too : what is your purpose of migration?

        I am working on something similar, for using scheme on
Beowulf-clusters with PVM. Here the purpose of migration is to
remotely-evaluate a sexp. But possible uses of this technology
could be to simply suspend/restart programs(see chpox), or for
delivering active web-content.

> Should a mutation on the new computer of a captured variable
> affect the old computer? - greg

        Basically, if your program is purely/mostly functional,
it might be easy to migrate, while maintaining referential
transparency - just byte-copy(with tags) all   data-types.
The semantics of mutable objects (when mutation does occur)
is for you to decide. It might make greater sense for example
for exceptions to be migrated and  handled remotely, than
say mutexes or ports.


>  Also, you'd need to check that any symbols referenced by
> the code or variables were actually in existence in the
> new guile instance's symbol table.  And that global
> variables referenced by the code got copied (as well as
> the lexical variables).
> So will be recursively nasty.

        I agree with Lynn. Guile's own procedures for
detecting free-variables in expressions may be inadequate for
this.  There is a technique called lambda-lifting meant for
converting sexps into pure combinators(a.k.a thunks). See
Jonhsson's paper on lambda-lifting. It was meant to be used
for compiler optimizations for ML. It was also used in the
Larceny Scheme compiler. But since you are not yet conversant
with SCM_API, I don't know....

        printer/reader is an elegant(read slow ;) solution for
plain Scheme code, but it obviously won't work for custom smobs,
which I am assuming that you might want to create(since you are
using C). 

So tell me more about it!!

- faraz 



[-- Attachment #2: Type: text/plain, Size: 139 bytes --]

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: to serialize/deserialize closures; and multithreading

[rm]

On Thu, Mar 25, 2004 at 10:38:00PM -0000, Faraz Shahbazker wrote:
> nick,
> 
> I am curious too : what is your purpose of migration?
> 
>         I am working on something similar, for using scheme on
> Beowulf-clusters with PVM. Here the purpose of migration is to
> remotely-evaluate a sexp. But possible uses of this technology
> could be to simply suspend/restart programs(see chpox), or for

Just out of _my_ curiosity: are you aware of 'kali' -- a reborn
MIT scheme that supports distributed computing?
For yet  another 'transport' layer for S-Expresive Data have 
a look at FramerD (former MIT project, now at sourceforge,
<http://www.framerd.org> -- erm, some basic Guile bindings for
that can be found at http://cvs.zeit.de (beware: highly experimental
code with bitrot!).
> 
> > Should a mutation on the new computer of a captured variable
> > affect the old computer? - greg
> 
>         Basically, if your program is purely/mostly functional,
> it might be easy to migrate, while maintaining referential
> transparency - just byte-copy(with tags) all   data-types.
> The semantics of mutable objects (when mutation does occur)
> is for you to decide. It might make greater sense for example
> for exceptions to be migrated and  handled remotely, than
> say mutexes or ports.

Autsch, that does get rather expensive when closures/continuation
are used (something that does happen occasionally in functional
programming ;-).


just my 0.02$, 

 Ralf Mattes


_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: to serialize/deserialize closures; and multithreading

[Nicholas Paul Johnson]

Wow,

This is almost exactly what I was looking for... thanks.

-- 
Nicholas Paul Johnson
nickjohnsonSPAM^H^H^H^H@virginia.edu
http://manjac.ath.cx/nick
 _
( ) ascii ribbon campaign - against html mail 
 X                        - against microsoft attachments
/ \ http://www.google.com/search?q=ascii+ribbon
--


On Fri, 26 Mar 2004 rm@fabula.de wrote:

> On Thu, Mar 25, 2004 at 10:38:00PM -0000, Faraz Shahbazker wrote:
> > nick,
> > 
> > I am curious too : what is your purpose of migration?
> > 
> >         I am working on something similar, for using scheme on
> > Beowulf-clusters with PVM. Here the purpose of migration is to
> > remotely-evaluate a sexp. But possible uses of this technology
> > could be to simply suspend/restart programs(see chpox), or for
> 
> Just out of _my_ curiosity: are you aware of 'kali' -- a reborn
> MIT scheme that supports distributed computing?
> For yet  another 'transport' layer for S-Expresive Data have 
> a look at FramerD (former MIT project, now at sourceforge,
> <http://www.framerd.org> -- erm, some basic Guile bindings for
> that can be found at http://cvs.zeit.de (beware: highly experimental
> code with bitrot!).
> > 
> > > Should a mutation on the new computer of a captured variable
> > > affect the old computer? - greg
> > 
> >         Basically, if your program is purely/mostly functional,
> > it might be easy to migrate, while maintaining referential
> > transparency - just byte-copy(with tags) all   data-types.
> > The semantics of mutable objects (when mutation does occur)
> > is for you to decide. It might make greater sense for example
> > for exceptions to be migrated and  handled remotely, than
> > say mutexes or ports.
> 
> Autsch, that does get rather expensive when closures/continuation
> are used (something that does happen occasionally in functional
> programming ;-).
> 
> 
> just my 0.02$, 
> 
>  Ralf Mattes
> 
> 



_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: Re: to serialize/deserialize closures; and multithreading

[Faraz Shahbazker]

[-- Attachment #1.1: Type: text/html, Size: 2934 bytes --]

[-- Attachment #1.2: Type: text/plain, Size: 2184 bytes --]


        The problem with Kali and other distributed Scheme
implementations (and there are quite a few) is that most of
them are built around the imperative style of parallel
computation -- result: things get ugly quickly for complex
programs. You could end-up writing C-like programs in
Scheme.

        IMHO the ideal system would be built around pure
functional style (using MultiLisp type parallel annotations
like future/touch/pcall). This category is also known as
para-functional languages. It has been explored reasonably in
ML/Haskell, but little work of this nature has been done in
Scheme/Lisp so far.....


> Autsch, that does get rather expensive when closures/continuation
> are used (something that does happen occasionally in functional
> programming ;-)

        That I accept. It can be speeded up by building the
transformation into the compiler, something which is not possible
with guile - bigloo perhaps would be a better choice for such work.
But it is difficult to leave guile for 2 reasons :

1. it's got the "GNU" in it's name
2. SCM API is too good - absolutely love it. I don't think anyone
can provide a better extensibility for Scheme than what we have here.



As an aside : for functional purists like me, we should built
future/touch constructs into guile - they are a lot of fun.
Try out these prototypes =>
(Note : these are not actually correct - the simple semantics fail
in the presence of mutations/continuations. A proper implementation
would have to be done in C using Guile-internals)

        It basically works like delay/force except that any-thing
that is "future"d returns a place-holder immediately and starts
concurrent computation. When the value is required, you "touch"
the place-holder, causing a block until the computation completes.


(defmacro future (expr)
  `(let ((ans (cons "#<future>" (cons #f #f))))
     (set-car! (cdr ans)
               (begin-thread
                (set-cdr! (cdr ans) ((lambda () ,expr)))))
     ans))

(defmacro touch (future-val)
  `(begin
     (join-thread (car (cdr ,future-val)))
     (cdr (cdr ,future-val))))


regards,
- faraz 



[-- Attachment #2: Type: text/plain, Size: 139 bytes --]

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: to serialize/deserialize closures; and multithreading

[Andreas Rottmann]

"Faraz Shahbazker" <faraz_ms@rediffmail.com> writes:

>         That I accept. It can be speeded up by building the
> transformation into the compiler, something which is not possible
> with guile - bigloo perhaps would be a better choice for such work.
> But it is difficult to leave guile for 2 reasons :
>
> 1. it's got the "GNU" in it's name
> 2. SCM API is too good - absolutely love it. I don't think anyone
> can provide a better extensibility for Scheme than what we have here.
>
Have a look at Pika's API - it's cleaner (IMHO) and more general, due
to passing only location of SCM values (which allows easier/better GC)
and passing an 'arena' (roughly a interpreter instance), which allows
for more than one Scheme "subsystem" in one program; each such
subsystem can have a different memory pool size, independent GC,
... OTOH, this tends to make the C code more verbose. Example:

t_scm_error
scm_ref_list_elt (t_scm_word * result, t_scm_arena arena, t_scm_word * list,
                  ssize_t elt)
{
  struct ref_locals
    {
      SCM_FRAME;
      t_scm_word tail;
    } l;
  t_scm_error status = 0;

  SCM_PROTECT_FRAME (l);

  status = scm_list_tail (&l.tail, arena, list, elt);
  if (status == 0)
    {
      scm_ref_pair_car (result, arena, &l.tail);
    }
   
  SCM_UNPROTECT_FRAME (l);
   
  return status;
}

Cheers, Andy
-- 
Andreas Rottmann         | Rotty@ICQ      | 118634484@ICQ | a.rottmann@gmx.at
http://yi.org/rotty      | GnuPG Key: http://yi.org/rotty/gpg.asc
Fingerprint              | DFB4 4EB4 78A4 5EEE 6219  F228 F92F CFC5 01FD 5B62

Make free software, not war!



_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user