Bytestructures, FFI

Bytestructures, FFI

[Taylan Ulrich Bayırlı/Kammer]

Hi there,

I had announced this project a while ago, here:

  https://lists.gnu.org/archive/html/guile-user/2015-08/msg00034.html

I just added support for converting a bytestructure descriptor into a
type descriptor as per Guile's FFI module.

E.g. the uint8 descriptor in (bytestructures guile numeric) converts to
the value of the uint8 variable in the (system foreign) module, and a
struct descriptor such as (bs:struct `((x ,uint8) (y ,uint16))) converts
to what one would get from evaluating (list uint8 uint16) after
importing (system foreign).  (Such lists are accepted by Guile's FFI to
denote structs; it's yet undocumented in the manual.)

For convenience, a bs:pointer->proc procedure is offered, that wraps
pointer->procedure and adds bytestructure related functionality.  Here's
a minimal example of how to use it:


--- start transcript ---

taylan@T420:~/src/scheme/bytestructures$ cat test.c
typedef struct {
  struct {
    int x;
    float y;
  } s;
  int z;
  float t[3];
} foo_t;

foo_t testfunc(foo_t arg)
{
  ++arg.s.x;
  ++arg.s.y;
  ++arg.z;
  ++arg.t[0];
  ++arg.t[1];
  ++arg.t[2];
  return arg;
}
taylan@T420:~/src/scheme/bytestructures$ gcc -shared -o libtest.so test.c
taylan@T420:~/src/scheme/bytestructures$ sudo mv libtest.so /usr/lib/
taylan@T420:~/src/scheme/bytestructures$ guile
GNU Guile 2.0.11
Copyright (C) 1995-2014 Free Software Foundation, Inc.

Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.

Enter `,help' for help.
scheme@(guile-user)> (import (bytestructures guile))
scheme@(guile-user)> (import (bytestructures guile ffi))
scheme@(guile-user)> (define foo_t (bs:struct `((s ,(bs:struct `((x ,int)
                                                                 (y ,float32))))
                                                (z ,int)
                                                (t ,(bs:vector 3 float32)))))
scheme@(guile-user)> (define foo (bytestructure foo_t '((s ((x 1)
                                                            (y 2.2)))
                                                        (z 3)
                                                        (t #(0 1 2)))))
scheme@(guile-user)> (bytestructure-ref foo 's 'x)
$1 = 1
scheme@(guile-user)> (bytestructure-ref foo 't 1)
$2 = 1.0
scheme@(guile-user)> (define testfunc
                       (bs:pointer->proc
                        foo_t  ;return value type
                        (dynamic-func "testfunc" (dynamic-link "libtest"))
                        (list foo_t)))  ;list of argument types
scheme@(guile-user)> (define foo2 (testfunc foo))
scheme@(guile-user)> (bytestructure-ref foo2 's 'x)
$3 = 2
scheme@(guile-user)> (bytestructure-ref foo2 't 1)
$4 = 2.0
scheme@(guile-user)>

--- end transcript ---


To recap:

We have a C file which defines a struct, and a function that
takes such a struct and returns a new one with some of the fields'
values modified.  We make this a shared library.

In Guile, we can express the same struct layout via a bytestructure
descriptor object, which we put in the variable foo_t.  We can also
instantiate such a struct in memory, reified in Scheme as a
bytestructure object, which we put into the variable foo.

We use bs:pointer->proc to create a Scheme procedure that wraps the
function in our shared library; bs:pointer->proc allows us to use the
foo_t descriptor in the return value and argument list declarations.

We can now call this procedure with the struct we had created from
Scheme and put in the variable foo as a bytestructure object.  The new
struct returned by the C function is automatically turned into a
bytestructure object that can be used from Scheme with the
bytestructures API.

Neat, isn't it?


If test.c instead contained:

void testfunc(foo_t *arg)
{
  ++arg->s.x;
  ++arg->s.y;
  ++arg->z;
  ++arg->t[0];
  ++arg->t[1];
  ++arg->t[2];
}

(i.e. testfunc now takes a struct pointer and modifies it in-place)

then we could create our procedure like:

scheme@(guile-user)> (import (prefix (system foreign) ffi:))
scheme@(guile-user)> (define testfunc
                       (bs:pointer->proc
                        ffi:void
                        (dynamic-func "testfunc" (dynamic-link "libtest"))
                        (list '*)))

and things would work like this:

scheme@(guile-user)> (testfunc foo)
scheme@(guile-user)> (bytestructure-ref foo 's 'x)
$3 = 2
scheme@(guile-user)> (bytestructure-ref foo 't 1)
$4 = 2.0

Magic!


Note that while bytestructures supports unions and bit-fields, libffi
and therefore (system foreign) does not, so the above won't work yet for
types in which unions or bitfields are involved.  Unions can possibly be
supported with a little hack, which I'll look into later.

I have not yet made the semantics of bs:pointer->proc concrete, and it's
yet undocumented.  Feel free to ask me questions here or on Freenode
under the nick 'taylan'.  Comments are welcome too!

Taylan

Re: Bytestructures, FFI

[Amirouche Boubekki]

On 2016-05-31 21:29, taylanbayirli@gmail.com wrote:
> Hi there,
> 
> I had announced this project a while ago, here:
> 
>   https://lists.gnu.org/archive/html/guile-user/2015-08/msg00034.html
> 
> I just added support for converting a bytestructure descriptor into a
> type descriptor as per Guile's FFI module.
> 
> E.g. the uint8 descriptor in (bytestructures guile numeric) converts to
> the value of the uint8 variable in the (system foreign) module, and a
> struct descriptor such as (bs:struct `((x ,uint8) (y ,uint16))) 
> converts
> to what one would get from evaluating (list uint8 uint16) after
> importing (system foreign).  (Such lists are accepted by Guile's FFI to
> denote structs; it's yet undocumented in the manual.)
> 
> For convenience, a bs:pointer->proc procedure is offered, that wraps
> pointer->procedure and adds bytestructure related functionality.  
> Here's
> a minimal example of how to use it:
> 
> 
> --- start transcript ---
> 
> taylan@T420:~/src/scheme/bytestructures$ cat test.c
> typedef struct {
>   struct {
>     int x;
>     float y;
>   } s;
>   int z;
>   float t[3];
> } foo_t;
> 
> foo_t testfunc(foo_t arg)
> {
>   ++arg.s.x;
>   ++arg.s.y;
>   ++arg.z;
>   ++arg.t[0];
>   ++arg.t[1];
>   ++arg.t[2];
>   return arg;
> }
> taylan@T420:~/src/scheme/bytestructures$ gcc -shared -o libtest.so 
> test.c
> taylan@T420:~/src/scheme/bytestructures$ sudo mv libtest.so /usr/lib/
> taylan@T420:~/src/scheme/bytestructures$ guile
> GNU Guile 2.0.11
> Copyright (C) 1995-2014 Free Software Foundation, Inc.
> 
> Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
> This program is free software, and you are welcome to redistribute it
> under certain conditions; type `,show c' for details.
> 
> Enter `,help' for help.
> scheme@(guile-user)> (import (bytestructures guile))
> scheme@(guile-user)> (import (bytestructures guile ffi))
> scheme@(guile-user)> (define foo_t (bs:struct `((s ,(bs:struct `((x 
> ,int)
>                                                                  (y 
> ,float32))))
>                                                 (z ,int)
>                                                 (t ,(bs:vector 3 
> float32)))))

IMO This is a better example:

  scheme@(guile-user)> (define contact_t (bs:struct `((address 
,(bs:struct `((street_id ,int)
                                                                          
    (number ,int))))
                                                      (age ,int))))

Or something

> scheme@(guile-user)> (define foo (bytestructure address_t '((address 
> ((street 42)
>                                                                 (number 
> 12345)))
>                                                             (age 66)))

> scheme@(guile-user)> (bytestructure-ref address 'address 'street)
> $1 = 42
> scheme@(guile-user)> (bytestructure-ref address 'age)
> $2 = 66


> scheme@(guile-user)> (define testfunc
>                        (bs:pointer->proc
>                         foo_t  ;return value type
>                         (dynamic-func "testfunc" (dynamic-link 
> "libtest"))
>                         (list foo_t)))  ;list of argument types

Why don't you use dynamic-link* signature like described in [1]:

   ((dynamic-link* [library-name]) return-type function-name . arguments)

[1] http://www.hyperdev.fr/notes/gnu-guile-ffi.html

> scheme@(guile-user)> (define foo2 (testfunc foo))

isn't foo2 `equal?' to foo?

> 
> Neat, isn't it?
> 

Awesome!

> 
> If test.c instead contained:
> 
> void testfunc(foo_t *arg)
> {
>   ++arg->s.x;
>   ++arg->s.y;
>   ++arg->z;
>   ++arg->t[0];
>   ++arg->t[1];
>   ++arg->t[2];
> }
> 
> (i.e. testfunc now takes a struct pointer and modifies it in-place)
> 
> then we could create our procedure like:
> 
> scheme@(guile-user)> (import (prefix (system foreign) ffi:))
> scheme@(guile-user)> (define testfunc
>                        (bs:pointer->proc
>                         ffi:void
>                         (dynamic-func "testfunc" (dynamic-link 
> "libtest"))
>                         (list '*)))
> 
> and things would work like this:
> 
> scheme@(guile-user)> (testfunc foo)
> scheme@(guile-user)> (bytestructure-ref foo 's 'x)
> $3 = 2
> scheme@(guile-user)> (bytestructure-ref foo 't 1)
> $4 = 2.0
> 
> Magic!
> 
> 
> Note that while bytestructures supports unions and bit-fields, libffi
> and therefore (system foreign) does not, so the above won't work yet 
> for
> types in which unions or bitfields are involved.  Unions can possibly 
> be
> supported with a little hack, which I'll look into later.
> 
> I have not yet made the semantics of bs:pointer->proc concrete, and 
> it's
> yet undocumented.  Feel free to ask me questions here or on Freenode
> under the nick 'taylan'.  Comments are welcome too!
> 
> Taylan

-- 
Amirouche ~ amz3 ~ http://www.hyperdev.fr

Re: Bytestructures, FFI

[Taylan Ulrich Bayırlı/Kammer]

Amirouche Boubekki <amirouche@hypermove.net> writes:

> On 2016-05-31 21:29, taylanbayirli@gmail.com wrote:
>> scheme@(guile-user)> (define testfunc
>>                        (bs:pointer->proc
>>                         foo_t  ;return value type
>>                         (dynamic-func "testfunc" (dynamic-link
>> "libtest"))
>>                         (list foo_t)))  ;list of argument types
>
> Why don't you use dynamic-link* signature like described in [1]:
>
>   ((dynamic-link* [library-name]) return-type function-name . arguments)
>
> [1] http://www.hyperdev.fr/notes/gnu-guile-ffi.html

Hmm, if one wishes, one can wrap bs:pointer->proc in a similar fashion;
I'll leave that up to the users of the library. :-)

>> scheme@(guile-user)> (define foo2 (testfunc foo))
>
> isn't foo2 `equal?' to foo?

I'm not sure how you mean exactly.  Firstly, foo and foo2 are separate
record instances, so they wouldn't be equal? even if they contained the
same values in the same fields (using equal? on record objects falls
back to using eqv?), but they also contain different values in their
fields, since foo2 is the one returned by 'testfunc'.  (Also remember
that structs are passed by-value in C, so testfunc receives a *copy* of
foo as its argument when it's called, which it then modifies and
returns.)

Hope that helps!
Taylan

Re: Bytestructures, FFI

[Ludovic Courtès]

Hi!

taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:

> scheme@(guile-user)> (import (bytestructures guile))
> scheme@(guile-user)> (import (bytestructures guile ffi))
> scheme@(guile-user)> (define foo_t (bs:struct `((s ,(bs:struct `((x ,int)
>                                                                  (y ,float32))))
>                                                 (z ,int)
>                                                 (t ,(bs:vector 3 float32)))))
> scheme@(guile-user)> (define foo (bytestructure foo_t '((s ((x 1)
>                                                             (y 2.2)))
>                                                         (z 3)
>                                                         (t #(0 1 2)))))

Gradually, without thinking much about it ;-), I ended up writing
something along these lines in (guix build syscalls), but with slightly
different goals.

Initially, that was because I was unsatisfied with ‘make-c-struct’ and
‘parse-c-struct’, notably the fact that they insisted on creating lists
and computing offsets etc. at run time.

The ‘define-c-struct’ macro in that module defines a variable containing
the size in bytes of the structure, a procedure to write an instance of
that structure to a bytevector, and one to read from a bytevector.
Example:

--8<---------------cut here---------------start------------->8---
(define-c-struct sockaddr-in                      ;<linux/in.h>
  sizeof-sockaddrin
  (lambda (family port address)
    (make-socket-address family address port))
  read-sockaddr-in
  write-sockaddr-in!
  (family    unsigned-short)
  (port      (int16 ~ big))
  (address   (int32 ~ big)))

;; …

(define (write-socket-address! sockaddr bv index)
  "Write SOCKADDR, a socket address as returned by 'make-socket-address', to
bytevector BV at INDEX."
  (let ((family (sockaddr:fam sockaddr)))
    (cond ((= family AF_INET)
           (write-sockaddr-in! bv index
                               family
                               (sockaddr:port sockaddr)
                               (sockaddr:addr sockaddr)))
          ((= family AF_INET6)
           (write-sockaddr-in6! bv index
                                family
                                (sockaddr:port sockaddr)
                                (sockaddr:flowinfo sockaddr)
                                (sockaddr:addr sockaddr)
                                (sockaddr:scopeid sockaddr)))
          (else
           (error "unsupported socket address" sockaddr)))))

(define* (read-socket-address bv #:optional (index 0))
  "Read a socket address from bytevector BV at INDEX."
  (let ((family (bytevector-u16-native-ref bv index)))
    (cond ((= family AF_INET)
           (read-sockaddr-in bv index))
          ((= family AF_INET6)
           (read-sockaddr-in6 bv index))
          (else
           ;; XXX: Unsupported address family, such as AF_PACKET.  Return a
           ;; vector such that the caller can at least use 'sockaddr:fam'.
           (vector family)))))
--8<---------------cut here---------------end--------------->8---

The expanded ‘read-sockaddr-in’ looks like this:

--8<---------------cut here---------------start------------->8---
(define* (read-sockaddr-in bv #:optional (offset 0))
  (let ((family
          (bytevector-uint-ref
            bv
            (#{1+}# (logior (#{1-}# offset) (#{1-}# 2)))
            (native-endianness)
            2))
        (port (bytevector-uint-ref
                bv
                (#{1+}# (logior
                          (#{1-}# (+ (#{1+}# (logior
                                               (#{1-}# offset)
                                               (#{1-}# 2)))
                                     2))
                          (#{1-}# 2)))
                'big
                2))
        (address
          (bytevector-uint-ref
            bv
            (#{1+}# (logior
                      (#{1-}# (+ (#{1+}# (logior
                                           (#{1-}# (+ (#{1+}# (logior
                                                                (#{1-}# offset)
                                                                (#{1-}# 2)))
                                                      2))
                                           (#{1-}# 2)))
                                 2))
                      (#{1-}# 4)))
            'big
            4)))
    (make-socket-address family address port)))
--8<---------------cut here---------------end--------------->8---

No extra allocations and computations.

http://git.savannah.gnu.org/cgit/guix.git/tree/guix/build/syscalls.scm#n264

I’m sure we could build a higher-level interface on top of that, for
instance for when we want a C struct to directly have a corresponding
Scheme record (as is the case of ‘%statfs’ in the file above.)

Ludo’.

Re: Bytestructures, FFI

[Taylan Ulrich Bayırlı/Kammer]

ludo@gnu.org (Ludovic Courtès) writes:

> Hi!
>
> taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:
>
>> scheme@(guile-user)> (import (bytestructures guile))
>> scheme@(guile-user)> (import (bytestructures guile ffi))
>> scheme@(guile-user)> (define foo_t (bs:struct `((s ,(bs:struct `((x ,int)
>>                                                                  (y ,float32))))
>>                                                 (z ,int)
>>                                                 (t ,(bs:vector 3 float32)))))
>> scheme@(guile-user)> (define foo (bytestructure foo_t '((s ((x 1)
>>                                                             (y 2.2)))
>>                                                         (z 3)
>>                                                         (t #(0 1 2)))))
>
> Gradually, without thinking much about it ;-), I ended up writing
> something along these lines in (guix build syscalls), but with slightly
> different goals.
>
> Initially, that was because I was unsatisfied with ‘make-c-struct’ and
> ‘parse-c-struct’, notably the fact that they insisted on creating lists
> and computing offsets etc. at run time.
>
> The ‘define-c-struct’ macro in that module defines a variable containing
> the size in bytes of the structure, a procedure to write an instance of
> that structure to a bytevector, and one to read from a bytevector.
> Example:
>
> (define-c-struct sockaddr-in                      ;<linux/in.h>
>   sizeof-sockaddrin
>   (lambda (family port address)
>     (make-socket-address family address port))
>   read-sockaddr-in
>   write-sockaddr-in!
>   (family    unsigned-short)
>   (port      (int16 ~ big))
>   (address   (int32 ~ big)))
>
> ;; …
>
> (define (write-socket-address! sockaddr bv index)
>   "Write SOCKADDR, a socket address as returned by 'make-socket-address', to
> bytevector BV at INDEX."
>   (let ((family (sockaddr:fam sockaddr)))
>     (cond ((= family AF_INET)
>            (write-sockaddr-in! bv index
>                                family
>                                (sockaddr:port sockaddr)
>                                (sockaddr:addr sockaddr)))
>           ((= family AF_INET6)
>            (write-sockaddr-in6! bv index
>                                 family
>                                 (sockaddr:port sockaddr)
>                                 (sockaddr:flowinfo sockaddr)
>                                 (sockaddr:addr sockaddr)
>                                 (sockaddr:scopeid sockaddr)))
>           (else
>            (error "unsupported socket address" sockaddr)))))
>
> (define* (read-socket-address bv #:optional (index 0))
>   "Read a socket address from bytevector BV at INDEX."
>   (let ((family (bytevector-u16-native-ref bv index)))
>     (cond ((= family AF_INET)
>            (read-sockaddr-in bv index))
>           ((= family AF_INET6)
>            (read-sockaddr-in6 bv index))
>           (else
>            ;; XXX: Unsupported address family, such as AF_PACKET.  Return a
>            ;; vector such that the caller can at least use 'sockaddr:fam'.
>            (vector family)))))
>
> The expanded ‘read-sockaddr-in’ looks like this:
>
> (define* (read-sockaddr-in bv #:optional (offset 0))
>   (let ((family
>           (bytevector-uint-ref
>             bv
>             (#{1+}# (logior (#{1-}# offset) (#{1-}# 2)))
>             (native-endianness)
>             2))
>         (port (bytevector-uint-ref
>                 bv
>                 (#{1+}# (logior
>                           (#{1-}# (+ (#{1+}# (logior
>                                                (#{1-}# offset)
>                                                (#{1-}# 2)))
>                                      2))
>                           (#{1-}# 2)))
>                 'big
>                 2))
>         (address
>           (bytevector-uint-ref
>             bv
>             (#{1+}# (logior
>                       (#{1-}# (+ (#{1+}# (logior
>                                            (#{1-}# (+ (#{1+}# (logior
>                                                                 (#{1-}# offset)
>                                                                 (#{1-}# 2)))
>                                                       2))
>                                            (#{1-}# 2)))
>                                  2))
>                       (#{1-}# 4)))
>             'big
>             4)))
>     (make-socket-address family address port)))
>
> No extra allocations and computations.
>
> http://git.savannah.gnu.org/cgit/guix.git/tree/guix/build/syscalls.scm#n264
>
> I’m sure we could build a higher-level interface on top of that, for
> instance for when we want a C struct to directly have a corresponding
> Scheme record (as is the case of ‘%statfs’ in the file above.)

Hi Ludo :-)

I'll try to grok your approach later and see how it compares, but let me
mention that the syntax-case based macro API of bytestructures lets one
hoist all computation to macro-expand time:

scheme@(guile-user)> (import (bytestructures guile))
scheme@(guile-user)> (define foo_t (bs:struct `((s ,(bs:struct `((x ,int)
                                                                 (y ,float32))))
                                                (z ,int)
                                                (t ,(bs:vector 3 float32)))))
scheme@(guile-user)> (define-bytestructure-accessors foo_t unwrapper getter setter)
scheme@(guile-user)> ,optimize (getter bv s y)
$3 = (bytevector-ieee-single-native-ref bv 4)
scheme@(guile-user)> ,optimize (getter bv t 2)
$4 = (bytevector-ieee-single-native-ref bv 20)
scheme@(guile-user)> ,optimize (setter bv t 2 42)
$5 = (bytevector-ieee-single-native-set! bv 20 42)

(In the REPL this works seamlessly; in a source file we would need to
make sure that foo_t is defined at macro expand time.)

> Ludo’.

Taylan

Re: Bytestructures, FFI

[Ludovic Courtès]

Hi,

taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:

> I'll try to grok your approach later and see how it compares, but let me
> mention that the syntax-case based macro API of bytestructures lets one
> hoist all computation to macro-expand time:

Ah, very nice!

Ludo’.