ffi helper

ffi helper

[Matt Wette]

I’m now working on a FFI helper based on the nyacc C99 parser.   
My current prototyping goal is to autogenerate the functions for cairo.h. 
I will assume going with scheme-bytestructures. 
I’ll keep this list posted on progress.

Matt

Re: ffi helper

[Taylan Ulrich Bayırlı/Kammer]

Matt Wette <matt.wette@gmail.com> writes:

> I’m now working on a FFI helper based on the nyacc C99 parser.   
> My current prototyping goal is to autogenerate the functions for cairo.h. 
> I will assume going with scheme-bytestructures. 
> I’ll keep this list posted on progress.
>
> Matt

Hi, I'm happy to hear my library is being made use of!

Reach me any time with questions or suggestions through this e-mail
address, the GitHub issues system, or 'taylan' on Freenode where I'll
usually respond fairly quickly during UTC+2 evenings and sometimes on
weekends.

I'm not very active in software development these days, but I'll try to
provide support where it's needed.

Taylan

Re: ffi helper

[David Pirotte]

[-- Attachment #1: Type: text/plain, Size: 815 bytes --]

Heya!

> > I’m now working on a FFI helper based on the nyacc C99 parser.   
> > My current prototyping goal is to autogenerate the functions for cairo.h. 
> > I will assume going with scheme-bytestructures. 
> > I’ll keep this list posted on progress.
> >
> > Matt  
> 
> Hi, I'm happy to hear my library is being made use of!
> 
> Reach me any time with questions or suggestions through this e-mail
> address, the GitHub issues system, or 'taylan' on Freenode where I'll
> usually respond fairly quickly during UTC+2 evenings and sometimes on
> weekends.

If you have questions or suggestions, please maintain the discussion here and/or on
freenode! If you use use the github issue system, we would not know ... at least I
would not know, and I'm interested to follow ...

Thanks!
David

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: ffi helper

[Matt Wette]

> On Mar 8, 2017, at 6:06 PM, Matt Wette <matt.wette@gmail.com> wrote:
> 
> I’m now working on a FFI helper based on the nyacc C99 parser.   

So, like, I think what might be useful is something that will convert a dot-h file to a spec file, with certain assumptions about function signatures (e.g., if passing a pointer to int, then the scheme implementation is a (system foreign) int*, or maybe a boxed integer).  

Then the spec is compiled, in a machine-dependent manner to deal with type sizes and alignment, to a .go file which provides scheme-callable functions.  I’m not sure the output of the dot-H reader can be machine independent, given all the spaghetti code that typically exists in /usr/include.  It may be worth a try, later.

Also, the module should support structures.  The bytestructures package looks promising here.

Enums should be provided also.

CPP constants (aka #define’s) should be supported.

Do CPP macros with arguments need to be supported?   This is not just converting to functions, because CPP macros do not recurse (e.g., `#define sqrt(F) sqrt(fabs(F))’ is perfectly legal C).

Some other tidbits: 

The nyacc c99 module provides a few neat functions to support this.  One is expand-typerefs (was expand-decl-typerefs).  This will convert an sxml representation of 

  typedef const int (*foo_t)(int a, double b);
  extern foo_t fctns[2];

into one of 

  extern const int (*fctns[2])(int a, double b);

Another function, udecl->mspec, helps with the conversion as follows.  Consider

  double x[10]; /* state vector */

The c99 parser, followed by tree->udict, followed by cadr will turn the above into

  (udecl (decl-spec-list
          (type-spec (float-type "double")))
        (init-declr
          (array-of (ident "x") (p-expr (fixed "10"))))
        (comment " state vector “))

And “udecl->mspec/comm” will convert the above into

  ("x"
   " state vector "
   (array-of "10")
   (float-type "double”))

I am in the process of cleaning up the above in order to convert some function signatures.  

Matt

Re: ffi helper

[Matt Wette]

> On Mar 13, 2017, at 5:53 PM, Matt Wette <matt.wette@gmail.com> wrote:
> 
> 
>> On Mar 8, 2017, at 6:06 PM, Matt Wette <matt.wette@gmail.com> wrote:
>> 
>> I’m now working on a FFI helper based on the nyacc C99 parser.   
> 

I am making some progress with some ragged prototype code.

Here is a top-level program, just pulling out three declarations to start:

(let* ((file "/opt/local/include/cairo/cairo-svg.h")
       (defs '())
       (incs '("/opt/local/include" "/opt/local/include/cairo" "/usr/include"))
       (tree (my-parse defs incs file))
       (file-decls (reverse (c99-trans-unit->udict tree #:filter cairo-filter)))
       (udecl-dict (c99-trans-unit->udict/deep tree)))

  (fold
   (lambda (pair type-list)
     (udecl->ffi-decl (cdr pair) type-list))
   fixed-width-int-names
   (filter
    (lambda (e)
      (member (car e)
	      '("cairo_matrix_t" "cairo_surface_t" "cairo_svg_surface_create")))
    file-decls)))

And here is the autogenerated output for the declarations:

(define-std-pointer-wrapper cairo_surface_t*)

(define cairo_matrix_t
  (bs:struct
    `(,(string->symbol "xx") ,double)
    `(,(string->symbol "yx") ,double)
    `(,(string->symbol "xy") ,double)
    `(,(string->symbol "yy") ,double)
    `(,(string->symbol "x0") ,double)
    `(,(string->symbol "y0") ,double)))

(define cairo_svg_surface_create
  (let ((f (pointer->procedure
             '*
             (libcairo-func "cairo_svg_surface_create")
             (list '* double double))))
    (lambda (filename width_in_points height_in_points)
      (let ((~filename (string->pointer filename))
            (~width_in_points width_in_points)
            (~height_in_points height_in_points)
            (~result
              (f ~filename ~width_in_points ~height_in_points)))
        (wrap-cairo_surface_t* ~result)))))

I was not able to come up with a procedure that would take arguments “xx” and “double” and generate the bs:struct pattern `(xx ,double).  Help is welcome.

Re: ffi helper

[Taylan Ulrich Bayırlı/Kammer]

Matt Wette <matt.wette@gmail.com> writes:

>> On Mar 13, 2017, at 5:53 PM, Matt Wette <matt.wette@gmail.com> wrote:
>> 
>> 
>>> On Mar 8, 2017, at 6:06 PM, Matt Wette <matt.wette@gmail.com> wrote:
>>> 
>>> I’m now working on a FFI helper based on the nyacc C99 parser.   
>> 
>
> I am making some progress with some ragged prototype code.
>
> Here is a top-level program, just pulling out three declarations to start:
>
> (let* ((file "/opt/local/include/cairo/cairo-svg.h")
>        (defs '())
>        (incs '("/opt/local/include" "/opt/local/include/cairo" "/usr/include"))
>        (tree (my-parse defs incs file))
>        (file-decls (reverse (c99-trans-unit->udict tree #:filter cairo-filter)))
>        (udecl-dict (c99-trans-unit->udict/deep tree)))
>
>   (fold
>    (lambda (pair type-list)
>      (udecl->ffi-decl (cdr pair) type-list))
>    fixed-width-int-names
>    (filter
>     (lambda (e)
>       (member (car e)
> 	      '("cairo_matrix_t" "cairo_surface_t" "cairo_svg_surface_create")))
>     file-decls)))
>
> And here is the autogenerated output for the declarations:
>
> (define-std-pointer-wrapper cairo_surface_t*)
>
> (define cairo_matrix_t
>   (bs:struct
>     `(,(string->symbol "xx") ,double)
>     `(,(string->symbol "yx") ,double)
>     `(,(string->symbol "xy") ,double)
>     `(,(string->symbol "yy") ,double)
>     `(,(string->symbol "x0") ,double)
>     `(,(string->symbol "y0") ,double)))
>
> (define cairo_svg_surface_create
>   (let ((f (pointer->procedure
>              '*
>              (libcairo-func "cairo_svg_surface_create")
>              (list '* double double))))
>     (lambda (filename width_in_points height_in_points)
>       (let ((~filename (string->pointer filename))
>             (~width_in_points width_in_points)
>             (~height_in_points height_in_points)
>             (~result
>               (f ~filename ~width_in_points ~height_in_points)))
>         (wrap-cairo_surface_t* ~result)))))

Neat!

> I was not able to come up with a procedure that would take arguments
> “xx” and “double” and generate the bs:struct pattern `(xx ,double).
> Help is welcome.

You mean you get "xx" and "double" as strings?  For xx, string->symbol
as you already do is OK.  As for double, that's a bit tricky.  I see a
few possibilities:

- Turn the string into a symbol and eval it:

    (eval (string->symbol str)
          (resolve-module '(bytestructures guile numeric)))

- Use the reflective module API:

    (module-ref (resolve-module '(bytestructures guile numeric))
                (string->symbol str))

- Create an explicit mapping from name (string or symbol) to object:

    (import (bytestructures guile numeric))

    (define descriptor-mapping
      `(("double" . ,double)
        ("float" . ,float)
        ...))

(The so-far undocumented module (bytestructures guile numeric-metadata)
contains a mapping similar to this, but it only contains name with
explicit bit width so e.g. 'double' isn't in it, so I guess it's not the
right thing here.)

While 'eval' is normally discouraged, I think it's fine in this case,
since we use string->symbol.  I don't see anything bad resulting from
evaluating an arbitrary symbol.

However, the second option could be said to be a tad bit more "correct"
than using eval, so I'd go with that and only use the eval method if I
had to write standards-compliant code where eval is available but not
something like module-ref.

The third option gives you full control but it shouldn't be necessary.

Hope that helps,
Taylan

Re: ffi helper

[Matt Wette]

> On Mar 19, 2017, at 10:23 AM, Matt Wette <matt.wette@gmail.com> wrote:
> 
> 
>> On Mar 13, 2017, at 5:53 PM, Matt Wette <matt.wette@gmail.com> wrote:
>> 
>> 
>>> On Mar 8, 2017, at 6:06 PM, Matt Wette <matt.wette@gmail.com> wrote:
>>> 
>>> I’m now working on a FFI helper based on the nyacc C99 parser.   
>> 
> 
> I am making some progress with some ragged prototype code.


I have issues with the bytestructures module: both it and (system foreign) use symbols like “double”.  Probably need to rename somewhere.

I can now autogenerate enough to successfully run the following program.

(use-modules (cairo cairo))

(define srf (cairo_svg_surface_create "abc.svg" 200.0 200.0))

(define cr (cairo_create srf))

(cairo_move_to cr 10.0 10.0)
(cairo_line_to cr 190.0 10.0)
(cairo_line_to cr 190.0 190.0)
(cairo_line_to cr 10.0 190.0)
(cairo_line_to cr 10.0 10.0)
(cairo_stroke cr)

;; need to add guardians?
(cairo_destroy cr)
(cairo_surface_destroy srf)

Re: ffi helper

[Amirouche]

Héllo Matt!


Le 09/03/2017 à 03:06, Matt Wette a écrit :
> I’m now working on a FFI helper based on the nyacc C99 parser.
> My current prototyping goal is to autogenerate the functions for cairo.h.
> I will assume going with scheme-bytestructures.
> I’ll keep this list posted on progress.
>
> Matt

Let me know, when it will be possible to try.
FWIW I started some bindings for libgit2 but
I got stuck... Basically because it's overwhelming.

I got stuck trying to bind "git clone" [1]. Here is the signature:

|int git_clone(git_repository **out, const char *url, const char 
*local_path, const git_clone_options *options);|

It requires a double pointer "out". Then it also
has a "options" which is a "git_clone_options" struct
which in turns requires |another struct named git_checkout_options.

I am still not sure how to declare function pointers in
guile-bytestructures and libgit2 make heavy use of those.
|
It's located at https://gitlab.com/amirouche/guile-git

Maybe it's too complicated, instead I can try to
regenerate my wiredtiger bindings [2]. What is
specific about it is that it has functions pointers
in struct.

Tx:)

[1] https://libgit2.github.com/libgit2/#HEAD/group/clone/git_clone
[2] which has unit tests, so it will be "easy" to know if it works or not

Re: ffi helper

[Matt Wette]

> On Mar 23, 2017, at 10:43 AM, Amirouche <amirouche@hypermove.net> wrote:
> 
> Héllo Matt!
> 
> 
> Le 09/03/2017 à 03:06, Matt Wette a écrit :
>> I’m now working on a FFI helper based on the nyacc C99 parser.
>> My current prototyping goal is to autogenerate the functions for cairo.h.
>> I will assume going with scheme-bytestructures.
>> I’ll keep this list posted on progress.
>> 
>> Matt
> 
> Let me know, when it will be possible to try.
> FWIW I started some bindings for libgit2 but
> I got stuck... Basically because it's overwhelming.
> 
> I got stuck trying to bind "git clone" [1]. Here is the signature:
> 
> |int git_clone(git_repository **out, const char *url, const char *local_path, const git_clone_options *options);|
> 
> It requires a double pointer "out". Then it also
> has a "options" which is a "git_clone_options" struct
> which in turns requires |another struct named git_checkout_options.
> 
> I am still not sure how to declare function pointers in
> guile-bytestructures and libgit2 make heavy use of those.
> |
> It's located at https://gitlab.com/amirouche/guile-git
> 
> Maybe it's too complicated, instead I can try to
> regenerate my wiredtiger bindings [2]. What is
> specific about it is that it has functions pointers
> in struct.
> 
> Tx:)
> 
> [1] https://libgit2.github.com/libgit2/#HEAD/group/clone/git_clone
> [2] which has unit tests, so it will be "easy" to know if it works or not

Lot of good items to chew on.   I will be looking as far as the pointer pointer I wonder if this will work for the out

(use-modules ((bytestructures guile) #:prefix bs:))
(use-modules ((system foreign) #:prefix ffi:))

;; int git_clone(git_repository **out, 
;;               const char *url, 
;;               const char *local_path, 
;;               const git_clone_options *options);
(define git_clone
 (let ((f (pointer->procedure
             int
             (lib-func "git_clone")
             (list '* '* '* '*))))
    (lambda (url local_path options)
      (let* ((out_ptr (make-pointer 0))
	     (out_ptr_ptr (scm->pointer out_ptr))
	     (_out_ptr_ptr (pointer-address))
	     (_url (string->pointer url))
	     (_local_path (string->pointer local_path))
	     (_options (make-pointer 0)) ;; todo
	     (res (f out_ptr_ptr _url _local_path _options)))
	(if (zero? res)
	    out_ptr
	    (error "call failed"))))))

Re: ffi helper

[Matt Wette]

> On Mar 21, 2017, at 3:40 PM, Matt Wette <matt.wette@gmail.com> wrote:
>> On Mar 19, 2017, at 10:23 AM, Matt Wette <matt.wette@gmail.com> wrote:
>>> On Mar 13, 2017, at 5:53 PM, Matt Wette <matt.wette@gmail.com> wrote:
>>>> On Mar 8, 2017, at 6:06 PM, Matt Wette <matt.wette@gmail.com> wrote:
>>>> I’m now working on a FFI helper based on the nyacc C99 parser.   

Still working on it.  I ran into some issues with the C preprocessor again so had to fix that.  


I have a lot to do still, but now I have a source file that looks like this:

(use-modules (ffi-help))

(define-ffi-helper (cairo cairo)
  #:pkg-config "cairo"
  #:include "cairo-svg.h"
  #:library "libcairo"
  #:filter (lambda (path) (string=? "cairo" (substring path 0 5)))
  )



and generates a 4800 line files that includes stuff like this:

(define-module (path-list)           <= should read (cairo cairo)
  #:(use-module (ffi-help)
  #:(use-module (system foreign)
  #:(use-module ((bytestructures guile) #:renamer bs-renamer)
  )

(define lib-link (dynamic-link "libcairo"))
(define (lib-func name) (dynamic-func name lib-link))

;; "cairo_version"
(define cairo_version
  (let ((f (pointer->procedure
             int
             (lib-func "cairo_version")
             (list))))
    (lambda () (let () (wrap-int (f))))))
(export cairo_version)

;; "cairo_version_string"
(define cairo_version_string
  (let ((f (pointer->procedure
             '*
             (lib-func "cairo_version_string")
             (list))))
    (lambda () (let () (wrap-char* (f))))))
(export cairo_version_string)

;; "cairo_t"
(define-std-pointer-wrapper cairo_t*)
(export cairo_t*)

;; "cairo_surface_t"
(define-std-pointer-wrapper cairo_surface_t*)
(export cairo_surface_t*)

;; "cairo_device_t"
(define-std-pointer-wrapper cairo_device_t*)
(export cairo_device_t*)

;; "cairo_matrix_t"
(define cairo_matrix_t
  (bs:struct
    `((xx ,double)
      (yx ,double)
      (xy ,double)
      (yy ,double)
      (x0 ,double)
      (y0 ,double))))
(export cairo_matrix_t)

;; "cairo_get_user_data"
(define cairo_get_user_data
  (let ((f (pointer->procedure
             '*
             (lib-func "cairo_get_user_data")
             (list '* '*))))
    (lambda (cr key)
      (let ((~cr (unwrap-cairo_t* cr))
            (~key (unwrap-cairo_user_data_key_t* key)))
        (wrap-void* (f ~cr ~key))))))
(export cairo_get_user_data)

;; "cairo_push_group_with_content"
(define cairo_push_group_with_content
  (let ((f (pointer->procedure
             void
             (lib-func "cairo_push_group_with_content")
             (list '* tbd:cairo_content_t))))
    (lambda (cr content)
      (let ((~cr (unwrap-cairo_t* cr)) (~content content))
        (f ~cr ~content)))))
(export cairo_push_group_with_content)

;; "cairo_set_source_rgba"
(define cairo_set_source_rgba
  (let ((f (pointer->procedure
             void
             (lib-func "cairo_set_source_rgba")
             (list '* double double double double))))
    (lambda (cr red green blue alpha)
      (let ((~cr (unwrap-cairo_t* cr))
            (~red red)
            (~green green)
            (~blue blue)
            (~alpha alpha))
        (f ~cr ~red ~green ~blue ~alpha)))))
(export cairo_set_source_rgba)

Re: ffi helper

[Matt Wette]

> On Apr 17, 2017, at 7:26 PM, Matt Wette <matt.wette@gmail.com> wrote:
> 
> 
>> On Mar 21, 2017, at 3:40 PM, Matt Wette <matt.wette@gmail.com> wrote:
>>> On Mar 19, 2017, at 10:23 AM, Matt Wette <matt.wette@gmail.com> wrote:
>>>> On Mar 13, 2017, at 5:53 PM, Matt Wette <matt.wette@gmail.com> wrote:
>>>>> On Mar 8, 2017, at 6:06 PM, Matt Wette <matt.wette@gmail.com> wrote:
>>>>> I’m now working on a FFI helper based on the nyacc C99 parser.   

Still grinding away.  I can generate wrappers for function pointers passed to C routines: see cairo_destroy_func_t below.  I don’t know if the converter should be called “wrap” or something else as, wrt the pointer wrappers, we are adding a “C” wrapper instead of a “scheme” wrapper.

I am deciding how to handle enums.  I will likely have wrapper to map symbols to integers and vice versa, but I will have to handle anonymous enums as well (e.g." enum { ABC = 1 };”). 

Matt

(use-modules (ffi-help))

(define-ffi-helper (cairo cairo)
  #:pkg-config "cairo"
  #:include "cairo-svg.h"
  #:library "libcairo"
  #:filter (lambda (path) (string=? "cairo" (substring path 0 5)))
  )

==[GENERATES]==>[filter samples]==>

;;
;; auto-generated by ffi-help.scm
;;

(define-module (cairo cairo)
  #:use-module (ffi-help)
  #:use-module ((system foreign) #:prefix ffi:)
  #:use-module ((bytestructures guile) #:prefix bs:)
  )
(define bs:struct bs:bs:struct)

(define lib-link (dynamic-link "libcairo"))
(define (lib-func name) (dynamic-func name lib-link))

;; typedef struct _cairo_surface cairo_surface_t;
(define-std-pointer-wrapper cairo_surface_t*)

;; cairo_matrix_t
(define cairo_matrix_t
  (bs:struct
    (list `(xx ,bs:double)
          `(yx ,bs:double)
          `(xy ,bs:double)
          `(yy ,bs:double)
          `(x0 ,bs:double)
          `(y0 ,bs:double))))
(export cairo_matrix_t)


;; typedef void  (*cairo_destroy_func_t)(void  *data);
(define (wrap-cairo_destroy_func_t proc) ;; => pointer
 (ffi:procedure->pointer ffi:void proc (list '*))
 )
(export wrap-cairo_destroy_func_t)

;; cairo_t *cairo_create(cairo_surface_t *target);
(define cairo_create
  (let ((f (ffi:pointer->procedure
             '*
             (lib-func "cairo_create")
             (list '*))))
    (lambda (target)
      (let ((~target ((unwrap-cairo_surface_t* target))))
        (identity (f ~target))))))
(export cairo_create)

;; typedef struct _cairo_region cairo_region_t;
(define-std-pointer-wrapper cairo_region_t*)

;; cairo_bool_t cairo_region_contains_point(const cairo_region_t *region, int 
;;     x, int y);
(define cairo_region_contains_point
  (let ((f (ffi:pointer->procedure
             ffi:int
             (lib-func "cairo_region_contains_point")
             (list '* ffi:int ffi:int))))
    (lambda (region x y)
      (let ((~region ((unwrap-cairo_region_t* region))))
        (f ~region x y)))))
(export cairo_region_contains_point)

;; --- last line ---