Calling back scheme from C

Calling back scheme from C

[divoplade]

[-- Attachment #1: Type: text/plain, Size: 1612 bytes --]

Hello guile,

I am having a hard time understanding what I do wrong when trying to
pass a guile function as a C callback (from C).

You should be able to trigger the bug by saving the 3 attached files
and running:

    guix build -L . pkg-with-the-bug

The bug disappears when ignoring the scheme callback, so I suspect it
has something to do with the callback and not further functions:

    guix build -L . pkg-without-the-bug

Does anyone see an error?

You will find the C code under libguile-nettle.c. guile-nettle.c is
just an ad-hoc interpreter with the primitives built in, to run the
test. The test is example-rsa-sign.in: it defines a callback that will
return a bytevector of length its argument, and that bytevector will be
processed to generate an RSA key pair with nettle. The function with-
rsa-generated-key-pair takes the callback plus other arguments and a
function, and it will call that function with the public key and the
private key.

(define (random-with-guile length)
  (use-modules (srfi srfi-1))
  (use-modules (rnrs bytevectors))
  (u8-list->bytevector
   (unfold (lambda (i) (>= i length))
	   (lambda (i) (random 256))
	   1+
	   0)))

(define (generate-rsa-keypair nbits random progress)
  (with-rsa-generated-key-pair random progress nbits 16
    (lambda (public private)
      (rsa-public-key-n public))))

(define keypair
  (generate-rsa-keypair 2048 random-with-guile #f))

(if keypair
    (format #t "The key pair has been generated.\n")
    (begin
      (format (current-error-port)
	      "The key pair could not be generated.\n")
      (exit 1)))

Best regards,

divoplade

[-- Attachment #2: bugs.scm --]
[-- Type: text/x-scheme, Size: 1498 bytes --]

(define-module (bugs)
  #:use-module (guix packages)
  #:use-module ((guix licenses) #:prefix license:)
  #:use-module (guix download)
  #:use-module (guix build-system gnu)
  #:use-module (gnu packages)
  #:use-module (gnu packages guile)
  #:use-module (gnu packages nettle)
  #:use-module (gnu packages pkg-config))

(define-public pkg-with-the-bug
  (package
   (name "pkg-with-the-bug")
   (version "0.3.0.8-c2e5")
   (source (origin
            (method url-fetch)
            (uri "guile-nettle-0.3.0.8-c2e5.tar.gz")
            (sha256
             (base32
              "0lmyqc4p6gjgkslkxs925ayh54g35mjn94zljynga1a0bvl7j48l"))))
   (build-system gnu-build-system)
   (arguments `())
   (native-inputs
    `(("pkg-config" ,pkg-config)
      ("guile" ,guile-3.0)))
   (inputs `(("guile" ,guile-3.0)
             ("nettle" ,nettle)))
   (propagated-inputs `())
   (native-search-paths
    (list (search-path-specification
	   (variable "LTDL_LIBRARY_PATH")
	   (files (list "lib/")))))
   (synopsis "")
   (description "")
   (home-page "https://divoplade.fr/guile-nettle.git")
   (license license:gpl3+)))

(define-public pkg-without-the-bug
  (package (inherit pkg-with-the-bug)
	   (name "pkg-without-the-bug")
	   (source
	    (origin
             (method url-fetch)
             (uri "guile-nettle-0.3.0.8-c2e5.tar.gz")
             (sha256
              (base32
               "0lmyqc4p6gjgkslkxs925ayh54g35mjn94zljynga1a0bvl7j48l"))
	     (patches '("disable-function-call.patch"))))))

[-- Attachment #3: disable-function-call.patch --]
[-- Type: text/x-patch, Size: 759 bytes --]

diff -u guile-nettle-0.3.0.8-c2e5/libguile-nettle.c patched/libguile-nettle.c
--- guile-nettle-0.3.0.8-c2e5/libguile-nettle.c	2020-09-22 07:38:24.000000000 +0200
+++ patched/libguile-nettle.c	2020-09-22 08:06:59.044627220 +0200
@@ -30,16 +30,10 @@
 static void
 guile_nettle_random_func (void *ctx, size_t length, uint8_t * dst)
 {
-  SCM *cb = ctx;
-  SCM scm_length = scm_from_size_t (length);
-  SCM random_data = scm_call_1 (*cb, scm_length);
-  size_t output_length = scm_c_bytevector_length (random_data);
   size_t i;
-  SCM_ASSERT (output_length == length,
-              *cb, SCM_ARGn, "generating random data");
   for (i = 0; i < length; i++)
     {
-      dst[i] = scm_c_bytevector_ref (random_data, i);
+      dst[i] = rand () % 256;
     }
 }
 

[-- Attachment #4: guile-nettle-0.3.0.8-c2e5.tar.gz --]
[-- Type: application/x-compressed-tar, Size: 399711 bytes --]

Re: Calling back scheme from C

[Chris Vine]

On Tue, 22 Sep 2020 08:25:25 +0200
divoplade <d@divoplade.fr> wrote:
> Hello guile,
> 
> I am having a hard time understanding what I do wrong when trying to
> pass a guile function as a C callback (from C).
> 
> You should be able to trigger the bug by saving the 3 attached files
> and running:
> 
>     guix build -L . pkg-with-the-bug
> 
> The bug disappears when ignoring the scheme callback, so I suspect it
> has something to do with the callback and not further functions:
> 
>     guix build -L . pkg-without-the-bug
> 
> Does anyone see an error?

I haven't looked at your code but note that GC'ed objects held only in
the C or C++ heap (say as closures of a C function) are not visible to
the GC and need protection with scm_gc_protect_object, otherwise they
can be collected while still in use by C code.  If you have one-shot C
callbacks taking guile objects as closures which are stored as
pointers in the C heap, you can call scm_gc_unprotect_object as the last
thing the callback does in order to get the object collected
subsequently.

This may not be your issue.  I offer it as something I had to deal with
in code of my own to prevent incorrect collection for closures of
callbacks executed by a C event loop.

Re: Calling back scheme from C

[divoplade]

Hello,

 Le mardi 22 septembre 2020 à 10:50 +0100, Chris Vine a écrit :
> This may not be your issue.  I offer it as something I had to deal
> with
> in code of my own to prevent incorrect collection for closures of
> callbacks executed by a C event loop.

It does not seem to fix it on my side. Could you share your callback
closure example?

divoplade

Re: Calling back scheme from C

[Chris Vine]

On Tue, 22 Sep 2020 12:36:15 +0200
divoplade <d@divoplade.fr> wrote:
> Hello,
> 
>  Le mardi 22 septembre 2020 à 10:50 +0100, Chris Vine a écrit :
> > This may not be your issue.  I offer it as something I had to deal
> > with
> > in code of my own to prevent incorrect collection for closures of
> > callbacks executed by a C event loop.
> 
> It does not seem to fix it on my side. Could you share your callback
> closure example?

I haven't got anything sufficiently compact to be postable.  However
the principle is fairly straightforward: when carrying out a mark, the
GC can see any memory dynamically alloced by scm_gc_malloc and
cognates (but _not_ SCM pointers held in memory allocated by
scm_gc_malloc_pointerless), and can also see global variables, static
data sections, function arguments, registers and variables on the C and
Scheme stacks, but not anything else, and in particular it cannot see
anything held on the C heap (memory formed by malloc and cognates).  So
if your code ends up keeping a SCM object by pointer in the C heap, and
there are no other live references to the object which the mark phase
can see, then it can be freed by the GC at any time and if so you are
left with a dangling pointer in the C heap.

Typically this can occur with SCM objects held by pointer as closures
for C callbacks.

If you have applied scm_gc_protect_object to any suspect SCM objects in
your program, and that doesn't resolve the issue, then that is not the
problem.  If it does resolve the issue then you need to decide how and
when subsequently to apply scm_gc_unprotect_object to them to avoid
memory leaks.  But that is a second stage operation from the diagnosis.

Re: Calling back scheme from C

[divoplade]

Hello all,

It seems that I'm making progress. 

    scm_call_* seems to corrupt something if the value is a lambda
calling (use-modules)

I tested with srfi srfi-1 or rnrs bytevectors. Without any of them,
calling the function and discarding its results does not crash the
program. With any or both of them, calling the function (still
discarding the result, it does not matter) corrupts some values (=>
assert failure in foreign library code).

I will make further tests...

divoplade


Le mardi 22 septembre 2020 à 08:25 +0200, divoplade a écrit :
> Hello guile,
> 
> I am having a hard time understanding what I do wrong when trying to
> pass a guile function as a C callback (from C).
> 
> You should be able to trigger the bug by saving the 3 attached files
> and running:
> 
>     guix build -L . pkg-with-the-bug
> 
> The bug disappears when ignoring the scheme callback, so I suspect it
> has something to do with the callback and not further functions:
> 
>     guix build -L . pkg-without-the-bug
> 
> Does anyone see an error?
> 
> You will find the C code under libguile-nettle.c. guile-nettle.c is
> just an ad-hoc interpreter with the primitives built in, to run the
> test. The test is example-rsa-sign.in: it defines a callback that
> will
> return a bytevector of length its argument, and that bytevector will
> be
> processed to generate an RSA key pair with nettle. The function with-
> rsa-generated-key-pair takes the callback plus other arguments and a
> function, and it will call that function with the public key and the
> private key.
> 
> (define (random-with-guile length)
>   (use-modules (srfi srfi-1))
>   (use-modules (rnrs bytevectors))
>   (u8-list->bytevector
>    (unfold (lambda (i) (>= i length))
> 	   (lambda (i) (random 256))
> 	   1+
> 	   0)))
> 
> (define (generate-rsa-keypair nbits random progress)
>   (with-rsa-generated-key-pair random progress nbits 16
>     (lambda (public private)
>       (rsa-public-key-n public))))
> 
> (define keypair
>   (generate-rsa-keypair 2048 random-with-guile #f))
> 
> (if keypair
>     (format #t "The key pair has been generated.\n")
>     (begin
>       (format (current-error-port)
> 	      "The key pair could not be generated.\n")
>       (exit 1)))
> 
> Best regards,
> 
> divoplade