* RFC: Foreign objects facility
@ 2014-04-27 13:17 Andy Wingo
2014-04-27 16:00 ` Mark H Weaver
` (2 more replies)
0 siblings, 3 replies; 14+ messages in thread
From: Andy Wingo @ 2014-04-27 13:17 UTC (permalink / raw)
To: guile-devel
[-- Attachment #1: Type: text/plain, Size: 4014 bytes --]
Hi,
SMOBs have a few problems.
1) They are limited in number to 255.
2) It's difficult to refer to a SMOB type from Scheme. You can use
class-of once you have an object, but the class-of isn't exactly
the same as the SMOB tc16, and getting the type beforehand is
gnarly. (See http://article.gmane.org/gmane.comp.gdb.patches/96857
for an example).
3) You can't create SMOB types from Scheme. This goes against our
general trend of making Scheme more powerful.
4) You can't create SMOB objects from Scheme.
5) Similarly, you can't access SMOB fields from Scheme.
6) You can't subclass SMOB types. (Some people would like this
ability.)
7) There is legacy code out there that uses e.g. SCM_SETCDR to set
smob fields. (This is terrible, but it exists:
https://github.com/search?q=SCM_SETCDR+smob&ref=cmdform&type=Code
for an example.)
8) The single/double SMOB thing is outdated and bogus. Objects should
be able to have any number of fields.
9) We document mark functions in the manual, even recommending them,
but they are really difficult to get right (see
https://lists.gnu.org/archive/html/guile-user/2011-11/msg00069.html),
and almost always a bad design -- the BDW GC can do a better job
without them.
And yet, if we consider the generic problem of wrapping C objects in
Scheme objects, it's clear that we have more solutions now than we used
to -- raw #<pointer> values, define-wrapped-pointer-type, etc. But
there's nothing that's accessible to C like SMOBs are, so people that
use the libguile interface to wrap C types and values are out of luck.
I propose to provide a new interface that will eventually make SMOBs
obsolete. This new interface is based on structs with raw fields -- the
'u' fields. (See
http://www.gnu.org/software/guile/docs/master/guile.html/Vtables.html#Vtables
for description of 'u' fields. Note that the documentation is wrong --
these fields are indeed traced by the GC.)
Here is the proposed C API:
SCM scm_make_foreign_object_type (SCM name, SCM slot_names,
scm_t_struct_finalize finalizer);
void scm_assert_foreign_object_type (SCM type, SCM val);
SCM scm_make_foreign_object_1 (SCM type, scm_t_bits val0);
SCM scm_make_foreign_object_2 (SCM type, scm_t_bits val0,
scm_t_bits val1);
SCM scm_make_foreign_object_3 (SCM type, scm_t_bits val0,
scm_t_bits val1, scm_t_bits val2);
SCM scm_make_foreign_object_n (SCM type, size_t n, scm_t_bits vals[]);
scm_t_bits scm_foreign_object_ref (SCM obj, size_t n);
void scm_foreign_object_set_x (SCM obj, size_t n, scm_t_bits val);
The finalizer may be NULL. The scm_make_foreign_object* functions are
just scm_make_struct without the no_tail arguments, and interpreting the
values as raw untagged values, not unpacked SCM values. Same thing with
scm_foreign_object_ref/set_x.
The overhead of a foreign object is two words -- the same as the
overhead on any struct. (Compare to SMOBs, which have a half-word
overhead.)
Here is the proposed Scheme API:
;; Exported from (system foreign-object):
(define* (make-foreign-object-type name slots #:key finalizer)
...)
(define-syntax-rule (define-foreign-object-type name constructor (slot ...)
kwarg ...)
(begin
(define name
(make-foreign-object-type 'name '(slot ...) kwarg ...))
(define slot [getter for slot])
...
(define constructor
(lambda (slot ...)
[...]))))
Foreign object types are GOOPS classes, although this is not really
exposed in the API. Foreign objects are GOOPS objects -- with no
additional overhead of course, compared to structs. You can subclass an
object type; see the test-foreign-object-scm test in the patch below.
So, what do people think? The patch below is against stable-2.0.
Andy
[-- Attachment #2: 0001-New-foreign-object-facility-to-replace-SMOBs.patch --]
[-- Type: text/plain, Size: 22569 bytes --]
From a12efcfaae1c65cc703616ea15106a88efba3f55 Mon Sep 17 00:00:00 2001
From: Andy Wingo <wingo@pobox.com>
Date: Sun, 27 Apr 2014 14:47:40 +0200
Subject: [PATCH] New foreign object facility, to replace SMOBs
* libguile/foreign-object.c:
* libguile/foreign-object.h:
* module/system/foreign-object.scm:
* test-suite/standalone/test-foreign-object-c.c:
* test-suite/standalone/test-foreign-object-scm: New files.
* test-suite/standalone/Makefile.am:
* module/Makefile.am:
* libguile/Makefile.am: Add new files.
* libguile.h: Add foreign-object.h.
* libguile/init.c (scm_i_init_guile): Call scm_register_foreign_object.
---
libguile.h | 3 +-
libguile/Makefile.am | 2 +
libguile/foreign-object.c | 187 ++++++++++++++++++++++++++
libguile/foreign-object.h | 48 +++++++
libguile/init.c | 1 +
module/Makefile.am | 1 +
module/system/foreign-object.scm | 88 ++++++++++++
test-suite/standalone/Makefile.am | 11 ++
test-suite/standalone/test-foreign-object-c.c | 116 ++++++++++++++++
test-suite/standalone/test-foreign-object-scm | 119 ++++++++++++++++
10 files changed, 575 insertions(+), 1 deletion(-)
create mode 100644 libguile/foreign-object.c
create mode 100644 libguile/foreign-object.h
create mode 100644 module/system/foreign-object.scm
create mode 100644 test-suite/standalone/test-foreign-object-c.c
create mode 100755 test-suite/standalone/test-foreign-object-scm
diff --git a/libguile.h b/libguile.h
index fefca43..48548c3 100644
--- a/libguile.h
+++ b/libguile.h
@@ -1,7 +1,7 @@
#ifndef SCM_LIBGUILE_H
#define SCM_LIBGUILE_H
-/* Copyright (C) 1995,1996,1997,1998,2000,2001, 2002, 2003, 2004, 2006, 2008, 2009, 2010, 2011, 2012 Free Software Foundation, Inc.
+/* Copyright (C) 1995,1996,1997,1998,2000,2001, 2002, 2003, 2004, 2006, 2008, 2009, 2010, 2011, 2012, 2014 Free Software Foundation, Inc.
*
* This library is free software; you can redistribute it and/or
* modify it under the terms of the GNU Lesser General Public License
@@ -52,6 +52,7 @@ extern "C" {
#include "libguile/finalizers.h"
#include "libguile/fluids.h"
#include "libguile/foreign.h"
+#include "libguile/foreign-object.h"
#include "libguile/fports.h"
#include "libguile/gc.h"
#include "libguile/gdbint.h"
diff --git a/libguile/Makefile.am b/libguile/Makefile.am
index 5decd99..2bdf71f 100644
--- a/libguile/Makefile.am
+++ b/libguile/Makefile.am
@@ -147,6 +147,7 @@ libguile_@GUILE_EFFECTIVE_VERSION@_la_SOURCES = \
finalizers.c \
fluids.c \
foreign.c \
+ foreign-object.c \
fports.c \
frames.c \
gc-malloc.c \
@@ -573,6 +574,7 @@ modinclude_HEADERS = \
filesys.h \
fluids.h \
foreign.h \
+ foreign-object.h \
fports.h \
frames.h \
gc.h \
diff --git a/libguile/foreign-object.c b/libguile/foreign-object.c
new file mode 100644
index 0000000..78b017a
--- /dev/null
+++ b/libguile/foreign-object.c
@@ -0,0 +1,187 @@
+/* Copyright (C) 2014 Free Software Foundation, Inc.
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public License
+ * as published by the Free Software Foundation; either version 3 of
+ * the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+ * 02110-1301 USA
+ */
+
+
+\f
+#ifdef HAVE_CONFIG_H
+# include <config.h>
+#endif
+
+#include "libguile/_scm.h"
+#include "libguile/goops.h"
+#include "libguile/foreign-object.h"
+
+
+\f
+
+static SCM make_fobj_type_var;
+
+static void
+init_make_fobj_type_var (void)
+{
+ make_fobj_type_var = scm_c_private_lookup ("system foreign-object",
+ "make-foreign-object-type");
+}
+
+SCM
+scm_make_foreign_object_type (SCM name, SCM slot_names,
+ scm_t_struct_finalize finalizer)
+{
+ SCM type;
+
+ static scm_i_pthread_once_t once = SCM_I_PTHREAD_ONCE_INIT;
+ scm_i_pthread_once (&once, init_make_fobj_type_var);
+
+ type = scm_call_2 (scm_variable_ref (make_fobj_type_var), name, slot_names);
+
+ if (finalizer)
+ SCM_SET_VTABLE_INSTANCE_FINALIZER (type, finalizer);
+
+ return type;
+}
+
+void
+scm_assert_foreign_object_type (SCM type, SCM val)
+{
+ if (!SCM_IS_A_P (val, type))
+ scm_error (scm_arg_type_key, NULL, "Wrong type (expecting ~A): ~S",
+ scm_list_2 (scm_class_name (type), val), scm_list_1 (val));
+}
+
+SCM
+scm_make_foreign_object_1 (SCM type, scm_t_bits val0)
+{
+ return scm_make_foreign_object_n (type, 1, &val0);
+}
+
+SCM
+scm_make_foreign_object_2 (SCM type, scm_t_bits val0, scm_t_bits val1)
+{
+ scm_t_bits vals[2] = { val0, val1 };
+
+ return scm_make_foreign_object_n (type, 2, vals);
+}
+
+SCM
+scm_make_foreign_object_3 (SCM type, scm_t_bits val0, scm_t_bits val1,
+ scm_t_bits val2)
+{
+ scm_t_bits vals[3] = { val0, val1, val2 };
+
+ return scm_make_foreign_object_n (type, 3, vals);
+}
+
+SCM
+scm_make_foreign_object_n (SCM type, size_t n, scm_t_bits vals[])
+#define FUNC_NAME "make-foreign-object"
+{
+ SCM obj;
+ SCM layout;
+ size_t i;
+
+ SCM_VALIDATE_VTABLE (SCM_ARG1, type);
+
+ layout = SCM_VTABLE_LAYOUT (type);
+
+ if (scm_i_symbol_length (layout) / 2 < n)
+ scm_out_of_range (FUNC_NAME, scm_from_size_t (n));
+
+ for (i = 0; i < n; i++)
+ if (scm_i_symbol_ref (layout, i * 2) != 'u')
+ scm_wrong_type_arg_msg (FUNC_NAME, 0, layout, "'u' field");
+
+ obj = scm_c_make_structv (type, 0, 0, NULL);
+
+ for (i = 0; i < n; i++)
+ SCM_STRUCT_DATA_SET (obj, i, vals[i]);
+
+ return obj;
+}
+#undef FUNC_NAME
+
+scm_t_bits
+scm_foreign_object_ref (SCM obj, size_t n)
+#define FUNC_NAME "foreign-object-ref"
+{
+ SCM layout;
+
+ SCM_VALIDATE_STRUCT (SCM_ARG1, obj);
+
+ layout = SCM_STRUCT_LAYOUT (obj);
+ if (scm_i_symbol_length (layout) / 2 < n)
+ scm_out_of_range (FUNC_NAME, scm_from_size_t (n));
+
+ if (scm_i_symbol_ref (layout, n * 2) != 'u')
+ scm_wrong_type_arg_msg (FUNC_NAME, 0, layout, "'u' field");
+
+ return SCM_STRUCT_DATA_REF (obj, n);
+}
+#undef FUNC_NAME
+
+void
+scm_foreign_object_set_x (SCM obj, size_t n, scm_t_bits val)
+#define FUNC_NAME "foreign-object-set!"
+{
+ SCM layout;
+
+ SCM_VALIDATE_STRUCT (SCM_ARG1, obj);
+
+ layout = SCM_STRUCT_LAYOUT (obj);
+ if (scm_i_symbol_length (layout) / 2 < n)
+ scm_out_of_range (FUNC_NAME, scm_from_size_t (n));
+
+ if (scm_i_symbol_ref (layout, n * 2) != 'u')
+ scm_wrong_type_arg_msg (FUNC_NAME, 0, layout, "'u' field");
+
+ SCM_STRUCT_DATA_SET (obj, n, val);
+}
+#undef FUNC_NAME
+
+static void
+invoke_finalizer (void *obj, void *data)
+{
+ scm_call_1 (PTR2SCM (data), PTR2SCM (obj));
+}
+
+static SCM
+sys_add_finalizer_x (SCM obj, SCM finalizer)
+#define FUNC_NAME "%add-finalizer!"
+{
+ SCM_VALIDATE_PROC (SCM_ARG2, finalizer);
+
+ scm_i_add_finalizer (SCM2PTR (obj), invoke_finalizer, SCM2PTR (finalizer));
+
+ return SCM_UNSPECIFIED;
+}
+#undef FUNC_NAME
+
+static void
+scm_init_foreign_object (void)
+{
+ scm_c_define_gsubr ("%add-finalizer!", 2, 0, 0,
+ (scm_t_subr) sys_add_finalizer_x);
+}
+
+void
+scm_register_foreign_object (void)
+{
+ scm_c_register_extension ("libguile-" SCM_EFFECTIVE_VERSION,
+ "scm_init_foreign_object",
+ (scm_t_extension_init_func)scm_init_foreign_object,
+ NULL);
+}
diff --git a/libguile/foreign-object.h b/libguile/foreign-object.h
new file mode 100644
index 0000000..fadb3b5
--- /dev/null
+++ b/libguile/foreign-object.h
@@ -0,0 +1,48 @@
+#ifndef SCM_FOREIGN_OBJECT_H
+#define SCM_FOREIGN_OBJECT_H
+
+/* Copyright (C) 2014 Free Software Foundation, Inc.
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public License
+ * as published by the Free Software Foundation; either version 3 of
+ * the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+ * 02110-1301 USA
+ */
+
+\f
+
+#include "libguile/__scm.h"
+#include "libguile/print.h"
+
+
+\f
+
+SCM_API SCM scm_make_foreign_object_type (SCM name, SCM slot_names,
+ scm_t_struct_finalize finalizer);
+
+SCM_API void scm_assert_foreign_object_type (SCM type, SCM val);
+
+SCM_API SCM scm_make_foreign_object_1 (SCM type, scm_t_bits val0);
+SCM_API SCM scm_make_foreign_object_2 (SCM type, scm_t_bits val0,
+ scm_t_bits val1);
+SCM_API SCM scm_make_foreign_object_3 (SCM type, scm_t_bits val0,
+ scm_t_bits val1, scm_t_bits val2);
+SCM_API SCM scm_make_foreign_object_n (SCM type, size_t n, scm_t_bits vals[]);
+
+SCM_API scm_t_bits scm_foreign_object_ref (SCM obj, size_t n);
+SCM_API void scm_foreign_object_set_x (SCM obj, size_t n, scm_t_bits val);
+
+SCM_INTERNAL void scm_register_foreign_object (void);
+
+
+#endif /* SCM_FOREIGN_OBJECT_H */
diff --git a/libguile/init.c b/libguile/init.c
index b320360..87a6988 100644
--- a/libguile/init.c
+++ b/libguile/init.c
@@ -397,6 +397,7 @@ scm_i_init_guile (void *base)
scm_bootstrap_vm ();
scm_register_r6rs_ports ();
scm_register_foreign ();
+ scm_register_foreign_object ();
scm_register_srfi_1 ();
scm_register_srfi_60 ();
scm_register_poll ();
diff --git a/module/Makefile.am b/module/Makefile.am
index fb9174b..521318b 100644
--- a/module/Makefile.am
+++ b/module/Makefile.am
@@ -361,6 +361,7 @@ SYSTEM_SOURCES = \
system/vm/trap-state.scm \
system/vm/vm.scm \
system/foreign.scm \
+ system/foreign-object.scm \
system/xref.scm \
system/repl/debug.scm \
system/repl/error-handling.scm \
diff --git a/module/system/foreign-object.scm b/module/system/foreign-object.scm
new file mode 100644
index 0000000..319b0f4
--- /dev/null
+++ b/module/system/foreign-object.scm
@@ -0,0 +1,88 @@
+;;; Wrapping foreign objects in Scheme
+
+;;; Copyright (C) 2014 Free Software Foundation, Inc.
+;;;
+;;; This library is free software; you can redistribute it and/or
+;;; modify it under the terms of the GNU Lesser General Public
+;;; License as published by the Free Software Foundation; either
+;;; version 3 of the License, or (at your option) any later version.
+;;;
+;;; This library is distributed in the hope that it will be useful,
+;;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+;;; Lesser General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU Lesser General Public
+;;; License along with this library; if not, write to the Free Software
+;;; Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+;;;
+
+;;; Commentary:
+;;
+;;
+;;; Code:
+
+(define-module (system foreign-object)
+ #:use-module (oop goops)
+ #:export (make-foreign-object-type
+ define-foreign-object-type))
+
+(eval-when (eval load expand)
+ (load-extension (string-append "libguile-" (effective-version))
+ "scm_init_foreign_object"))
+
+(define-class <finalizer-class> (<class>)
+ (finalizer #:init-keyword #:finalizer #:init-value #f
+ #:getter finalizer))
+
+(define-method (allocate-instance (class <finalizer-class>) initargs)
+ (let ((instance (next-method))
+ (finalizer (finalizer class)))
+ (when finalizer
+ (%add-finalizer! instance finalizer))
+ instance))
+
+(define (getter-method class slot-name existing)
+ (let ((getter (ensure-generic existing slot-name))
+ (slot-def (or (assq slot-name (slot-ref class 'getters-n-setters))
+ (slot-missing class slot-name))))
+ (add-method! getter (compute-getter-method class slot-def))
+ getter))
+
+(define* (make-foreign-object-type name slots #:key finalizer)
+ (unless (symbol? name)
+ (error "type name should be a symbol" name))
+ (unless (or (not finalizer) (procedure? finalizer))
+ (error "finalizer should be a procedure" finalizer))
+ (let ((dslots (map (lambda (slot)
+ (unless (symbol? slot)
+ (error "slot name should be a symbol" slot))
+ (list slot #:class <foreign-slot>
+ #:init-keyword (symbol->keyword slot)
+ #:init-value 0))
+ slots)))
+ (if finalizer
+ (make-class '() dslots #:name name
+ #:finalizer finalizer #:metaclass <finalizer-class>)
+ (make-class '() dslots #:name name))))
+
+(define-syntax define-foreign-object-type
+ (lambda (x)
+ (define (kw-apply slots)
+ (syntax-case slots ()
+ (() #'())
+ ((slot . slots)
+ (let ((kw (symbol->keyword (syntax->datum #'slot))))
+ #`(#,kw slot . #,(kw-apply #'slots))))))
+
+ (syntax-case x ()
+ ((_ name constructor (slot ...) kwarg ...)
+ #`(begin
+ (define name
+ (make-foreign-object-type 'name '(slot ...) kwarg ...))
+ (define slot
+ (getter-method name 'slot (and (defined? 'slot) slot)))
+ ...
+ (define constructor
+ (lambda (slot ...)
+ (make name #,@(kw-apply #'(slot ...))))))))))
diff --git a/test-suite/standalone/Makefile.am b/test-suite/standalone/Makefile.am
index 7c4633a..9360f69 100644
--- a/test-suite/standalone/Makefile.am
+++ b/test-suite/standalone/Makefile.am
@@ -129,6 +129,17 @@ TESTS += test-ffi
endif HAVE_SHARED_LIBRARIES
+# test-foreign-object-scm
+check_SCRIPTS += test-foreign-object-scm
+TESTS += test-foreign-object-scm
+
+# test-foreign-object-c
+test_foreign_object_c_SOURCES = test-foreign-object-c.c
+test_foreign_object_c_CFLAGS = ${test_cflags}
+test_foreign_object_c_LDADD = $(LIBGUILE_LDADD)
+check_PROGRAMS += test-foreign-object-c
+TESTS += test-foreign-object-c
+
# test-list
test_list_SOURCES = test-list.c
test_list_CFLAGS = ${test_cflags}
diff --git a/test-suite/standalone/test-foreign-object-c.c b/test-suite/standalone/test-foreign-object-c.c
new file mode 100644
index 0000000..9cd8d67
--- /dev/null
+++ b/test-suite/standalone/test-foreign-object-c.c
@@ -0,0 +1,116 @@
+/* test-foreign-object-c.c - exercise C foreign object interface */
+
+/* Copyright (C) 2014 Free Software Foundation, Inc.
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public License
+ * as published by the Free Software Foundation; either version 3 of
+ * the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+ * 02110-1301 USA
+ */
+
+#ifdef HAVE_CONFIG_H
+# include <config.h>
+#endif
+
+#include <libguile.h>
+
+#include <stdlib.h>
+#include <stdio.h>
+#include <string.h>
+
+enum
+ {
+ CSTR_SLOT_ADDR,
+ CSTR_SLOT_LEN,
+ CSTR_SLOT_COUNT
+ };
+
+static void
+finalizer (SCM obj)
+{
+ scm_t_bits addr = scm_foreign_object_ref (obj, CSTR_SLOT_ADDR);
+ free ((void *) addr);
+}
+
+static SCM
+make_cstr_from_static (SCM type, const char *str)
+{
+ char *ours = strdup (str);
+
+ if (!ours)
+ abort ();
+
+ return scm_make_foreign_object_2 (type, (scm_t_bits) ours, strlen (ours));
+}
+
+static int
+cstr_equals_static_p (SCM cstr, const char *str)
+{
+ const char *addr;
+ size_t len;
+
+ addr = (const char *) scm_foreign_object_ref (cstr, CSTR_SLOT_ADDR);
+ len = scm_foreign_object_ref (cstr, CSTR_SLOT_LEN);
+
+ if (strlen (str) != len)
+ return 0;
+
+ return strncmp (addr, str, len) == 0;
+}
+
+static void
+test_scm_foreign_object (void)
+{
+ SCM type_name, slot_names, type, cstr;
+
+ type_name = scm_from_utf8_symbol ("<cstr>");
+ slot_names = scm_list_2 (scm_from_utf8_symbol ("addr"),
+ scm_from_utf8_symbol ("len"));
+ type = scm_make_foreign_object_type (type_name, slot_names, finalizer);
+
+ cstr = make_cstr_from_static (type, "Hello, world!");
+ scm_assert_foreign_object_type (type, cstr);
+
+ if (!cstr_equals_static_p (cstr, "Hello, world!"))
+ {
+ fprintf (stderr, "fail: test-foreign-object 1\n");
+ exit (EXIT_FAILURE);
+ }
+
+ {
+ int i;
+ for (i = 0; i < 5000; i++)
+ cstr = make_cstr_from_static (type, "Hello, world!");
+ cstr = SCM_BOOL_F;
+ }
+
+ scm_gc ();
+ scm_gc ();
+ scm_gc ();
+
+ /* Allow time for the finalizer thread to run. */
+ scm_usleep (scm_from_uint (50 * 1000));
+}
+
+static void
+tests (void *data, int argc, char **argv)
+{
+ test_scm_foreign_object ();
+}
+
+int
+main (int argc, char *argv[])
+{
+ scm_boot_guile (argc, argv, tests, NULL);
+ return 0;
+}
diff --git a/test-suite/standalone/test-foreign-object-scm b/test-suite/standalone/test-foreign-object-scm
new file mode 100755
index 0000000..7e4bd85
--- /dev/null
+++ b/test-suite/standalone/test-foreign-object-scm
@@ -0,0 +1,119 @@
+#!/bin/sh
+exec guile -q -s "$0" "$@"
+!#
+;;; test-foreign-object-scm --- Foreign object interface. -*- Scheme -*-
+;;;
+;;; Copyright (C) 2014 Free Software Foundation, Inc.
+;;;
+;;; This library is free software; you can redistribute it and/or
+;;; modify it under the terms of the GNU Lesser General Public
+;;; License as published by the Free Software Foundation; either
+;;; version 3 of the License, or (at your option) any later version.
+;;;
+;;; This library is distributed in the hope that it will be useful,
+;;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+;;; Lesser General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU Lesser General Public
+;;; License along with this library; if not, write to the Free Software
+;;; Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+(use-modules (system foreign)
+ (system foreign-object)
+ (rnrs bytevectors)
+ (oop goops))
+
+(define (libc-ptr name)
+ (catch #t
+ (lambda () (dynamic-pointer name (dynamic-link)))
+ (lambda (k . args)
+ (print-exception (current-error-port) #f k args)
+ (write "Skipping test.\n" (current-error-port))
+ (exit 0))))
+
+(define malloc (pointer->procedure '* (libc-ptr "malloc") (list size_t)))
+(define memcpy (pointer->procedure void (libc-ptr "memcpy") (list '* '* size_t)))
+(define free (pointer->procedure void (libc-ptr "free") '(*)))
+
+(define (finalize-cstr cstr)
+ (free (make-pointer (addr cstr))))
+
+(define-foreign-object-type <cstr> make-cstr (addr len)
+ #:finalizer finalize-cstr)
+
+(define (cstr->string cstr)
+ (pointer->string (make-pointer (addr cstr)) (len cstr) "UTF-8"))
+
+(define* (string->cstr str #:optional (k make-cstr))
+ (let* ((bv (string->utf8 str))
+ (len (bytevector-length bv))
+ (mem (malloc len)))
+ (when (null-pointer? mem)
+ (error "Out of memory."))
+ (memcpy mem (bytevector->pointer bv) len)
+ (k (pointer-address mem) len)))
+
+(define-method (write (cstr <cstr>) port)
+ (format port "<<cstr> ~s>" (cstr->string cstr)))
+
+(define-method (display (cstr <cstr>) port)
+ (display (cstr->string cstr) port))
+
+(define-method (+ (a <cstr>) (b <cstr>))
+ (string->cstr (string-append (cstr->string a) (cstr->string b))))
+
+(define-method (equal? (a <cstr>) (b <cstr>))
+ (equal? (cstr->string a) (cstr->string b)))
+
+(define failed? #f)
+(define-syntax test
+ (syntax-rules ()
+ ((_ exp res)
+ (let ((expected res)
+ (actual exp))
+ (if (not (equal? actual expected))
+ (begin
+ (set! failed? #t)
+ (format (current-error-port)
+ "bad return from expression `~a': expected ~A; got ~A~%"
+ 'exp expected actual)))))))
+
+(test (string->cstr "Hello, world!")
+ (+ (string->cstr "Hello, ") (string->cstr "world!")))
+
+;; GOOPS construction syntax instead of make-cstr.
+(test (string->cstr "Hello, world!")
+ (string->cstr "Hello, world!"
+ (lambda (addr len)
+ (make <cstr> #:addr addr #:len len))))
+
+;; Subclassing.
+(define-class <wrapped-cstr> (<cstr>)
+ (wrapped-string #:init-keyword #:wrapped-string
+ #:getter wrapped-string
+ #:init-form (error "missing #:wrapped-string")))
+
+(define (string->wrapped-cstr string)
+ (string->cstr string (lambda (addr len)
+ (make <wrapped-cstr> #:addr addr #:len len
+ #:wrapped-string string))))
+
+(let ((wrapped-cstr (string->wrapped-cstr "Hello, world!")))
+ ;; Tests that <cst> methods work on <wrapped-cstr>.
+ (test "Hello, world!" (cstr->string wrapped-cstr))
+ ;; Test the additional #:wrapped-string slot.
+ (test "Hello, world!" (wrapped-string wrapped-cstr)))
+
+(gc) (gc) (gc)
+
+;; Sleep 50 milliseconds to allow the finalization thread to run.
+(usleep #e50e3)
+
+;; But we don't really know if it ran. Oh well.
+
+(exit (if failed? 1 0))
+
+;; Local Variables:
+;; mode: scheme
+;; End:
--
2.0.0.rc0
[-- Attachment #3: Type: text/plain, Size: 25 bytes --]
--
http://wingolog.org/
^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: RFC: Foreign objects facility
2014-04-27 13:17 RFC: Foreign objects facility Andy Wingo
@ 2014-04-27 16:00 ` Mark H Weaver
2014-04-27 16:46 ` Stefan Israelsson Tampe
` (2 more replies)
2014-04-28 8:24 ` Ludovic Courtès
2014-04-29 15:56 ` Doug Evans
2 siblings, 3 replies; 14+ messages in thread
From: Mark H Weaver @ 2014-04-27 16:00 UTC (permalink / raw)
To: Andy Wingo; +Cc: guile-devel
Hi Andy,
Andy Wingo <wingo@pobox.com> writes:
> I propose to provide a new interface that will eventually make SMOBs
> obsolete. This new interface is based on structs with raw fields -- the
> 'u' fields. (See
> http://www.gnu.org/software/guile/docs/master/guile.html/Vtables.html#Vtables
> for description of 'u' fields. Note that the documentation is wrong --
> these fields are indeed traced by the GC.)
Sounds like a good idea, in general.
> Here is the proposed C API:
>
> SCM scm_make_foreign_object_type (SCM name, SCM slot_names,
> scm_t_struct_finalize finalizer);
Shouldn't it be 'scm_t_struct_finalizer', with trailing 'r'?
> void scm_assert_foreign_object_type (SCM type, SCM val);
>
> SCM scm_make_foreign_object_1 (SCM type, scm_t_bits val0);
> SCM scm_make_foreign_object_2 (SCM type, scm_t_bits val0,
> scm_t_bits val1);
> SCM scm_make_foreign_object_3 (SCM type, scm_t_bits val0,
> scm_t_bits val1, scm_t_bits val2);
If we include an interface like this, I think we should have more of
these, but see below.
> SCM scm_make_foreign_object_n (SCM type, size_t n, scm_t_bits vals[]);
>
> scm_t_bits scm_foreign_object_ref (SCM obj, size_t n);
> void scm_foreign_object_set_x (SCM obj, size_t n, scm_t_bits val);
I'm worried about the type-punning that's implied by this interface.
The C standards are now very strict about this sort of thing, and modern
compilers are becoming increasingly aggressive about taking advantage of
these restrictions to derive assumptions about the code.
For example, having recently researched this, I know that converting
unsigned integers to signed integers is very awkward and potentially
inefficient to do portably, so using 'scm_foreign_object_ref' to extract
a signed integer will be a pain. It would look something like this
(untested):
--8<---------------cut here---------------start------------->8---
scm_t_signed_bits
scm_foreign_object_signed_ref (SCM obj, size_t n)
{
scm_t_bits bits = scm_foreign_object_ref (obj, n);
/* GNU C specifies that when casting to a signed integer of width N, the
value is reduced modulo 2^N to be within range of the type. Neither
C99 nor C11 make any guarantees about casting an out-of-range value
to a signed integer type. */
#ifndef __GNUC__
if (bits > SCM_T_SIGNED_BITS_MAX)
return -1 - (scm_t_signed_bits) ~bits;
#endif
return (scm_t_signed_bits) bits;
}
--8<---------------cut here---------------end--------------->8---
Portability is more problematic for pointer types. The C standards make
no guarantees about the semantics of converting between pointers and
integers, and it's not clear to me how future proof this will be. One
related issue is that I'd like to leave open the possibility that
'scm_t_bits' might be larger than a pointer.
For slot refs and sets, I suggest that we need to have multiple variants
for different types. At the very least, we'll need variants for signed
integers, unsigned integers, and pointers.
For constructors, I think we should provide a macro that handles the
conversion from pointer to scm_t_bits. (Converting from signed to
unsigned is portable, so there's no issue there).
> The overhead of a foreign object is two words -- the same as the
> overhead on any struct. (Compare to SMOBs, which have a half-word
> overhead.)
It would be good to think about how we might someday reduce this
overhead in the future, and to make sure we don't make decisions that
would prevent us from doing that.
> From a12efcfaae1c65cc703616ea15106a88efba3f55 Mon Sep 17 00:00:00 2001
> From: Andy Wingo <wingo@pobox.com>
> Date: Sun, 27 Apr 2014 14:47:40 +0200
> Subject: [PATCH] New foreign object facility, to replace SMOBs
>
> * libguile/foreign-object.c:
> * libguile/foreign-object.h:
> * module/system/foreign-object.scm:
> * test-suite/standalone/test-foreign-object-c.c:
> * test-suite/standalone/test-foreign-object-scm: New files.
> * test-suite/standalone/Makefile.am:
Something should be added to the line above.
> diff --git a/libguile/foreign-object.c b/libguile/foreign-object.c
> new file mode 100644
> index 0000000..78b017a
> --- /dev/null
> +++ b/libguile/foreign-object.c
> @@ -0,0 +1,187 @@
> +/* Copyright (C) 2014 Free Software Foundation, Inc.
> + *
> + * This library is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU Lesser General Public License
> + * as published by the Free Software Foundation; either version 3 of
> + * the License, or (at your option) any later version.
> + *
> + * This library is distributed in the hope that it will be useful, but
> + * WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> + * Lesser General Public License for more details.
> + *
> + * You should have received a copy of the GNU Lesser General Public
> + * License along with this library; if not, write to the Free Software
> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
> + * 02110-1301 USA
> + */
> +
> +
> +\f
> +#ifdef HAVE_CONFIG_H
> +# include <config.h>
> +#endif
> +
> +#include "libguile/_scm.h"
> +#include "libguile/goops.h"
> +#include "libguile/foreign-object.h"
> +
> +
> +\f
> +
> +static SCM make_fobj_type_var;
> +
> +static void
> +init_make_fobj_type_var (void)
> +{
> + make_fobj_type_var = scm_c_private_lookup ("system foreign-object",
> + "make-foreign-object-type");
> +}
> +
> +SCM
> +scm_make_foreign_object_type (SCM name, SCM slot_names,
> + scm_t_struct_finalize finalizer)
> +{
> + SCM type;
> +
> + static scm_i_pthread_once_t once = SCM_I_PTHREAD_ONCE_INIT;
> + scm_i_pthread_once (&once, init_make_fobj_type_var);
Good!
> +
> + type = scm_call_2 (scm_variable_ref (make_fobj_type_var), name, slot_names);
> +
> + if (finalizer)
> + SCM_SET_VTABLE_INSTANCE_FINALIZER (type, finalizer);
> +
> + return type;
> +}
> +
> +void
> +scm_assert_foreign_object_type (SCM type, SCM val)
> +{
> + if (!SCM_IS_A_P (val, type))
> + scm_error (scm_arg_type_key, NULL, "Wrong type (expecting ~A): ~S",
> + scm_list_2 (scm_class_name (type), val), scm_list_1 (val));
> +}
> +
> +SCM
> +scm_make_foreign_object_1 (SCM type, scm_t_bits val0)
> +{
> + return scm_make_foreign_object_n (type, 1, &val0);
> +}
> +
> +SCM
> +scm_make_foreign_object_2 (SCM type, scm_t_bits val0, scm_t_bits val1)
> +{
> + scm_t_bits vals[2] = { val0, val1 };
This non-constant initializer depends on C99. We discussed this on IRC,
and IIRC we agreed that we should transition to requiring C99 for 2.2,
but making that change in 2.0 is potentially disruptive.
> +
> + return scm_make_foreign_object_n (type, 2, vals);
> +}
> +
> +SCM
> +scm_make_foreign_object_3 (SCM type, scm_t_bits val0, scm_t_bits val1,
> + scm_t_bits val2)
> +{
> + scm_t_bits vals[3] = { val0, val1, val2 };
Ditto.
> +
> + return scm_make_foreign_object_n (type, 3, vals);
> +}
> +
> +SCM
> +scm_make_foreign_object_n (SCM type, size_t n, scm_t_bits vals[])
> +#define FUNC_NAME "make-foreign-object"
> +{
> + SCM obj;
> + SCM layout;
> + size_t i;
> +
> + SCM_VALIDATE_VTABLE (SCM_ARG1, type);
> +
> + layout = SCM_VTABLE_LAYOUT (type);
> +
> + if (scm_i_symbol_length (layout) / 2 < n)
> + scm_out_of_range (FUNC_NAME, scm_from_size_t (n));
> +
> + for (i = 0; i < n; i++)
> + if (scm_i_symbol_ref (layout, i * 2) != 'u')
> + scm_wrong_type_arg_msg (FUNC_NAME, 0, layout, "'u' field");
This looks inefficient. How about using 'scm_i_symbol_chars'?
> +
> + obj = scm_c_make_structv (type, 0, 0, NULL);
> +
> + for (i = 0; i < n; i++)
> + SCM_STRUCT_DATA_SET (obj, i, vals[i]);
> +
> + return obj;
> +}
> +#undef FUNC_NAME
[...]
Thanks!
Mark
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: RFC: Foreign objects facility
2014-04-27 16:00 ` Mark H Weaver
@ 2014-04-27 16:46 ` Stefan Israelsson Tampe
2014-04-28 17:47 ` Andy Wingo
2014-04-27 17:51 ` Andy Wingo
2014-04-28 16:08 ` Andy Wingo
2 siblings, 1 reply; 14+ messages in thread
From: Stefan Israelsson Tampe @ 2014-04-27 16:46 UTC (permalink / raw)
To: Mark H Weaver; +Cc: Andy Wingo, guile-devel
[-- Attachment #1: Type: text/plain, Size: 9137 bytes --]
Currently only having 256 smob types is enough arguments for the additions.
This can only be a good thing even if I need to the current smob for
modelling prolog variables (the overhead of 2 words is a bit heavy if
you only have one data slot). So it is brilliant to dispatch all the
multitude of larger smob types to the slightly
heavier interface.
FYI I need need the gc mark interface for my prolog variables in order to
hope for garbage collecting of prolog programs, e.g. clean the variables
bindings in the prolog stack for variables that no longer is referenced. It
is possible to even skip this if I could probe if a SCM variable is
unreferenced (but still gc protected)
/Stefan
On Sun, Apr 27, 2014 at 6:00 PM, Mark H Weaver <mhw@netris.org> wrote:
> Hi Andy,
>
> Andy Wingo <wingo@pobox.com> writes:
>
> > I propose to provide a new interface that will eventually make SMOBs
> > obsolete. This new interface is based on structs with raw fields -- the
> > 'u' fields. (See
> >
> http://www.gnu.org/software/guile/docs/master/guile.html/Vtables.html#Vtables
> > for description of 'u' fields. Note that the documentation is wrong --
> > these fields are indeed traced by the GC.)
>
> Sounds like a good idea, in general.
>
> > Here is the proposed C API:
> >
> > SCM scm_make_foreign_object_type (SCM name, SCM slot_names,
> > scm_t_struct_finalize finalizer);
>
> Shouldn't it be 'scm_t_struct_finalizer', with trailing 'r'?
>
> > void scm_assert_foreign_object_type (SCM type, SCM val);
> >
> > SCM scm_make_foreign_object_1 (SCM type, scm_t_bits val0);
> > SCM scm_make_foreign_object_2 (SCM type, scm_t_bits val0,
> > scm_t_bits val1);
> > SCM scm_make_foreign_object_3 (SCM type, scm_t_bits val0,
> > scm_t_bits val1, scm_t_bits val2);
>
> If we include an interface like this, I think we should have more of
> these, but see below.
>
> > SCM scm_make_foreign_object_n (SCM type, size_t n, scm_t_bits
> vals[]);
> >
> > scm_t_bits scm_foreign_object_ref (SCM obj, size_t n);
> > void scm_foreign_object_set_x (SCM obj, size_t n, scm_t_bits val);
>
> I'm worried about the type-punning that's implied by this interface.
> The C standards are now very strict about this sort of thing, and modern
> compilers are becoming increasingly aggressive about taking advantage of
> these restrictions to derive assumptions about the code.
>
> For example, having recently researched this, I know that converting
> unsigned integers to signed integers is very awkward and potentially
> inefficient to do portably, so using 'scm_foreign_object_ref' to extract
> a signed integer will be a pain. It would look something like this
> (untested):
>
> --8<---------------cut here---------------start------------->8---
> scm_t_signed_bits
> scm_foreign_object_signed_ref (SCM obj, size_t n)
> {
> scm_t_bits bits = scm_foreign_object_ref (obj, n);
>
> /* GNU C specifies that when casting to a signed integer of width N, the
> value is reduced modulo 2^N to be within range of the type. Neither
> C99 nor C11 make any guarantees about casting an out-of-range value
> to a signed integer type. */
>
> #ifndef __GNUC__
> if (bits > SCM_T_SIGNED_BITS_MAX)
> return -1 - (scm_t_signed_bits) ~bits;
> #endif
> return (scm_t_signed_bits) bits;
> }
> --8<---------------cut here---------------end--------------->8---
>
> Portability is more problematic for pointer types. The C standards make
> no guarantees about the semantics of converting between pointers and
> integers, and it's not clear to me how future proof this will be. One
> related issue is that I'd like to leave open the possibility that
> 'scm_t_bits' might be larger than a pointer.
>
> For slot refs and sets, I suggest that we need to have multiple variants
> for different types. At the very least, we'll need variants for signed
> integers, unsigned integers, and pointers.
>
> For constructors, I think we should provide a macro that handles the
> conversion from pointer to scm_t_bits. (Converting from signed to
> unsigned is portable, so there's no issue there).
>
> > The overhead of a foreign object is two words -- the same as the
> > overhead on any struct. (Compare to SMOBs, which have a half-word
> > overhead.)
>
> It would be good to think about how we might someday reduce this
> overhead in the future, and to make sure we don't make decisions that
> would prevent us from doing that.
>
> > From a12efcfaae1c65cc703616ea15106a88efba3f55 Mon Sep 17 00:00:00 2001
> > From: Andy Wingo <wingo@pobox.com>
> > Date: Sun, 27 Apr 2014 14:47:40 +0200
> > Subject: [PATCH] New foreign object facility, to replace SMOBs
> >
> > * libguile/foreign-object.c:
> > * libguile/foreign-object.h:
> > * module/system/foreign-object.scm:
> > * test-suite/standalone/test-foreign-object-c.c:
> > * test-suite/standalone/test-foreign-object-scm: New files.
> > * test-suite/standalone/Makefile.am:
>
> Something should be added to the line above.
>
> > diff --git a/libguile/foreign-object.c b/libguile/foreign-object.c
> > new file mode 100644
> > index 0000000..78b017a
> > --- /dev/null
> > +++ b/libguile/foreign-object.c
> > @@ -0,0 +1,187 @@
> > +/* Copyright (C) 2014 Free Software Foundation, Inc.
> > + *
> > + * This library is free software; you can redistribute it and/or
> > + * modify it under the terms of the GNU Lesser General Public License
> > + * as published by the Free Software Foundation; either version 3 of
> > + * the License, or (at your option) any later version.
> > + *
> > + * This library is distributed in the hope that it will be useful, but
> > + * WITHOUT ANY WARRANTY; without even the implied warranty of
> > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> > + * Lesser General Public License for more details.
> > + *
> > + * You should have received a copy of the GNU Lesser General Public
> > + * License along with this library; if not, write to the Free Software
> > + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
> > + * 02110-1301 USA
> > + */
> > +
> > +
> > +
> > +#ifdef HAVE_CONFIG_H
> > +# include <config.h>
> > +#endif
> > +
> > +#include "libguile/_scm.h"
> > +#include "libguile/goops.h"
> > +#include "libguile/foreign-object.h"
> > +
> > +
> > +
> > +
> > +static SCM make_fobj_type_var;
> > +
> > +static void
> > +init_make_fobj_type_var (void)
> > +{
> > + make_fobj_type_var = scm_c_private_lookup ("system foreign-object",
> > +
> "make-foreign-object-type");
> > +}
> > +
> > +SCM
> > +scm_make_foreign_object_type (SCM name, SCM slot_names,
> > + scm_t_struct_finalize finalizer)
> > +{
> > + SCM type;
> > +
> > + static scm_i_pthread_once_t once = SCM_I_PTHREAD_ONCE_INIT;
> > + scm_i_pthread_once (&once, init_make_fobj_type_var);
>
> Good!
>
> > +
> > + type = scm_call_2 (scm_variable_ref (make_fobj_type_var), name,
> slot_names);
> > +
> > + if (finalizer)
> > + SCM_SET_VTABLE_INSTANCE_FINALIZER (type, finalizer);
> > +
> > + return type;
> > +}
> > +
> > +void
> > +scm_assert_foreign_object_type (SCM type, SCM val)
> > +{
> > + if (!SCM_IS_A_P (val, type))
> > + scm_error (scm_arg_type_key, NULL, "Wrong type (expecting ~A): ~S",
> > + scm_list_2 (scm_class_name (type), val), scm_list_1
> (val));
> > +}
> > +
> > +SCM
> > +scm_make_foreign_object_1 (SCM type, scm_t_bits val0)
> > +{
> > + return scm_make_foreign_object_n (type, 1, &val0);
> > +}
> > +
> > +SCM
> > +scm_make_foreign_object_2 (SCM type, scm_t_bits val0, scm_t_bits val1)
> > +{
> > + scm_t_bits vals[2] = { val0, val1 };
>
> This non-constant initializer depends on C99. We discussed this on IRC,
> and IIRC we agreed that we should transition to requiring C99 for 2.2,
> but making that change in 2.0 is potentially disruptive.
>
> > +
> > + return scm_make_foreign_object_n (type, 2, vals);
> > +}
> > +
> > +SCM
> > +scm_make_foreign_object_3 (SCM type, scm_t_bits val0, scm_t_bits val1,
> > + scm_t_bits val2)
> > +{
> > + scm_t_bits vals[3] = { val0, val1, val2 };
>
> Ditto.
>
> > +
> > + return scm_make_foreign_object_n (type, 3, vals);
> > +}
> > +
> > +SCM
> > +scm_make_foreign_object_n (SCM type, size_t n, scm_t_bits vals[])
> > +#define FUNC_NAME "make-foreign-object"
> > +{
> > + SCM obj;
> > + SCM layout;
> > + size_t i;
> > +
> > + SCM_VALIDATE_VTABLE (SCM_ARG1, type);
> > +
> > + layout = SCM_VTABLE_LAYOUT (type);
> > +
> > + if (scm_i_symbol_length (layout) / 2 < n)
> > + scm_out_of_range (FUNC_NAME, scm_from_size_t (n));
> > +
> > + for (i = 0; i < n; i++)
> > + if (scm_i_symbol_ref (layout, i * 2) != 'u')
> > + scm_wrong_type_arg_msg (FUNC_NAME, 0, layout, "'u' field");
>
> This looks inefficient. How about using 'scm_i_symbol_chars'?
>
> > +
> > + obj = scm_c_make_structv (type, 0, 0, NULL);
> > +
> > + for (i = 0; i < n; i++)
> > + SCM_STRUCT_DATA_SET (obj, i, vals[i]);
> > +
> > + return obj;
> > +}
> > +#undef FUNC_NAME
> [...]
>
> Thanks!
> Mark
>
>
[-- Attachment #2: Type: text/html, Size: 11546 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: RFC: Foreign objects facility
2014-04-27 16:00 ` Mark H Weaver
2014-04-27 16:46 ` Stefan Israelsson Tampe
@ 2014-04-27 17:51 ` Andy Wingo
2014-05-03 4:57 ` Doug Evans
2014-04-28 16:08 ` Andy Wingo
2 siblings, 1 reply; 14+ messages in thread
From: Andy Wingo @ 2014-04-27 17:51 UTC (permalink / raw)
To: Mark H Weaver; +Cc: guile-devel
Hi Mark,
Thanks for the review!
On Sun 27 Apr 2014 18:00, Mark H Weaver <mhw@netris.org> writes:
> Andy Wingo <wingo@pobox.com> writes:
>
>> Here is the proposed C API:
>>
>> SCM scm_make_foreign_object_type (SCM name, SCM slot_names,
>> scm_t_struct_finalize finalizer);
>
> Shouldn't it be 'scm_t_struct_finalizer', with trailing 'r'?
Probably it should, but note that it's already defined -- see struct.h.
(My bad there, I think!)
>> SCM scm_make_foreign_object_n (SCM type, size_t n, scm_t_bits vals[]);
>>
>> scm_t_bits scm_foreign_object_ref (SCM obj, size_t n);
>> void scm_foreign_object_set_x (SCM obj, size_t n, scm_t_bits val);
>
> I'm worried about the type-punning that's implied by this interface.
This issue exists already with the SMOB and struct interfaces. I don't
think that the proposed API adds new hazards, though it does extend the
old ones.
> The C standards are now very strict about this sort of thing,
It's an incredible shame that C is making itself semantically less
capable as time goes on :-(
> For example, having recently researched this, I know that converting
> unsigned integers to signed integers is very awkward and potentially
> inefficient to do portably, so using 'scm_foreign_object_ref' to extract
> a signed integer will be a pain. It would look something like this
> (untested):
>
> scm_t_signed_bits
> scm_foreign_object_signed_ref (SCM obj, size_t n)
> {
> scm_t_bits bits = scm_foreign_object_ref (obj, n);
>
> /* GNU C specifies that when casting to a signed integer of width N, the
> value is reduced modulo 2^N to be within range of the type. Neither
> C99 nor C11 make any guarantees about casting an out-of-range value
> to a signed integer type. */
>
> #ifndef __GNUC__
> if (bits > SCM_T_SIGNED_BITS_MAX)
> return -1 - (scm_t_signed_bits) ~bits;
> #endif
> return (scm_t_signed_bits) bits;
> }
True sadness.
> Portability is more problematic for pointer types. The C standards make
> no guarantees about the semantics of converting between pointers and
> integers, and it's not clear to me how future proof this will be.
Don't they make some guarantees wrt uintptr_t and intptr_t ?
> One related issue is that I'd like to leave open the possibility that
> 'scm_t_bits' might be larger than a pointer.
You prefer to have three interface then: one for uintptr_t, one for
intptr_t, and one for void*?
> For constructors, I think we should provide a macro that handles the
> conversion from pointer to scm_t_bits. (Converting from signed to
> unsigned is portable, so there's no issue there).
This is really gross...
>> The overhead of a foreign object is two words -- the same as the
>> overhead on any struct. (Compare to SMOBs, which have a half-word
>> overhead.)
>
> It would be good to think about how we might someday reduce this
> overhead in the future, and to make sure we don't make decisions that
> would prevent us from doing that.
This is an orthogonal issue, I think, and would like to punt on that for
now.
>> + scm_t_bits vals[2] = { val0, val1 };
>
> This non-constant initializer depends on C99.
Will fix.
>> + if (scm_i_symbol_ref (layout, i * 2) != 'u')
>> + scm_wrong_type_arg_msg (FUNC_NAME, 0, layout, "'u' field");
>
> This looks inefficient. How about using 'scm_i_symbol_chars'?
OK.
Cheers,
Andy
--
http://wingolog.org/
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: RFC: Foreign objects facility
2014-04-27 13:17 RFC: Foreign objects facility Andy Wingo
2014-04-27 16:00 ` Mark H Weaver
@ 2014-04-28 8:24 ` Ludovic Courtès
2014-04-28 18:05 ` Andy Wingo
2014-04-29 15:56 ` Doug Evans
2 siblings, 1 reply; 14+ messages in thread
From: Ludovic Courtès @ 2014-04-28 8:24 UTC (permalink / raw)
To: guile-devel
Hello!
Andy Wingo <wingo@pobox.com> skribis:
> 7) There is legacy code out there that uses e.g. SCM_SETCDR to set
> smob fields. (This is terrible, but it exists:
> https://github.com/search?q=SCM_SETCDR+smob&ref=cmdform&type=Code
> for an example.)
[...]
> I propose to provide a new interface that will eventually make SMOBs
> obsolete. This new interface is based on structs with raw fields -- the
> 'u' fields. (See
> http://www.gnu.org/software/guile/docs/master/guile.html/Vtables.html#Vtables
> for description of 'u' fields. Note that the documentation is wrong --
> these fields are indeed traced by the GC.)
I like the idea or basic the new facility on structs.
I would have preferred to rebase SMOBs on top of structs, with the added
documentation as to how they can be accessed from Scheme, but I suppose
this is ruled out by the SCM_SETCDR issue above?
> So, what do people think? The patch below is against stable-2.0.
The API looks good to me. The C code could use comments above
functions, but otherwise nothing more than what Mark said about the
patch.
Thanks!
Ludo’.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: RFC: Foreign objects facility
2014-04-27 16:00 ` Mark H Weaver
2014-04-27 16:46 ` Stefan Israelsson Tampe
2014-04-27 17:51 ` Andy Wingo
@ 2014-04-28 16:08 ` Andy Wingo
2014-04-29 16:08 ` Doug Evans
2 siblings, 1 reply; 14+ messages in thread
From: Andy Wingo @ 2014-04-28 16:08 UTC (permalink / raw)
To: Mark H Weaver; +Cc: guile-devel
[-- Attachment #1: Type: text/plain, Size: 1794 bytes --]
On Sun 27 Apr 2014 18:00, Mark H Weaver <mhw@netris.org> writes:
> Andy Wingo <wingo@pobox.com> writes:
>
>> I propose to provide a new interface that will eventually make SMOBs
>> obsolete. This new interface is based on structs with raw fields -- the
>> 'u' fields. (See
>> http://www.gnu.org/software/guile/docs/master/guile.html/Vtables.html#Vtables
>> for description of 'u' fields. Note that the documentation is wrong --
>> these fields are indeed traced by the GC.)
>
> Sounds like a good idea, in general.
Thanks; with this note, Ludo's approbation, and a doc update I feel OK
with pushing to stable-2.0. Let's hammer out the details there.
>> SCM scm_make_foreign_object_1 (SCM type, scm_t_bits val0);
>> SCM scm_make_foreign_object_2 (SCM type, scm_t_bits val0,
>> scm_t_bits val1);
>> SCM scm_make_foreign_object_3 (SCM type, scm_t_bits val0,
>> scm_t_bits val1, scm_t_bits val2);
>
> If we include an interface like this, I think we should have more of
> these, but see below.
I changed this interface to take void* pointers, with the idea that
that's the common thing. Same with scm_foreign_object_ref. But I added
scm_foreign_object_signed_ref, unsigned_set_x, etc, with notes in the
manual that you can indeed provide fewer initializers than fields, and
then initialize the fields yourself later.
> #ifndef __GNUC__
> if (bits > SCM_T_SIGNED_BITS_MAX)
> return -1 - (scm_t_signed_bits) ~bits;
> #endif
> return (scm_t_signed_bits) bits;
> }
I think I forgot to actually implement this bit though :P
Tutorial-style documentation (in the Programming in C chapter, replacing
the SMOB tutorial) attached. There is also reference-style
documentation in the API chapter.
Andy
[-- Attachment #2: libguile-foreign-objects.txt --]
[-- Type: text/plain, Size: 18311 bytes --]
1 Defining New Foreign Object Types
===================================
The "foreign object type" facility is Guile's mechanism for importing
object and types from C or other languages into Guile's system. If you
have a C 'struct foo' type, for example, you can define a corresponding
Guile foreign object type that allows Scheme code to handle 'struct foo
*' objects.
To define a new foreign object type, the programmer provides Guile
with some essential information about the type -- what its name is, how
many fields it has, and its finalizer (if any) -- and Guile allocates a
fresh type for it. Foreign objects can be accessed from Scheme or from
C.
1.1 Defining Foreign Object Types
---------------------------------
To create a new foreign object type from C, call
'scm_make_foreign_object_type'. It returns a value of type 'SCM' which
identifies the new type.
Here is how one might declare a new type representing eight-bit
gray-scale images:
#include <libguile.h>
struct image {
int width, height;
char *pixels;
/* The name of this image */
SCM name;
/* A function to call when this image is
modified, e.g., to update the screen,
or SCM_BOOL_F if no action necessary */
SCM update_func;
};
static SCM image_type image_type;
void
init_image_type (void)
{
SCM name, slots;
scm_t_struct_finalize finalizer;
name = scm_from_utf8_symbol ("image");
slots = scm_list_1 (scm_from_utf8_symbol ("data"));
finalizer = NULL;
image_type =
scm_make_foreign_object_type (name, slots, finalizer);
}
The result is an initialized 'image_type' value that identifies the
new foreign object type. The next section describes how to create
foreign objects and how to access their slots.
1.2 Creating Foreign Objects
----------------------------
Foreign objects contain zero or more "slots" of data. A slot can hold a
pointer, an integer that fits into a 'size_t' or 'ssize_t', or a 'SCM'
value.
All objects of a given foreign type have the same number of slots.
In the example from the previous section, the 'image' type has one slot,
because the slots list passed to 'scm_make_foreign_object_type' is of
length one. (The actual names given to slots are unimportant for most
users of the C interface, but can be used on the Scheme side to
introspect on the foreign object.)
To construct a foreign object and initialize its first slot, call
'scm_make_foreign_object_1 (TYPE, FIRST_SLOT_VALUE)'. There are
similarly named constructors for initializing 0, 1, 2, or 3 slots, or
initializing N slots via an array. *Note Foreign Objects::, for full
details. Any fields that are not explicitly initialized are set to 0.
To get or set the value of a slot by index, you can use the
'scm_foreign_object_ref' and 'scm_foreign_object_set_x' functions.
These functions take and return values as 'void *' pointers; there are
corresponding convenience procedures like '_signed_ref',
'_unsigned_set_x' and so on for dealing with slots as signed or unsigned
integers.
Foreign objects fields that are pointers can be tricky to manage. If
possible, it is best that all memory that is referenced by a foreign
object be managed by the garbage collector. That way, the GC can
automatically ensure that memory is accessible when it is needed, and
freed when it becomes inaccessible. If this is not the case for your
program - for example, if you are exposing an object to Scheme that was
allocated by some other, Guile-unaware part of your program - then you
will probably need to implement a finalizer. *Note Foreign Object
Memory Management::, for more.
Continuing the example from the previous section, if the global
variable 'image_type' contains the type returned by
'scm_make_foreign_object_type', here is how we could construct a foreign
object whose "data" field contains a pointer to a freshly allocated
'struct image':
SCM
make_image (SCM name, SCM s_width, SCM s_height)
{
struct image *image;
int width = scm_to_int (s_width);
int height = scm_to_int (s_height);
/* Allocate the `struct image'. Because we
use scm_gc_malloc, this memory block will
be automatically reclaimed when it becomes
inaccessible, and its members will be traced
by the garbage collector. */
image = (struct image *)
scm_gc_malloc (sizeof (struct image), "image");
image->width = width;
image->height = height;
/* Allocating the pixels with
scm_gc_malloc_pointerless means that the
pixels data is collectable by GC, but
that GC shouldn't spend time tracing its
contents for nested pointers because there
aren't any. */
image->pixels =
scm_gc_malloc_pointerless (width * height, "image pixels");
image->name = name;
image->update_func = SCM_BOOL_F;
/* Now wrap the struct image* in a new foreign
object, and return that object. */
return scm_make_foreign_object_1 (image_type, image);
}
We use 'scm_gc_malloc_pointerless' for the pixel buffer to tell the
garbage collector not to scan it for pointers. Calls to
'scm_gc_malloc', 'scm_make_foreign_object_1', and
'scm_gc_malloc_pointerless' raise an exception in out-of-memory
conditions; the garbage collector is able to reclaim previously
allocated memory if that happens.
1.3 Type Checking of Foreign Objects
------------------------------------
Functions that operate on foreign objects should check that the passed
'SCM' value indeed is of the correct type before accessing its data.
They can do this with 'scm_assert_foreign_object_type'.
For example, here is a simple function that operates on an image
object, and checks the type of its argument.
SCM
clear_image (SCM image_obj)
{
int area;
struct image *image;
scm_assert_foreign_object_type (image_type, image_obj);
image = scm_foreign_object_ref (image_obj, 0);
area = image->width * image->height;
memset (image->pixels, 0, area);
/* Invoke the image's update function. */
if (scm_is_true (image->update_func))
scm_call_0 (image->update_func);
return SCM_UNSPECIFIED;
}
1.4 Foreign Object Memory Management
------------------------------------
Once a foreign object has been released to the tender mercies of the
Scheme system, it must be prepared to survive garbage collection. In
the example above, all the memory associated with the foreign object is
managed by the garbage collector because we used the 'scm_gc_'
allocation functions. Thus, no special care must be taken: the garbage
collector automatically scans them and reclaims any unused memory.
However, when data associated with a foreign object is managed in
some other way--e.g., 'malloc''d memory or file descriptors--it is
possible to specify a "finalizer" function to release those resources
when the foreign object is reclaimed.
As discussed in *note Garbage Collection::, Guile's garbage collector
will reclaim inaccessible memory as needed. This reclamation process
runs concurrently with the main program. When Guile analyzes the heap
and determines that an object's memory can be reclaimed, that memory is
put on a "free list" of objects that can be reclaimed. Usually that's
the end of it--the object is available for immediate re-use. However
some objects can have "finalizers" associated with them--functions that
are called on reclaimable objects to effect any external cleanup
actions.
Finalizers are tricky business and it is best to avoid them. They
can be invoked at unexpected times, or not at all--for example, they are
not invoked on process exit. They don't help the garbage collector do
its job; in fact, they are a hindrance. Furthermore, they perturb the
garbage collector's internal accounting. The GC decides to scan the
heap when it thinks that it is necessary, after some amount of
allocation. Finalizable objects almost always represent an amount of
allocation that is invisible to the garbage collector. The effect can
be that the actual resource usage of a system with finalizable objects
is higher than what the GC thinks it should be.
All those caveats aside, some foreign object types will need
finalizers. For example, if we had a foreign object type that wrapped
file descriptors--and we aren't suggesting this, as Guile already has
ports --then you might define the type like this:
static SCM file_type;
static void
finalize_file (SCM file)
{
int fd = scm_foreign_object_signed_ref (file, 0);
if (fd >= 0)
{
scm_foreign_object_signed_set_x (file, 0, -1);
close (fd);
}
}
static void
init_file_type (void)
{
SCM name, slots;
scm_t_struct_finalize finalizer;
name = scm_from_utf8_symbol ("file");
slots = scm_list_1 (scm_from_utf8_symbol ("fd"));
finalizer = finalize_file;
image_type =
scm_make_foreign_object_type (name, slots, finalizer);
}
static SCM
make_file (int fd)
{
return scm_make_foreign_object_1 (file_type, (void *) fd);
}
Note that the finalizer may be invoked in ways and at times you might
not expect. In particular, if the user's Guile is built with support
for threads, the finalizer may be called from any thread that is running
Guile. In Guile 2.0, finalizers are invoked via "asyncs", which
interleaves them with running Scheme code; *note System asyncs::. In
Guile 2.2 there will be a dedicated finalization thread, to ensure that
the finalization doesn't run within the critical section of any other
thread known to Guile.
In either case, finalizers run concurrently with the main program,
and so they need to be async-safe and thread-safe. If for some reason
this is impossible, perhaps because you are embedding Guile in some
application that is not itself thread-safe, you have a few options. One
is to use guardians instead of finalizers, and arrange to pump the
guardians for finalizable objects. *Note Guardians::, for more
information. The other option is to disable automatic finalization
entirely, and arrange to call 'scm_run_finalizers ()' at appropriate
points. *Note Foreign Objects::, for more on these interfaces.
Finalizers are allowed to allocate memory, access GC-managed memory,
and in general can do anything any Guile user code can do. This was not
the case in Guile 1.8, where finalizers were much more restricted. In
particular, in Guile 2.0, finalizers can resuscitate objects. We do not
recommend that users avail themselves of this possibility, however, as a
resuscitated object can re-expose other finalizable objects that have
been already finalized back to Scheme. These objects will not be
finalized again, but they could cause use-after-free problems to code
that handles objects of that particular foreign object type. To guard
against this possibility, robust finalization routines should clear
state from the foreign object, as in the above 'free_file' example.
One final caveat. Foreign object finalizers are associated with the
lifetime of a foreign object, not of its fields. If you access a field
of a finalizable foreign object, and do not arrange to keep a reference
on the foreign object itself, it could be that the outer foreign object
gets finalized while you are working with its field.
For example, consider a procedure to read some data from a file, from
our example above.
SCM
read_bytes (SCM file, SCM n)
{
int fd;
SCM buf;
size_t len, pos;
scm_assert_foreign_object_type (file_type, file);
fd = scm_foreign_object_signed_ref (file, 0);
if (fd < 0)
scm_wrong_type_arg_msg ("read-bytes", SCM_ARG1,
file, "open file");
len = scm_to_size_t (n);
SCM buf = scm_c_make_bytevector (scm_to_size_t (n));
pos = 0;
while (pos < len)
{
char *bytes = SCM_BYTEVECTOR_CONTENTS (buf);
ssize_t count = read (fd, bytes + pos, len - pos);
if (count < 0)
scm_syserror ("read-bytes");
if (count == 0)
break;
pos += count;
}
scm_remember_upto_here_1 (file);
return scm_values (scm_list_2 (buf, scm_from_size_t (pos)));
}
After the prelude, only the 'fd' value is used and the C compiler has
no reason to keep the 'file' object around. If 'scm_c_make_bytevector'
results in a garbage collection, 'file' might not be on the stack or
anywhere else and could be finalized, leaving 'read' to read a closed
(or, in a multi-threaded program, possibly re-used) file descriptor.
The use of 'scm_remember_upto_here_1' prevents this, by creating a
reference to 'file' after all data accesses. *Note Garbage Collection
Functions::.
'scm_remember_upto_here_1' is only needed on finalizable objects,
because garbage collection of other values is invisible to the program -
it happens when needed, and is not observable. But if you can, save
yourself the headache and build your program in such a way that it
doesn't need finalization.
1.5 Foreign Objects and Scheme
------------------------------
It is also possible to create foreign objects and object types from
Scheme, and to access fields of foreign objects from Scheme. For
example, the file example from the last section could be equivalently
expressed as:
(define-module (my-file)
#:use-module (system foreign-object)
#:use-module ((oop goops) #:select (make))
#:export (make-file))
(define (finalize-file file)
(let ((fd (struct-ref file 0)))
(unless (< fd 0)
(struct-set! file 0 -1)
(close-fdes fd))))
(define <file>
(make-foreign-object-type '<file> '(fd)
#:finalizer finalize-file))
(define (make-file fd)
(make <file> #:fd fd))
Here we see that the result of 'make-foreign-object-type', which is
the equivalent of 'scm_make_foreign_object_type', is a struct vtable.
*Note Vtables::, for more information. To instantiate the foreign
object, which is really a Guile struct, we use 'make'. (We could have
used 'make-struct/no-tail', but as an implementation detail, finalizers
are attached in the 'initialize' method called by 'make'). To access
the fields, we use 'struct-ref' and 'struct-set!'. *Note Structure
Basics::.
There is a convenience syntax, 'define-foreign-object-type', that
defines a type along with a constructor, and getters for the fields. An
appropriate invocation of 'define-foreign-object-type' for the file
object type could look like this:
(use-modules (system foreign-object))
(define-foreign-object-type <file>
make-file
(fd)
#:finalizer finalize-file)
This defines the '<file>' type with one field, a 'make-file'
constructor, and a getter for the 'fd' field, bound to 'fd'.
Foreign object types are not only vtables but are actually GOOPS
classes, as hinted at above. *Note GOOPS::, for more on Guile's
object-oriented programming system. Thus one can define print and
equality methods using GOOPS:
(use-modules (oop goops))
(define-method (write (file <file>) port)
;; Assuming existence of the `fd' getter
(format port "#<<file> ~a>" (fd file)))
(define-method (equal? (a <file>) (b <file>))
(eqv? (fd a) (fd b)))
One can even sub-class foreign types.
(define-class <named-file> (<file>)
(name #:init-keyword #:name #:init-value #f #:accessor name))
The question arises of how to construct these values, given that
'make-file' returns a plain old '<file>' object. It turns out that you
can use the GOOPS construction interface, where every field of the
foreign object has an associated initialization keyword argument.
(define* (my-open-file name #:optional (flags O_RDONLY))
(make <named-file> #:fd (open-fdes name flags) #:name name))
(define-method (write (file <named-file>) port)
(format port "#<<file> ~s ~a>" (name file) (fd file)))
*Note Foreign Objects::, for full documentation on the Scheme
interface to foreign objects. *Note GOOPS::, for more on GOOPS.
As a final note, you might wonder how this system supports
encapsulation of sensitive values. First, we have to recognize that
some facilities are essentially unsafe and have global scope. For
example, in C, the integrity and confidentiality of a part of a program
is at the mercy of every other part of that program - because any part
of the program can read and write anything in its address space. At the
same time, principled access to structured data is organized in C on
lexical boundaries; if you don't expose accessors for your object, you
trust other parts of the program not to work around that barrier.
The situation is not dissimilar in Scheme. Although Scheme's unsafe
constructs are fewer in number than in C, they do exist. The '(system
foreign)' module can be used to violate confidentiality and integrity,
and shouldn't be exposed to untrusted code. Although 'struct-ref' and
'struct-set!' are less unsafe, they still have a cross-cutting
capability of drilling through abstractions. Performing a 'struct-set!'
on a foreign object slot could cause unsafe foreign code to crash.
Ultimately, structures in Scheme are capabilities for abstraction, and
not abstractions themselves.
That leaves us with the lexical capabilities, like constructors and
accessors. Here is where encapsulation lies: the practical degree to
which the innards of your foreign objects are exposed is the degree to
which their accessors are lexically available in user code. If you want
to allow users to reference fields of your foreign object, provide them
with a getter. Otherwise you should assume that the only access to your
object may come from your code, which has the relevant authority, or via
code with access to cross-cutting 'struct-ref' and such, which also has
the cross-cutting authority.
[-- Attachment #3: Type: text/plain, Size: 26 bytes --]
--
http://wingolog.org/
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: RFC: Foreign objects facility
2014-04-27 16:46 ` Stefan Israelsson Tampe
@ 2014-04-28 17:47 ` Andy Wingo
0 siblings, 0 replies; 14+ messages in thread
From: Andy Wingo @ 2014-04-28 17:47 UTC (permalink / raw)
To: Stefan Israelsson Tampe; +Cc: Mark H Weaver, guile-devel
On Sun 27 Apr 2014 18:46, Stefan Israelsson Tampe <stefan.itampe@gmail.com> writes:
> FYI I need need the gc mark interface for my prolog variables in order
> to hope for garbage collecting of prolog programs, e.g. clean the
> variables bindings in the prolog stack for variables that no longer is
> referenced. It is possible to even skip this if I could probe if a SCM
> variable is unreferenced (but still gc protected)
Whatever we do, this sort of thing will remain supported as long as we
are using the BDW GC, via the BDW GC API. You might look there for your
needs in this case; perhaps they have something that Guile doesn't
expose.
Andy
--
http://wingolog.org/
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: RFC: Foreign objects facility
2014-04-28 8:24 ` Ludovic Courtès
@ 2014-04-28 18:05 ` Andy Wingo
0 siblings, 0 replies; 14+ messages in thread
From: Andy Wingo @ 2014-04-28 18:05 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: guile-devel
On Mon 28 Apr 2014 10:24, ludo@gnu.org (Ludovic Courtès) writes:
> I would have preferred to rebase SMOBs on top of structs, with the added
> documentation as to how they can be accessed from Scheme, but I suppose
> this is ruled out by the SCM_SETCDR issue above?
We can think of doing this in the future, at least as a
mostly-compatible shim. The fundamental differences are really the mark
functions and the lack of "apply" capability. Getting "apply" working
does sound possible though, given applicable structs.
Regards,
Andy
--
http://wingolog.org/
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: RFC: Foreign objects facility
2014-04-27 13:17 RFC: Foreign objects facility Andy Wingo
2014-04-27 16:00 ` Mark H Weaver
2014-04-28 8:24 ` Ludovic Courtès
@ 2014-04-29 15:56 ` Doug Evans
2014-04-29 18:25 ` Andy Wingo
2 siblings, 1 reply; 14+ messages in thread
From: Doug Evans @ 2014-04-29 15:56 UTC (permalink / raw)
To: Andy Wingo; +Cc: guile-devel
On Sun, Apr 27, 2014 at 6:17 AM, Andy Wingo <wingo@pobox.com> wrote:
> Hi,
>
> SMOBs have a few problems.
>
> 1) They are limited in number to 255.
>
> 2) It's difficult to refer to a SMOB type from Scheme. You can use
> class-of once you have an object, but the class-of isn't exactly
> the same as the SMOB tc16, and getting the type beforehand is
> gnarly. (See http://article.gmane.org/gmane.comp.gdb.patches/96857
> for an example).
>
> 3) You can't create SMOB types from Scheme. This goes against our
> general trend of making Scheme more powerful.
>
> 4) You can't create SMOB objects from Scheme.
>
> 5) Similarly, you can't access SMOB fields from Scheme.
>
> 6) You can't subclass SMOB types. (Some people would like this
> ability.)
>
> 7) There is legacy code out there that uses e.g. SCM_SETCDR to set
> smob fields. (This is terrible, but it exists:
> https://github.com/search?q=SCM_SETCDR+smob&ref=cmdform&type=Code
> for an example.)
>
> 8) The single/double SMOB thing is outdated and bogus. Objects should
> be able to have any number of fields.
>
> 9) We document mark functions in the manual, even recommending them,
> but they are really difficult to get right (see
> https://lists.gnu.org/archive/html/guile-user/2011-11/msg00069.html),
> and almost always a bad design -- the BDW GC can do a better job
> without them.
>
> And yet, if we consider the generic problem of wrapping C objects in
> Scheme objects, it's clear that we have more solutions now than we used
> to -- raw #<pointer> values, define-wrapped-pointer-type, etc. But
> there's nothing that's accessible to C like SMOBs are, so people that
> use the libguile interface to wrap C types and values are out of luck.
>
> I propose to provide a new interface that will eventually make SMOBs
> obsolete. This new interface is based on structs with raw fields -- the
> 'u' fields. (See
> http://www.gnu.org/software/guile/docs/master/guile.html/Vtables.html#Vtables
> for description of 'u' fields. Note that the documentation is wrong --
> these fields are indeed traced by the GC.)
>
> Here is the proposed C API:
>
> SCM scm_make_foreign_object_type (SCM name, SCM slot_names,
> scm_t_struct_finalize finalizer);
>
> void scm_assert_foreign_object_type (SCM type, SCM val);
>
> SCM scm_make_foreign_object_1 (SCM type, scm_t_bits val0);
> SCM scm_make_foreign_object_2 (SCM type, scm_t_bits val0,
> scm_t_bits val1);
> SCM scm_make_foreign_object_3 (SCM type, scm_t_bits val0,
> scm_t_bits val1, scm_t_bits val2);
> SCM scm_make_foreign_object_n (SCM type, size_t n, scm_t_bits vals[]);
>
> scm_t_bits scm_foreign_object_ref (SCM obj, size_t n);
> void scm_foreign_object_set_x (SCM obj, size_t n, scm_t_bits val);
>
> The finalizer may be NULL. The scm_make_foreign_object* functions are
> just scm_make_struct without the no_tail arguments, and interpreting the
> values as raw untagged values, not unpacked SCM values. Same thing with
> scm_foreign_object_ref/set_x.
The struct interface, contrary to what the documentation says, takes a
stdarg list beginning with the number of fields (and not terminated
with SCM_UNDEFINED), instead of _1, _2, _3. fwiw, I would use that
over the _n version. I'd probably even use it instead of _1,_2,_3,
but I know some people like the comfortableness of having the compiler
verify the number of args is correct. I'm not sure if gcc could be
extended to handle verifying stdarg functions where one parameter
specifies the number of stdarg parms (assuming it doesn't support that
already).
fwiw, structs already provide most of what is needed.
I have a few gdb objects using them, and it doesn't seem too bad.
I think you should fix/extend the struct interface instead of
inventing something new: given that structs already have
"uninterpreted" it is most of the way there already. It's already
intended to solve the problem you're trying to solve.
Its API just needs a way to set/ref "uninterpreted" values directly.
Plus making foreign objects completely usable from Scheme kinda
removes any usefulness of having "foreign" in the name.
With a bit of tweaking of the struct interface you're done.
So why invent something new?
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: RFC: Foreign objects facility
2014-04-28 16:08 ` Andy Wingo
@ 2014-04-29 16:08 ` Doug Evans
2014-04-29 18:27 ` Andy Wingo
0 siblings, 1 reply; 14+ messages in thread
From: Doug Evans @ 2014-04-29 16:08 UTC (permalink / raw)
To: Andy Wingo; +Cc: Mark H Weaver, guile-devel
On Mon, Apr 28, 2014 at 9:08 AM, Andy Wingo <wingo@pobox.com> wrote:
> [...]
> 1.4 Foreign Object Memory Management
> ------------------------------------
>
> Once a foreign object has been released to the tender mercies of the
> Scheme system, it must be prepared to survive garbage collection. In
> the example above, all the memory associated with the foreign object is
> managed by the garbage collector because we used the 'scm_gc_'
> allocation functions. Thus, no special care must be taken: the garbage
> collector automatically scans them and reclaims any unused memory.
>
> However, when data associated with a foreign object is managed in
> some other way--e.g., 'malloc''d memory or file descriptors--it is
> possible to specify a "finalizer" function to release those resources
> when the foreign object is reclaimed.
>
> As discussed in *note Garbage Collection::, Guile's garbage collector
> will reclaim inaccessible memory as needed. This reclamation process
> runs concurrently with the main program. When Guile analyzes the heap
> and determines that an object's memory can be reclaimed, that memory is
> put on a "free list" of objects that can be reclaimed. Usually that's
> the end of it--the object is available for immediate re-use. However
> some objects can have "finalizers" associated with them--functions that
> are called on reclaimable objects to effect any external cleanup
> actions.
>
> Finalizers are tricky business and it is best to avoid them. They
> can be invoked at unexpected times, or not at all--for example, they are
> not invoked on process exit. They don't help the garbage collector do
> its job; in fact, they are a hindrance. Furthermore, they perturb the
> garbage collector's internal accounting. The GC decides to scan the
> heap when it thinks that it is necessary, after some amount of
> allocation. Finalizable objects almost always represent an amount of
> allocation that is invisible to the garbage collector. The effect can
> be that the actual resource usage of a system with finalizable objects
> is higher than what the GC thinks it should be.
>
> All those caveats aside, some foreign object types will need
> finalizers. For example, if we had a foreign object type that wrapped
> file descriptors--and we aren't suggesting this, as Guile already has
> ports --then you might define the type like this:
>
> static SCM file_type;
>
> static void
> finalize_file (SCM file)
> {
> int fd = scm_foreign_object_signed_ref (file, 0);
> if (fd >= 0)
> {
> scm_foreign_object_signed_set_x (file, 0, -1);
> close (fd);
> }
> }
>
> static void
> init_file_type (void)
> {
> SCM name, slots;
> scm_t_struct_finalize finalizer;
>
> name = scm_from_utf8_symbol ("file");
> slots = scm_list_1 (scm_from_utf8_symbol ("fd"));
> finalizer = finalize_file;
>
> image_type =
> scm_make_foreign_object_type (name, slots, finalizer);
> }
>
> static SCM
> make_file (int fd)
> {
> return scm_make_foreign_object_1 (file_type, (void *) fd);
> }
>
> Note that the finalizer may be invoked in ways and at times you might
> not expect. In particular, if the user's Guile is built with support
> for threads, the finalizer may be called from any thread that is running
> Guile. In Guile 2.0, finalizers are invoked via "asyncs", which
> interleaves them with running Scheme code; *note System asyncs::. In
> Guile 2.2 there will be a dedicated finalization thread, to ensure that
> the finalization doesn't run within the critical section of any other
> thread known to Guile.
>
> In either case, finalizers run concurrently with the main program,
> and so they need to be async-safe and thread-safe. If for some reason
> this is impossible, perhaps because you are embedding Guile in some
> application that is not itself thread-safe, you have a few options. One
> is to use guardians instead of finalizers, and arrange to pump the
> guardians for finalizable objects. *Note Guardians::, for more
> information. The other option is to disable automatic finalization
> entirely, and arrange to call 'scm_run_finalizers ()' at appropriate
> points. *Note Foreign Objects::, for more on these interfaces.
>
> Finalizers are allowed to allocate memory, access GC-managed memory,
> and in general can do anything any Guile user code can do. This was not
> the case in Guile 1.8, where finalizers were much more restricted. In
> particular, in Guile 2.0, finalizers can resuscitate objects. We do not
> recommend that users avail themselves of this possibility, however, as a
> resuscitated object can re-expose other finalizable objects that have
> been already finalized back to Scheme. These objects will not be
> finalized again, but they could cause use-after-free problems to code
> that handles objects of that particular foreign object type. To guard
> against this possibility, robust finalization routines should clear
> state from the foreign object, as in the above 'free_file' example.
>
> One final caveat. Foreign object finalizers are associated with the
> lifetime of a foreign object, not of its fields. If you access a field
> of a finalizable foreign object, and do not arrange to keep a reference
> on the foreign object itself, it could be that the outer foreign object
> gets finalized while you are working with its field.
>
> For example, consider a procedure to read some data from a file, from
> our example above.
>
> SCM
> read_bytes (SCM file, SCM n)
> {
> int fd;
> SCM buf;
> size_t len, pos;
>
> scm_assert_foreign_object_type (file_type, file);
>
> fd = scm_foreign_object_signed_ref (file, 0);
> if (fd < 0)
> scm_wrong_type_arg_msg ("read-bytes", SCM_ARG1,
> file, "open file");
>
> len = scm_to_size_t (n);
> SCM buf = scm_c_make_bytevector (scm_to_size_t (n));
>
> pos = 0;
> while (pos < len)
> {
> char *bytes = SCM_BYTEVECTOR_CONTENTS (buf);
> ssize_t count = read (fd, bytes + pos, len - pos);
> if (count < 0)
> scm_syserror ("read-bytes");
> if (count == 0)
> break;
> pos += count;
> }
>
> scm_remember_upto_here_1 (file);
>
> return scm_values (scm_list_2 (buf, scm_from_size_t (pos)));
> }
>
> After the prelude, only the 'fd' value is used and the C compiler has
> no reason to keep the 'file' object around. If 'scm_c_make_bytevector'
> results in a garbage collection, 'file' might not be on the stack or
> anywhere else and could be finalized, leaving 'read' to read a closed
> (or, in a multi-threaded program, possibly re-used) file descriptor.
> The use of 'scm_remember_upto_here_1' prevents this, by creating a
> reference to 'file' after all data accesses. *Note Garbage Collection
> Functions::.
>
> 'scm_remember_upto_here_1' is only needed on finalizable objects,
> because garbage collection of other values is invisible to the program -
> it happens when needed, and is not observable. But if you can, save
> yourself the headache and build your program in such a way that it
> doesn't need finalization.
Hi. fwiw, this is all great stuff (and welcome!), but I think it's in
the wrong place in the docs.
The issue comes up in multiple places, so I would write it ("it" being
the general prose regarding issues with finalizers) once and refer to
it from all the places that use finalizers.
That would make this section shorter (more likely to be read) and
*everywhere* that has to deal with finalizers benefits from far more
easily finding all this great text.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: RFC: Foreign objects facility
2014-04-29 15:56 ` Doug Evans
@ 2014-04-29 18:25 ` Andy Wingo
2014-05-03 5:45 ` Doug Evans
0 siblings, 1 reply; 14+ messages in thread
From: Andy Wingo @ 2014-04-29 18:25 UTC (permalink / raw)
To: Doug Evans; +Cc: guile-devel
Hi!
Thanks for the feedback, it's really useful.
On Tue 29 Apr 2014 17:56, Doug Evans <xdje42@gmail.com> writes:
> The struct interface, contrary to what the documentation says, takes a
> stdarg list beginning with the number of fields (and not terminated
> with SCM_UNDEFINED), instead of _1, _2, _3
This is what the documentation says (in the info node 'Structure
Basics'):
-- C Function: SCM scm_make_struct (SCM vtable, SCM tail_size, SCM
init_list)
-- C Function: SCM scm_c_make_struct (SCM vtable, SCM tail_size, SCM
init, ...)
-- C Function: SCM scm_c_make_structv (SCM vtable, SCM tail_size,
size_t n_inits, scm_t_bits init[])
I believe this to be correct.
> fwiw, I would use that over the _n version.
I thought so too and so I used this in my first attempt ;) However
scm_make_struct expects the scm_t_bits to actually be unpacked SCM
values -- which is to say, it expects them to be tagged. A bit wonky.
SCM is a perfectly valid varargs type, as evinced by scm_list_n and
friends. But that's how it is and for ABI reasons we can't really
change that.
Then we had the points that Mark brought up about portable type-casting,
and that really we should provide separate signed/unsigned integer
interfaces and also pointer interfaces. It gets into a combinatoric
mess on the constructor level, though perhaps for N<=3 it's manageable.
Anyway then you also have to match the types that are passed as
initializers to the field type (unboxed 'u' or tagged 'p') and it's just
a big mess. I thought that the given foreign object API would cover the
majority of cases, like SMOB cases, and that for multi-field types the
ref/set accessors would be sufficient.
> fwiw, structs already provide most of what is needed.
> I have a few gdb objects using them, and it doesn't seem too bad.
Yes. The API doesn't support this use case so well. It also doesn't
support GOOPS method dispatch, subclassing, or named fields (unless you
start getting into records territory).
> I think you should fix/extend the struct interface instead of
> inventing something new: given that structs already have
> "uninterpreted" it is most of the way there already. It's already
> intended to solve the problem you're trying to solve.
> Its API just needs a way to set/ref "uninterpreted" values directly.
Good point. Though, it can hard to square with the existing smob/struct
cases. You could have scm_struct_unsigned_ref -- would it then untag a
'p' value? It seems to me that there are some combinatorics that we do
away with by saying that "this is a foreign object. It's also a struct,
and a GOOPS instance, and you can treat it that way, but the API we
expose to C is the most useful way to treat these objects."
WDYT?
Andy
p.s. Will reply to your other mail, but see the updated master docs
here:
http://www.gnu.org/software/guile/docs/master/guile.html/Programming-in-C.html#Programming-in-C
http://www.gnu.org/software/guile/docs/master/guile.html/API-Reference.html#API-Reference
--
http://wingolog.org/
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: RFC: Foreign objects facility
2014-04-29 16:08 ` Doug Evans
@ 2014-04-29 18:27 ` Andy Wingo
0 siblings, 0 replies; 14+ messages in thread
From: Andy Wingo @ 2014-04-29 18:27 UTC (permalink / raw)
To: Doug Evans; +Cc: Mark H Weaver, guile-devel
Hi!
On Tue 29 Apr 2014 18:08, Doug Evans <xdje42@gmail.com> writes:
> Hi. fwiw, this is all great stuff (and welcome!), but I think it's in
> the wrong place in the docs.
> The issue comes up in multiple places, so I would write it ("it" being
> the general prose regarding issues with finalizers) once and refer to
> it from all the places that use finalizers.
> That would make this section shorter (more likely to be read) and
> *everywhere* that has to deal with finalizers benefits from far more
> easily finding all this great text.
Perhaps this text fragment was misinterpreted; the stable-2.0 docs have
been updated in git to the same thing, but the master HTML docs have
been pushed to the web site. Note that in:
http://www.gnu.org/software/guile/docs/master/guile.html/Programming-in-C.html#Programming-in-C
I've removed the tutorial-like section on smobs, and that later in the
API reference the SMOB documentation links to "Foreign Object Memory
Management" when discussing the pitfalls of finalizers.
Does that cover your needs?
A
--
http://wingolog.org/
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: RFC: Foreign objects facility
2014-04-27 17:51 ` Andy Wingo
@ 2014-05-03 4:57 ` Doug Evans
0 siblings, 0 replies; 14+ messages in thread
From: Doug Evans @ 2014-05-03 4:57 UTC (permalink / raw)
To: Andy Wingo; +Cc: Mark H Weaver, guile-devel
On Sun, Apr 27, 2014 at 10:51 AM, Andy Wingo <wingo@pobox.com> wrote:
>[...]
>> Portability is more problematic for pointer types. The C standards make
>> no guarantees about the semantics of converting between pointers and
>> integers, and it's not clear to me how future proof this will be.
>
> Don't they make some guarantees wrt uintptr_t and intptr_t ?
Data point: Yes. One can convert from void * to either of these and
back "and the result will compare equal to the original pointer".
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: RFC: Foreign objects facility
2014-04-29 18:25 ` Andy Wingo
@ 2014-05-03 5:45 ` Doug Evans
0 siblings, 0 replies; 14+ messages in thread
From: Doug Evans @ 2014-05-03 5:45 UTC (permalink / raw)
To: Andy Wingo; +Cc: guile-devel
On Tue, Apr 29, 2014 at 11:25 AM, Andy Wingo <wingo@pobox.com> wrote:
> Hi!
>
> Thanks for the feedback, it's really useful.
>
> On Tue 29 Apr 2014 17:56, Doug Evans <xdje42@gmail.com> writes:
>
>> The struct interface, contrary to what the documentation says, takes a
>> stdarg list beginning with the number of fields (and not terminated
>> with SCM_UNDEFINED), instead of _1, _2, _3
>
> This is what the documentation says (in the info node 'Structure
> Basics'):
>
> -- C Function: SCM scm_make_struct (SCM vtable, SCM tail_size, SCM
> init_list)
> -- C Function: SCM scm_c_make_struct (SCM vtable, SCM tail_size, SCM
> init, ...)
> -- C Function: SCM scm_c_make_structv (SCM vtable, SCM tail_size,
> size_t n_inits, scm_t_bits init[])
>
> I believe this to be correct.
Almost. git struct.c has:
SCM
scm_c_make_struct (SCM vtable, size_t n_tail, size_t n_init,
scm_t_bits init, ...)
>> fwiw, I would use that over the _n version.
>
> I thought so too and so I used this in my first attempt ;) However
> scm_make_struct expects the scm_t_bits to actually be unpacked SCM
> values -- which is to say, it expects them to be tagged. A bit wonky.
> SCM is a perfectly valid varargs type, as evinced by scm_list_n and
> friends. But that's how it is and for ABI reasons we can't really
> change that.
I'm not suggesting changing the API of course. :-)
Plus scm_make_struct is callable from Scheme so it doesn't really
support passing "uninterpretered" values (except hackily).
> Then we had the points that Mark brought up about portable type-casting,
> and that really we should provide separate signed/unsigned integer
> interfaces and also pointer interfaces. It gets into a combinatoric
> mess on the constructor level, though perhaps for N<=3 it's manageable.
> Anyway then you also have to match the types that are passed as
> initializers to the field type (unboxed 'u' or tagged 'p') and it's just
> a big mess. I thought that the given foreign object API would cover the
> majority of cases, like SMOB cases, and that for multi-field types the
> ref/set accessors would be sufficient.
Technically, one could provide just intptr/uintptr and skip void*, but
the latter would be sufficiently convenient I think.
re: The combinatoric mess, I'm not suggesting adding that.
Plus, if an app wants to have an SCM in their "foreign object" they
can't store it in a slot, whereas they can with structs.
>> fwiw, structs already provide most of what is needed.
>> I have a few gdb objects using them, and it doesn't seem too bad.
>
> Yes. The API doesn't support this use case so well. It also doesn't
> support GOOPS method dispatch, subclassing, or named fields (unless you
> start getting into records territory).
IIUC, goops already had <foreign-slot>, which is what the foreign
object API uses.
So I still don't see what creating a whole new class of data structure
provides that can't be provided by extending existing ones. And if
there is something, doesn't that mean that the existing ones
(goops/structs) are still broken?
>> I think you should fix/extend the struct interface instead of
>> inventing something new: given that structs already have
>> "uninterpreted" it is most of the way there already. It's already
>> intended to solve the problem you're trying to solve.
>> Its API just needs a way to set/ref "uninterpreted" values directly.
>
> Good point. Though, it can hard to square with the existing smob/struct
> cases. You could have scm_struct_unsigned_ref -- would it then untag a
> 'p' value?
I would flag referencing or setting a 'p' field via the
"uninterpreted" accessors as errors.
> It seems to me that there are some combinatorics that we do
> away with by saying that "this is a foreign object. It's also a struct,
> and a GOOPS instance, and you can treat it that way, but the API we
> expose to C is the most useful way to treat these objects."
OTOH, yet another compound data structure with its own API now exists.
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2014-05-03 5:45 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-04-27 13:17 RFC: Foreign objects facility Andy Wingo
2014-04-27 16:00 ` Mark H Weaver
2014-04-27 16:46 ` Stefan Israelsson Tampe
2014-04-28 17:47 ` Andy Wingo
2014-04-27 17:51 ` Andy Wingo
2014-05-03 4:57 ` Doug Evans
2014-04-28 16:08 ` Andy Wingo
2014-04-29 16:08 ` Doug Evans
2014-04-29 18:27 ` Andy Wingo
2014-04-28 8:24 ` Ludovic Courtès
2014-04-28 18:05 ` Andy Wingo
2014-04-29 15:56 ` Doug Evans
2014-04-29 18:25 ` Andy Wingo
2014-05-03 5:45 ` Doug Evans
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).