Guile foreign object interface

Guile foreign object interface

[Mike Gran]

Hi All-
I wanted to make a quick post about the foreign object interface.
This is a bit of a placeholder because I haven't had time to
investigate the interface properly, yet. But I intend to poke at
it soon.
But for there record, there are some problematic design patterns
that I want to make sure can be covered by the new interface.
Again, I'll try to make proper examples soon, but, with Andy wanting
to release soon, I want to get this down now.

1. First off is a Lilypond-like pattern.  A C++ vector
is used to dynamically add or remove elements.  The memory
management of those elements lives in the C++ world.  Those elements
are boxed up as SCM.  GCing the SCM should not free its boxed
contents, but, deleting the boxed contents from the C++ side
should render those SCMs invalid in some sense. 
2. Mutually owned information. Two structures (A and B) both
mutually contain a non-GC-malloc'd structure C. C must exist
so long as one of A or B exist.
3. Here's one from guile-ncurses. An element ITEM is created with
some contents. Default memory management suffices: if ITEM is GC'd,
its contents are GC'd.  But the contents of ITEM can later be
attached to a COLLECTION. If ITEM is GC'd when attached to
COLLECTION, its contents are not GC'd.
5. Gobject-like subclassing.  In Gtk, an application window is
a type of a window which is a type of a widget which is a type of
object. If we were to resurrect an effort on GTK3 binding,
how would it work for foreign objects?
Again, forgive this random dump. Wanted to get something down now,
but I'll try to make a more sensible post later.
-Mike Gran

Re: Guile foreign object interface

[Andy Wingo]

Hi :)

On Thu 02 Mar 2017 16:47, Mike Gran <spk121@yahoo.com> writes:

> I wanted to make a quick post about the foreign object interface.
> This is a bit of a placeholder because I haven't had time to
> investigate the interface properly, yet. But I intend to poke at
> it soon.
> But for there record, there are some problematic design patterns
> that I want to make sure can be covered by the new interface.

I am not sure that all cases need to be covered by the new interface to
be able to include it.  I think it would be sufficient to simply replace
SMOBs, unrecommend mark procedures and all that, and get the benefits I
mentioned in my mail.  However that said...

> 1. First off is a Lilypond-like pattern.  A C++ vector
> is used to dynamically add or remove elements.  The memory
> management of those elements lives in the C++ world.  Those elements
> are boxed up as SCM.  GCing the SCM should not free its boxed
> contents, but, deleting the boxed contents from the C++ side
> should render those SCMs invalid in some sense. 

Not really helped by foreign objects AFAIU.

> 2. Mutually owned information. Two structures (A and B) both
> mutually contain a non-GC-malloc'd structure C. C must exist
> so long as one of A or B exist.

Likewise, if I understand A and B to be scheme objects of some kind; the
pattern here would be a gc-managed wrapper around C with a finalizer,
and A and B both use the wrapper.

> 3. Here's one from guile-ncurses. An element ITEM is created with
> some contents. Default memory management suffices: if ITEM is GC'd,
> its contents are GC'd.  But the contents of ITEM can later be
> attached to a COLLECTION. If ITEM is GC'd when attached to
> COLLECTION, its contents are not GC'd.

I understand you mean that ITEM has the only reference on its contents,
and that if it becomes unreachable, the contents also become
unreachable, and may be collected (and possibly finalized if they have a
finalizer and finalizers have a chance to run and all the other
finalizer caveats).  Sure, foreign objects work here, as do any other
Scheme data type, right?  As long as the ITEM finalizer doesn't finalize
its contents.  Each contained object needs its own finalizer I think, if
finalization is necessary.

> 5. Gobject-like subclassing.  In Gtk, an application window is
> a type of a window which is a type of a widget which is a type of
> object. If we were to resurrect an effort on GTK3 binding,
> how would it work for foreign objects?

This is very tricky; there are lots of considerations here.  A new
guile-gobject might well want its own scheme.  However, yes, foreign
objects are subclassable.

Cheers,

Andy

Re: Guile foreign object interface

[Andy Wingo]

Hi :)

On Thu 02 Mar 2017 16:47, Mike Gran <spk121@yahoo.com> writes:

> I wanted to make a quick post about the foreign object interface.
> This is a bit of a placeholder because I haven't had time to
> investigate the interface properly, yet. But I intend to poke at
> it soon.
> But for there record, there are some problematic design patterns
> that I want to make sure can be covered by the new interface.

I am not sure that all cases need to be covered by the new interface to
be able to include it.  I think it would be sufficient to simply replace
SMOBs, unrecommend mark procedures and all that, and get the benefits I
mentioned in my mail.  However that said...

> 1. First off is a Lilypond-like pattern.  A C++ vector
> is used to dynamically add or remove elements.  The memory
> management of those elements lives in the C++ world.  Those elements
> are boxed up as SCM.  GCing the SCM should not free its boxed
> contents, but, deleting the boxed contents from the C++ side
> should render those SCMs invalid in some sense. 

Not really helped by foreign objects AFAIU.

> 2. Mutually owned information. Two structures (A and B) both
> mutually contain a non-GC-malloc'd structure C. C must exist
> so long as one of A or B exist.

Likewise, if I understand A and B to be scheme objects of some kind; the
pattern here would be a gc-managed wrapper around C with a finalizer,
and A and B both use the wrapper.

> 3. Here's one from guile-ncurses. An element ITEM is created with
> some contents. Default memory management suffices: if ITEM is GC'd,
> its contents are GC'd.  But the contents of ITEM can later be
> attached to a COLLECTION. If ITEM is GC'd when attached to
> COLLECTION, its contents are not GC'd.

I understand you mean that ITEM has the only reference on its contents,
and that if it becomes unreachable, the contents also become
unreachable, and may be collected (and possibly finalized if they have a
finalizer and finalizers have a chance to run and all the other
finalizer caveats).  Sure, foreign objects work here, as do any other
Scheme data type, right?  As long as the ITEM finalizer doesn't finalize
its contents.  Each contained object needs its own finalizer I think, if
finalization is necessary.

> 5. Gobject-like subclassing.  In Gtk, an application window is
> a type of a window which is a type of a widget which is a type of
> object. If we were to resurrect an effort on GTK3 binding,
> how would it work for foreign objects?

This is very tricky; there are lots of considerations here.  A new
guile-gobject might well want its own scheme.  However, yes, foreign
objects are subclassable.

Cheers,

Andy

Re: Guile foreign object interface

[Ludovic Courtès]

Hello!

I think Mark made two kinds of comments back then:

  1. There were suggestions about the API itself, nothing deep:
     <https://lists.gnu.org/archive/html/guile-devel/2014-04/msg00070.html>.
     Andy, do you know if they’ve been addressed?

  2. A general concern about “API churn” and our ability to support the
     SMOB API in the future, which I can sympathize with:
     <https://lists.gnu.org/archive/html/guile-devel/2015-11/msg00005.html>.

     Essentially, (1) it should be easy for people to migrate from SMOBs
     to foreign objects (I think it’s largely a matter of
     documentation), and (2) existing code that uses the documented SMOB
     API should just work in 2.2, possibly with a deprecation warning.

It may be that we’re already mostly there (?).  If we get these two
things right, then I have nothing against adding foreign objects in 2.2.

Mark?

Thanks,
Ludo’.

Re: Guile foreign object interface

[Andy Wingo]

Hi :)

Thanks for taking the time to look at the issue!

On Tue 07 Mar 2017 14:44, ludo@gnu.org (Ludovic Courtès) writes:

> I think Mark made two kinds of comments back then:
>
>   1. There were suggestions about the API itself, nothing deep:
>      <https://lists.gnu.org/archive/html/guile-devel/2014-04/msg00070.html>.
>      Andy, do you know if they’ve been addressed?

There were a couple points I think:

In

  SCM scm_make_foreign_object_type (SCM name, SCM slot_names,
                                    scm_t_struct_finalize finalizer);

Mark suggested it should be "scm_t_struct_finalizer".  I.e. with an r.
I can agree but scm_t_struct_finalize is a type that's already in 2.0;
it's just being re-used here.  That said we can do the new name +
typedef + eventually deprecate the old name dance.

He also suggested to get rid of some type punning, and indeed now we
have:

  void* scm_foreign_object_ref (SCM obj, size_t n);
  void scm_foreign_object_set_x (SCM obj, size_t n, void *val);

  scm_t_bits scm_foreign_object_unsigned_ref (SCM obj, size_t n);
  void scm_foreign_object_unsigned_set_x (SCM obj, size_t n,
                                          scm_t_bits val);

  scm_t_signed_bits scm_foreign_object_signed_ref (SCM obj, size_t n);
  void scm_foreign_object_signed_set_x (SCM obj, size_t n,
                                        scm_t_signed_bits val);

I think that was it from an API POV.

>   2. A general concern about “API churn” and our ability to support the
>      SMOB API in the future, which I can sympathize with:
>      <https://lists.gnu.org/archive/html/guile-devel/2015-11/msg00005.html>.
>
>      Essentially, (1) it should be easy for people to migrate from SMOBs
>      to foreign objects (I think it’s largely a matter of
>      documentation), and (2) existing code that uses the documented SMOB
>      API should just work in 2.2, possibly with a deprecation warning.

I am not so sure about about this one.  I think it's not accurate to
characterize beginning to replace a 25-year-old C API (SMOBs) as
"churn".  I will get to Mark's point specifically in a minute but
regarding your points I believe that (1) is somewhat under-documented;
the documentation is more oriented towards new users than migrating
users; we can improve here; and (2) should work fine (the old API is
still there, not even any deprecation warnings currently).

I think Mark's desire (if I understand it) was for the new API to
completely replace the old in all forms, specifically in support for
mark procedures.  I really think that (a) we should not support the SMOB
mark procedure interface, as it's both horrible and insufficient for a
moving collector, and (b) that until we have a moving collector (?), we
shouldn't attempt to speculatively standardize anything in this regard.
Basically the arguments from here:

  https://lists.gnu.org/archive/html/guile-devel/2015-11/msg00003.html

There's still the SMOB API if you need mark procedures, basically, but
hopefully you don't (e.g. when porting GDB to 2.0 I was able to remove
them; they're less necessary in 2.x than they were in 1.8).  Some users
still need them, which is why SMOBs are still around :)

Andy

Re: Guile foreign object interface

[Ludovic Courtès]

Howdy!

Andy Wingo <wingo@pobox.com> skribis:

> On Tue 07 Mar 2017 14:44, ludo@gnu.org (Ludovic Courtès) writes:
>
>> I think Mark made two kinds of comments back then:
>>
>>   1. There were suggestions about the API itself, nothing deep:
>>      <https://lists.gnu.org/archive/html/guile-devel/2014-04/msg00070.html>.
>>      Andy, do you know if they’ve been addressed?
>
> There were a couple points I think:
>
> In
>
>   SCM scm_make_foreign_object_type (SCM name, SCM slot_names,
>                                     scm_t_struct_finalize finalizer);
>
> Mark suggested it should be "scm_t_struct_finalizer".  I.e. with an r.
> I can agree but scm_t_struct_finalize is a type that's already in 2.0;
> it's just being re-used here.  That said we can do the new name +
> typedef + eventually deprecate the old name dance.

Good point; given the type is already there in 2.0, I’d be in favor of
keeping it as is (without the ‘r’).

> He also suggested to get rid of some type punning, and indeed now we
> have:

Perfect.

>>   2. A general concern about “API churn” and our ability to support the
>>      SMOB API in the future, which I can sympathize with:
>>      <https://lists.gnu.org/archive/html/guile-devel/2015-11/msg00005.html>.
>>
>>      Essentially, (1) it should be easy for people to migrate from SMOBs
>>      to foreign objects (I think it’s largely a matter of
>>      documentation), and (2) existing code that uses the documented SMOB
>>      API should just work in 2.2, possibly with a deprecation warning.
>
> I am not so sure about about this one.  I think it's not accurate to
> characterize beginning to replace a 25-year-old C API (SMOBs) as
> "churn".

I think the point is that there’s lots of code out there that rely on
SMOBs and we shouldn’t break it overnight, precisely because that API is
this old.

Of course, I agree that pushing users towards an improved API is the
right thing long term, no argument here.

> I will get to Mark's point specifically in a minute but regarding your
> points I believe that (1) is somewhat under-documented; the
> documentation is more oriented towards new users than migrating users;
> we can improve here; and (2) should work fine (the old API is still
> there, not even any deprecation warnings currently).

Sounds good.

> I think Mark's desire (if I understand it) was for the new API to
> completely replace the old in all forms, specifically in support for
> mark procedures.  I really think that (a) we should not support the SMOB
> mark procedure interface, as it's both horrible and insufficient for a
> moving collector, and (b) that until we have a moving collector (?), we
> shouldn't attempt to speculatively standardize anything in this regard.
> Basically the arguments from here:
>
>   https://lists.gnu.org/archive/html/guile-devel/2015-11/msg00003.html
>
> There's still the SMOB API if you need mark procedures, basically, but
> hopefully you don't (e.g. when porting GDB to 2.0 I was able to remove
> them; they're less necessary in 2.x than they were in 1.8).  Some users
> still need them, which is why SMOBs are still around :)

Understood.

I think we’ve achieved our goal if code that is not ready to migrate
works the same as with 2.0, which should be the case AIUI.

So, green lights for me.  :-)

Thank you,
Ludo’.