managing external memory

managing external memory

[Tom Schouten]

hi all,

i'm integrating guile in my 'pdp' project
(http://zwizwa.fartit.com/pd/pdp) and i've run into a problem where i
don't really understand how to properly tame the garbage collector.

basicly, libpdp has it's own reference count and reuse pool based memory
manager which is used to allocate and reuse memory buffers for storing
video image frames. data packets are referenced by a positive integer and
the manager is a global object.

what i want to do with the guile extensions is to have the entire system,
which is implemented as a forth-like script language language, to be
available from scheme in a straightforward way, trading the tight
real-time and low memory usage operation mode for a more convenient gc'd
scheme implementation.


the basic idea is that i don't need the system to behave in a real-time
environment when it is accessed from scheme, but i do want to have garbage
collector to behave properly.

right now, i use this interface:


/* packet wrapper */
typedef struct
{
    int packet;
    SCM update_func;
} pdp_scm_t;

static SCM mark_pdp (SCM pdp_smob)
{
    /* Mark the image's name and update function.  */
    pdp_scm_t *wrapper = (pdp_scm_t *) SCM_SMOB_DATA (pdp_smob);
    scm_gc_mark (wrapper->update_func);
    return SCM_BOOL_F;
}

static size_t free_pdp (SCM pdp_smob)
{
    pdp_scm_t *wrapper = (pdp_scm_t *) SCM_SMOB_DATA (pdp_smob);
    pdp_packet_unregister(wrapper->packet);
    scm_must_free(wrapper);
    return sizeof(pdp_scm_t);
}

scm_t_bits pdp_tag = scm_make_smob_type ("pdp", sizeof(pdp_scm_t));
scm_set_smob_free (pdp_tag, free_pdp);
scm_set_smob_mark (pdp_tag, mark_pdp);



the problem with this approach is that guile doesn't know the real
size of the object, but just the smob size. this can make the system run
out of memory if the gc is not called manually from time to time.

obviously i'm doing something terribly wrong, so i wondered if there is a
way to tell the gc how much 'external memory' a smob refers to, without
changing the way the memory allocation in libpdp works.

maybe it is better to let guile do all the memory management, but my
question then would be: will used memory be moved by the gc? in pdp, it is
required a packet's buffer will not be moved as long as it is registered,
and there's no way i can change that.

maybe someone can shed some light on this?

regards,
tom



_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://lists.gnu.org/mailman/listinfo/guile-user

Re: managing external memory

[Rob Browning]

Tom Schouten <doelie@zzz.kotnet.org> writes:

> obviously i'm doing something terribly wrong, so i wondered if there is a
> way to tell the gc how much 'external memory' a smob refers to, without
> changing the way the memory allocation in libpdp works.

Hmm.  Which version of guile are you using?  I seem to recall some
memory management problems in earlier versions.  If you check the NEWS
in CVS for 1.6 and for HEAD, you'll see some potentially relevant
changes.

Offhand I'm not exactly sure what the effect is supposed to be if you
can't report accurate sizes.

> maybe it is better to let guile do all the memory management, but my
> question then would be: will used memory be moved by the gc? in pdp,
> it is required a packet's buffer will not be moved as long as it is
> registered, and there's no way i can change that.

Guile doesn't use a copying collector, so your memory should stay put,
and even if it did, I suspect it wouldn't be moving any non-SCM
pointers.

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org; previously @cs.utexas.edu
GPG starting 2002-11-03 = 14DD 432F AE39 534D B592  F9A0 25C8 D377 8C7E 73A4


_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://lists.gnu.org/mailman/listinfo/guile-user

Re: managing external memory

[Kevin Ryde]

Tom Schouten <doelie@zzz.kotnet.org> writes:
>
> obviously i'm doing something terribly wrong, so i wondered if there is a
> way to tell the gc how much 'external memory' a smob refers to, without
> changing the way the memory allocation in libpdp works.

In 1.6, scm_done_malloc and scm_done_free can tell the gc how much is
in use.  Those funcs don't actually do any malloc or free, they just
let the gc keep score of how much is in use.  When it gets above a
threshold a gc is run to try to reduce.

But these funcs aren't actually documented, and in the cvs head I
think they're changing to scm_gc_register_collectable_memory and
scm_gc_unregister_collectable_memory.


_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://lists.gnu.org/mailman/listinfo/guile-user

Re: managing external memory

[Tom Schouten]

On Tue, 10 Aug 2004, Kevin Ryde wrote:

> Tom Schouten <doelie@zzz.kotnet.org> writes:
> >
> > obviously i'm doing something terribly wrong, so i wondered if there is a
> > way to tell the gc how much 'external memory' a smob refers to, without
> > changing the way the memory allocation in libpdp works.
>
> In 1.6, scm_done_malloc and scm_done_free can tell the gc how much is
> in use.  Those funcs don't actually do any malloc or free, they just
> let the gc keep score of how much is in use.  When it gets above a
> threshold a gc is run to try to reduce.
>

that worked like a charm! thanks.

in case anyone needs a guiled up image processing tool, just call :)

tom



_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://lists.gnu.org/mailman/listinfo/guile-user