[d.love@dl.ac.uk: dynamic loading of native code modules]

[d.love@dl.ac.uk: dynamic loading of native code modules]

[Thien-Thi Nguyen]

   From: Dave Love <d.love@dl.ac.uk>
   Subject: dynamic loading of native code modules
   Date: 10 Apr 2002 23:22:27 +0100
   To: gnu-emacs-sources@gnu.org
   
   This patch for Emacs 21.1 basically extends `load' to allow loading
   compiled C modules and provides a script to build them.  See also the
   attached README.
   
   [...]
   
   * Features
   
   These changes add the ability to dynamically load modules coded in C,
   like the Emacs interpreter core using Libtool facilities.  This works
   just by extending `load' to accept file names with extension `.la'
   which describe Libtool-built modules.  These are found on `load-path',
   like Lisp files, taking precedence over Lisp of the same base name if
   an extension isn't specified.  They work with `require' (if the module
   provides a feature) and `eval-after-load' (should that ever be
   useful).  They can't be loaded from remote filesystems.  There is a
   feature test `dynamic-modules-p' added for the facility, but I'm not
   sure that's actually useful.  Modules can only be loaded once and
   cannot be unloaded; it may be possible to remove these constraints.
   The script `build-emacs-module' cans the Libtool and doc-snarfing
   steps for building modules from source.
   
   XEmacs has a rather more complicated native code module system.  I'm
   not sure the complication is necessary, and it doesn't build on
   libtool.  (Bill Perry, who did the original XEmacs implementation,
   agrees that using Libtool is the right thing.)
    
   [...]

guile-1.4 supports extending module name semantics to allow mapping also
to .so in addition to .scm files (albeit low level uses dyn* directly --
could use update (patches welcome)).  1.6 does not at the moment (it was
removed).  grep "dyn" reveals 1537 hits since 2000-01.  anyone have
specific pointers for removal rationale?  where is that refbot?

it is possible to provide support again in `resolve-interface', but this
support should not be put in boot-9.scm.  instead, resolve-interface
ought to add a user hook somewhere, or be tunable in some other way.
this allows users to customize their concept of "interface" in a
well-defined environment (entirely :-) suited for such customization.
instantiable modules can be supported, etc.

in parallel w/ this change is of course revival of (use-modules FOO)
possibly resolving to .../libfoo.so.x.y.z, using the above hook and the
modern load-extension interface.  the previous mapping proc needs
rationalization and some design to keep weird use-cases in check.

another runner in the race is replacing lowest levels of the module
system w/ environments, using the above hook (or similar internal hook)
for implementation.  getting load-extension and environments together,
basically.

these all are branches that can be pursued relatively independently
after discussion yields a design.  in the end, they are all part of
defining the execution model, if i understand that term correctly, and
moving things to "user space" is what it's all about (slack!).

alternatively, we need to document *why* 1.6 chooses to rob the users
so, at least to ourselves.  "This has been found to be too tricky, and
is no longer supported" is, although not dis-honest, still pretty lame.

thi

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Neil Jerram]

>>>>> "Thien-Thi" == Thien-Thi Nguyen <ttn@giblet.glug.org> writes:

    Thien-Thi> guile-1.4 supports extending module name semantics to allow mapping also
    Thien-Thi> to .so in addition to .scm files (albeit low level uses dyn* directly --
    Thien-Thi> could use update (patches welcome)).  1.6 does not at the moment (it was
    Thien-Thi> removed).  grep "dyn" reveals 1537 hits since 2000-01.  anyone have

grep "dyn" where?

    Thien-Thi> specific pointers for removal rationale?  where is that refbot?

I think just that the scm_init_xxx - scm_register_module_xxx -
scm_init_module_xxx was thought rather awkward, and that too much of
the boot-9.scm code was trying to be cleverer than it could justify.
I recall an email from Marius saying this, but I don't have a specific
pointer.

    Thien-Thi> it is possible to provide support again in `resolve-interface', but this
    Thien-Thi> support should not be put in boot-9.scm.  instead, resolve-interface
    Thien-Thi> ought to add a user hook somewhere, or be tunable in some other way.
    Thien-Thi> this allows users to customize their concept of "interface" in a
    Thien-Thi> well-defined environment (entirely :-) suited for such customization.
    Thien-Thi> instantiable modules can be supported, etc.

    Thien-Thi> in parallel w/ this change is of course revival of (use-modules FOO)
    Thien-Thi> possibly resolving to .../libfoo.so.x.y.z, using the above hook and the
    Thien-Thi> modern load-extension interface.  the previous mapping proc needs
    Thien-Thi> rationalization and some design to keep weird use-cases in check.

Yes, I think this would be good, actually, and that the
mechanism/hooks that you suggest are exactly the right approach.

The description you gave of the Emacs patch glossed over one detail -
what's the name of the function that gets called to initialize the
dynamically loaded module?  I think it would be acceptable to derive
it algorithmically from the module name (and obviously impose this as
a requirement on the module coder).

If we can agree this, it would be good to do it in 1.6, for
continuity.  (Of interface, I mean; module coding would change
slightly, as just stated.)

    Thien-Thi> another runner in the race is replacing lowest levels of the module
    Thien-Thi> system w/ environments, using the above hook (or similar internal hook)
    Thien-Thi> for implementation.  getting load-extension and environments together,
    Thien-Thi> basically.

Sounds orthogonal to me; is it?

    Thien-Thi> alternatively, we need to document *why* 1.6 chooses to rob the users
    Thien-Thi> so, at least to ourselves.  "This has been found to be too tricky, and
    Thien-Thi> is no longer supported" is, although not dis-honest, still pretty lame.

Upon reflection, I agree.

More generally, looking back through mailing list history, it's
actually astonishing how much support for various stuff that Guile has
_lost_ along the way.  My overall impression is that we (collectively)
have been too glib about this.

        Neil




_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Rob Browning]

Thien-Thi Nguyen <ttn@giblet.glug.org> writes:

> alternatively, we need to document *why* 1.6 chooses to rob the
> users so, at least to ourselves.  "This has been found to be too
> tricky, and is no longer supported" is, although not dis-honest,
> still pretty lame.

This was not done to "rob" anyone.  It was done because (after a lot
of heavy use of the shared lib system), the previous "automagic"
loading was in some cases a little too smart for it's own good.  It
made it difficult to create a module that needed several shared libs,
or a shared lib that served several modules.  It also didn't allow any
easy way to work around problems in libtool (which are still serious)
from the scheme level.  I'll comment more on this later, but for part
of the issue see workbook/bugs/versioning-of-extensions.

Don't get me wrong, I'm not opposed to a more user-friendly,
higher-level interface, but IMO we need a sufficiently flexible
alternate (or lower-level) interface first, and in the process we need
to come up with a coherent solution that includes shared-lib-esque
versioning for scheme level modules (i.e. via use-modules).  We also
need to make sure that our interface abstracts the lowest levels
enough so that we can work around any libtool "issues".  I have a good
idea of how I think most of this should look, but was planning that
this wait until 1.8.

-- 
Rob Browning
rlb @defaultvalue.org, @linuxdevel.com, and @debian.org
Previously @cs.utexas.edu
GPG=1C58 8B2C FB5E 3F64 EA5C  64AE 78FE E5FE F0CB A0AD

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Rob Browning]

Neil Jerram <neil@ossau.uklinux.net> writes:

> The description you gave of the Emacs patch glossed over one detail -
> what's the name of the function that gets called to initialize the
> dynamically loaded module?  I think it would be acceptable to derive
> it algorithmically from the module name (and obviously impose this as
> a requirement on the module coder).

Some of the higher level abstractions and flexibility sound good, but
I'm a little concerned about automatically generating the init
function from the module name -- this makes it hard (as I mentioned
before) to create modules that share the same C lib or modules that
require multiple C libs.  It may also make it difficult to easily wrap
up existing libs without having to create unnecessary "dummy stub"
libs, and feels a little like overspecification.

Of course this would only be a problem if it was the *only* interface.
If we also provided a flexible enough low-level interface for loading
shared libs, then having a more automagic, optional higher-level
interface might be fine.

> If we can agree this, it would be good to do it in 1.6, for
> continuity.  (Of interface, I mean; module coding would change
> slightly, as just stated.)

My current plan is for 1.8 to have versioned scheme level modules
(i.e. use-modules modules), and some workaround that allows for
versioned dynamic-link'ing (in part to avoid libtool problems with
installations of multiple guile versions), so I don't think there's
much chance for seamless continuity between 1.6 and 1.8 anyway.  Given
that, I don't think it's worth holding up 1.6 for this, and in truth,
my personal feeling (given my experiences with the module system and
shared libraries while working on gnucash, g-wrap, mainstream guile,
and other projects) is that the current 1.6 system is a lot easier to
follow.  In the normal case, by looking at one .scm file, you know
exactly what's going on, and what's part of a module's interface.

  (define-module (foo bar))
  (dynamic-call "my_init_func" (dynamic-link "libmylib"))
  (export func1)
  (export func2)
  ...

(load-extension might also (should?) be used here).  To me this code
is *really* clear, but other people's mileage may vary :>

> More generally, looking back through mailing list history, it's
> actually astonishing how much support for various stuff that Guile
> has _lost_ along the way.  My overall impression is that we
> (collectively) have been too glib about this.

I guess I'd have to disagree here too.  Most of the stuff that I can
think of that we've "lost" actually makes guile cleaner and less
confusing to me, and the stuff that we've added (or are adding) makes
guile much more useful.  As examples of things we've done or planned
since 1.4:

  drop gh_* -> less confusing

  add documentation for scm_* -> more useful.

  add goops -> *much* more useful

  adding GMP -> (for bignums -- on the way to rationals?) more useful

  fixing libtool versioning issues -> more useful (more packagable)

  planning to drop certain macro "flexibility" so we can support a
  clear evaluation model and perhaps eventually support compilation ->
  more useful.

etc., but do you have particular things you think we've dropped that
have actually hurt substantially?

Overall I think guile may have (had?) a bit of excess baggage, and
I've felt like there's likely to be a decent amount of pruning,
pruning which should make guile stronger, before it becomes the
extension languge (and perhaps often scheme implementation) of choice
if it *is* going to.  I guess rightly or wrongly I've been considering
1.6 and 1.8 to be fairly serious "housecleaning" releases...

Of course - IMO, etc.

-- 
Rob Browning
rlb @defaultvalue.org, @linuxdevel.com, and @debian.org
Previously @cs.utexas.edu
GPG=1C58 8B2C FB5E 3F64 EA5C  64AE 78FE E5FE F0CB A0AD

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Rob Browning]

Rob Browning <rlb@defaultvalue.org> writes:

> I have a good idea of how I think most of this should look, but was
> planning that this wait until 1.8.

  s/that this wait/to hold off on raising the discussion/

(Didn't mean to imply that I was trying to be the arbiter of all things
 shared lib here :>)

-- 
Rob Browning
rlb @defaultvalue.org, @linuxdevel.com, and @debian.org
Previously @cs.utexas.edu
GPG=1C58 8B2C FB5E 3F64 EA5C  64AE 78FE E5FE F0CB A0AD

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Marius Vollmer]

Neil Jerram <neil@ossau.uklinux.net> writes:

> If we can agree this, it would be good to do it in 1.6,

No. :-)

>     Thien-Thi> alternatively, we need to document *why* 1.6 chooses
>     Thien-Thi> to rob the users so, at least to ourselves.  "This
>     Thien-Thi> has been found to be too tricky, and is no longer
>     Thien-Thi> supported" is, although not dis-honest, still pretty
>     Thien-Thi> lame.
> 
> Upon reflection, I agree.

Yeah, ok.  Let me try to explain.  (This is the new
workbook/modules/modules-and-shared-libs.text.)

Guile used to be able to automatically find and link a shared
library to satisfy requests for a module.  For example, the module
`(foo bar)' could be implemented by placing a shared library named
"foo/libbar.so" (or with a different extension) in a directory on the
load path of Guile.

This has been found to be too tricky, and is no longer supported.  The
shared libraries are now called "extensions".  You should now write a
small Scheme file that calls `load-extension' to load the shared
library and initialize it explicitely.


Here is more about why "this has been found to be to tricky".  It is
about the way it was done, not about why it can't possible at all.

While this support was still present, modules could be either
implemented by Scheme source files, or by shared libraries compiled
from C.  These two forms are two very different things: one is
platform independent and installed somewhere in 'prefix', the other is
platform dependent and is installed in 'exec-prefix'.  However, Guile
had no platform dependent locations in its default search path.

Moreover, the search algorithm required shared libraries that were to
be autoloaded as modules to reside not in the usual library
directories (like /usr/local/lib), but in Guile search path as
.../foo/libbar.so for example for module (foo bar).  This will not
really work for shared libraries that are also to be used from C code.
Guile usually provides a C API for its features that are written in C.
This should be encouraged for extensions as well.  However, the Unix
shared library does not deal well with shared libraries that don't
come from standard locations or are referenced by multiple names
(symlinks).

Additionally, module boundaries are not necessarily language
boundaries.  That is, a module can be a mix of Scheme and C (and one
file might want to provide more than one module).  Therefore, we need
a good way to load shared libraries independently from modules anyway.

Restricting module system operations and autoloading to Scheme code
only provided an immediate and significant simplification, without
much hassle to the user.  The simplified setup should also be easier
to understand.

> More generally, looking back through mailing list history, it's
> actually astonishing how much support for various stuff that Guile
> has _lost_ along the way.

I'd say we didn't lose it, we freed us from it.

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Neil Jerram]

>>>>> "Rob" == Rob Browning <rlb@defaultvalue.org> writes:

    Rob> Neil Jerram <neil@ossau.uklinux.net> writes:
    >> The description you gave of the Emacs patch glossed over one detail -
    >> what's the name of the function that gets called to initialize the
    >> dynamically loaded module?  I think it would be acceptable to derive
    >> it algorithmically from the module name (and obviously impose this as
    >> a requirement on the module coder).

    Rob> Some of the higher level abstractions and flexibility sound
    Rob> good, but I'm a little concerned about automatically
    Rob> generating the init function from the module name -- this
    Rob> makes it hard (as I mentioned before) to create modules that
    Rob> share the same C lib or modules that require multiple C libs.
    Rob> It may also make it difficult to easily wrap up existing libs
    Rob> without having to create unnecessary "dummy stub" libs, and
    Rob> feels a little like overspecification.

Ah yes, I remember the discussion now.

    Rob> Of course this would only be a problem if it was the *only*
    Rob> interface.  If we also provided a flexible enough low-level
    Rob> interface for loading shared libs, then having a more
    Rob> automagic, optional higher-level interface might be fine.

I'm pretty sure that Thi was wanting to reinstate (use-modules ...)
support _in addition to_ keeping load-extension and dynamic-call etc.

    >> If we can agree this, it would be good to do it in 1.6, for
    >> continuity.  (Of interface, I mean; module coding would change
    >> slightly, as just stated.)

    Rob> My current plan is for 1.8 to have versioned scheme level
    Rob> modules (i.e. use-modules modules), and some workaround that
    Rob> allows for versioned dynamic-link'ing (in part to avoid
    Rob> libtool problems with installations of multiple guile
    Rob> versions), so I don't think there's much chance for seamless
    Rob> continuity between 1.6 and 1.8 anyway.  Given that, I don't
    Rob> think it's worth holding up 1.6 for this, and in truth, my
    Rob> personal feeling (given my experiences with the module system
    Rob> and shared libraries while working on gnucash, g-wrap,
    Rob> mainstream guile, and other projects) is that the current 1.6
    Rob> system is a lot easier to follow.  In the normal case, by
    Rob> looking at one .scm file, you know exactly what's going on,
    Rob> and what's part of a module's interface.

    Rob>   (define-module (foo bar))
    Rob>   (dynamic-call "my_init_func" (dynamic-link "libmylib"))
    Rob>   (export func1)
    Rob>   (export func2)
    Rob>   ...

Using this kind of approach, is it always possible to emulate the
effect of the old use-modules behaviour by installing a .scm file that
loads the required library?

If it is, then that's probably sufficient for 1.6, and I'll add
something to the docs to make that clear.

    Rob> (load-extension might also (should?) be used here).  To me
    Rob> this code is *really* clear, but other people's mileage may
    Rob> vary :>

Yes, load-extension should be used here.  The difference is that
load-extension provides a way of handling the case where the LIB is
already statically linked in.  (And the case where it has previously
been dynamically linked?)  I think the code is still pretty explicit:

(load-extension "libmylib" "my_init_func")

    >> More generally, looking back through mailing list history, it's
    >> actually astonishing how much support for various stuff that Guile
    >> has _lost_ along the way.  My overall impression is that we
    >> (collectively) have been too glib about this.

    Rob> I guess I'd have to disagree here too.  Most of the stuff that I can
    Rob> think of that we've "lost" actually makes guile cleaner and less
    Rob> confusing to me, and the stuff that we've added (or are adding) makes
    Rob> guile much more useful.  As examples of things we've done or planned
    Rob> since 1.4:

    Rob>   drop gh_* -> less confusing

    Rob>   add documentation for scm_* -> more useful.

    Rob>   add goops -> *much* more useful

    Rob>   adding GMP -> (for bignums -- on the way to rationals?) more useful

    Rob>   fixing libtool versioning issues -> more useful (more packagable)

    Rob>   planning to drop certain macro "flexibility" so we can support a
    Rob>   clear evaluation model and perhaps eventually support compilation ->
    Rob>   more useful.

    Rob> etc., but do you have particular things you think we've dropped that
    Rob> have actually hurt substantially?

    Rob> Overall I think guile may have (had?) a bit of excess baggage, and
    Rob> I've felt like there's likely to be a decent amount of pruning,
    Rob> pruning which should make guile stronger, before it becomes the
    Rob> extension languge (and perhaps often scheme implementation) of choice
    Rob> if it *is* going to.  I guess rightly or wrongly I've been considering
    Rob> 1.6 and 1.8 to be fairly serious "housecleaning" releases...

Well, I agree with all the examples you give here, but what about the
following?

- dropped support for multibyte strings  [unless I'm misunderstanding
  the old mailing lists, Guile used to have these !]

- dropped/lost support for Tcl/Tk

- dropped/lost support for Ctax and other things (parser, rx etc.)
  in the guile-lang-allover package (or guile-rgx-ctax in CVS)

- dropped/lost support for Hobbit compilation

Now, apart from multibyte strings, none of these should be in the core
(and I haven't checked whether they were ever distributed as such).
Most of the examples here are to do with compatibility of Guile core
with its surrounding packages, and so shouldn't conflict with the
wonderful pruning/cleanups that have been going on in the last couple
of years (and which I totally support).  So how did we lose them?

I shall resist the temptation to quote Lady Bracknell.

        Neil


_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Rob Browning]

Neil Jerram <neil@ossau.uklinux.net> writes:

> I'm pretty sure that Thi was wanting to reinstate (use-modules ...)
> support _in addition to_ keeping load-extension and dynamic-call
> etc.

OK, that's somewhat different, though there would still be some issues
to work out like where the libs should go -- i.e. you used to put them
in %load-path, but if we're using libtool, then they have to go in
LD_LIBRARY_PATH or LTDL_LIBRARY_PATH which somewhat complicates the
semantics of use-modules.  This isn't a *big* deal, though, and to
some extent I may just be coming from a perspective of feeling like
shared library loading and modules are better treated more
orthogonally than others might.

> Using this kind of approach, is it always possible to emulate the
> effect of the old use-modules behaviour by installing a .scm file
> that loads the required library?
>
> If it is, then that's probably sufficient for 1.6, and I'll add
> something to the docs to make that clear.

More or less, the primary differences that I can think of are that in
the old approach you would put the shared libs in a /usr/share
directory (which isn't quite right according to the FHS and causes
problems on shared NFS volumes which expect only arch-independent data
in /usr/share), where with the new approach the libs need to be
located somewhere that libltdl can find them (system lib path,
LD_LIBRARY_PATH or LDTL_LIBRARY_PATH).  Also with the old approach,
any functions defined in your shared lib would automatically and
unavoidably be exported from the module that the shared lib
represented -- with the new approach you need to either add an export
for each function to the .scm file by hand, or write some module code
to export all symbols defined in the module after the shared lib has
been initialized -- i.e.

  (for-each
    (lambda (sym) (export sym))
    (module-bindings (current-module)))

or similar...

> Yes, load-extension should be used here.  The difference is that
> load-extension provides a way of handling the case where the LIB is
> already statically linked in.  (And the case where it has previously
> been dynamically linked?)  I think the code is still pretty explicit:
>
> (load-extension "libmylib" "my_init_func")

OK, right -- I just haven't used load-extension much yet, but that
seems good to me.

> Well, I agree with all the examples you give here, but what about the
> following?

OK, here goes nothing :> (Oh, and I am by no means implying here that
I know we haven't ever dropped anything we shouldn't have or that I'm
*certain* that none of these things should have been dropped -- what's
below is just my current impression regarding each.)

> - dropped support for multibyte strings  [unless I'm misunderstanding
>   the old mailing lists, Guile used to have these !]

Hmm.  I not sure about whether or not guile used to have them, but my
main impression is that guile has taken a long time even figuring out
what it thinks a reasonable answer to this problem is, much less
designing a solution with good documentation that is intended to be
supported for the long-haul.  However, I could easily be mistaken
here.

> - dropped/lost support for Tcl/Tk

That strikes me as an "add on" that is only going to stick around for
as long as there are enough people interested in guile's tcl/tk
support to continue maintaining it, and if there aren't enough people
around interested in supporting it, then perhaps it *should* go away.

> - dropped/lost support for Ctax and other things (parser, rx etc.)
>   in the guile-lang-allover package (or guile-rgx-ctax in CVS)

I can't really speak to this having never seen, used, or worked on any
of this -- I like the *idea* of translating multiple languages into
guile, but I doubt that it's likely to happen on a large scale until
guile is a persuasive enough platform with respect to the basics to
attract groups wanting to work on the languages in question.
Otherwise I doubt there are enough people here, just given us, to both
maintain the core of guile well *and* develop and maintain any
significant number of font ends.

I guess in the end I feel that in the absence of infinite resources,
extensions from the guile core that don't have enough demand to create
and sustain the communities that develop and support them often will
(and likely should) fade away.

Things people want and need *will* get worked on.  As an example, I
don't think guile-pg has had a lot of development lately, but recently
we've started using it heavily here, and Dale has also been working
with it.  As a result, I've contacted Ian to see what his current
status is and to see how we might be able to help -- Dale and I have
already fixed it to work at least crudely with the newer autotools and
the stable branch.

> - dropped/lost support for Hobbit compilation

This one I've talked to Marius about off and on at some length, and
have actually worked on a bit myself (Hobbit and CVS in particular).
I think the conclusion was that guile's evaluator needs to be reworked
in some of the ways that Marius, Lynn, and others, including myself
have talked about, or any work on compilation is likely to be way too
much effort for too little gain.  Guile is not currently built to
support compilation cleanly -- the addition of syntax-case and the
ways in which it breaks hobbit (among other things) points this out.

As Marius has mentioned we likely need a cleaner evaluation model
(with clear separations between "read time", "compile time", and
"execution time" before we're going to be able to make a lot of
sustainable headway here.  The good thing is that if we do manage to
seprate these things in a resonable way, we may end up with a lot of
interesting flexibility wrt to offline-compilation, JIT compilation,
byte-compilation, etc.

-- 
Rob Browning
rlb @defaultvalue.org, @linuxdevel.com, and @debian.org
Previously @cs.utexas.edu
GPG=1C58 8B2C FB5E 3F64 EA5C  64AE 78FE E5FE F0CB A0AD

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Marius Vollmer]

Neil Jerram <neil@ossau.uklinux.net> writes:

> Well, I agree with all the examples you give here, but what about the
> following?
> 
> - dropped support for multibyte strings  [unless I'm misunderstanding
>   the old mailing lists, Guile used to have these !]

Yes, we had them, but nobody was really supporting them in C code.
Multi-byte strings were a distinct type from uni-byte strings, but
almost everybody just wrote code to deal with uni-byte strings.

(Rob has answered the rest already.)

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Andreas Rottmann]

Marius Vollmer <mvo@zagadka.ping.de> writes:

> Neil Jerram <neil@ossau.uklinux.net> writes:
> 
> > If we can agree this, it would be good to do it in 1.6,
> 
> No. :-)
> 
> >     Thien-Thi> alternatively, we need to document *why* 1.6 chooses
> >     Thien-Thi> to rob the users so, at least to ourselves.  "This
> >     Thien-Thi> has been found to be too tricky, and is no longer
> >     Thien-Thi> supported" is, although not dis-honest, still pretty
> >     Thien-Thi> lame.
> > 
> > Upon reflection, I agree.
> 
> Yeah, ok.  Let me try to explain.  (This is the new
> workbook/modules/modules-and-shared-libs.text.)
> 
<snip/>

I'm OK with the new policy not to have *.so loaded automatically. 

However, I have a suggestion: If the lib is only to be used by the
.scm module (using load-extension), I see no point in putting it in
${libdir}. Instead it would be cleaner to be able to put it somewhere
in ${libdir}/guile to avoid namespace cluttering in ${libdir} (which
is, for example, (likely to be) indexed by the runtime linker on
linux).

Regards, Andy
-- 
Andreas Rottmann         | Dru@ICQ        | 118634484@ICQ | a.rottmann@gmx.at
http://www.8ung.at/rotty | GnuPG Key: http://www.8ung.at/rotty/gpg.asc
Fingerprint              | DFB4 4EB4 78A4 5EEE 6219  F228 F92F CFC5 01FD 5B62

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Marius Vollmer]

Andreas Rottmann <a.rottmann@gmx.at> writes:

> However, I have a suggestion: If the lib is only to be used by the
> .scm module (using load-extension), I see no point in putting it in
> ${libdir}.  Instead it would be cleaner to be able to put it
> somewhere in ${libdir}/guile to avoid namespace cluttering in
> ${libdir} (which is, for example, (likely to be) indexed by the
> runtime linker on linux).

I don't see much risk in name collisions when we name our libs like
libguile-foo-bar.

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Neil Jerram]

>>>>> "Rob" == Rob Browning <rlb@defaultvalue.org> writes:

    >> Using this kind of approach, is it always possible to emulate the
    >> effect of the old use-modules behaviour by installing a .scm file
    >> that loads the required library?

    Rob> More or less, the primary differences that I can think of are that in
    Rob> the old approach you would put the shared libs in a /usr/share
    Rob> directory (which isn't quite right according to the FHS and causes
    Rob> problems on shared NFS volumes which expect only arch-independent data
    Rob> in /usr/share), where with the new approach the libs need to be
    Rob> located somewhere that libltdl can find them (system lib path,
    Rob> LD_LIBRARY_PATH or LDTL_LIBRARY_PATH).  Also with the old approach,
    Rob> any functions defined in your shared lib would automatically and
    Rob> unavoidably be exported from the module that the shared lib
    Rob> represented -- with the new approach you need to either add an export
    Rob> for each function to the .scm file by hand, or write some module code
    Rob> to export all symbols defined in the module after the shared lib has
    Rob> been initialized -- i.e.

    Rob>   (for-each
    Rob>     (lambda (sym) (export sym))
    Rob>     (module-bindings (current-module)))

    Rob> or similar...

In summary, then, if a module author previously packaged his/her
library so that it could be loaded by (use-modules (pkg whatnot)),
there is a way that he/she can continue to make (use-modules (pkg
whatnot)) work in 1.6.  That sounds OK to me.

    >> - dropped/lost support for Tcl/Tk, [also Ctax etc.]

    Rob> That strikes me as an "add on" that is only going to stick around for
    Rob> as long as there are enough people interested in guile's tcl/tk
    Rob> support to continue maintaining it [...]

    Rob> I guess in the end I feel that in the absence of infinite resources,
    Rob> extensions from the guile core that don't have enough demand to create
    Rob> and sustain the communities that develop and support them often will
    Rob> (and likely should) fade away.

    Rob> Things people want and need *will* get worked on.  [...]

True, but I'm still slightly worried that one of the influences on
{the set of people interested enough to maintain surrounding packages}
might be a gradual trickle of incompatible changes in the core.

So (although I didn't state it clearly before) I guess my point is
that the bitrotting of such modules might be pointing to a problem
that results from an accumulation of small changes like the
use-modules one here.

On the other hand, a project that doesn't change is a dead project,
and packages that can't cope with a small amount of change may not be
worth coddling, and I agree with you that the utility of recent
additions and cleanups far exceeds that of Tcl/Tk and Ctax support.

    >> - dropped/lost support for Hobbit compilation

    Rob> This one I've talked to Marius about off and on at some length, and
    Rob> have actually worked on a bit myself (Hobbit and CVS in particular).
    Rob> I think the conclusion was that guile's evaluator needs to be reworked
    Rob> in some of the ways that Marius, Lynn, and others, including myself
    Rob> have talked about, or any work on compilation is likely to be way too
    Rob> much effort for too little gain.  Guile is not currently built to
    Rob> support compilation cleanly -- the addition of syntax-case and the
    Rob> ways in which it breaks hobbit (among other things) points this out.

    Rob> As Marius has mentioned we likely need a cleaner evaluation model
    Rob> (with clear separations between "read time", "compile time", and
    Rob> "execution time" before we're going to be able to make a lot of
    Rob> sustainable headway here.  The good thing is that if we do manage to
    Rob> seprate these things in a resonable way, we may end up with a lot of
    Rob> interesting flexibility wrt to offline-compilation, JIT compilation,
    Rob> byte-compilation, etc.

All very reasonable, and yet... something used to work, and now it
doesn't.  But on this point I'm unqualified to understand what the
issues really were, so I'll shut up now! :-)

        Neil


_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Neil Jerram]

>>>>> "Marius" == Marius Vollmer <mvo@zagadka.ping.de> writes:

    Marius> Neil Jerram <neil@ossau.uklinux.net> writes:
    >> Well, I agree with all the examples you give here, but what about the
    >> following?
    >> 
    >> - dropped support for multibyte strings  [unless I'm misunderstanding
    >> the old mailing lists, Guile used to have these !]

    Marius> Yes, we had them, but nobody was really supporting them in
    Marius> C code.  Multi-byte strings were a distinct type from
    Marius> uni-byte strings, but almost everybody just wrote code to
    Marius> deal with uni-byte strings.

But only _almost_ everybody?

(OK, I think that's enough devil's advocate from me on this general
argument.  In fact I'm as much in favour of simplifying and cleaning
up the core as everyone else, and I realize that sometimes we have to
compromise compatibility in order to move forward.  Let's just be as
considerate as we can in the process.)

        Neil


_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[NIIBE Yutaka]

Neil Jerram wrote:
 > But only _almost_ everybody?
 > 
 > (OK, I think that's enough devil's advocate from me on this general
 > argument.  In fact I'm as much in favour of simplifying and cleaning
 > up the core as everyone else, and I realize that sometimes we have to
 > compromise compatibility in order to move forward.  Let's just be as
 > considerate as we can in the process.)

I don't think guile had multi-byte support.  IIRC, Cygnus version (I
mean, before guile 1.0) had a feature of handling multibyte text
(convert it wide char text), and I've tested the feature for Japanese
text.  But I don't used it, as it's not well implemented.

I think that cleaning up is fair.  Wwe have the design guide written
by Jim Blandy, we could implement along that way.
-- 

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Rob Browning]

Neil Jerram <neil@ossau.uklinux.net> writes:

> In summary, then, if a module author previously packaged his/her
> library so that it could be loaded by (use-modules (pkg whatnot)),
> there is a way that he/she can continue to make (use-modules (pkg
> whatnot)) work in 1.6.  That sounds OK to me.

Right.  They may just have to create a fairly trivial .scm file and
move their shared lib to one of the "proper" lib directories.

> True, but I'm still slightly worried that one of the influences on
> {the set of people interested enough to maintain surrounding packages}
> might be a gradual trickle of incompatible changes in the core.

Well worth keeping an eye on as a possibility, but I suspect we're
probably helping offset this now by active assistance, better
documentation, help on the lists, etc., all things that are crucial
going forward.

-- 
Rob Browning
rlb @defaultvalue.org, @linuxdevel.com, and @debian.org
Previously @cs.utexas.edu
GPG=1C58 8B2C FB5E 3F64 EA5C  64AE 78FE E5FE F0CB A0AD

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Rob Browning]

Neil Jerram <neil@ossau.uklinux.net> writes:

> (OK, I think that's enough devil's advocate from me on this general
> argument.  In fact I'm as much in favour of simplifying and cleaning
> up the core as everyone else, and I realize that sometimes we have to
> compromise compatibility in order to move forward.  Let's just be as
> considerate as we can in the process.)

No problem -- this kind of self-examination is a good idea.

However I do feel like guile probably needs to be thought of as a
project still pretty "deep in the design phase", so while we need to
make a strong effort not to be gratuitous with our incompatibilities,
I suspect there will still be plenty going forward, for another few
releases at least.

-- 
Rob Browning
rlb @defaultvalue.org, @linuxdevel.com, and @debian.org
Previously @cs.utexas.edu
GPG=1C58 8B2C FB5E 3F64 EA5C  64AE 78FE E5FE F0CB A0AD

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Rob Browning]

Rob Browning <rlb@defaultvalue.org> writes:

> project still pretty "deep in the design phase", so while we need to

(Attribution: paraphrasing Marius, I think)

-- 
Rob Browning
rlb @defaultvalue.org, @linuxdevel.com, and @debian.org
Previously @cs.utexas.edu
GPG=1C58 8B2C FB5E 3F64 EA5C  64AE 78FE E5FE F0CB A0AD

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Thien-Thi Nguyen]

   From: Rob Browning <rlb@defaultvalue.org>
   Date: Wed, 17 Apr 2002 00:36:48 -0500

   However I do feel like guile probably needs to be thought of as a
   project still pretty "deep in the design phase", so while we need to
   make a strong effort not to be gratuitous with our incompatibilities,
   I suspect there will still be plenty going forward, for another few
   releases at least.

we can make incompatibilities non-gratuituous by reflecting them in the
version numbering (more "descriptive" thinking here).  by this method,
the next official guile should be "2.x" -- what do people think of that?

thi

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Thien-Thi Nguyen]

   From: NIIBE Yutaka <gniibe@m17n.org>
   Date: Wed, 17 Apr 2002 09:53:33 +0900 (JST)

   But I don't used it, as it's not well implemented.

how would you rate its design?  is the proposed design very different?

thi

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Thien-Thi Nguyen]

   From: Neil Jerram <neil@ossau.uklinux.net>
   Date: 16 Apr 2002 21:23:24 +0100

   True, but I'm still slightly worried that one of the influences on
   {the set of people interested enough to maintain surrounding packages}
   might be a gradual trickle of incompatible changes in the core.

different users have different sensitivities to (interface) change.  for
some, any drop of incompatibility is like poison in the waterworks or
shameful rust buildup on an old tool, but others are not so allergic.
as a guile user, i tend to value the "hammer is clean and works the same
Always" approach (headbangers unite ;-).

   On the other hand, a project that doesn't change is a dead project,
   and packages that can't cope with a small amount of change may not be
   worth coddling, and I agree with you that the utility of recent
   additions and cleanups far exceeds that of Tcl/Tk and Ctax support.

it is useful to separate interface and implementation when discussion
change.  i would agree implementation change is inevitable and mostly
desirable, but not so easily taken wrt interface change.

guile users are all programmers who only marginally like hacking guile
itself.  interest can be motivated towards guile development by allowing
more guile users write privs, and practicing some mature process.

thi

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Thien-Thi Nguyen]

   From: Neil Jerram <neil@ossau.uklinux.net>
   Date: 13 Apr 2002 09:50:22 +0100

   grep "dyn" where?

m.l. archives.

   The description you gave of the Emacs patch glossed over one detail -
   what's the name of the function that gets called to initialize the
   dynamically loaded module?  I think it would be acceptable to derive
   it algorithmically from the module name (and obviously impose this as
   a requirement on the module coder).

the renaming mechanism can be user supplied.  the system can supply
convenience renamers, including the old renamer.

	    [another runner is replacing lowest levels.
	     getting load-extension and environments together,
	     basically.]

   Sounds orthogonal to me; is it?

which elements are you referring to?  the tasks or the areas?

thi

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Neil Jerram]

>>>>> "thi" == Thien-Thi Nguyen <ttn@giblet.glug.org> writes:

    thi> and practicing some mature process.

While I kind of agree, I also have to say that absence of process is
one of the things that I like about free software work, and which
makes it a different challenge from my day job (which I also enjoy,
BTW).

        Neil


_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Thien-Thi Nguyen]

   From: Neil Jerram <neil@ossau.uklinux.net>
   Date: 20 Apr 2002 13:21:56 +0100

   Both, I think, but I don't see the distinction that you're making.

the tasks can be done independently because the implementation of a
"low-level module" (aka obarray) can be changed if the functional
interface to this data structure is maintained.  load-extension and
environments are independent areas in of themselves and only join when
they interact in implementing the module system at the module system
internal interfaces ("together" w/in resolve-interface).

the node "Modules and Environments" in $w/modules/module-snippets.texi
explains some of the overloaded uses of the word "module".  i think it
would be a helpful change to use different names for data structures at
different use-scopes, to give ourselves some semantic cues.  otherwise,
confusion naturally reigns.

thi

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Rob Browning]

Thien-Thi Nguyen <ttn@giblet.glug.org> writes:

> we can make incompatibilities non-gratuituous by reflecting them in the
> version numbering (more "descriptive" thinking here).  by this method,
> the next official guile should be "2.x" -- what do people think of that?

Not sure I follow what you mean by "by this method".

IMO we should bump the major version to 2.0 whenever we made changes
that we feel warrant it.  Things I would consider warranting such a
change:

  - Restructuring of the evaluation system and macros so that we can
    add compilation (and by compilation I mean anything like VM, JIT,
    offline->c, whatever).

  - Support for elisp that's ready for public use.

  - Performance improvement of large factors due to overhaul of XYZ.

  - Full numeric tower.

etc.

-- 
Rob Browning
rlb @defaultvalue.org, @linuxdevel.com, and @debian.org
Previously @cs.utexas.edu
GPG=1C58 8B2C FB5E 3F64 EA5C  64AE 78FE E5FE F0CB A0AD

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Robert A. Uhl]

On Sun, Apr 21, 2002 at 10:20:58AM -0500, Rob Browning wrote:
> 
>   - Full numeric tower.

Speaking of, anyone know what the status on that is?  On the one hand
it's nice when using Guile as a quick calculator not to need to type
e.g. (/ 456786958 1024 1024.) instead of (/ 456786958 1024 1024), but
on the other it's something of a nuisance when dealing with other
Scheme code.

Were I a better programmer, I'd work on it myself, but I'm not:-(

-- 
Robert Uhl <ruhl@4dv.net>
When anti-gun activists list the number of deaths per year from
firearms, they neglect to mention that 60 percent of the 30,000 figure
they often use are suicides.  They also fail to mention that at least
three-quarters of the 12,000 homicides are criminals killing other
criminals in disputes over illicit drugs,+or police shooting criminals
engaged in felonies.  Subtracting those, we are left with no more than
3,000 deaths that I think most would consider truly lamentable.

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Rob Browning]

"Robert A. Uhl" <ruhl@4dv.net> writes:

> Speaking of, anyone know what the status on that is?  On the one
> hand it's nice when using Guile as a quick calculator not to need to
> type e.g. (/ 456786958 1024 1024.) instead of (/ 456786958 1024
> 1024), but on the other it's something of a nuisance when dealing
> with other Scheme code.

I have GMP working here for bignums, but I have a lot of FIXME's in
the code that I have to work on, and I'll need to run some more tests
before I'd be comfortable committing it.  I'm planning to work on this
in the unstable branch once 1.6.1 is out, and there are some other
things I'd like to check in the process -- I've come across a couple
of bits in the code where I'm not sure we're RNRS compliant (or is
that what you're asking?)

Thanks

-- 
Rob Browning
rlb @defaultvalue.org, @linuxdevel.com, and @debian.org
Previously @cs.utexas.edu
GPG=1C58 8B2C FB5E 3F64 EA5C  64AE 78FE E5FE F0CB A0AD

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[news]

Rob Browning <rlb@defaultvalue.org> writes:

> However I do feel like guile probably needs to be thought of as a
> project still pretty "deep in the design phase", so while we need to

Guile is an implementation of scheme, which is specified
already, in about 50 pages, plus SRFIs.  It started with the SCM
codebase. What's being designed deeply?  What will follow the
deep design phase?

> make a strong effort not to be gratuitous with our incompatibilities,
> I suspect there will still be plenty going forward, for another few
> releases at least.

Why not implement guile to have generous backward compatiblity
from version to version?  Are you aware that lots of existing
code written for guile is broken and un(re)usable?

I'm just a wannabe user, who has a hard time using guile for
anything beyond SICP exercises.

Chris

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Thien-Thi Nguyen]

   From: Neil Jerram <neil@ossau.uklinux.net>
   Date: 13 Apr 2002 09:50:22 +0100

   If we can agree this, it would be good to do it in 1.6, for
   continuity.  (Of interface, I mean; module coding would change
   slightly, as just stated.)

would you like to do this rejuvenation job?  it seems like you
understand the motivation and the implementation approach.

   More generally, looking back through mailing list history, it's
   actually astonishing how much support for various stuff that Guile
   has _lost_ along the way.  My overall impression is that we
   (collectively) have been too glib about this.

basically, removing stuff from the interface w/o taking prior use into
account has the same effect as forking.  you see this a lot w/ newbie
designers.  other telltale signs are "the new way is better because
things *should* be used in this new way" and other circular thinking.

thi

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Thien-Thi Nguyen]

   From: Rob Browning <rlb@defaultvalue.org>
   Date: Sat, 13 Apr 2002 19:34:59 -0500

   Don't get me wrong, I'm not opposed to a more user-friendly,
   higher-level interface, but IMO we need a sufficiently flexible
   alternate (or lower-level) interface first,

this is the problem: there already was an interface first.  actions to
date are viewed by users as changes to that interface.  if a higher (or
different) interface is required that takes time to develop, that's no
problem.  but don't drop the one that's being used currently.  if the
implementation is to change, the least you can do is to support the old
interface.

the users don't care about the intention.  instead, they just see pain
and put the blame (rightly) on those who did that change.  it is the
users who would typify this kind of change as "rob".

   and in the process we need to come up with a coherent solution that
   includes shared-lib-esque versioning for scheme level modules
   (i.e. via use-modules).  We also need to make sure that our interface
   abstracts the lowest levels enough so that we can work around any
   libtool "issues".  I have a good idea of how I think most of this
   should look, but was planning that this wait until 1.8.

you should write down your good idea under workbook/ so that it can be
refined w/ input from all stakeholders (notably the users!).  keeping it
a secret doesn't help.  delaying "discussion" (which is lost in the spam
ridden archives) is not recommended.

thi

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Thien-Thi Nguyen]

   From: Marius Vollmer <mvo@zagadka.ping.de>
   Date: 14 Apr 2002 23:30:52 +0200

   No. :-)

better answer would be: "how?".

   I'd say we didn't lose it, we freed us from it.

as a user of this feature, i would say its loss is not appreciated.
please try not to speak for all users.

thi

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Rob Browning]

Thien-Thi Nguyen <ttn@giblet.glug.org> writes:

> you should write down your good idea under workbook/ so that it can be
> refined w/ input from all stakeholders (notably the users!).  keeping it
> a secret doesn't help.  delaying "discussion" (which is lost in the spam
> ridden archives) is not recommended.

With the time I have for guile between now and next Monday, what I
actually *will* be doing is trying to finish up my two release
critical bugs, and checking the other release critical bugs.  If I'm
finished with my bugs by Monday, and if I haven't heard anything else
with respect to the remaining release critical issues from anyone else
by then, I'm planning to handle those remaining issues myself.
Without further input, this will result in them being moved to
"Eventually" since I'll presume that no one disagreed with my
assesment in the last "Items blocking release 1.6.1" email.  After all
that I will release the next guile unstable beta.

Then, if no serious, agreed upon bugs or todo items crop up within a
week of *that* release, I will release 1.6.1.

During this process, if I have time, I may also integrate my release
management proposals into the more permanent workbook docs.

*After* the stable release I will get to other guile related items
that are on my more general to do list, a list which includes plenty
of non-guile, real-life items as well.  The guile items on that list
already included: working on the versioning issue (discussion,
documentation, and implementation) and revising various things in
workbook/.

BTW: have you read workbook/bugs/versioning-of-extensions?  It's not
nearly complete, but it's what I've had time to write down as yet.

-- 
Rob Browning
rlb @defaultvalue.org, @linuxdevel.com, and @debian.org
Previously @cs.utexas.edu
GPG=1C58 8B2C FB5E 3F64 EA5C  64AE 78FE E5FE F0CB A0AD

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Thien-Thi Nguyen]

   From: Andreas Rottmann <a.rottmann@gmx.at>
   Date: 15 Apr 2002 19:58:08 +0200

   However, I have a suggestion: If the lib is only to be used by the
   .scm module (using load-extension), I see no point in putting it in
   ${libdir}. Instead it would be cleaner to be able to put it somewhere
   in ${libdir}/guile to avoid namespace cluttering in ${libdir} (which
   is, for example, (likely to be) indexed by the runtime linker on
   linux).

well i raised this point two (3?) years ago and the counter-point was
that all shared object libraries are the same and should be treated the
same.  of course, this is not entirely true; all the object libraries
we're interested in playing w/ via a scheme interface must depend on
libguile, minimally, and are loaded into a more restricted environment
than say "appfoo dynloads libbar.so".

placing all these libguile-dependent thingies in libdir directly means
we need to implement and enforce some kind of name flattening algorithm
should the thingies be hierarchical instead of using an already existing
file hierarchy organization (the filesystem hierarcy).  it also means we
are subject to the vagaries of the thingy-access program (libtool).

these are points i should have stressed more at that time, but i guess
when it comes to appreciating clutter, nothing beats experience...  so
now that dealing w/ libraries has proven to be a royal nightmare, i'm
hoping we can wake up and design the "guile object library" and its
access protocols and reverse this stupidity.

thi

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Rob Browning]

Thien-Thi Nguyen <ttn@giblet.glug.org> writes:

> well i raised this point two (3?) years ago and the counter-point
> was that all shared object libraries are the same and should be
> treated the same.  of course, this is not entirely true; all the
> object libraries we're interested in playing w/ via a scheme
> interface must depend on libguile, minimally, and are loaded into a
> more restricted environment than say "appfoo dynloads libbar.so".

And if we put them in some non-lib dir, do you propose that we
internally mangle the user's LD_LIBRARY_PATH in order to allow ldso to
find the libs?  Many people consider this unacceptable application
behavior, so that's another point that somewhat weighs in favor of
using the standard directories.

Also, you do have to worry about distribution packaging systems --
make sure that everyone's going to be ok, inter-package
dependency-wise with us placing libs in non-standard locations.

> placing all these libguile-dependent thingies in libdir directly
> means we need to implement and enforce some kind of name flattening
> algorithm should the thingies be hierarchical instead of using an
> already existing file hierarchy organization (the filesystem
> hierarcy).  it also means we are subject to the vagaries of the
> thingy-access program (libtool).

It also means that we don't have to worry about name collisions.  If
you put all the libs in /usr/lib, then it's (filesystem wise)
impossible to accidentally have shadowing of one lib by another with
the same name and to have to deal with the mysterious errors that can
cause.

> these are points i should have stressed more at that time, but i
> guess when it comes to appreciating clutter, nothing beats
> experience...  so now that dealing w/ libraries has proven to be a
> royal nightmare, i'm hoping we can wake up and design the "guile
> object library" and its access protocols and reverse this stupidity.

Any reason you have to be both condescending and insulting about this?

Please feel free to post your obviously better solution, but don't
forget to make sure it works across all the supported architectures
and operating systems (I hear AIX is particularly nasty), both on the
build side and on the runtime side, and that it doesn't cause serious
packaging or dependency problems for any of the major distributions.

While I get fairly aggravated with libtool (and libltdl) myself from
time to time, I fully recognize the complexity of the problem they're
trying to solve.

Oh, and I'm not saying there isn't a better solution -- I'd be more
than happy to explore the possibilities, but I consider your
suggestion that "these people just chose this stupid solution because
they like to be stupid" disingenuous at best, and it certainly doesn't
inspire me, nor I suspect others, to want to work *with* you on the
problem.

-- 
Rob Browning
rlb @defaultvalue.org, @linuxdevel.com, and @debian.org
Previously @cs.utexas.edu
GPG=1C58 8B2C FB5E 3F64 EA5C  64AE 78FE E5FE F0CB A0AD

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[rm]

On Wed, Apr 24, 2002 at 09:33:33AM -0500, Rob Browning wrote:
> Thien-Thi Nguyen <ttn@giblet.glug.org> writes:
> 
> > well i raised this point two (3?) years ago and the counter-point
> > was that all shared object libraries are the same and should be
> > treated the same.  of course, this is not entirely true; all the
> > object libraries we're interested in playing w/ via a scheme
> > interface must depend on libguile, minimally, and are loaded into a
> > more restricted environment than say "appfoo dynloads libbar.so".
> 
> And if we put them in some non-lib dir, do you propose that we
> internally mangle the user's LD_LIBRARY_PATH in order to allow ldso to
> find the libs?  Many people consider this unacceptable application
> behavior, so that's another point that somewhat weighs in favor of
> using the standard directories.

I'd say "follow the masses". What  do applications like Apache, Perl,
Python etc. do? They all come with their 'private' locations for
application specific libraries (let's call them plug-ins, since this
seems to describe their function).
Why is manipulation of an _applications_ LD_LIBRARY_PATH an un-
acceptable behaviour (only applications exec'ed by guile would 
notice this, anyway) ?
 
> Also, you do have to worry about distribution packaging systems --
> make sure that everyone's going to be ok, inter-package
> dependency-wise with us placing libs in non-standard locations.

Again, it doesn't seem to be a problem in other languages that support
FFI. One _major_ benefit would be the possibility to have multiple versions
of guile not interfere with each other. As an example:
 i currently have both /usr/lib/python1.5, /usr/lib/python2.0 and
 /usr/lib/python2.1 on my box. Trying to do the same with guile turned
 out to be rather complex.

 Ralf Mattes

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Andreas Rottmann]

rm@fabula.de writes:

> On Wed, Apr 24, 2002 at 09:33:33AM -0500, Rob Browning wrote:
>
> > And if we put them in some non-lib dir, do you propose that we
> > internally mangle the user's LD_LIBRARY_PATH in order to allow ldso to
> > find the libs?  Many people consider this unacceptable application
> > behavior, so that's another point that somewhat weighs in favor of
> > using the standard directories.
> 
> I'd say "follow the masses". What  do applications like Apache, Perl,
> Python etc. do? They all come with their 'private' locations for
> application specific libraries (let's call them plug-ins, since this
> seems to describe their function).
> Why is manipulation of an _applications_ LD_LIBRARY_PATH an un-
> acceptable behaviour (only applications exec'ed by guile would 
> notice this, anyway) ?
> 
I fully agree. However, I don't think manipulation of LD_LIBRARY_PATH
is even necessary, since one can compile the directory
(e.g. /usr/local/lib/guile1.7.0) into the guile interpreter/shared
lib.

Andy
-- 
Andreas Rottmann         | Dru@ICQ        | 118634484@ICQ | a.rottmann@gmx.at
http://www.8ung.at/rotty | GnuPG Key: http://www.8ung.at/rotty/gpg.asc
Fingerprint              | DFB4 4EB4 78A4 5EEE 6219  F228 F92F CFC5 01FD 5B62

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Rob Browning]

rm@fabula.de writes:

> I'd say "follow the masses". What  do applications like Apache, Perl,
> Python etc. do? They all come with their 'private' locations for
> application specific libraries (let's call them plug-ins, since this
> seems to describe their function).
> Why is manipulation of an _applications_ LD_LIBRARY_PATH an un-
> acceptable behaviour (only applications exec'ed by guile would 
> notice this, anyway) ?

Aside from hearing people claim that "it's for user use only", I think
one of the primary issues was with whether or not you put the
additions at the beginning or the end -- i.e. do you allow the user to
override your value for debugging purposes, etc?  Another complaint if
I recall correctly is that if you're inserting directories that aren't
very *specific* you can cause other unwanted libs to be pulled in.
i.e. inserting /usr/local/lib somewhere might be *bad*, but if we were
only inserting /usr/lib/guile/1.8/ then that concern would be somewhat
alleviated.

One other concern would be what to do about add-ons -- i.e. you'd have
to be careful to work out a strategy whereby things like guile-www,
guile-pg, etc would have a place to put their libs that guile could
find.  Further, what if guile ships with a version of something like
guile-pg built in, but now then user wants to use a new version of
guile-pg.  If we aren't careful, i.e. if we just append
/usr/lib/guile/1.8 to the front of the LD_LIBRARY_PATH to be safe,
we've now prevented the user from loading their newer version unless
they install it in a way that clobbers the one inside the guile
install dir, a no-no as far as most distributions are concerned.

I can't recall off the top of my head if there are any stronger
arguments against it.

>> Also, you do have to worry about distribution packaging systems --
>> make sure that everyone's going to be ok, inter-package
>> dependency-wise with us placing libs in non-standard locations.
>
> Again, it doesn't seem to be a problem in other languages that support
> FFI. One _major_ benefit would be the possibility to have multiple versions
> of guile not interfere with each other. As an example:
>  i currently have both /usr/lib/python1.5, /usr/lib/python2.0 and
>  /usr/lib/python2.1 on my box. Trying to do the same with guile turned
>  out to be rather complex.

Guile has planned to allow users to link directly against guile's
internal libs, i.e. the user is supposed to compile an app and link
against -llibguile-srfi-srfi-4 if they need that functionality[1].  I
was under the (possibly mistaken) impression that python and perl
didn't intend to support that, and if not, that affects how much
flexibility you have.  [1] i.e. other apps are supposed to be able to
link appropriately and then call scm_make_u8vector directly.  Bear in
mind that allowing direct linking also means that you have to fully
version all your shared libs in the traditional manner.  This is
complex, but does have a number of benefits, including being well
understood on the OS side and allowing, if the distributions packaging
guile handle things right, easy upgrades of guile independent of all
the apps compiled against guile and it's various shared libs.

That said, I totally agree that when we sit down to work on versioning
scheme-level "use-modules modules", and on a better solution to the
current situation wrt lt_dlopen and versioned shared libs, we should
survey what the other languages are doing as part of the process.

It may well be that the subdirectory approach is suitable, and in fact
its one I originally leaned toward, but I was shifted away after
talking to Marius about direct linking and realizing that if we were
going to properly version all our shared libs to accmodate  etc.

-- 
Rob Browning
rlb @defaultvalue.org, @linuxdevel.com, and @debian.org
Previously @cs.utexas.edu
GPG=1C58 8B2C FB5E 3F64 EA5C  64AE 78FE E5FE F0CB A0AD

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Rob Browning]

Andreas Rottmann <a.rottmann@gmx.at> writes:

> I fully agree. However, I don't think manipulation of
> LD_LIBRARY_PATH is even necessary, since one can compile the
> directory (e.g. /usr/local/lib/guile1.7.0) into the guile
> interpreter/shared lib.

You mean -rpath?  That might be a possibility, though it's (at least
used to be) against Debian Policy -- there's usually a good reason,
but I'd have to go check to be sure what the rationale was, and of
course it might no longer apply...

-- 
Rob Browning
rlb @defaultvalue.org, @linuxdevel.com, and @debian.org
Previously @cs.utexas.edu
GPG=1C58 8B2C FB5E 3F64 EA5C  64AE 78FE E5FE F0CB A0AD

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Bill Gribble]

On Wed, 2002-04-24 at 10:48, Rob Browning wrote:
 > You mean -rpath?  That might be a possibility, though it's (at least
> used to be) against Debian Policy -- there's usually a good reason,
> but I'd have to go check to be sure what the rationale was, and of
> course it might no longer apply...

I think this is part of the whole "install tree != Real Install Tree"
thing, isn't it?   i.e. doing 'make install' into a sandbox, then
un-rooting and packaging up the resulting tree?   seems like -rpath into
the final destination could cause some problems with that ...

b.g.





_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Rob Browning]

Bill Gribble <grib@linuxdevel.com> writes:

> I think this is part of the whole "install tree != Real Install Tree"
> thing, isn't it?   i.e. doing 'make install' into a sandbox, then
> un-rooting and packaging up the resulting tree?   seems like -rpath into
> the final destination could cause some problems with that ...

Hmm, might be it -- I can't recall offhand.

-- 
Rob Browning
rlb @defaultvalue.org, @linuxdevel.com, and @debian.org
Previously @cs.utexas.edu
GPG=1C58 8B2C FB5E 3F64 EA5C  64AE 78FE E5FE F0CB A0AD

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Andreas Rottmann]

Rob Browning <rlb@defaultvalue.org> writes:

> Andreas Rottmann <a.rottmann@gmx.at> writes:
> 
> > I fully agree. However, I don't think manipulation of
> > LD_LIBRARY_PATH is even necessary, since one can compile the
> > directory (e.g. /usr/local/lib/guile1.7.0) into the guile
> > interpreter/shared lib.
> 
> You mean -rpath?  That might be a possibility, though it's (at least
> used to be) against Debian Policy -- there's usually a good reason,
> but I'd have to go check to be sure what the rationale was, and of
> course it might no longer apply...
> 
No, here we talk about extension .so files, don't we? These are loaded
via libltdl (which has ugly error messages, by the way) and thus one
can specify the directory to load the lib from.

Andy
-- 
Andreas Rottmann         | Dru@ICQ        | 118634484@ICQ | a.rottmann@gmx.at
http://www.8ung.at/rotty | GnuPG Key: http://www.8ung.at/rotty/gpg.asc
Fingerprint              | DFB4 4EB4 78A4 5EEE 6219  F228 F92F CFC5 01FD 5B62

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Andreas Rottmann]

Bill Gribble <grib@linuxdevel.com> writes:

> On Wed, 2002-04-24 at 10:48, Rob Browning wrote:
>  > You mean -rpath?  That might be a possibility, though it's (at least
> > used to be) against Debian Policy -- there's usually a good reason,
> > but I'd have to go check to be sure what the rationale was, and of
> > course it might no longer apply...
> 
> I think this is part of the whole "install tree != Real Install Tree"
> thing, isn't it?   i.e. doing 'make install' into a sandbox, then
> un-rooting and packaging up the resulting tree?   seems like -rpath into
> the final destination could cause some problems with that ...
> 
About -rpath: se my other mail.

I think this is not really an issue, since you can ./configure with
--prefix=/real/install/prefix, meaning /real/install/prefix is
compiled into the library (via #define) and then do 'make install
prefix=/stage/dir' without problems, given that in the final
installation (using a .deb or .rpm, for instance) the thing is put
into /real/install/prefix.

Regards, Andy
-- 
Andreas Rottmann         | Dru@ICQ        | 118634484@ICQ | a.rottmann@gmx.at
http://www.8ung.at/rotty | GnuPG Key: http://www.8ung.at/rotty/gpg.asc
Fingerprint              | DFB4 4EB4 78A4 5EEE 6219  F228 F92F CFC5 01FD 5B62

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Thien-Thi Nguyen]

   From: Rob Browning <rlb@defaultvalue.org>
   Date: Wed, 24 Apr 2002 09:33:33 -0500

   Any reason you have to be both condescending and insulting about this?

i'm condescending because i'm a User.  i'm insulting because i'm a
Programmer.  i'm both because i'm a subversive revolutionary (and hacker
wannabe) w/ a selectively long memory.  ymmv.

   Please feel free to post your obviously better solution [...]

others are wielding the clue stick, so i'll refer you to their comments.

it's not stupid to make mistakes, but it's stupid to not understand
those mistakes, and it's really stupid to ignore how other people have
previously made/fixed/avoided those mistakes.

thi

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Rob Browning]

Thien-Thi Nguyen <ttn@giblet.glug.org> writes:

> i'm condescending because i'm a User.  i'm insulting because i'm a
> Programmer.  i'm both because i'm a subversive revolutionary (and hacker
> wannabe) w/ a selectively long memory.  ymmv.
>
>    Please feel free to post your obviously better solution [...]
>
> others are wielding the clue stick, so i'll refer you to their comments.
>
> it's not stupid to make mistakes, but it's stupid to not understand
> those mistakes, and it's really stupid to ignore how other people have
> previously made/fixed/avoided those mistakes.

For the time being, I think I had better respond to you on technical
points if I think I can help, and limit my communications otherwise.
The "Have you stopped beating your wife lately?"  communication
strategy just makes me tired.  It makes work that would otherwise be
enjoyable more of a chore.  OPMMV.

-- 
Rob Browning
rlb @defaultvalue.org, @linuxdevel.com, and @debian.org
Previously @cs.utexas.edu
GPG=1C58 8B2C FB5E 3F64 EA5C  64AE 78FE E5FE F0CB A0AD

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Rob Browning]

Andreas Rottmann <a.rottmann@gmx.at> writes:

> I think this is not really an issue, since you can ./configure with
> --prefix=/real/install/prefix, meaning /real/install/prefix is
> compiled into the library (via #define) and then do 'make install
> prefix=/stage/dir' without problems, given that in the final
> installation (using a .deb or .rpm, for instance) the thing is put
> into /real/install/prefix.

Actually this won't necessarily work by itself if you're using
libtool.  In addition to prefix=/stage/dir, you may need to specify
LIBRARY_PATH=/stage/dir/lib.

-- 
Rob Browning
rlb @defaultvalue.org, @linuxdevel.com, and @debian.org
Previously @cs.utexas.edu
GPG=1C58 8B2C FB5E 3F64 EA5C  64AE 78FE E5FE F0CB A0AD

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Rob Browning]

Andreas Rottmann <a.rottmann@gmx.at> writes:

> No, here we talk about extension .so files, don't we? These are
> loaded via libltdl (which has ugly error messages, by the way) and
> thus one can specify the directory to load the lib from.

Current policy is that guile's extension libs are also to be
dynamically linkable directly into applications.  This means that
lt_dlopen is not the only way they're loaded.  Hence the need for
LD_LIBRARY_PATH or similar.

i.e. if you write an app that uses scm_make_u8vector from srfi-4, then
you have to "-l link" against libguilesrfi-srfi4.  For your app to ru,
libguilesrfi-srfi4 will have to either have an appropriate rpath, an
appropriate LD_LIBRARY_PATH, or be in one of the standard locations.

-- 
Rob Browning
rlb @defaultvalue.org, @linuxdevel.com, and @debian.org
Previously @cs.utexas.edu
GPG=1C58 8B2C FB5E 3F64 EA5C  64AE 78FE E5FE F0CB A0AD

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Andreas Rottmann]

Rob Browning <rlb@defaultvalue.org> writes:

> Andreas Rottmann <a.rottmann@gmx.at> writes:
> 
> > No, here we talk about extension .so files, don't we? These are
> > loaded via libltdl (which has ugly error messages, by the way) and
> > thus one can specify the directory to load the lib from.
> 
> Current policy is that guile's extension libs are also to be
> dynamically linkable directly into applications.  This means that
> lt_dlopen is not the only way they're loaded.  Hence the need for
> LD_LIBRARY_PATH or similar.
> 
Ok, I didn't know that.

Regards, Andy
-- 
Andreas Rottmann         | Dru@ICQ        | 118634484@ICQ | a.rottmann@gmx.at
http://www.8ung.at/rotty | GnuPG Key: http://www.8ung.at/rotty/gpg.asc
Fingerprint              | DFB4 4EB4 78A4 5EEE 6219  F228 F92F CFC5 01FD 5B62

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Joshua Judson Rosen]

On Wed, Apr 24, 2002 at 03:36:39PM -0500, Rob Browning wrote:
> Andreas Rottmann <a.rottmann@gmx.at> writes:
> 
> > I think this is not really an issue, since you can ./configure with
> > --prefix=/real/install/prefix, meaning /real/install/prefix is
> > compiled into the library (via #define) and then do 'make install
> > prefix=/stage/dir' without problems, given that in the final
> > installation (using a .deb or .rpm, for instance) the thing is put
> > into /real/install/prefix.
> 
> Actually this won't necessarily work by itself if you're using
> libtool.  In addition to prefix=/stage/dir, you may need to specify
> LIBRARY_PATH=/stage/dir/lib.

Why? Building the package does not involve executing things in /stage/dir....

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Rob Browning]

Joshua Judson Rosen <rozzin@geekspace.com> writes:

> Why? Building the package does not involve executing things in /stage/dir....

libtool insists on running a re-link during make install, and it has
to succeed.  Without LIBRARY_PATH, it will fail because it will only
specify -L/install/lib and the libraries won't be there yet (or worse,
the wrong versions *will* be there).

-- 
Rob Browning
rlb @defaultvalue.org, @linuxdevel.com, and @debian.org
Previously @cs.utexas.edu
GPG=1C58 8B2C FB5E 3F64 EA5C  64AE 78FE E5FE F0CB A0AD

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Thien-Thi Nguyen]

   From: Rob Browning <rlb@defaultvalue.org>
   Date: Wed, 24 Apr 2002 13:58:20 -0500

   For the time being, I think I had better respond to you on technical
   points if I think I can help, and limit my communications otherwise.
   The "Have you stopped beating your wife lately?"  communication
   strategy just makes me tired.  It makes work that would otherwise be
   enjoyable more of a chore.  OPMMV.

fair enough.  technically, there are several plug-in architectures in
existence to study from.  internal interfaces documentation is available
for some of them (typically, the good ones :-).  our job as a community
of guile programmers is to know guile well enough (including through
using guile) to be able to precisely describe our guile usage patterns
wrt other objects on the system.  these patterns most likely would be
influenced (in practice) by usage of these existing plug-in
implementations.

next, those who factor break down the patterns into attributes of the
general usage.  attributes may map to variables and procedures, which
are slammed into source by those who code, and explained thoroughly by
those who write about code.

this code and documentation is the toolkit and mini-language for
expressing an FFI.  it could be delivered alone as an "extensible
attribute mapping framework" or whatever.

interesting FFIs (ltdl, bfd, *.out, network streams, source in some vfs,
loopback dev, etc) can be expressed using this toolkit and (here's the
slackful part for maintainers :-) experimentation at that level is a
user concern and can be done completely in user land.  no worries, oop
ack!

for the purposes of bundled object libraries (aside from libguile),
guile itself uses the FFI api, in 1.4 or 1.6 or 1.8 or ... fashion, or
perhaps in the fashion of one of the users' creations.

so, this is what needs to be done to really put the active design phase
back in the users' hands and leave the maintainers w/ the core to tend,
for FFI.

bonus: this kind of process can run parallel w/ maintainers' own
experimentation w/ FFI architectures, which upon reflection, cannot be
called stupid because experimentation is a path w/ unknown ending (the
only stupidity would be not seeing this ;-).

to kick this off, can someone post an exhaustive summary of plug-in
architectures?  i will add that (and any further additions) under
workbook/ somewhere, no questions asked.

some seed attributes to think about:
 is-2-phase
 has-hook
 is-hooks-based
 depends-on-env-vars
 passes-through-env-vars-prefixing
 passes-through-env-vars-modifying-in-some-weird-way
 unloadable
 requires-system
 part-of-ABSTRACTION-PROGRAM-feature-list
 [your attributes here]

coders feel free to post code (if you know what you're doing :-).
everyone elese, please ask questions and point out flaws in the
approach.  i think we can get this ffi-toolkit api prepared and in use
for 1.8.

thi

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Thien-Thi Nguyen]

   From: Lynn Winebarger <owinebar@free-expression.org>
   Date: Mon, 29 Apr 2002 19:26:19 -0500

	This is an interesting policy.  Is it documented somewhere?  I
   take it the reason is because it's easier to develop the C libraries
   that way.

dude, you buzzkill cuz-we-can (bfd).

what makes C libraries easy to develop and install?

[cc guile-user]

thi

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Lynn Winebarger]

On Monday 29 April 2002 20:35, Thien-Thi Nguyen wrote:
>    From: Lynn Winebarger <owinebar@free-expression.org>
>    Date: Mon, 29 Apr 2002 19:26:19 -0500
> 
> 	This is an interesting policy.  Is it documented somewhere?  I
>    take it the reason is because it's easier to develop the C libraries
>    that way.
> 
> dude, you buzzkill cuz-we-can (bfd).
      I don't know how to take this.
> 
> what makes C libraries easy to develop and install?

      Being easily integrated with C programs is Guile's raison d'etre
(or it was originally, anyway).  Producing multiple libraries or putting
constraints on the format/contents of the library would impose more
work, and hence less ease, on the integrator. That's just my guess,
though.
     It's not immediately clear to me what a better format would be
without a guile-specific compiler/linker/loader  (a scheme compiler
shouldn't have to comply with C ABI requirements except where it, uh,
has to).
     Anyway, I don't have any immediate complaint about the policy,
just think it's useful to know.

Lynn

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Lynn Winebarger]

On Wednesday 24 April 2002 09:33, Rob Browning wrote:
> Thien-Thi Nguyen <ttn@giblet.glug.org> writes:
> 
> > well i raised this point two (3?) years ago and the counter-point
> > was that all shared object libraries are the same and should be
> > treated the same.  of course, this is not entirely true; all the
> > object libraries we're interested in playing w/ via a scheme
> > interface must depend on libguile, minimally, and are loaded into a
> > more restricted environment than say "appfoo dynloads libbar.so".
> 
> And if we put them in some non-lib dir, do you propose that we
> internally mangle the user's LD_LIBRARY_PATH in order to allow ldso to
> find the libs?  Many people consider this unacceptable application
> behavior, so that's another point that somewhat weighs in favor of
> using the standard directories.

      I don't think this is exactly accurate.  The documentation I have for 
libltdl states (note the "in the order as follows"!):
    
     If libltdl cannot find the library and the file name FILENAME does
     not have a directory component it will additionally search in the
     following search paths for the module (in the order as follows):

       1. user-defined search path: This search path can be set by the
          program using the functions `lt_dlsetsearchpath' and
          `lt_dladdsearchdir'.

       2. libltdl's search path: This search path is the value of the
          environment variable LTDL_LIBRARY_PATH.

       3. system library search path: The system dependent library
          search path (e.g. on Linux it is LD_LIBRARY_PATH).

     Each search path must be a colon-separated list of absolute
     directories, for example, `"/usr/lib/mypkg:/lib/foo"'.

     If the same module is loaded several times, the same handle is
     returned.  If `lt_dlopen' fails for any reason, it returns `NULL'.

which means you have 2 ways of searching that won't mess with normal
shared library behaviour.  Although it does look as though it won't play 
well when linking to library/application that uses libltdl for its own modules. 
You'd have to work around it by having a wrapper function switch the 
paths back and forth, and beware threads.  

Lynn

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Rob Browning]

Lynn Winebarger <owinebar@free-expression.org> writes:

>       I don't think this is exactly accurate.  The documentation I have for 
> libltdl states (note the "in the order as follows"!):
>     
>      If libltdl cannot find the library and the file name FILENAME does
>      not have a directory component it will additionally search in the
>      following search paths for the module (in the order as follows):
>
>        1. user-defined search path: This search path can be set by the
>           program using the functions `lt_dlsetsearchpath' and
>           `lt_dladdsearchdir'.

Ahh, I had forgotten about that.

> which means you have 2 ways of searching that won't mess with normal
> shared library behaviour.  Although it does look as though it won't
> play well when linking to library/application that uses libltdl for
> its own modules.  You'd have to work around it by having a wrapper
> function switch the paths back and forth, and beware threads.

And AFAIK adding a path via the lt_dl*search* functions above wouldn't
help other apps or app libs that are directly (i.e. ldso, not libltdl)
linked against the guile libs.

BTW thanks for the lt_dl*search* reminder.

-- 
Rob Browning
rlb @defaultvalue.org, @linuxdevel.com, and @debian.org
Previously @cs.utexas.edu
GPG=1C58 8B2C FB5E 3F64 EA5C  64AE 78FE E5FE F0CB A0AD

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Thien-Thi Nguyen]

   From: Lynn Winebarger <owinebar@free-expression.org>
   Date: Mon, 29 Apr 2002 21:33:14 -0500

   > dude, you buzzkill cuz-we-can (bfd).
	 I don't know how to take this.

this was my underhanded way of shaking my head in disgust at how poorly
guile users have been treated, presumably because guile maintainers have
been high on "joy of implementation" and changing things seemingly for
the hell of it ("because we can").

a more constructive way to say this would be to show how "because we
can" is probably meant in the right spirit (making things better), and
to encourage guile developers to expand their scope of operations and
interest to make changes that are both better and not burdensome to the
user.  there are more community-oriented ways to power-trip, basically.

unfortunately, i let several years of pent-up anger and resentment build
up and this was one of the results.  luckily i recently revisited Aubrey
Jaffer's homepage and saw something new (since i last checked) which was
immensely helpful in releasing the dams of disquietude.  maybe some of
that attitude readjustment will bear quiet fruit as i mature.

thi

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Thien-Thi Nguyen]

   From: Rob Browning <rlb@defaultvalue.org>
   Date: Sun, 21 Apr 2002 10:20:58 -0500

   Thien-Thi Nguyen <ttn@giblet.glug.org> writes:

   > we can make incompatibilities non-gratuituous by reflecting them in the
   > version numbering (more "descriptive" thinking here).  by this method,
   > the next official guile should be "2.x" -- what do people think of that?

   Not sure I follow what you mean by "by this method".

   IMO we should bump the major version to 2.0 whenever we made changes
   that we feel warrant it.  Things I would consider warranting such a
   change:

     - Restructuring of the evaluation system and macros so that we can
       add compilation (and by compilation I mean anything like VM, JIT,
       offline->c, whatever).

     - Support for elisp that's ready for public use.

     - Performance improvement of large factors due to overhaul of XYZ.

     - Full numeric tower.

"by this method" means considering major incompatiblities as another
thing that warrants a major version change (in addition to new features,
as you've suggested).

thi

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Thien-Thi Nguyen]

   From: Rob Browning <rlb@defaultvalue.org>
   Date: Wed, 24 Apr 2002 00:25:22 -0500

   BTW: have you read workbook/bugs/versioning-of-extensions?  It's not
   nearly complete, but it's what I've had time to write down as yet.

it looks like a plan to just implement something and throw it against
the wall to see if it sticks.  in particular, having the interface
number encoded in the name doesn't sound like fun for anyone.

the proposal is not detailed enough to be taken seriously.  when you say
"One fairly simple possibility that *might* work ..." there is a lot of
room for mis-design, mis-implementation and mis-understanding.  perhaps
what is there can be filled out w/ some use-cases that show how the
design holds up to both normal use and weird boundary conditions (don't
forget to handle ignorant non-compliance).

i've just claimed "write modules/arch-survey.text" from TODO and should
be checking in something in the next day or so.  probably you can get
some ideas from there.

another example to look at: i just noticed in guile-rgx-ctax there are
PLUGIN subdirectories, w/ simple interface-definition directives.

thi

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Bill Gribble]

On Tue, 2002-05-14 at 05:57, Thien-Thi Nguyen wrote:
> it looks like a plan to just implement something and throw it against
> the wall to see if it sticks.  in particular, having the interface
> number encoded in the name doesn't sound like fun for anyone.

For someone who spends so much time hopping up and down about process
and development culture, you sure don't hesitate to start throwing
around derogatory and inflammatory language.  Put a cork in it, please! 
Language like this makes you part of the problem, not part of the
solution. 

The whole idea of encoding interface numbers in the name was explicitly
and exclusively a temporary hack.  It certainly wasn't proposed by
_anybody_ as a final solution. 

b.g.



_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Thien-Thi Nguyen]

   From: Bill Gribble <grib@linuxdevel.com>
   Date: 14 May 2002 11:11:12 -0500

   On Tue, 2002-05-14 at 05:57, Thien-Thi Nguyen wrote:
   > it looks like a plan to just implement something and throw it against
   > the wall to see if it sticks.  in particular, having the interface
   > number encoded in the name doesn't sound like fun for anyone.

   For someone who spends so much time hopping up and down about process
   and development culture, you sure don't hesitate to start throwing
   around derogatory and inflammatory language.  Put a cork in it, please! 
   Language like this makes you part of the problem, not part of the
   solution. 

   The whole idea of encoding interface numbers in the name was explicitly
   and exclusively a temporary hack.  It certainly wasn't proposed by
   _anybody_ as a final solution. 

the two areas you mention are independent; i don't see the connection.
good process demands open exchange of different ideas, including the
evaluation of "is this idea sound or will there be problems?" by people
besides one's nanny.

i don't see how hacks that touch /usr/local/lib and require third party
cooperation (by some who are extremely vociferous in their disgust of
bad design, trust me) can be called "temporary".  but let's say that
this does go into code and (less discerning) people buy into it.  what
exactly is the final form of the "guile plugin"?  will it be compatible
w/ this scheme?  if not, what help will you provide to keep me from
cursing your name and switching to SCM+SLIB+Hobbit?  etc.

if you want to dissuade discerning people from characterizing your work
as "throwing it against a wall to see if it sticks", you have to answer
these kinds of questions (by asking them of yourself, or finding some
unpleasant person to ask them for you ;-).  which means you have to know
how your current scheme relates to its final form.  which means you have
to design for the long term.  this is not easy to do, granted.

the general principle applicable here is (once again) encapsulation.
some of these annoying questions can be rendered moot (you could be free
to encode favorite color and zodiac sign in the name, why not?) if the
shared objects need not live in a public dir.  other questions (like how
you help ford incompatibilites) are always in play.

thi

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user

Re: [d.love@dl.ac.uk: dynamic loading of native code modules]

[Thien-Thi Nguyen]

   From: Rob Browning <rlb@defaultvalue.org>
   Date: Wed, 24 Apr 2002 10:28:14 -0500

   i.e. other apps are supposed to be able to
   link appropriately and then call scm_make_u8vector directly.

"directly linkable" is less important an attribute to consider than
"independently linkable".  to continue w/ this example, you still need
libguile around to make sense of `scm_make_u8vector' return value, that
is, libguile-srfi-srfi-4 is not independent of libguile.  understanding
this is crucial to understanding how much freedom you can gain by not
subscribing to system-wide conventions.

   It may well be that the subdirectory approach is suitable, and in fact
   its one I originally leaned toward, but I was shifted away after
   talking to Marius about direct linking and realizing that if we were
   going to properly version all our shared libs to accmodate  etc.

well, then i say you were swayed by fuzzy thinking -- "direct linking"
by 3rd party programs is a red herring for the plug-in system designer.
it's a nice feature to provide, but its implementation is best done
through the interface provided by libguile, aka `load-extension'.

btw, i think your ideas on extending `load-extension' argument list are
a good start.  concentrate on designing this abstraction interface in a
general way and you'll be fine -- do not confuse extension namespace and
sofile namespace.

thi

_______________________________________________
Guile-user mailing list
Guile-user@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-user