ccall mechanism in Julia

ccall mechanism in Julia

[Ricardo Wurmus]

Hi Guix,

I'm currently packaging up Julia, a programming language for technical
computing.  When I configure the build process such that system
libraries are used instead of the bundled copies I run into a problem.

Julia's bindings to these libraries use the ccall mechanism; ccall
builds a map from library names to paths by parsing the output of
"ldconfig -p" at runtime.  I worked around this problem by patching the
sources to include a static map of library names to store paths.  Then I
noticed, however, that my patch effectively cripples the FFI.  It is no
longer possible to call a function that is exported by a shared library
unless it happens to be in the static map that is created at build time.

Before I hack a little more on the sources to implement a solution that
also considers libraries that happen to lie around in
~/.guix-profile/lib I would like to ask for comments.

Would it be a very bad idea if I wrote a simple replacement for
"ldconfig -p" that returned all libraries in the user's profile?  I
don't think this would be as bad as breaking the FFI and hard-coding
build-time paths to inputs.

What do you think?

~~ Ricardo

Re: ccall mechanism in Julia

[Andreas Enge]

On Fri, Mar 06, 2015 at 04:48:59PM +0100, Ricardo Wurmus wrote:
> Julia's bindings to these libraries use the ccall mechanism; ccall
> builds a map from library names to paths by parsing the output of
> "ldconfig -p" at runtime.  I worked around this problem by patching the
> sources to include a static map of library names to store paths.  Then I
> noticed, however, that my patch effectively cripples the FFI.  It is no
> longer possible to call a function that is exported by a shared library
> unless it happens to be in the static map that is created at build time.

This looks like a good approach for guix, where we try to not use random
libraries lying around in the user profile. Setting the rpath with our
ld-wrapper behaves also like a static map. Otherwise, updating some other
library in the user profile would modify the behaviour of julia, which
would contradict our functional approach.

Or did I misunderstand anything?

Andreas

Re: ccall mechanism in Julia

[Ricardo Wurmus]

Andreas Enge writes:

> On Fri, Mar 06, 2015 at 04:48:59PM +0100, Ricardo Wurmus wrote:
>> Julia's bindings to these libraries use the ccall mechanism; ccall
>> builds a map from library names to paths by parsing the output of
>> "ldconfig -p" at runtime.  I worked around this problem by patching the
>> sources to include a static map of library names to store paths.  Then I
>> noticed, however, that my patch effectively cripples the FFI.  It is no
>> longer possible to call a function that is exported by a shared library
>> unless it happens to be in the static map that is created at build time.
>
> This looks like a good approach for guix, where we try to not use random
> libraries lying around in the user profile. Setting the rpath with our
> ld-wrapper behaves also like a static map. Otherwise, updating some other
> library in the user profile would modify the behaviour of julia, which
> would contradict our functional approach.

The problem I see is that a user won't be able to make foreign function
calls to installed libraries at runtime in an interactive Julia
session.

With Guile I'm not limited in this manner because I can arbitrarily
extend the GUILE_LOAD_PATH and use stuff that's not installed to the
store.  With C I can extend the paths as well and compile stuff that are
not in the store, in an exploratory fashion.

If we went with a static map of only the most rudamentary compile-time
dependencies users won't be able to use the flexible FFI in an ad-hoc
manner, severely limiting expression.

I would still want to hard-code the paths of input librarie providing
core features so that the behaviour of Julia would not change dependent
on what libraries are installed, but I think that we should avoid
disabling the FFI for anything but the core libraries.  I would like to
extend the ccall mechanism such that one could make foreign function
calls to installed libraries (or libraries available in some list of
paths).

From what I can tell Julia's FFI requires no boilerplate.  A user just
needs the library name (or rather the key in the sonameMap resolving to
a library path) and the function name to access.  I would very much like
to keep this feature.

How are Guile's or Python's FFI handled?

~~ Ricardo

Re: ccall mechanism in Julia

[Ludovic Courtès]

I think you want to things:

  1. That the libraries that Julia requires are automatically resolved
     to those that were available at build time.  The static map you
     propose is the right thing for that, as Andreas notes.

  2. That any other libraries users may want to load is search for using
     the normal dynamic linker mechanism–essentially LD_LIBRARY_PATH in
     our case.

In Guile, ‘dynamic-link’ is essentially a wrapper around lt_dlopen,
itself a wrapper around dlopen, so LTDL_LIBRARY_PATH and LD_LIBRARY_PATH
are honored (and if libc is configured with ldconfig support, then
/etc/ld*.conf are honored as well.)  This addresses #2.

In Guile applications, such as Guix, we hard-code the absolute path to
libraries that are dlopened, as is the case with libgcrypt in (guix
config) (specifically, see the recipe for Guix in (gnu packages
package-management).)

Yet, Guile knows where to find its own extensions, such as
guile-readline.  For that it has ‘load-extension’, which is sort-of like
‘dynamic-link’ except that it first searches $libdir (roughly.)  That
addresses #1.

HTH!

Ludo’.