Re: [PATCH] load-foreign-library: perform substring match on library files

["Sören Tempel" <soeren@soeren-tempel.net>]

Hi,

Thanks for your feedback, comments below.

Maxime Devos <maximedevos@telenet.be> wrote:
> Long term, I think it would be ideal for Guile to decide upon a major 
> version (and maybe even location, depending on the choices of the 
> distro) at _compile_ time instead of runtime, not unlike how other 
> compilers work.

Sure, that could work too.

> If the Guile module is being compiled with --rpath, it searches
> $CROSS_LIBRARY_PATH or $LIBRARY_PATH and encodes the full file name
> (/usr/lib/.../libguile-... or /gnu/store/.../lib/...) in the .go,
> which avoids some manual patching we have to do in Guix.

What kind of manual patching do you do on Guix? Could you refer me to
the code for that? Maybe that's something we could also do on Alpine in
the meantime.

> IIUC, the string-prefix? search is non-deterministic, which can make 
> debugging complicated when multiple versions are installed. 

Well, on Linux readdir(3) file name order depends on the file system
implementation. Since the algorithm returns the first file with a
substring match, it depends on the readdir order. As long as the same
filesystem is used, it should be deterministic.

> I think it would be better to bail out if there are multiple matches
> instead of a risk of guessing incorrectly.

Many packages provide multiple .so files with different version
granularity so bailing out if there are multiple substring matches
doesn't really work. For example, for libgit2 I have the following files
installed on my system:

	$ ls /usr/lib/libgit2*
	/usr/lib/libgit2.so.1.4
	/usr/lib/libgit2.so.1.4.4

Where the former is a symlink to the latter. However, it would be
possible to collect all substring matches and prioritize them according
to some algorithm (e.g. alphabetical order). This would also make the
algorithm independent of the readdir(3) order.

> The prefixing behaviour is also not documented, so some documentation is 
> needed in the manual.

I can add some documentation if there is interest in merging this.

>   * Does it go scanning the dir even if libfoo.so could be found?
>     Otherwise, there are some possible performance gains by checking for
>     libfoo.so first -- consider the case where /usr/lib is huge.

Yes, a fast path could be added though you probably really need to have
a lot of files installed in /usr/lib for this to be worth it.

>   * When doing (load-foreign-library "/gnu/store/.../libfoo.so")
>     (absolute file name!), would it search for
>     /gnu/store/.../libfoo.so.N?  If so, that would be surprising,
>     especially if libfoo.so.N exists.

Yep, it does. I originally didn't want to modify the handling of
absolute paths but unfortunately during testing I noticed that Guile
extensions seem to be loaded with an absolute path and hence don't
work without the libfoo.so symlink [1].

>   * If doing libfoo.so.N, will it search for libfoo.so.N.M?

Yes, since libfoo.so.N is a prefix of libfoo.so.N.M.

>   * Does it only apply to the system paths, or also to
>     GUILE_SYSTEM_EXTENSION_PATH, LTDL_LIBRARY_PATH and
>     GUILE_EXTENSION_PATH? The latter would be surprising to me, as
>     versioning is more of a system thing.

If those paths are also searched using the load-foreign-library
procedure then they are affected by this change. Also, I am not a Guile
expert but on Alpine, Guile extensions such as guile-reader also ship
versioned sonames [1].

>   * To test it, and avoid breaking things later with future changes to
>     load-foreign-library, could some tests be added?

Probably, though I am not familiar with the Guile test setup and there
don't seem to be any existing tests for foreign-library.

>  * Is this change desirable?  I mean, this is an FFI API, so the ABI of
>    the library is rather important. If a Guile module links to
>    libfoo.so, and they had version N in mind, then it's important it
>    doesn't link to N-1 or N+1 instead, because of ABI
>    incompatibilities. As such, to me it seems _good_ that you got some
>    errors, as now you get a reminder to explicitly state which ABI
>    version is needed. (YMMV, and the mileage of the Guile maintainers
>    might vary, etc.)

In my experience, most languages which don't link against shared
libraries directly but instead load them at run-time don't hardcode ABI
versions (for example, refer to Python's ctypes.util.find_library).
Also, the current implementation of load-foreign-library does not force
you to specify an ABI version but instead loads whatever the libfoo.so
symlink refers to.

> Also, this seems like a non-trivial change to me, so a copyright line 
> might be in order, unless you did the copyright assignment.

I didn't do any copyright assignment yet but if there is actually any
interest in merging this then I can do it once we agreed on changes to
the algorithm.

Greetings,
Sören

[1]: https://gitlab.alpinelinux.org/alpine/aports/-/issues/12783