bug#21076: dynamic-link often fails to load libraries

bug#21076: dynamic-link often fails to load libraries

[Frank Webster]

On my Debian system, `dynamic-link' often fails to load shared
libraries:

Loading using a versioned soname never works:

scheme@(guile-user)> (dynamic-link "libGL.so.1")
ERROR: In procedure dynamic-link:
ERROR: In procedure dynamic-link: file: "libGL.so.1", message: "file not found"

Loading without a library extension doesn't work unless the
corresponding *-dev package is installed:

scheme@(guile-user)> (dynamic-link "libGL")
ERROR: In procedure dynamic-link:
ERROR: In procedure dynamic-link: file: "libGL", message: "file not found"

It is supposed to work with or without the extension, however because
of limitations in the underlying lt_dlopenext function in libtool,
neither works.

The first doesn't work because contrary to its
documentation, lt_dlopenext doesn't always try loading
the bare filename first, without appending an extention.

The second doesn't work because the versioned soname extension (.so.1)
isn't among the ones that lt_dlopenext tries to append. It does try the
unversioned extension (.so), but in Debian the unversioned symlink
is only available when the corresponding *-dev package is installed.

Here is a related libtool bug report: https://debbugs.gnu.org/8976

Note that there are arguably two distinct issues:
(1) lt_dlopenext not trying the bare filename in all cases
(2) lt_dlopenext not trying versioned soname extensions.

Obviously one way to address this would be to fix these two issues in
lt_dlopenext in libtool.

Alternatively, the (system foreign) module in guile could provide a
simple low-level wrapper around lt_dlopen, and possibly implement
the higher level functionality of `dynamic-link' (additional search
paths, guessing extensions etc.) in scheme.

bug#21076: dynamic-link often fails to load libraries

[Andreas Rottmann]

Frank Webster <f.webster@yandex.com> writes:

> On my Debian system, `dynamic-link' often fails to load shared
> libraries:
>
> Loading using a versioned soname never works:
>
> scheme@(guile-user)> (dynamic-link "libGL.so.1")
> ERROR: In procedure dynamic-link:
> ERROR: In procedure dynamic-link: file: "libGL.so.1", message: "file not found"
>
> Loading without a library extension doesn't work unless the
> corresponding *-dev package is installed:
>
> scheme@(guile-user)> (dynamic-link "libGL")
> ERROR: In procedure dynamic-link:
> ERROR: In procedure dynamic-link: file: "libGL", message: "file not found"
>
> It is supposed to work with or without the extension, however because
> of limitations in the underlying lt_dlopenext function in libtool,
> neither works.
>
I'd argue that using the second variant (or using `(dynamic-link
"libGL.so")') is kinda inherently broken in an application loading
general-use (public) shared libraries, as opposed to plugins specific to
an application: you have no idea what ABI the library you just loaded
has. This means you may later use of symbols from that library in a way
that violates the ABI.

Unfortunatly, how the ABI is encoded into the shared library name is
platform-dependent, and this is also (as I read between the lines of the
bug thread referenced by you [0]) one of the reasons why the libtool
developers are not sure how a solution to this problem should look
like.

I think to write an application that is somewhat protected against ABI
changes, you need to have a mapping from ABIs supported by your
application to shared library file names (including the version
extension). You might have only one supported ABI, but the mapping is
still platform-dependent, so you have to either have a run-time or
build/install-time switch choosing the appriopriate shared library name.

Now, to make matters worse, you can't do that using libltdl, as you
noticed:

> The first doesn't work because contrary to its
> documentation, lt_dlopenext doesn't always try loading
> the bare filename first, without appending an extention.
>
Yes, this is a really annoying bug, and is what kept me from (trying to)
port sbank[1] (a gobject-introspection binding) to Guile a few years
back. In gobject-introspection, you get the full shared library file
name, along with a machine-readable ABI description, but you can't open
the shared library using the versioned name in Guile.

[1] https://github.com/rotty/sbank

> The second doesn't work because the versioned soname extension (.so.1)
> isn't among the ones that lt_dlopenext tries to append. It does try the
> unversioned extension (.so), but in Debian the unversioned symlink
> is only available when the corresponding *-dev package is installed.
>
Yeah, but IMHO, this kind of use with a bare shared library name
(without SONAME/ABI information) is already misguided in the first
place, unless you have some other way of knowing the ABI. Debian and the
likes not installing .so files (or .la files) by default only exposes
the inherent potential breakage.

[0]
> Here is a related libtool bug report: https://debbugs.gnu.org/8976
>
> Note that there are arguably two distinct issues:
> (1) lt_dlopenext not trying the bare filename in all cases
> (2) lt_dlopenext not trying versioned soname extensions.
>
> Obviously one way to address this would be to fix these two issues in
> lt_dlopenext in libtool.
>
As mentioned, solving problem (2) is not a good idea IMO -- the versions
are there for a reason, and you cannot (should not) guess them.

> Alternatively, the (system foreign) module in guile could provide a
> simple low-level wrapper around lt_dlopen, and possibly implement
> the higher level functionality of `dynamic-link' (additional search
> paths, guessing extensions etc.) in scheme.
>
+1

Regards, Rotty
-- 
Andreas Rottmann -- <http://rotty.xx.vu/>

bug#21076: dynamic-link often fails to load libraries

[Andy Wingo]

On Fri 17 Jul 2015 01:00, Andreas Rottmann <a.rottmann@gmx.at> writes:

>> Alternatively, the (system foreign) module in guile could provide a
>> simple low-level wrapper around lt_dlopen, and possibly implement
>> the higher level functionality of `dynamic-link' (additional search
>> paths, guessing extensions etc.) in scheme.
>>
> +1

Either one of yall want to give a shot at this or something like it?  I
think in an ideal world we could lose the ltdl dependency.  I gave a
first crack at it but got mired in C; I never considered implementing it
in Scheme, which seems like a nice idea.

Andy

bug#21076: dynamic-link often fails to load libraries

[Isaac Jurado]

[-- Attachment #1: Type: text/plain, Size: 473 bytes --]

I've experienced this issue as well, but the fix is very simple
(attaching Git patch).

The fix does NOT break foreign.test but I have no idea, yet, on how to
test the new behaviour in a platform independent or continuous
integration friendly way.

The commit has been applied to the stable-2.2 branch, but it would
also apply cleanly to master.

Any feedback welcomed.

Best regards.

-- 
Isaac Jurado

"The noblest pleasure is the joy of understanding"
Leonardo da Vinci

[-- Attachment #2: 0001-Interpret-dynamic-library-name-as-literal-path-first.patch --]
[-- Type: text/x-patch, Size: 2402 bytes --]

From b337d412b3c10cabe355df07d2295c4d0a560b10 Mon Sep 17 00:00:00 2001
From: Isaac Jurado <diptongo@gmail.com>
Date: Sat, 8 Jun 2019 14:00:29 +0200
Subject: [PATCH] Interpret dynamic library name as literal path first.

Fixes <https://bugs.gnu.org/21076>.

* libguile/dynl.c (sysdep_dyn_link): Try plain lt_dlopen first, to
  interpret fname as a literal path.
* doc/ref/api-foreign.texi: Update explanation to describe the new
  behavior.
---
 doc/ref/api-foreign.texi | 13 ++++++++-----
 libguile/dynl.c          |  7 +++----
 2 files changed, 11 insertions(+), 9 deletions(-)

diff --git a/doc/ref/api-foreign.texi b/doc/ref/api-foreign.texi
index d99a33300..94fabf23c 100644
--- a/doc/ref/api-foreign.texi
+++ b/doc/ref/api-foreign.texi
@@ -75,11 +75,14 @@ Scheme object suitable for representing the linked object file.
 Otherwise an error is thrown.  How object files are searched is system
 dependent.
 
-Normally, @var{library} is just the name of some shared library file
-that will be searched for in the places where shared libraries usually
-reside, such as in @file{/usr/lib} and @file{/usr/local/lib}.
-
-@var{library} should not contain an extension such as @code{.so}.  The
+Guile first tries to load @var{library} as a full path to a shared
+library file.  If that fails, then it falls back to interpret
+@var{library} as just the name of some shared library that will be
+searched for in the places where shared libraries usually reside, such
+as @file{/usr/lib} and @file{/usr/local/lib}.
+
+@var{library} should not contain an extension such as @code{.so}, unless
+@var{library} represents the full path to the shared library file.  The
 correct file name extension for the host operating system is provided
 automatically, according to libltdl's rules (@pxref{Libltdl interface,
 lt_dlopenext, @code{lt_dlopenext}, libtool, Shared Library Support for
diff --git a/libguile/dynl.c b/libguile/dynl.c
index 2a25e5d2e..0096e8cf5 100644
--- a/libguile/dynl.c
+++ b/libguile/dynl.c
@@ -79,10 +79,9 @@ sysdep_dynl_link (const char *fname, const char *subr)
 {
   lt_dlhandle handle;
 
-  if (fname == NULL)
-    /* Return a handle for the program as a whole.  */
-    handle = lt_dlopen (NULL);
-  else
+  /* Try the literal filename first or, if NULL, the program itself */
+  handle = lt_dlopen (fname);
+  if (handle == NULL)
     {
       handle = lt_dlopenext (fname);
 
-- 
2.17.1

bug#21076: dynamic-link often fails to load libraries

[Isaac Jurado]

[-- Attachment #1: Type: text/plain, Size: 473 bytes --]

I've experienced this issue as well, but the fix is very simple
(attaching Git patch).

The fix does NOT break foreign.test but I have no idea, yet, on how to
test the new behaviour in a platform independent or continuous
integration friendly way.

The commit has been applied to the stable-2.2 branch, but it would
also apply cleanly to master.

Any feedback welcomed.

Best regards.

-- 
Isaac Jurado

"The noblest pleasure is the joy of understanding"
Leonardo da Vinci

[-- Attachment #2: 0001-Interpret-dynamic-library-name-as-literal-path-first.patch --]
[-- Type: text/x-patch, Size: 2402 bytes --]

From b337d412b3c10cabe355df07d2295c4d0a560b10 Mon Sep 17 00:00:00 2001
From: Isaac Jurado <diptongo@gmail.com>
Date: Sat, 8 Jun 2019 14:00:29 +0200
Subject: [PATCH] Interpret dynamic library name as literal path first.

Fixes <https://bugs.gnu.org/21076>.

* libguile/dynl.c (sysdep_dyn_link): Try plain lt_dlopen first, to
  interpret fname as a literal path.
* doc/ref/api-foreign.texi: Update explanation to describe the new
  behavior.
---
 doc/ref/api-foreign.texi | 13 ++++++++-----
 libguile/dynl.c          |  7 +++----
 2 files changed, 11 insertions(+), 9 deletions(-)

diff --git a/doc/ref/api-foreign.texi b/doc/ref/api-foreign.texi
index d99a33300..94fabf23c 100644
--- a/doc/ref/api-foreign.texi
+++ b/doc/ref/api-foreign.texi
@@ -75,11 +75,14 @@ Scheme object suitable for representing the linked object file.
 Otherwise an error is thrown.  How object files are searched is system
 dependent.
 
-Normally, @var{library} is just the name of some shared library file
-that will be searched for in the places where shared libraries usually
-reside, such as in @file{/usr/lib} and @file{/usr/local/lib}.
-
-@var{library} should not contain an extension such as @code{.so}.  The
+Guile first tries to load @var{library} as a full path to a shared
+library file.  If that fails, then it falls back to interpret
+@var{library} as just the name of some shared library that will be
+searched for in the places where shared libraries usually reside, such
+as @file{/usr/lib} and @file{/usr/local/lib}.
+
+@var{library} should not contain an extension such as @code{.so}, unless
+@var{library} represents the full path to the shared library file.  The
 correct file name extension for the host operating system is provided
 automatically, according to libltdl's rules (@pxref{Libltdl interface,
 lt_dlopenext, @code{lt_dlopenext}, libtool, Shared Library Support for
diff --git a/libguile/dynl.c b/libguile/dynl.c
index 2a25e5d2e..0096e8cf5 100644
--- a/libguile/dynl.c
+++ b/libguile/dynl.c
@@ -79,10 +79,9 @@ sysdep_dynl_link (const char *fname, const char *subr)
 {
   lt_dlhandle handle;
 
-  if (fname == NULL)
-    /* Return a handle for the program as a whole.  */
-    handle = lt_dlopen (NULL);
-  else
+  /* Try the literal filename first or, if NULL, the program itself */
+  handle = lt_dlopen (fname);
+  if (handle == NULL)
     {
       handle = lt_dlopenext (fname);
 
-- 
2.17.1

bug#21076: dynamic-link often fails to load libraries

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 939 bytes --]

Hi Isaac,

Isaac Jurado <diptongo@gmail.com> skribis:

> From b337d412b3c10cabe355df07d2295c4d0a560b10 Mon Sep 17 00:00:00 2001
> From: Isaac Jurado <diptongo@gmail.com>
> Date: Sat, 8 Jun 2019 14:00:29 +0200
> Subject: [PATCH] Interpret dynamic library name as literal path first.
>
> Fixes <https://bugs.gnu.org/21076>.
>
> * libguile/dynl.c (sysdep_dyn_link): Try plain lt_dlopen first, to
>   interpret fname as a literal path.
> * doc/ref/api-foreign.texi: Update explanation to describe the new
>   behavior.

It’s taken a while (!), but thanks to Matt Wette’s ping, it’s now
applied on ‘master’ and will make it into 3.0.2.

I slightly modified the documentation wording as shown in the patch
below (the GNU convention is to use “file name” for file names and
“path” for search paths).

Like you wrote, it’s hard to test, but I think it’s OK to leave it as is.

Thanks!

Ludo’.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 1227 bytes --]

diff --git a/doc/ref/api-foreign.texi b/doc/ref/api-foreign.texi
index 94fabf23c..b0d6c249b 100644
--- a/doc/ref/api-foreign.texi
+++ b/doc/ref/api-foreign.texi
@@ -75,14 +75,14 @@ Scheme object suitable for representing the linked object file.
 Otherwise an error is thrown.  How object files are searched is system
 dependent.
 
-Guile first tries to load @var{library} as a full path to a shared
-library file.  If that fails, then it falls back to interpret
+Guile first tries to load @var{library} as the absolute file name of a shared
+library.  If that fails, it then falls back to interpret
 @var{library} as just the name of some shared library that will be
 searched for in the places where shared libraries usually reside, such
 as @file{/usr/lib} and @file{/usr/local/lib}.
 
 @var{library} should not contain an extension such as @code{.so}, unless
-@var{library} represents the full path to the shared library file.  The
+@var{library} represents the absolute file name to the shared library.  The
 correct file name extension for the host operating system is provided
 automatically, according to libltdl's rules (@pxref{Libltdl interface,
 lt_dlopenext, @code{lt_dlopenext}, libtool, Shared Library Support for

bug#21076: dynamic-link

[Matt Wette]

I have encountered this issue with dynamic-link.
I propose a small patch to Guile (works on v3.0.1) to address it.
At least this allows me to load the file if I know the exact name.

This patch fixes issue with libtool and OSes which use
non-standard extensions for shared libraries.  The libltdl
function lt_dlopenext expects the argument to have a standard
extension or no extention.  Arguments reflecting shared object
files with non-standard extensions will fail to load.  For
example, my system has /usr/lib64/libyaml-0.so.2.  w/o this
patch (dynamic-link "libyaml-0.so.2") does not work; with it,
it does.  Go figure.

--- libguile/dynl.c-orig    2020-03-20 05:56:42.101214929 -0700
+++ libguile/dynl.c    2020-03-20 05:57:40.432893330 -0700
@@ -69,7 +69,10 @@
      handle = lt_dlopen (NULL);
    else
      {
-      handle = lt_dlopenext (fname);
+      handle = lt_dlopen (fname);
+
+      if (handle == NULL)
+         handle = lt_dlopenext (fname);

        if (handle == NULL
  #ifdef LT_DIRSEP_CHAR