Re: LuaJIT ffi issue with dynamic libraries

[Ekaitz Zarraga <ekaitz@elenq.tech>]

Hi,

I'm pushing this again just in case anyone has an answer.

Thanks!

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On Thursday, November 25th, 2021 at 5:46 PM, Ekaitz Zarraga <ekaitz@elenq.tech> wrote:

> Hi all,
>
> I've been having some issues with LuaJIT's FFI. More specifically with the
> `load` function, that loads a library in the "search path for dynamic
> libraries".
>
> The LuaJIT documentation says the following:
>
> > On POSIX systems, if the name contains no dot, the extension .so is appended.
> > Also, the lib prefix is prepended if necessary. So ffi.load("z") looks for
> > "libz.so" in the default shared library search path.
>
>
> In my case I'm unable to load a library, in this case `librec.so` simply by
> calling `ffi.load` even if both are part of the same environment. I read a
> little bit of the LuaJIT code and I found that it is doing a `dlopen` inside so
> I made a test:
>
> First I made a simple C file with a call to `dlopen` and straced its result. It
> simply finds the library after proving some folders. I also noted that the
> environment automatically set the `LIBRARY_PATH` variable for me (even if the
> value of the variable does not match the patch where librec.so is being
> searched):
>
> --8<---------------cut here---------------start------------->8---
> $ guix shell gcc-toolchain strace recutils
> $ cat <<EOF > dl.c
> > #include<dlfcn.h>
> >
> > int main(int argc, char *argv[]){
> >     void * i = dlopen("librec.so", RTLD_LAZY);
> >     return (int)i;
> > }
> EOF
> $ gcc dl.c -ldl
> $ strace ./a.out 2>&1 | grep librec.so
> openat(AT_FDCWD, "/gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/lib/librec.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
> openat(AT_FDCWD, "/gnu/store/28bcmy08ki5krvr2g9hbm3bhys822fvn-gcc-11.2.0-lib/lib/librec.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
> openat(AT_FDCWD, "/gnu/store/fz52krs37spvca6q0g0zmx6jmc1n388g-profile/lib/tls/haswell/x86_64/librec.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
> openat(AT_FDCWD, "/gnu/store/fz52krs37spvca6q0g0zmx6jmc1n388g-profile/lib/tls/haswell/librec.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
> openat(AT_FDCWD, "/gnu/store/fz52krs37spvca6q0g0zmx6jmc1n388g-profile/lib/tls/x86_64/librec.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
> openat(AT_FDCWD, "/gnu/store/fz52krs37spvca6q0g0zmx6jmc1n388g-profile/lib/tls/librec.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
> openat(AT_FDCWD, "/gnu/store/fz52krs37spvca6q0g0zmx6jmc1n388g-profile/lib/haswell/x86_64/librec.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
> openat(AT_FDCWD, "/gnu/store/fz52krs37spvca6q0g0zmx6jmc1n388g-profile/lib/haswell/librec.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
> openat(AT_FDCWD, "/gnu/store/fz52krs37spvca6q0g0zmx6jmc1n388g-profile/lib/x86_64/librec.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
> openat(AT_FDCWD, "/gnu/store/fz52krs37spvca6q0g0zmx6jmc1n388g-profile/lib/librec.so", O_RDONLY|O_CLOEXEC) = 3
> $ echo $LIBRARY_PATH
> /gnu/store/hiiyy2b8pw2cq9mw8iik9xi9lpclcv7b-profile/lib
> --8<---------------cut here---------------end--------------->8---
>
> When I do a similar approach with LuaJIT I get the following:
>
> - The paths the `dlopen` call is proving are different to the previous case and
>   they are pretty weird too.
> - There's no `LIBRARY_PATH` set.
> - Even if I set `LIBRARY_PATH` myself it's ignored.
> - It works correctly when I set `LD_LIBRARY_PATH`, but should I?
>
>
> --8<---------------cut here---------------start------------->8---
> $ guix shell luajit strace recutils
> $ cat <<EOF > dl.lua
> > ffi = require "ffi"
> > ffi.load "rec"
> > EOF
> $ strace luajit dl.lua 2>&1  | grep librec.so
> openat(AT_FDCWD, "/gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/lib/librec.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
> openat(AT_FDCWD, "/gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/librec.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
> openat(AT_FDCWD, "/gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/7.5.0/../../../tls/haswell/x86_64/librec.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
> openat(AT_FDCWD, "/gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/7.5.0/../../../tls/haswell/librec.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
> openat(AT_FDCWD, "/gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/7.5.0/../../../tls/x86_64/librec.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
> openat(AT_FDCWD, "/gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/7.5.0/../../../tls/librec.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
> openat(AT_FDCWD, "/gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/7.5.0/../../../haswell/x86_64/librec.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
> openat(AT_FDCWD, "/gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/7.5.0/../../../haswell/librec.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
> openat(AT_FDCWD, "/gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/7.5.0/../../../x86_64/librec.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
> openat(AT_FDCWD, "/gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/7.5.0/../../../librec.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
> openat(AT_FDCWD, "/gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/lib/librec.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
> $ echo $LIBRARY_PATH
>
> $ export LIBRARY_PATH=$GUIX_ENVIRONMENT/lib
> $ strace luajit dl.lua 2>&1  | grep librec.so
> #### SAME AS BEFORE ####
> $ export LD_LIBRARY_PATH=$GUIX_ENVIRONMENT/lib
> $ echo $LD_LIBRARY_PATH
> /gnu/store/5v09kbdda7966x263cgnva2z15yrdbfb-profile/lib
> $ strace luajit dl.lua 2>&1  | grep librec.so
> openat(AT_FDCWD, "/gnu/store/5v09kbdda7966x263cgnva2z15yrdbfb-profile/lib/librec.so", O_RDONLY|O_CLOEXEC) = 3
> --8<---------------cut here---------------end--------------->8---
>
> Does any of this make sense?
>
> The easiest conclusion is that LuaJIT is not working out the box and we should
> fix it, but I'm not sure where to start because I'm assuming both examples here
> come from the same concept: making a similar call to `dlopen`. There must be
> some context I'm missing here to solve this.
>
> Any ideas?
>
>
> ElenQ Technology
> Ethical Innovation