bug#74532: with-extensions does not add native extensions to the load path

bug#74532: with-extensions does not add native extensions to the load path

[Tomas Volf]

[-- Attachment #1: Type: text/plain, Size: 2778 bytes --]


The documentation for `with-extensions' says:

> In the same vein, sometimes you want to import not just pure-Scheme
> modules, but also “extensions” such as Guile bindings to C libraries
> or other “full-blown” packages.

However it does not actually add those C libraries into
(guile-extensions-path), which means that trying to actually use a
library imported this way will lead to an error.

I guess `load-path-expression' in (guix gexp) needs to also append to
the `guile-extensions-path' based on the search-paths of the imported
packages?

Reproduction is simple.  Make sure you *do not* have guile-yamlpp
installed in your profile.  Then build the `test' into a store item.

--8<---------------cut here---------------start------------->8---
(use-modules (gnu packages guile-xyz))
(define test
  (with-extensions (list guile-yamlpp)
    (program-file "test" #~(use-modules (yamlpp)))))
--8<---------------cut here---------------end--------------->8---

And try to run it:

--8<---------------cut here---------------start------------->8---
$ /gnu/store/nq0sahcnph5sxms4irq4inv2hq2hfbzr-test
Backtrace:
          19 (primitive-load "/gnu/store/nq0sahcnph5sxms4irq4inv2hq2hfbzr-test")
In ice-9/eval.scm:
   721:20 18 (primitive-eval _)
In ice-9/psyntax.scm:
  1229:36 17 (expand-top-sequence _ _ _ #f _ _ _)
  1221:19 16 (parse _ (("placeholder" placeholder)) ((top) #(ribcage () () ())) _ e (eval) #)
   259:10 15 (parse _ (("placeholder" placeholder)) (()) _ c&e (eval) (hygiene guile-user))
In ice-9/boot-9.scm:
  3935:20 14 (process-use-modules _)
   222:17 13 (map1 (((yamlpp))))
  3936:31 12 (_ ((yamlpp)))
  3327:17 11 (resolve-interface (yamlpp) #:select _ #:hide _ #:prefix _ #:renamer _ # _)
In ice-9/threads.scm:
    390:8 10 (_ _)
In ice-9/boot-9.scm:
  3253:13  9 (_)
In ice-9/threads.scm:
    390:8  8 (_ _)
In ice-9/boot-9.scm:
  3544:20  7 (_)
   2836:4  6 (save-module-excursion _)
  3564:26  5 (_)
In unknown file:
           4 (primitive-load-path "yamlpp" #<procedure 7f405e4fd560 at ice-9/boot-9.scm:35…>)
In yamlpp.scm:
     65:1  3 (_)
In unknown file:
           2 (load-extension "libguile-yamlpp" "init")
In system/foreign-library.scm:
   190:25  1 (load-foreign-library _ #:extensions _ #:search-ltdl-library-path? _ # _ # _ # …)
In unknown file:
           0 (dlopen "libguile-yamlpp.so" 1)

ERROR: In procedure dlopen:
In procedure dlopen: file "libguile-yamlpp.so", message "libguile-yamlpp.so: cannot open shared object file: No such file or directory"
--8<---------------cut here---------------end--------------->8---

Have a nice day,
Tomas

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 853 bytes --]

bug#74532: with-extensions does not add native extensions to the load path

[Ludovic Courtès]

Hi,

Tomas Volf <~@wolfsden.cz> skribis:

> The documentation for `with-extensions' says:
>
>> In the same vein, sometimes you want to import not just pure-Scheme
>> modules, but also “extensions” such as Guile bindings to C libraries
>> or other “full-blown” packages.

It’s not just bindings but also pure Scheme libraries like Guile-JSON.

> However it does not actually add those C libraries into
> (guile-extensions-path), which means that trying to actually use a
> library imported this way will lead to an error.

[...]

>    190:25  1 (load-foreign-library _ #:extensions _ #:search-ltdl-library-path? _ # _ # _ # …)
> In unknown file:
>            0 (dlopen "libguile-yamlpp.so" 1)
>
> ERROR: In procedure dlopen:
> In procedure dlopen: file "libguile-yamlpp.so", message "libguile-yamlpp.so: cannot open shared object file: No such file or directory"

Usually, packages like these (guile-gnutls, guile-git, guile-ssh, etc.)
have their .so absolute file name hard-coded, which sidesteps this
problem entirely.  I recommend doing that for guile-yamlpp as well.

That said, it would probably make sense to arrange for ‘with-extensions’
to set GUILE_EXTENSIONS_PATH.

Ludo’.

PS: Your MUA sets “Mail-Followup-To: bug-guix@gnu.org”, which is kinda
    annoying because that’s the wrong address when replying to a bug.
    :-)

bug#74532: with-extensions does not add native extensions to the load path

[Tomas Volf]

[-- Attachment #1: Type: text/plain, Size: 3408 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> Hi,
>
> Tomas Volf <~@wolfsden.cz> skribis:
>
>> The documentation for `with-extensions' says:
>>
>>> In the same vein, sometimes you want to import not just pure-Scheme
>>> modules, but also “extensions” such as Guile bindings to C libraries
>>> or other “full-blown” packages.
>
> It’s not just bindings but also pure Scheme libraries like Guile-JSON.

True, and for those it works well.  However the text documentation
explicitly mentions "bindings to C libraries" as one of the use
cases. :)

>
>> However it does not actually add those C libraries into
>> (guile-extensions-path), which means that trying to actually use a
>> library imported this way will lead to an error.
>
> [...]
>
>>    190:25  1 (load-foreign-library _ #:extensions _ #:search-ltdl-library-path? _ # _ # _ # …)
>> In unknown file:
>>            0 (dlopen "libguile-yamlpp.so" 1)
>>
>> ERROR: In procedure dlopen:
>> In procedure dlopen: file "libguile-yamlpp.so", message "libguile-yamlpp.so: cannot open shared object file: No such file or directory"
>
> Usually, packages like these (guile-gnutls, guile-git, guile-ssh, etc.)
> have their .so absolute file name hard-coded, which sidesteps this
> problem entirely.  I recommend doing that for guile-yamlpp as well.

How it that usually done?  I cannot do that *before* build, because then
tests would not run (the library is not yet installed into the absolute
path), and I cannot do it after, because I would need to rebuild the .go
files after patching the source code.

Assuming I have a guile library that creates a new .so library during
the build, and the .so library must be loadable by the other modules in
the library during the build (and for the tests), how should I approach
that?

I am pretty sure I cannot just patch the source code, since the library
would not be installed into the absolute path yet during the build.

Honestly, setting GUILE_EXTENSIONS_PATH via pre-inst-env seemed like
fairly elegant solution.

>
> That said, it would probably make sense to arrange for ‘with-extensions’
> to set GUILE_EXTENSIONS_PATH.

That would be great.  My current work-around is

  (with-extensions (list guile-wolfsden)
    (program-file
     "audio-cycle-sinks"
     #~(begin
         ;; Bug 74532: Native extensions are not added to the load path.
         (eval-when (expand load eval)
           (let ((ext-path (@ (system foreign-library) guile-extensions-path)))
             ;; Just a temporary hack, we can live with duplicates in the path.
             (ext-path (cons #$(file-append guile-wolfsden
                                            "/lib/guile/3.0/extensions")
                             (ext-path)))))
         ...)))

which is anything but elegant.

>
> Ludo’.
>
> PS: Your MUA sets “Mail-Followup-To: bug-guix@gnu.org”, which is kinda
>     annoying because that’s the wrong address when replying to a bug.
>     :-)

Thanks for letting me know, I was not aware of that.  After reading
(message)Mailing Lists bit more carefully, adjusting the Posting Styles
and sending a bug fix to the Emacs' bug tracker, I believe it should not
happen anymore.  Sorry for the annoyance.

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 853 bytes --]

bug#74532: with-extensions does not add native extensions to the load path

[Z572]

[-- Attachment #1: Type: text/plain, Size: 3628 bytes --]

Tomas Volf <~@wolfsden.cz> writes:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Hi,
>>
>> Tomas Volf <~@wolfsden.cz> skribis:
>>
>>> The documentation for `with-extensions' says:
>>>
>>>> In the same vein, sometimes you want to import not just pure-Scheme
>>>> modules, but also “extensions” such as Guile bindings to C libraries
>>>> or other “full-blown” packages.
>>
>> It’s not just bindings but also pure Scheme libraries like Guile-JSON.
>
> True, and for those it works well.  However the text documentation
> explicitly mentions "bindings to C libraries" as one of the use
> cases. :)
>
>>
>>> However it does not actually add those C libraries into
>>> (guile-extensions-path), which means that trying to actually use a
>>> library imported this way will lead to an error.
>>
>> [...]
>>
>>>    190:25  1 (load-foreign-library _ #:extensions _ #:search-ltdl-library-path? _ # _ # _ # …)
>>> In unknown file:
>>>            0 (dlopen "libguile-yamlpp.so" 1)
>>>
>>> ERROR: In procedure dlopen:
>>> In procedure dlopen: file "libguile-yamlpp.so", message "libguile-yamlpp.so: cannot open shared object file: No such file or directory"
>>
>> Usually, packages like these (guile-gnutls, guile-git, guile-ssh, etc.)
>> have their .so absolute file name hard-coded, which sidesteps this
>> problem entirely.  I recommend doing that for guile-yamlpp as well.
>
> How it that usually done?  I cannot do that *before* build, because then
> tests would not run (the library is not yet installed into the absolute
> path), and I cannot do it after, because I would need to rebuild the .go
> files after patching the source code.

you can use substitute* to adjust source.
e.g.

(or (false-if-exception (load-extension "/path/to/lib-some-object-file.so"))
    (load-extension "lib-some-object-file.so"))

>
> Assuming I have a guile library that creates a new .so library during
> the build, and the .so library must be loadable by the other modules in
> the library during the build (and for the tests), how should I approach
> that?
>
> I am pretty sure I cannot just patch the source code, since the library
> would not be installed into the absolute path yet during the build.
>
> Honestly, setting GUILE_EXTENSIONS_PATH via pre-inst-env seemed like
> fairly elegant solution.
>
>>
>> That said, it would probably make sense to arrange for ‘with-extensions’
>> to set GUILE_EXTENSIONS_PATH.
>
> That would be great.  My current work-around is
>
>   (with-extensions (list guile-wolfsden)
>     (program-file
>      "audio-cycle-sinks"
>      #~(begin
>          ;; Bug 74532: Native extensions are not added to the load path.
>          (eval-when (expand load eval)
>            (let ((ext-path (@ (system foreign-library) guile-extensions-path)))
>              ;; Just a temporary hack, we can live with duplicates in the path.
>              (ext-path (cons #$(file-append guile-wolfsden
>                                             "/lib/guile/3.0/extensions")
>                              (ext-path)))))
>          ...)))
>
> which is anything but elegant.
>
>>
>> Ludo’.
>>
>> PS: Your MUA sets “Mail-Followup-To: bug-guix@gnu.org”, which is kinda
>>     annoying because that’s the wrong address when replying to a bug.
>>     :-)
>
> Thanks for letting me know, I was not aware of that.  After reading
> (message)Mailing Lists bit more carefully, adjusting the Posting Styles
> and sending a bug fix to the Emacs' bug tracker, I believe it should not
> happen anymore.  Sorry for the annoyance.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

bug#74532: with-extensions does not add native extensions to the load path

[Tomas Volf]

[-- Attachment #1: Type: text/plain, Size: 1355 bytes --]


Hello,

Z572 <zhengjunjie@iscas.ac.cn> writes:

>> How it that usually done?  I cannot do that *before* build, because then
>> tests would not run (the library is not yet installed into the absolute
>> path), and I cannot do it after, because I would need to rebuild the .go
>> files after patching the source code.
>
> you can use substitute* to adjust source.
> e.g.
>
> (or (false-if-exception (load-extension "/path/to/lib-some-object-file.so"))
>     (load-extension "lib-some-object-file.so"))
>

Thank you for the suggestion.  I am not sure I like the
false-if-exception part, but temporary adjustment of the (@ (system
foreign-library) guile-extensions-path) should do the trick as well.

One thing I am curious about, I can see you have put the load from
absolute path as the first alternative.  But if it fails, it still falls
back to lookup based on the guile-extensions-path.  I wonder whether
putting it at the end of the list would not be better, that way you
could still replace the library in the spirit of LD_PRELOAD by setting
the appropriate environment variable.  And if reproducibility is the
priority, then even the `or' might not make sense.

Any thoughts on that?

Thanks,
Tomas

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 853 bytes --]

bug#74532: with-extensions does not add native extensions to the load path

[Ludovic Courtès]

Hi,

Tomas Volf <~@wolfsden.cz> skribis:

> Z572 <zhengjunjie@iscas.ac.cn> writes:
>
>>> How it that usually done?  I cannot do that *before* build, because then
>>> tests would not run (the library is not yet installed into the absolute
>>> path), and I cannot do it after, because I would need to rebuild the .go
>>> files after patching the source code.
>>
>> you can use substitute* to adjust source.
>> e.g.
>>
>> (or (false-if-exception (load-extension "/path/to/lib-some-object-file.so"))
>>     (load-extension "lib-some-object-file.so"))
>>
>
> Thank you for the suggestion.  I am not sure I like the
> false-if-exception part, but temporary adjustment of the (@ (system
> foreign-library) guile-extensions-path) should do the trick as well.

I think you can just do:

  (load-extension "/path/to/lib-some-object-file.so")

but you need to (1) do that in a post-install phase, and (2) re-run
‘make’ before ‘make install’ if necessary.

See ‘guile-gdbm-ffi’ as one example.

Ludo’.