unofficial mirror of guix-devel@gnu.org 
 help / color / mirror / code / Atom feed
* Installing a wrapper guile script in <out>/bin
@ 2021-02-01 11:28 elaexuotee
  2021-02-01 16:04 ` Ludovic Courtès
  0 siblings, 1 reply; 9+ messages in thread
From: elaexuotee @ 2021-02-01 11:28 UTC (permalink / raw)
  To: guix-devel

Hello Guix,

Writing package definition, I have need of a non-trivial wrapper script that
decides how to execute the installed binary. How do I accomplish this?

With my vague understanding, I am envisioning writing a gexp directly in the
install phase and would like to somehow reify this into a guile script and
install that file under <out>/bin. Is this correct, at the high level?

More specifically, the package I have builds separate libraries for CPUs with
AVX, AVX2, and no AVX support. Since build-type isn't sufficiently specific to
distinguish such CPU features, I have, so far, opted to just build all three
libs and stuff them under <out>/lib/<foo>.

My idea is to have the linker script check CPU features at runtime (by parsing
/proc/cpuinfo or something) and executing the binary with the parameters to
load the correct binary.

Perhaps there is a better overall approach?

Appreciate your thoughts!


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Installing a wrapper guile script in <out>/bin
@ 2021-02-01 12:16 Leo Prikler
  0 siblings, 0 replies; 9+ messages in thread
From: Leo Prikler @ 2021-02-01 12:16 UTC (permalink / raw)
  To: elaexuotee; +Cc: guix-devel

Hi elaexuotee,

> More specifically, the package I have builds separate libraries for
> CPUs with
> AVX, AVX2, and no AVX support. Since build-type isn't sufficiently
> specific to
> distinguish such CPU features, I have, so far, opted to just build
> all three
> libs and stuff them under <out>/lib/<foo>.
That's certainly one approach to solving the issue of not knowing which
CPU your code runs on.  Another approach is described in [1] under the
section "Dependency graph rewriting".  TL;DR, just offer the base
package with no AVX features, an AVX package and an AVX2 package.  Then
build any dependants against the base package.  Users can afterwards
decide to build against optimized versions on their own.

Regards,
Leo

[1] 
https://hpc.guix.info/blog/2018/01/pre-built-binaries-vs-performance/



^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Installing a wrapper guile script in <out>/bin
  2021-02-01 11:28 Installing a wrapper guile script in <out>/bin elaexuotee
@ 2021-02-01 16:04 ` Ludovic Courtès
  2021-02-03  5:50   ` elaexuotee
  0 siblings, 1 reply; 9+ messages in thread
From: Ludovic Courtès @ 2021-02-01 16:04 UTC (permalink / raw)
  To: elaexuotee; +Cc: guix-devel

Hi!

elaexuotee@wilsonb.com skribis:

> More specifically, the package I have builds separate libraries for CPUs with
> AVX, AVX2, and no AVX support. Since build-type isn't sufficiently specific to
> distinguish such CPU features, I have, so far, opted to just build all three
> libs and stuff them under <out>/lib/<foo>.
>
> My idea is to have the linker script check CPU features at runtime (by parsing
> /proc/cpuinfo or something) and executing the binary with the parameters to
> load the correct binary.
>
> Perhaps there is a better overall approach?

I wrote about this topic in the past:

  https://hpc.guix.info/blog/2018/01/pre-built-binaries-vs-performance/

I you’re the upstream author, I recommend using one of the techniques
given above to provide so-called “fat binaries” that contain several
implementations of the performance-sensitive functions; the loader
will pick the right implementation when the program starts.

If you’re downstream… it depends on the specifics.  The loader is also
able to pick a .so from the right lib/ sub-directory depending on the
micro-architecture.  You can try:

  LD_DEBUG=files your program

to see where the loader looks for shared libraries.

HTH!

Ludo’.


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Installing a wrapper guile script in <out>/bin
  2021-02-01 16:04 ` Ludovic Courtès
@ 2021-02-03  5:50   ` elaexuotee
  2021-02-03 10:00     ` Maxime Devos
  0 siblings, 1 reply; 9+ messages in thread
From: elaexuotee @ 2021-02-03  5:50 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: guix-devel

Ludovic Courtès <ludovic.courtes@inria.fr> wrote:
> I wrote about this topic in the past:
> 
>   https://hpc.guix.info/blog/2018/01/pre-built-binaries-vs-performance/

Very nice overview! Thanks for sharing (and writing!) that article. Definitely
feeling the urge to jump down this rabbit hole and patch upstream now. Hehe.

> I you’re the upstream author, I recommend using one of the techniques
> given above to provide so-called “fat binaries” that contain several
> implementations of the performance-sensitive functions; the loader
> will pick the right implementation when the program starts.
> 
> If you’re downstream… it depends on the specifics.  The loader is also
> able to pick a .so from the right lib/ sub-directory depending on the
> micro-architecture.  You can try:

I'm downstream, unfortunately. However, the final executable actually provides
a flag to explicitly specify a path to the lib, so that's not really a hurdle
in this case.

Given the small size of the build products, I feel like it would be nice to
fake a fat binary at the filesystem level. Mind if we just entertain this idea
for a second?

Say I have a script that reads /proc/cpuinfo and runs my executable with the
correct flags to load the library with the best CPU features possible. How can
I embed such a script in the package definition (as a gexp?) and install it
under <out>/bin/?


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Installing a wrapper guile script in <out>/bin
  2021-02-03  5:50   ` elaexuotee
@ 2021-02-03 10:00     ` Maxime Devos
  2021-02-03 11:04       ` elaexuotee--- via Development of GNU Guix and the GNU System distribution.
  0 siblings, 1 reply; 9+ messages in thread
From: Maxime Devos @ 2021-02-03 10:00 UTC (permalink / raw)
  To: elaexuotee, Ludovic Courtès; +Cc: guix-devel

[-- Attachment #1: Type: text/plain, Size: 631 bytes --]

> Say I have a script that reads /proc/cpuinfo and runs my executable with the
> correct flags to load the library with the best CPU features possible. How can
> I embed such a script in the package definition (as a gexp?) and install it
> under <out>/bin/?

Let's presume the binary is called $X.

What I would do: add a build phase after the "install" phase that renames
<out>/bin/$X to <out>/bin/.$X-real using the rename-file procedure.  Create
your wrapper script at <out>/bin/.$X-real with call-with-output-file, some
I/O procedures and chmod (to make the wrapper script executable).

I hope that helps, Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Installing a wrapper guile script in <out>/bin
  2021-02-03 10:00     ` Maxime Devos
@ 2021-02-03 11:04       ` elaexuotee--- via Development of GNU Guix and the GNU System distribution.
  2021-02-03 12:12         ` Maxime Devos
  0 siblings, 1 reply; 9+ messages in thread
From: elaexuotee--- via Development of GNU Guix and the GNU System distribution. @ 2021-02-03 11:04 UTC (permalink / raw)
  To: Maxime Devos; +Cc: guix-devel, Ludovic Courtès

Maxime Devos <maximedevos@telenet.be> wrote:
> Let's presume the binary is called $X.
> 
> What I would do: add a build phase after the "install" phase that renames
> <out>/bin/$X to <out>/bin/.$X-real using the rename-file procedure.  Create
> your wrapper script at <out>/bin/.$X-real with call-with-output-file, some
> I/O procedures and chmod (to make the wrapper script executable).
> 
> I hope that helps, Maxime.

Thanks for the pointers.

The script contents are not what I'm confused about. I don't know how to turn
my gexp script into a file under <out>/bin/. This is conceptually what I want:

    (package
      (name "foo")
      ...
      (arguments
       `(...
         #:phases
         (modify-phases %standard-phases
         ...
         (replace 'install
           (lambda* (#:key outputs #:allow-other-keys)
             (let ((out (assoc-ref outputs "out"))
                   (wrapper-script #~(...)))
             ... ; Do normal install stuff
             (copy-file wrapper-script (string-append out "/bin/foo"))
             ... ; Finish install stuff
             )))))))

Of course `copy-file` doesn't work here because `wrapper-script` is a gexp not
a file. What code do I replace this with?

I am vaguely aware things like `build-expression->derivation` to reify a gexp
into a derivation; however, I'm not sure what to do with the derivation object
in this case or if this is even on the right track.

Cheers!


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Installing a wrapper guile script in <out>/bin
  2021-02-03 11:04       ` elaexuotee--- via Development of GNU Guix and the GNU System distribution.
@ 2021-02-03 12:12         ` Maxime Devos
  2021-02-03 19:17           ` elaexuotee
  0 siblings, 1 reply; 9+ messages in thread
From: Maxime Devos @ 2021-02-03 12:12 UTC (permalink / raw)
  To: elaexuotee; +Cc: guix-devel, Ludovic Courtès

[-- Attachment #1: Type: text/plain, Size: 2009 bytes --]

> The script contents are not what I'm confused about. I don't know how to turn
> my gexp script into a file under <out>/bin/. This is conceptually what I want:
> 
>     (package
>       (name "foo")
>       ...
>       (arguments
>        `(...
>          #:phases
>          (modify-phases %standard-phases
>          ...
>          (replace 'install
>            (lambda* (#:key outputs #:allow-other-keys)
>              (let ((out (assoc-ref outputs "out"))
>                    (wrapper-script #~(...)))
>              ... ; Do normal install stuff
>              (copy-file wrapper-script (string-append out "/bin/foo"))
>              ... ; Finish install stuff
>              )))))))
> 
> Of course `copy-file` doesn't work here because `wrapper-script` is a gexp not
> a file. What code do I replace this with?

Three options I have in mind:

* forego representing `wrapper-script` as a gexp (using #~), instead represent
  `wrapper-script` as something quasiquoted.  Then write this expression to
  a file (with 'write', and include an appropriate shebang line)

  Something like this:

  (let ((wrapper-script-contents `(exec* ,(string-append out "/bin/.foo-real") "--extra-library-stuff" other-arguments)))
    rename-upstream-binary
    write-wrapper-binary
  )

* The procedure ‘program-file’ turns a gexp into a representation of a store item.

  Write your script to accept arguments like this:
  YOUR-SCRIPT REAL-BINARY ARGUMENTS
  
  And write a small wrapper shell script to <out>/bin/foo. Maybe with something like this
  (format PORT "#!~a~%exec ~a ~a #$@" bin-sh-something #$THE-SCRIPT-GEXP <out>/bin/.foo-real)

  (Warning: the above shell script might be incorrect.  I'm not very familiar
  with using the shell as a programming language.)

* Define a wrapper package.  See wrap-python3 in packages/python.scm
  for an example. (This option probably has to be combined with one of the
  first two options.) 

Greetings,
Maxime

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Installing a wrapper guile script in <out>/bin
  2021-02-03 12:12         ` Maxime Devos
@ 2021-02-03 19:17           ` elaexuotee
  2021-02-04  8:56             ` Maxime Devos
  0 siblings, 1 reply; 9+ messages in thread
From: elaexuotee @ 2021-02-03 19:17 UTC (permalink / raw)
  To: Maxime Devos; +Cc: Ludovic Courtès, guix-devel

Thank you for the thoughtful input.

Maxime Devos <maximedevos@telenet.be> wrote:
> * forego representing `wrapper-script` as a gexp (using #~), instead represent
>   `wrapper-script` as something quasiquoted.  Then write this expression to
>   a file (with 'write', and include an appropriate shebang line)

> * The procedure ‘program-file’ turns a gexp into a representation of a store item.

After spending most of the day figuring out how to manually create and build
derivations. I hit a fundamental problem including that code in a package def
which made me realize that manually embedding scripts is a flawed approach.

The issue is that a script itself has dependencies, e.g. guile, bash, whatever.
Those dependencies also need to show up in the transitive closure of whatever
package embeds this script.

> * Define a wrapper package.  See wrap-python3 in packages/python.scm
>   for an example. (This option probably has to be combined with one of the
>   first two options.) 

Your suggestion here eventually set me on the right track. Say we want the
final executable to end up at <out>/bin/X. The basic idea is this:

1) Creat original package with a name like X-lib, and leave <out>/bin/X empty,
2) Write a wrapper script as a gexp argument to program-file. In this gexp we
   reference the X-lib package, and finally
3) Create an trivial-build-system package X that includes the program-file
   wrapper as an input. The build step will copy this input script to
   <out>/bin/X!

This way dependencies are correctly tracked, and there's no need for manual
derivation building hackery.

Anyway, thanks for the pointers! They motivated me to keep pushing until
something worked.


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Installing a wrapper guile script in <out>/bin
  2021-02-03 19:17           ` elaexuotee
@ 2021-02-04  8:56             ` Maxime Devos
  0 siblings, 0 replies; 9+ messages in thread
From: Maxime Devos @ 2021-02-04  8:56 UTC (permalink / raw)
  To: elaexuotee; +Cc: guix-devel, Ludovic Courtès

[-- Attachment #1: Type: text/plain, Size: 123 bytes --]

> Anyway, thanks for the pointers! They motivated me to keep pushing until
> something worked.

Glad I could help you!

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2021-02-04  8:56 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-02-01 11:28 Installing a wrapper guile script in <out>/bin elaexuotee
2021-02-01 16:04 ` Ludovic Courtès
2021-02-03  5:50   ` elaexuotee
2021-02-03 10:00     ` Maxime Devos
2021-02-03 11:04       ` elaexuotee--- via Development of GNU Guix and the GNU System distribution.
2021-02-03 12:12         ` Maxime Devos
2021-02-03 19:17           ` elaexuotee
2021-02-04  8:56             ` Maxime Devos
  -- strict thread matches above, loose matches on Subject: below --
2021-02-01 12:16 Leo Prikler

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/guix.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).