bug#27003: [PATCH 0/3] Generalized wrap phase for perl, python.

bug#27003: [PATCH 0/3] Generalized wrap phase for perl, python.

[Alex Sassmannshausen]

Hello,

This patch series goes back some time, but the need for it has not
disappeared.

It implements a general `wrap` phase that can be used in individual build
systems to make sure binaries in those languages have access to all their
propagated inputs at runtime, by setting an appropriate environment variable.

I tested it against perl and python packages.

It will need to go in core-updates due to the number of packages that will
need to be rebuilt.

Best wishes,

Alex

Alex Sassmannshausen (3):
  build/utils: Add 'program-wrapper'.
  build/perl-build-system: Add 'wrap' phase.
  build/python-build-system: Refactor 'wrap'.

 guix/build/perl-build-system.scm   | 15 +++++++++++++--
 guix/build/python-build-system.scm | 36 ++++++++----------------------------
 guix/build/utils.scm               | 36 ++++++++++++++++++++++++++++++++++++
 3 files changed, 57 insertions(+), 30 deletions(-)

-- 
2.12.2

bug#27003: [PATCH 1/3] build/utils: Add 'program-wrapper'.

[Alex Sassmannshausen]

* guix/build/utils.scm (program-wrapper): New procedure.
---
 guix/build/utils.scm | 36 ++++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)

diff --git a/guix/build/utils.scm b/guix/build/utils.scm
index e8efb0653..af5583651 100644
--- a/guix/build/utils.scm
+++ b/guix/build/utils.scm
@@ -78,6 +78,7 @@
             patch-/usr/bin/file
             fold-port-matches
             remove-store-references
+            program-wrapper
             wrap-program
 
             locale-category->string))
@@ -956,6 +957,41 @@ known as `nuke-refs' in Nixpkgs."
                              (put-u8 out (char->integer char))
                              result))))))
 
+(define (program-wrapper path-proc env-var)
+  "Return a procedure, which, invoked as part of a 'wrap' phase, is capable of
+wrapping executables inside an environment in which ENV-VAR is correctly set.
+
+The string ENV-VAR is the name of the environmental variable we are setting
+for the executable we are wrapping.  PATH-PROC is a procedure of 2 arguments,
+'inputs' and 'outputs', returning the value that we should set ENV-VAR to.
+
+This is a specialized version of 'wrap-program' below, intended specifically
+to grant all executables that are part of our output access to all libraries
+that were declared in our inputs.  This is of use for languages such as Perl,
+Python and Guile."
+  (define (list-of-files dir)
+    (map (cut string-append dir "/" <>)
+         (or (scandir dir (lambda (f)
+                            (let ((s (stat (string-append dir "/" f))))
+                              (eq? 'regular (stat:type s)))))
+             '())))
+  (lambda* (#:key inputs outputs #:allow-other-keys)
+    (define bindirs
+      (append-map (match-lambda
+                    ((_ . dir)
+                     (list (string-append dir "/bin")
+                           (string-append dir "/sbin"))))
+                  outputs))
+    (define vars
+      `(,env-var prefix ,(cons (path-proc inputs outputs)
+                               (search-path-as-string->list
+                                (or (getenv env-var) "")))))
+    (for-each (lambda (dir)
+                (let ((files (list-of-files dir)))
+                  (for-each (cut wrap-program <> vars)
+                            files)))
+              bindirs)))
+
 (define* (wrap-program prog #:rest vars)
   "Make a wrapper for PROG.  VARS should look like this:
 
-- 
2.12.2

bug#27003: [PATCH 2/3] build/perl-build-system: Add 'wrap' phase.

[Alex Sassmannshausen]

* guix/build/perl-build-system.scm (wrap): New procedure.
  (%standard-phases): Add 'wrap' phase.
---
 guix/build/perl-build-system.scm | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/guix/build/perl-build-system.scm b/guix/build/perl-build-system.scm
index b2024e440..5bf5b2523 100644
--- a/guix/build/perl-build-system.scm
+++ b/guix/build/perl-build-system.scm
@@ -19,7 +19,7 @@
 (define-module (guix build perl-build-system)
   #:use-module ((guix build gnu-build-system) #:prefix gnu:)
   #:use-module (guix build utils)
-  #:use-module (ice-9 match)
+  #:use-module (srfi srfi-1)
   #:export (%standard-phases
             perl-build))
 
@@ -51,6 +51,15 @@
     (format #t "running `perl' with arguments ~s~%" args)
     (zero? (apply system* "perl" args))))
 
+;; Use 'program-wrapper' to return an executable wrapper for perl.
+(define wrap
+  (program-wrapper
+   (lambda (inputs outputs)
+     (string-append (assoc-ref outputs "out") "/lib/perl5/site_perl/"
+                    ;; As in python, assume version at end of `perl' string.
+                    (last (string-split (assoc-ref inputs "perl") #\-))))
+   "PERL5LIB"))
+
 (define-syntax-rule (define-w/gnu-fallback* (name args ...) body ...)
   (define* (name args ... #:rest rest)
     (if (access? "Build" X_OK)
@@ -74,9 +83,11 @@
 
 (define %standard-phases
   ;; Everything is as with the GNU Build System except for the `configure',
-  ;; `build', `check', and `install' phases.
+  ;; `build', `check', and `install' phases.  We also add a `wrap' phase to
+  ;; wrap perl binaries with a complete PERL5LIB path.
   (modify-phases gnu:%standard-phases
     (replace 'install install)
+    (add-after 'install 'wrap wrap)
     (replace 'check check)
     (replace 'build build)
     (replace 'configure configure)))
-- 
2.12.2

bug#27003: [PATCH 3/3] build/python-build-system: Refactor 'wrap'.

[Alex Sassmannshausen]

* guix/build/python-build-system.scm (wrap): Use
  'wrap-language-program'.
---
 guix/build/python-build-system.scm | 36 ++++++++----------------------------
 1 file changed, 8 insertions(+), 28 deletions(-)

diff --git a/guix/build/python-build-system.scm b/guix/build/python-build-system.scm
index dd07986b9..30d01f8cb 100644
--- a/guix/build/python-build-system.scm
+++ b/guix/build/python-build-system.scm
@@ -184,34 +184,14 @@ when running checks after installing the package."
                          configure-flags)))
     (call-setuppy "install" params use-setuptools?)))
 
-(define* (wrap #:key inputs outputs #:allow-other-keys)
-  (define (list-of-files dir)
-    (map (cut string-append dir "/" <>)
-         (or (scandir dir (lambda (f)
-                            (let ((s (stat (string-append dir "/" f))))
-                              (eq? 'regular (stat:type s)))))
-             '())))
-
-  (define bindirs
-    (append-map (match-lambda
-                 ((_ . dir)
-                  (list (string-append dir "/bin")
-                        (string-append dir "/sbin"))))
-                outputs))
-
-  (let* ((out  (assoc-ref outputs "out"))
-         (python (assoc-ref inputs "python"))
-         (var `("PYTHONPATH" prefix
-                ,(cons (string-append out "/lib/python"
-                                      (get-python-version python)
-                                      "/site-packages")
-                       (search-path-as-string->list
-                        (or (getenv "PYTHONPATH") ""))))))
-    (for-each (lambda (dir)
-                (let ((files (list-of-files dir)))
-                  (for-each (cut wrap-program <> var)
-                            files)))
-              bindirs)))
+;; Use 'program-wrapper' to return an executable wrapper for python.
+(define wrap
+  (program-wrapper
+   (lambda (inputs outputs)
+     (string-append (assoc-ref outputs "out") "/lib/python"
+                    (get-python-version (assoc-ref inputs "python"))
+                    "/site-packages"))
+   "PYTHONPATH"))
 
 (define* (rename-pth-file #:key name inputs outputs #:allow-other-keys)
   "Rename easy-install.pth to NAME.pth to avoid conflicts between packages
-- 
2.12.2

bug#27003: [PATCH 0/3] Generalized wrap phase for perl, python.

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 571 bytes --]

Alex Sassmannshausen <alex.sassmannshausen@gmail.com> writes:

> Hello,
>
> This patch series goes back some time, but the need for it has not
> disappeared.
>
> It implements a general `wrap` phase that can be used in individual build
> systems to make sure binaries in those languages have access to all their
> propagated inputs at runtime, by setting an appropriate environment variable.

Cool, thanks a lot for working on this. CCing Arun who was working on
something similar.

AFAICT this is also subject to <https://bugs.gnu.org/25235>. Can you
have a look at it?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

bug#27003: [PATCH 0/3] Generalized wrap phase for perl, python.

[Arun Isaac]

>> It implements a general `wrap` phase that can be used in individual build
>> systems to make sure binaries in those languages have access to all their
>> propagated inputs at runtime, by setting an appropriate environment variable.
>
> CCing Arun who was working on something similar.

I haven't actually worked on this problem for a long while. I'll test
Alex's patches, and get back to you soon.

> AFAICT this is also subject to <https://bugs.gnu.org/25235>. Can you
> have a look at it?

Yes, that's the python wrap phase bug report. And this is the one for
perl: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=26275

bug#27003: [PATCH 0/3] Generalized wrap phase for perl, python.

[Arun Isaac]

Building on my system is taking a long time. I'll need some more time to
properly test these patches. But, if I understand correctly, these
patches still wrap binaries with paths of native-inputs. That problem
remains to be solved.

bug#27003: [PATCH 0/3] Generalized wrap phase for perl, python.

[Alex Sassmannshausen]

Hi Arun,

Thank you for testing my patches.

Arun Isaac writes:

> Building on my system is taking a long time. I'll need some more time to
> properly test these patches. But, if I understand correctly, these
> patches still wrap binaries with paths of native-inputs. That problem
> remains to be solved.

Yes, I certainly did nothing to resolve that particular issue, so indeed
it remains to be solved.

I cannot currently make guarantees on when I'll be able to wrap my head
around the entire problem, so if you come to conclusions for next steps
on the basis of your testing, we should go with those.

Else I will revisit this issue when I have time (which might be a
while).

Best wishes,

Alex

bug#27003: [PATCH 0/3] Generalized wrap phase for perl, python.

[Arun Isaac]

> I cannot currently make guarantees on when I'll be able to wrap my head
> around the entire problem, so if you come to conclusions for next steps
> on the basis of your testing, we should go with those.
>
> Else I will revisit this issue when I have time (which might be a
> while).

I don't have any ideas on how to fix this. And, I don't understand Guix
internals very well. So, I'll wait for you or someone else to come up
with a solution. But, I can help with testing patches.

bug#27003: [PATCH 0/3] Generalized wrap phase for perl, python.

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 824 bytes --]

Arun Isaac <arunisaac@systemreboot.net> writes:

>> I cannot currently make guarantees on when I'll be able to wrap my head
>> around the entire problem, so if you come to conclusions for next steps
>> on the basis of your testing, we should go with those.
>>
>> Else I will revisit this issue when I have time (which might be a
>> while).
>
> I don't have any ideas on how to fix this. And, I don't understand Guix
> internals very well. So, I'll wait for you or someone else to come up
> with a solution. But, I can help with testing patches.

I'm not sure if #25235 should block this issue though, since it's a
useful change regardless. But, not handling it adds "technical debt",
and a (minor) problem that is not currently present in perl modules.

We do have until the next 'core-updates' to think about it though ;-)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

bug#27003: [PATCH 0/3] Generalized wrap phase for perl, python.

[Brice Waegeneire]

Hello,

ludo@gnu.org (Ludovic Courtès) writes:
> BTW, there’s also this patch series on this topic:
> 
>   https://issues.guix.info/issue/27003
> 
> What to do?

A `wrap-script' procedure has been merged as of 
0fb9a8df429a7b9f40610ff15baaff0d8e31e8cf
thanks to Ricardo's patch from https://issues.guix.info/issue/27003, it 
seems to
provide similar functionality as the `program-wrapper' suggested here. 
Should we close
this issue?

Brice.

bug#27003: [PATCH 0/3] Generalized wrap phase for perl, python.

[Ludovic Courtès]

Hi,

Brice Waegeneire <brice@waegenei.re> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>> BTW, there’s also this patch series on this topic:
>>
>>   https://issues.guix.info/issue/27003
>>
>> What to do?
>
> A `wrap-script' procedure has been merged as of
> 0fb9a8df429a7b9f40610ff15baaff0d8e31e8cf
> thanks to Ricardo's patch from https://issues.guix.info/issue/27003,
> it seems to
> provide similar functionality as the `program-wrapper' suggested
> here. Should we close
> this issue?

Yes, it seems that the patch that Alex initially submitted shares the
same goals as Ricardo’s ‘wrap-script’, so I guess we can close it.

Thanks for your work on bug triage!

Ludo’.