[PATCH 0/2] native-search-paths for GHC

[PATCH 0/2] native-search-paths for GHC

[ericbavier]

From: Eric Bavier <bavier@member.fsf.org>

The first of these patches lets search-path-as-list function correctly when a
pattern is given and the 'directory file type is specified.

The second adds a native-search-paths field to our ghc package.  It modifies
our haskell-build-system to install a package-specific package database with
each haskell library.  GHC insists on a binary package cache file called
'package.cache' to be in each directory listed in GHC_PACKAGE PATH, so we
uniquely name the package database directories, and use a file-pattern to add
those to GHC_PACKAGE_PATH.

The benefit of this over the current situation is that one gets the benefit of
`guix package --search-paths`.  Currently, after installing ghc packages, the
user needs to know to manually add
~/.guix-profile/lib/ghc-7.8.4/package.conf.d to their GHC_PACKAGE_PATH.  GHC
package recipes would no longer need to propagate runtime dependencies, and
'guix environment' also works nicely out-of-the-box::

$ guix environment --ad-hoc ghc ghc-attoparsec
$ ghc-pkg list
/gnu/store/...-ghc-mtl-2.1.3.1/lib/ghc-7.8.4/ghc-mtl-2.1.3.1.conf.d
   base-4.7.0.2
   ghc-prim-0.3.1.0
   integer-gmp-0.5.1.0
   mtl-2.1.3.1
   rts-1.0
   transformers-0.3.0.0
/gnu/store/...-ghc-regex-base-0.93.2/lib/ghc-7.8.4/ghc-regex-base-0.93.2.conf.d
   array-0.5.0.0
   base-4.7.0.2
   bytestring-0.10.4.0
...
/gnu/store/4vvmngz1w8ccm7v7mk4f4dxk45834464-ghc-attoparsec-0.13.0.0/lib/ghc-7.8.4/ghc-attoparsec-0.13.0.0.conf.d
   array-0.5.0.0
   attoparsec-0.13.0.0
...

Though, as you can see in this example, libraries may be listed more than
once.  As far as I can tell at this point, that is just an aesthetic detail.

Future work might involve filtering build-only library dependencies from the
generated package database.  We could probably also remove the ghc package
database creation during profile generation.

I'd be insterested in hearing others' thoughts on this approach.

Eric Bavier (2):
  utils: Have search-path-as-list pattern search for directories.
  gnu: ghc: Add GHC_PACKAGE_PATH native search path.

 gnu/packages/haskell.scm            |   6 +++
 guix/build/haskell-build-system.scm | 100 +++++++++++++++++++++++++++++-------
 guix/build/utils.scm                |   9 ++--
 guix/search-paths.scm               |   6 ---
 4 files changed, 93 insertions(+), 28 deletions(-)

-- 
2.4.3

[PATCH 1/2] utils: Have search-path-as-list pattern search for directories.

[ericbavier]

From: Eric Bavier <bavier@member.fsf.org>

* guix/build/utils.scm (search-path-as-list)[pattern]: Check requested file
  type.  Check pattern against directory names.
* guix/search-paths.scm (evaluate-search-paths)[pattern]: Remove symlink hack.
---
 guix/build/utils.scm  | 9 ++++++---
 guix/search-paths.scm | 6 ------
 2 files changed, 6 insertions(+), 9 deletions(-)

diff --git a/guix/build/utils.scm b/guix/build/utils.scm
index 676a012..d4686c0 100644
--- a/guix/build/utils.scm
+++ b/guix/build/utils.scm
@@ -337,10 +337,13 @@ for under the directories designated by FILES.  For example:
   (append-map (lambda (input)
                 (append-map (lambda (file)
                               (let ((file (string-append input "/" file)))
-                                ;; XXX: By using 'find-files', we implicitly
-                                ;; assume #:type 'regular.
                                 (if pattern
-                                    (find-files file pattern)
+                                    (find-files file (lambda (file stat)
+                                                       (and stat
+                                                            (eq? type (stat:type stat))
+                                                            ((file-name-predicate pattern) file stat)))
+                                                #:stat stat
+                                                #:directories? #t)
                                     (let ((stat (stat file #f)))
                                       (if (and stat (eq? type (stat:type stat)))
                                           (list file)
diff --git a/guix/search-paths.scm b/guix/search-paths.scm
index 7fd15d4..7a6fe67 100644
--- a/guix/search-paths.scm
+++ b/guix/search-paths.scm
@@ -139,12 +139,6 @@ report only settings not already effective."
        (let* ((values (or (and=> (getenv variable)
                                  (cut string-tokenize* <> separator))
                           '()))
-              ;; Add a trailing slash to force symlinks to be treated as
-              ;; directories when 'find-files' traverses them.
-              (files  (if pattern
-                          (map (cut string-append <> "/") files)
-                          files))
-
               ;; XXX: Silence 'find-files' when it stumbles upon non-existent
               ;; directories (see
               ;; <http://lists.gnu.org/archive/html/guix-devel/2015-01/msg00269.html>.)
-- 
2.4.3

[PATCH 2/2] gnu: ghc: Add GHC_PACKAGE_PATH native search path.

[ericbavier]

From: Eric Bavier <bavier@member.fsf.org>

Benefits include: 'guix environment' more useful for ghc libraries, more
useful 'guix package --search-paths' for installed ghc libraries, cleaner
package recipes: no longer need to propagate runtime package dependencies.

* guix/build/haskell-build-system.scm (configure): Unset GHC_PACKAGE_PATH
  around cabal configure.
  (make-ghc-package-database): Use pattern directory search.
  (register): Install complete package database for the current package.
* gnu/packages/haskell.scm (ghc): Add native-search-paths field.
---
 gnu/packages/haskell.scm            |   6 +++
 guix/build/haskell-build-system.scm | 100 +++++++++++++++++++++++++++++-------
 2 files changed, 87 insertions(+), 19 deletions(-)

diff --git a/gnu/packages/haskell.scm b/gnu/packages/haskell.scm
index 34cad87..652409b 100644
--- a/gnu/packages/haskell.scm
+++ b/gnu/packages/haskell.scm
@@ -234,6 +234,12 @@
                    (string-append ghc-bootstrap-path "/" ,name "-" ,version)
                  (zero? (system* "make" "install"))))
              %standard-phases)))))))
+    (native-search-paths (list (search-path-specification
+                                (variable "GHC_PACKAGE_PATH")
+                                (files (list
+                                        (string-append "lib/ghc-" version)))
+                                (file-pattern ".*\\.conf\\.d$")
+                                (file-type 'directory))))
     (home-page "https://www.haskell.org/ghc")
     (synopsis "The Glasgow Haskell Compiler")
     (description
diff --git a/guix/build/haskell-build-system.scm b/guix/build/haskell-build-system.scm
index c0cb789..59a91d5 100644
--- a/guix/build/haskell-build-system.scm
+++ b/guix/build/haskell-build-system.scm
@@ -24,6 +24,7 @@
   #:use-module (ice-9 rdelim)
   #:use-module (ice-9 regex)
   #:use-module (ice-9 match)
+  #:use-module (ice-9 vlist)
   #:export (%standard-phases
             haskell-build))
 
@@ -77,6 +78,7 @@ and parameters ~s~%"
                        (((_ . dir) ...)
                         dir)
                        (_ '())))
+         (ghc-path (getenv "GHC_PACKAGE_PATH"))
          (params (append `(,(string-append "--prefix=" out))
                          `(,(string-append "--libdir=" (or lib out) "/lib"))
                          `(,(string-append "--bindir=" (or bin out) "/bin"))
@@ -96,7 +98,11 @@ and parameters ~s~%"
                              '("--enable-tests")
                              '())
                          configure-flags)))
-    (run-setuphs "configure" params)))
+    ;; Cabal errors if GHC_PACKAGE_PATH is set during 'configure', so unset
+    ;; and restore it.
+    (unsetenv "GHC_PACKAGE_PATH")
+    (run-setuphs "configure" params)
+    (setenv "GHC_PACKAGE_PATH" ghc-path)))
 
 (define* (build #:rest empty)
   "Build a given Haskell package."
@@ -134,6 +140,12 @@ first match and return the content of the group."
       (format #t
               "Compiler ~a not supported~%" name-version)))))
 
+;;; TODO: Move this to (guix build utils)?
+(define-syntax-rule (with-null-error-port exp)
+  "Evaluate EXP with the error port pointing to the bit bucket."
+  (with-error-to-port (%make-void-port "w")
+    (lambda () exp)))
+
 (define (make-ghc-package-database system inputs outputs)
   "Generate the GHC package database."
   (let* ((haskell  (assoc-ref inputs "haskell"))
@@ -141,44 +153,94 @@ first match and return the content of the group."
                        (((_ . dir) ...)
                         dir)
                        (_ '())))
-         (conf-dirs (search-path-as-list
-                     `(,(string-append "lib/"
-                                       (package-name-version haskell)
-                                       "/package.conf.d"))
-                     input-dirs))
+         ;; Silence 'find-files' (see 'evaluate-search-paths')
+         (conf-dirs (with-null-error-port
+                     (search-path-as-list
+                      `(,(string-append "lib/" (package-name-version haskell)))
+                      input-dirs #:pattern ".*\\.conf.d$")))
          (conf-files (append-map (cut find-files <> "\\.conf$") conf-dirs)))
     (mkdir-p %tmp-db-dir)
     (for-each (lambda (file)
-                (copy-file file
-                           (string-append %tmp-db-dir "/" (basename file))))
+                (let ((dest (string-append %tmp-db-dir "/" (basename file))))
+                  (unless (file-exists? dest)
+                    (copy-file file dest))))
               conf-files)
     (zero? (system* "ghc-pkg"
                     (string-append "--package-db=" %tmp-db-dir)
                     "recache"))))
 
 (define* (register #:key name system inputs outputs #:allow-other-keys)
-  "Generate the compiler registration file for a given Haskell package.  Don't
-generate the cache as it would clash in user profiles."
+  "Generate the compiler registration and binary package database files for a
+given Haskell package."
+
+  (define (line-items str)
+    ;; Return a list of the whitespace-delimited "words" in a string
+    (remove string-null?
+            (map string-trim-both (string-split str char-set:whitespace))))
+
+  (define (conf-depends conf-file)
+    ;; Return a list of pkg-ids from the "depends" field in CONF-FILE
+    (let ((port (open-input-file conf-file))
+          (field-rx (make-regexp "^(.*):")))
+      (let loop ((collecting #f)
+                 (deps '()))
+        (let* ((line (read-line port))
+               (field (and=> (regexp-exec field-rx line)
+                             (cut match:substring <> 1))))
+          (cond
+           ((and=> field (cut string=? <> "depends"))
+            ;; The first dependency is listed on the same line as "depends:",
+            ;; so drop those characters.  A line may list more than one .conf.
+            (let ((d (line-items (string-drop line 8))))
+              (loop #t (append d deps))))
+           ((and collecting field)
+            (begin
+              (close-port port)
+              (reverse! deps)))
+           (collecting
+            (loop #t (append (line-items line) deps)))
+           (else (loop #f deps)))))))
+
+  (define (install-transitive-deps conf-file src dest)
+    ;; Copy .conf files from SRC to DEST for dependencies in CONF-FILE, and
+    ;; their dependencies, etc.
+    (let loop ((seen vlist-null)
+               (lst (conf-depends conf-file)))
+      (match lst
+        (() #t)                         ;done
+        ((id . tail)
+         (if (not (vhash-assoc id seen))
+             (let ((dep-conf  (string-append src  "/" id ".conf"))
+                   (dep-conf* (string-append dest "/" id ".conf")))
+               (copy-file (pk 'from dep-conf) (pk 'to dep-conf*))
+               (loop (vhash-cons id #t seen)
+                     (append lst (conf-depends dep-conf))))
+             (loop seen tail))))))
+
   (let* ((out (assoc-ref outputs "out"))
          (haskell  (assoc-ref inputs "haskell"))
          (lib (string-append out "/lib"))
          (config-dir (string-append lib "/"
                                     (package-name-version haskell)
-                                    "/package.conf.d"))
+                                    "/" name ".conf.d"))
          (id-rx (make-regexp "^id: *(.*)$"))
          (config-file (string-append out "/" name ".conf"))
          (params
           (list (string-append "--gen-pkg-config=" config-file))))
     (run-setuphs "register" params)
     ;; The conf file is created only when there is a library to register.
-    (when (file-exists? config-file)
-      (mkdir-p config-dir)
-      (let ((config-file-name+id
-             (call-with-ascii-input-file config-file (cut grep id-rx <>))))
-        (rename-file config-file
-                     (string-append config-dir "/" config-file-name+id
-                                    ".conf"))))
-    #t))
+    (or (not (file-exists? config-file))
+        (begin
+          (mkdir-p config-dir)
+          (let* ((config-file-name+id
+                  (call-with-ascii-input-file config-file (cut grep id-rx <>))))
+            (install-transitive-deps config-file %tmp-db-dir config-dir)
+            (rename-file config-file
+                         (string-append config-dir "/"
+                                        config-file-name+id ".conf"))
+            (zero? (system* "ghc-pkg"
+                            (string-append "--package-db=" config-dir)
+                            "recache")))))))
 
 (define* (check #:key tests? test-target #:allow-other-keys)
   "Run the test suite of a given Haskell package."
-- 
2.4.3

Re: [PATCH 2/2] gnu: ghc: Add GHC_PACKAGE_PATH native search path.

[Ian Denhardt]

[-- Attachment #1: Type: text/plain, Size: 403 bytes --]

Quoting ericbavier@openmailbox.org (2015-10-06 17:26:17)
> +  (define (line-items str)
> +    ;; Return a list of the whitespace-delimited "words" in a string
> +    (remove string-null?
> +            (map string-trim-both (string-split str char-set:whitespace))))

This strikes me as something generally useful enough to go with general
string manipulation code somewhere. Maybe (guix utils)?

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAABCAAGBQJWFKMWAAoJEPZUuMfUyjy4/QsP/3zFidD7pQ6i6IWohjyM6k+F
g31UL3i1IPFdrd7kixYzg4DRJ2U/rzraTCdPYqcKwExIY5OVjYSH/L7Nd3sCl0iT
lZY/NEPBAtsuwz7GbB1vttWZLkrVEuYCMdECHOsqNomKgPwGXqrHboAOGY9qG1Mg
xjXTvOvbkpnjHNXvxi/GuuKlfHTshccWfQIQ0tAY3Izs60oEtpokclRe5Bt29uaw
C1EB3D4OonCOjEUd0XZpklV4QHBp6G73v598PdXFmHVDdVNWZjmsnPTHU/plWLEn
o40Z0XlKkL5MWX4JFByvNrFniLT3wU37OJqA4EnV7tHTyLTmbUtJ4RiSlK+niH2o
7yw0zJb/Sgp+MMCc9gi7TfUwgAn0IsUiW+xmEVIAvGbdIknCEpQTusjAbhJRuair
tw4bYNJmJ8KDlItZ7n1rGiaDwxmHjPWLPEeyJpdSRHqAIuGLKOWyM3oqCvhl4sTO
3VeHb1tYsyUO9Jg6fxlUD0dMRadCKZ6PJvX7tJR0TYVMhamuTUPvEPgvMvKT7AYG
rMJAWYuc/T00h8H+aaJ1BRTD1qxMPqSAjHnSRFclNuHT+Ghnai3TmPOJGi5X0h0Q
9Xds+FMBe1kGctK9VIDwbMpfUZdUbwYLG+PTBJHbEPu5PlhV5egjdgyWl17D/c4Z
7TFyp16ayUHuSXc5iV2E
=BcH0
-----END PGP SIGNATURE-----

Re: [PATCH 0/2] native-search-paths for GHC

[Eric Bavier]

Hello Guix,

With all the Haskell packages that have made their way to the list, I 
was wondering if anyone had any further comments on these patches.

`~Eric

On 2015-10-06 16:26, ericbavier@openmailbox.org wrote:
> From: Eric Bavier <bavier@member.fsf.org>
> 
> The first of these patches lets search-path-as-list function correctly 
> when a
> pattern is given and the 'directory file type is specified.
> 
> The second adds a native-search-paths field to our ghc package.  It 
> modifies
> our haskell-build-system to install a package-specific package database 
> with
> each haskell library.  GHC insists on a binary package cache file 
> called
> 'package.cache' to be in each directory listed in GHC_PACKAGE PATH, so 
> we
> uniquely name the package database directories, and use a file-pattern 
> to add
> those to GHC_PACKAGE_PATH.
> 
> The benefit of this over the current situation is that one gets the 
> benefit of
> `guix package --search-paths`.  Currently, after installing ghc 
> packages, the
> user needs to know to manually add
> ~/.guix-profile/lib/ghc-7.8.4/package.conf.d to their GHC_PACKAGE_PATH. 
>  GHC
> package recipes would no longer need to propagate runtime dependencies, 
> and
> 'guix environment' also works nicely out-of-the-box::
> 
> $ guix environment --ad-hoc ghc ghc-attoparsec
> $ ghc-pkg list
> /gnu/store/...-ghc-mtl-2.1.3.1/lib/ghc-7.8.4/ghc-mtl-2.1.3.1.conf.d
>    base-4.7.0.2
>    ghc-prim-0.3.1.0
>    integer-gmp-0.5.1.0
>    mtl-2.1.3.1
>    rts-1.0
>    transformers-0.3.0.0
> /gnu/store/...-ghc-regex-base-0.93.2/lib/ghc-7.8.4/ghc-regex-base-0.93.2.conf.d
>    array-0.5.0.0
>    base-4.7.0.2
>    bytestring-0.10.4.0
> ...
> /gnu/store/4vvmngz1w8ccm7v7mk4f4dxk45834464-ghc-attoparsec-0.13.0.0/lib/ghc-7.8.4/ghc-attoparsec-0.13.0.0.conf.d
>    array-0.5.0.0
>    attoparsec-0.13.0.0
> ...
> 
> Though, as you can see in this example, libraries may be listed more 
> than
> once.  As far as I can tell at this point, that is just an aesthetic 
> detail.
> 
> Future work might involve filtering build-only library dependencies 
> from the
> generated package database.  We could probably also remove the ghc 
> package
> database creation during profile generation.
> 
> I'd be insterested in hearing others' thoughts on this approach.
> 
> Eric Bavier (2):
>   utils: Have search-path-as-list pattern search for directories.
>   gnu: ghc: Add GHC_PACKAGE_PATH native search path.
> 
>  gnu/packages/haskell.scm            |   6 +++
>  guix/build/haskell-build-system.scm | 100 
> +++++++++++++++++++++++++++++-------
>  guix/build/utils.scm                |   9 ++--
>  guix/search-paths.scm               |   6 ---
>  4 files changed, 93 insertions(+), 28 deletions(-)

-- 
`~Eric