[PATCH] profiles: Generate GHC's package database cache.

[PATCH] profiles: Generate GHC's package database cache.

[Federico Beffa]

[-- Attachment #1: Type: text/plain, Size: 66 bytes --]

As discussed on the 'haskell-build-system' thread.

Regards,
Fede

[-- Attachment #2: 0001-profiles-Generate-GHC-s-package-database-cache.patch --]
[-- Type: text/x-diff, Size: 7016 bytes --]

From dfe3b875267731006512b8a9803aaa56f07db12e Mon Sep 17 00:00:00 2001
From: Federico Beffa <beffa@fbengineering.ch>
Date: Sat, 4 Apr 2015 22:51:13 +0200
Subject: [PATCH] profiles: Generate GHC's package database cache.

* guix/profiles.scm (ghc-package-cache-file): New procedure.
  (profile-derivation): Add 'ghc-package-cache?' keyword argument.  If true
  (the default), add the result of 'ghc-package-cache-file' to 'inputs'.
* guix/scripts/package.scm (guix-package)[process-actions]: Pass
  #:ghc-package-cache? to 'profile-generation'.
* tests/packages.scm ("--search-paths with pattern"): Likewise.
* tests/profiles.scm ("profile-derivation"): Likewise.
---
 guix/profiles.scm        | 59 ++++++++++++++++++++++++++++++++++++++++++++++--
 guix/scripts/package.scm |  1 +
 tests/packages.scm       |  1 +
 tests/profiles.scm       |  2 ++
 4 files changed, 61 insertions(+), 2 deletions(-)

diff --git a/guix/profiles.scm b/guix/profiles.scm
index 465aaf9..74e8390 100644
--- a/guix/profiles.scm
+++ b/guix/profiles.scm
@@ -404,6 +404,54 @@ MANIFEST."
       (gexp->derivation "info-dir" build
                         #:modules '((guix build utils)))))
 
+(define (ghc-package-cache-file manifest)
+  "Return a derivation that builds the GHC 'package.cache' file for all the
+entries of MANIFEST."
+  (define ghc                                 ;lazy reference
+    (module-ref (resolve-interface '(gnu packages haskell)) 'ghc))
+
+  (define build
+    #~(begin 
+        (use-modules (guix build utils)
+                     (srfi srfi-1) (srfi srfi-26)
+                     (ice-9 ftw))
+
+        (define ghc-name-version
+          (let* ((base (basename #+ghc)))
+            (string-drop base
+                         (+ 1 (string-index base #\-)))))
+        
+        (define db-subdir
+          (string-append "lib/" ghc-name-version "/package.conf.d"))
+
+        (define db-dir
+          (string-append #$output "/" db-subdir))
+        
+        (define (conf-files top)
+          (find-files (string-append top "/" db-subdir) "\\.conf$"))
+
+        (define (copy-conf-file conf)
+          (let ((base (basename conf)))
+            (copy-file conf (string-append db-dir "/" base))))
+        
+        (system* (string-append #+ghc "/bin/ghc-pkg") "init" db-dir)
+        (for-each copy-conf-file
+                  (append-map conf-files
+                              '#$(manifest-inputs manifest)))
+        (let ((success
+               (zero?
+                (system* (string-append #+ghc "/bin/ghc-pkg") "recache"
+                         (string-append "--package-db=" db-dir)))))
+          (for-each delete-file (find-files db-dir "\\.conf$"))
+          success)))
+
+  ;; Don't depend on GHC when there's nothing to do.
+  (if (null? (manifest-entries manifest))
+      (gexp->derivation "ghc-package-cache" #~(mkdir #$output))
+      (gexp->derivation "ghc-package-cache" build
+                        #:modules '((guix build utils))
+                        #:local-build? #t)))
+
 (define (ca-certificate-bundle manifest)
   "Return a derivation that builds a single-file bundle containing the CA
 certificates in the /etc/ssl/certs sub-directories of the packages in
@@ -465,14 +513,18 @@ MANIFEST.  Single-file bundles are required by programs such as Git and Lynx."
 (define* (profile-derivation manifest
                              #:key
                              (info-dir? #t)
+                             (ghc-package-cache? #t)
                              (ca-certificate-bundle? #t))
   "Return a derivation that builds a profile (aka. 'user environment') with
 the given MANIFEST.  The profile includes a top-level Info 'dir' file unless
-INFO-DIR? is #f, and a single-file CA certificate bundle unless
-CA-CERTIFICATE-BUNDLE? is #f."
+INFO-DIR? is #f, a GHC 'package.cache' file unless GHC-PACKAGE-CACHE? is #f
+and a single-file CA certificate bundle unless CA-CERTIFICATE-BUNDLE? is #f."
   (mlet %store-monad ((info-dir (if info-dir?
                                     (info-dir-file manifest)
                                     (return #f)))
+                      (ghc-package-cache (if ghc-package-cache?
+                                             (ghc-package-cache-file manifest)
+                                             (return #f)))
                       (ca-cert-bundle (if ca-certificate-bundle?
                                           (ca-certificate-bundle manifest)
                                           (return #f))))
@@ -480,6 +532,9 @@ CA-CERTIFICATE-BUNDLE? is #f."
       (append (if info-dir
                   (list (gexp-input info-dir))
                   '())
+              (if ghc-package-cache
+                  (list (gexp-input ghc-package-cache))
+                  '())
               (if ca-cert-bundle
                   (list (gexp-input ca-cert-bundle))
                   '())
diff --git a/guix/scripts/package.scm b/guix/scripts/package.scm
index 3cc7ae7..e7fe879 100644
--- a/guix/scripts/package.scm
+++ b/guix/scripts/package.scm
@@ -829,6 +829,7 @@ more information.~%"))
                                   (profile-derivation
                                    new
                                    #:info-dir? (not bootstrap?)
+                                   #:ghc-package-cache? (not bootstrap?)
                                    #:ca-certificate-bundle? (not bootstrap?))))
                       (prof     (derivation->output-path prof-drv)))
                  (show-manifest-transaction (%store) manifest transaction
diff --git a/tests/packages.scm b/tests/packages.scm
index c9dd5d8..4e3a116 100644
--- a/tests/packages.scm
+++ b/tests/packages.scm
@@ -600,6 +600,7 @@
                   (manifest (map package->manifest-entry
                                  (list p1 p2)))
                   #:info-dir? #f
+                  #:ghc-package-cache? #f
                   #:ca-certificate-bundle? #f)
                  #:guile-for-build (%guile-for-build))))
     (build-derivations %store (list prof))
diff --git a/tests/profiles.scm b/tests/profiles.scm
index 7b942e3..d20cb9d 100644
--- a/tests/profiles.scm
+++ b/tests/profiles.scm
@@ -184,6 +184,7 @@
        (guile      (package->derivation %bootstrap-guile))
        (drv        (profile-derivation (manifest (list entry))
                                        #:info-dir? #f
+                                       #:ghc-package-cache? #f
                                        #:ca-certificate-bundle? #f))
        (profile -> (derivation->output-path drv))
        (bindir ->  (string-append profile "/bin"))
@@ -197,6 +198,7 @@
       ((entry ->   (package->manifest-entry packages:glibc "debug"))
        (drv        (profile-derivation (manifest (list entry))
                                        #:info-dir? #f
+                                       #:ghc-package-cache? #f
                                        #:ca-certificate-bundle? #f)))
     (return (derivation-inputs drv))))
 
-- 
2.2.1

Re: [PATCH] profiles: Generate GHC's package database cache.

[Mark H Weaver]

Federico Beffa <beffa@ieee.org> writes:

> From dfe3b875267731006512b8a9803aaa56f07db12e Mon Sep 17 00:00:00 2001
> From: Federico Beffa <beffa@fbengineering.ch>
> Date: Sat, 4 Apr 2015 22:51:13 +0200
> Subject: [PATCH] profiles: Generate GHC's package database cache.

[...]

> +(define (ghc-package-cache-file manifest)
> +  "Return a derivation that builds the GHC 'package.cache' file for all the
> +entries of MANIFEST."
> +  (define ghc                                 ;lazy reference
> +    (module-ref (resolve-interface '(gnu packages haskell)) 'ghc))
> +
> +  (define build
> +    #~(begin 
> +        (use-modules (guix build utils)
> +                     (srfi srfi-1) (srfi srfi-26)
> +                     (ice-9 ftw))
> +
> +        (define ghc-name-version
> +          (let* ((base (basename #+ghc)))
> +            (string-drop base
> +                         (+ 1 (string-index base #\-)))))
> +        
> +        (define db-subdir
> +          (string-append "lib/" ghc-name-version "/package.conf.d"))
> +
> +        (define db-dir
> +          (string-append #$output "/" db-subdir))
> +        
> +        (define (conf-files top)
> +          (find-files (string-append top "/" db-subdir) "\\.conf$"))
> +
> +        (define (copy-conf-file conf)
> +          (let ((base (basename conf)))
> +            (copy-file conf (string-append db-dir "/" base))))
> +        
> +        (system* (string-append #+ghc "/bin/ghc-pkg") "init" db-dir)
> +        (for-each copy-conf-file
> +                  (append-map conf-files
> +                              '#$(manifest-inputs manifest)))
> +        (let ((success
> +               (zero?
> +                (system* (string-append #+ghc "/bin/ghc-pkg") "recache"
> +                         (string-append "--package-db=" db-dir)))))
> +          (for-each delete-file (find-files db-dir "\\.conf$"))
> +          success)))
> +
> +  ;; Don't depend on GHC when there's nothing to do.
> +  (if (null? (manifest-entries manifest))
> +      (gexp->derivation "ghc-package-cache" #~(mkdir #$output))
> +      (gexp->derivation "ghc-package-cache" build
> +                        #:modules '((guix build utils))
> +                        #:local-build? #t)))

I believe this will require GHC to build *any* non-empty profile.  Since
GHC is only available on Intel platforms, this will break profile
building on MIPS and ARM.

Even on Intel platforms, I don't want to have to install GHC to build
profiles that don't contain any Haskell packages, especially since it
involves trusting the upstream binaries from GHC.

What if I want to use Hugs instead, or some other Haskell implementation
that is capable of being bootstrapped from source code.  Is GHC the only
tool that can do this job?

      Mark

Re: [PATCH] profiles: Generate GHC's package database cache.

[Federico Beffa]

On Sun, Apr 5, 2015 at 7:21 AM, Mark H Weaver <mhw@netris.org> wrote:
> I believe this will require GHC to build *any* non-empty profile.  Since
> GHC is only available on Intel platforms, this will break profile
> building on MIPS and ARM.
>
> Even on Intel platforms, I don't want to have to install GHC to build
> profiles that don't contain any Haskell packages, especially since it
> involves trusting the upstream binaries from GHC.

Good point. Would this check be satisfactory for you?

  ;; Don't depend on GHC when there's nothing to do.
  (if (any (cut string-prefix? "ghc" <>)
           (map manifest-entry-name (manifest-entries manifest)))
      (gexp->derivation "ghc-package-cache" build
                        #:modules '((guix build utils))
                        #:local-build? #t)
      (gexp->derivation "ghc-package-cache" #~(mkdir #$output))))

>
> What if I want to use Hugs instead, or some other Haskell implementation
> that is capable of being bootstrapped from source code.  Is GHC the only
> tool that can do this job?

As far as I know, the library database is compiler specific. Here we
are handling GHC only. But, if you know better, don't hesitate to make
a concrete suggestion.

Regards,
Fede

Re: [PATCH] profiles: Generate GHC's package database cache.

[Mark H Weaver]

Federico Beffa <beffa@ieee.org> writes:

> On Sun, Apr 5, 2015 at 7:21 AM, Mark H Weaver <mhw@netris.org> wrote:
>> I believe this will require GHC to build *any* non-empty profile.  Since
>> GHC is only available on Intel platforms, this will break profile
>> building on MIPS and ARM.
>>
>> Even on Intel platforms, I don't want to have to install GHC to build
>> profiles that don't contain any Haskell packages, especially since it
>> involves trusting the upstream binaries from GHC.
>
> Good point. Would this check be satisfactory for you?
>
>   ;; Don't depend on GHC when there's nothing to do.
>   (if (any (cut string-prefix? "ghc" <>)
>            (map manifest-entry-name (manifest-entries manifest)))
>       (gexp->derivation "ghc-package-cache" build
>                         #:modules '((guix build utils))
>                         #:local-build? #t)
>       (gexp->derivation "ghc-package-cache" #~(mkdir #$output))))

Sure, this would be fine, although I wonder if we could replace the
'string-prefix?' with 'string=?'.  If so, the conditional could be
changed to:

  (if (member "ghc" (map manifest-entry-name (manifest-entries manifest)))
      ...)

What do you think?

>> What if I want to use Hugs instead, or some other Haskell implementation
>> that is capable of being bootstrapped from source code.  Is GHC the only
>> tool that can do this job?
>
> As far as I know, the library database is compiler specific. Here we
> are handling GHC only.

Okay, makes sense.

    Thank you!
       Mark

Re: [PATCH] profiles: Generate GHC's package database cache.

[Federico Beffa]

On Sun, Apr 5, 2015 at 10:24 PM, Mark H Weaver <mhw@netris.org> wrote:
>>
>>   ;; Don't depend on GHC when there's nothing to do.
>>   (if (any (cut string-prefix? "ghc" <>)
>>            (map manifest-entry-name (manifest-entries manifest)))
>>       (gexp->derivation "ghc-package-cache" build
>>                         #:modules '((guix build utils))
>>                         #:local-build? #t)
>>       (gexp->derivation "ghc-package-cache" #~(mkdir #$output))))
>
> Sure, this would be fine, although I wonder if we could replace the
> 'string-prefix?' with 'string=?'.  If so, the conditional could be

I used 'string-prefix?' to catch any GHC library (that we are
prefixing with 'ghc-...'). However, to compile such a library we need
GHC, so I think that 'string=?' should do.

> changed to:
>
>   (if (member "ghc" (map manifest-entry-name (manifest-entries manifest)))
>       ...)
>
> What do you think?

Thanks, I will update the patch.
Regards,
Fede

Re: [PATCH] profiles: Generate GHC's package database cache.

[Federico Beffa]

On Sun, Apr 5, 2015 at 10:33 PM, Federico Beffa <beffa@ieee.org> wrote:
> On Sun, Apr 5, 2015 at 10:24 PM, Mark H Weaver <mhw@netris.org> wrote:
>>>
>>>   ;; Don't depend on GHC when there's nothing to do.
>>>   (if (any (cut string-prefix? "ghc" <>)
>>>            (map manifest-entry-name (manifest-entries manifest)))
>>>       (gexp->derivation "ghc-package-cache" build
>>>                         #:modules '((guix build utils))
>>>                         #:local-build? #t)
>>>       (gexp->derivation "ghc-package-cache" #~(mkdir #$output))))
>>
>> Sure, this would be fine, although I wonder if we could replace the
>> 'string-prefix?' with 'string=?'.  If so, the conditional could be
>
> I used 'string-prefix?' to catch any GHC library (that we are
> prefixing with 'ghc-...'). However, to compile such a library we need
> GHC, so I think that 'string=?' should do.


Thinking again about this: In principle a user can install a 'ghc-...'
library without having GHC in his profile. I'm not sure how practical
this would be, but I don't see a good reason to not support this by
not creating the cache.  For this reason I would prefer to keep the
'(any ...)' variant.

Regards,
Fede

Re: [PATCH] profiles: Generate GHC's package database cache.

[Mark H Weaver]

Federico Beffa <beffa@ieee.org> writes:

> On Sun, Apr 5, 2015 at 10:33 PM, Federico Beffa <beffa@ieee.org> wrote:
>> On Sun, Apr 5, 2015 at 10:24 PM, Mark H Weaver <mhw@netris.org> wrote:
>>>>
>>>>   ;; Don't depend on GHC when there's nothing to do.
>>>>   (if (any (cut string-prefix? "ghc" <>)
>>>>            (map manifest-entry-name (manifest-entries manifest)))
>>>>       (gexp->derivation "ghc-package-cache" build
>>>>                         #:modules '((guix build utils))
>>>>                         #:local-build? #t)
>>>>       (gexp->derivation "ghc-package-cache" #~(mkdir #$output))))
>>>
>>> Sure, this would be fine, although I wonder if we could replace the
>>> 'string-prefix?' with 'string=?'.  If so, the conditional could be
>>
>> I used 'string-prefix?' to catch any GHC library (that we are
>> prefixing with 'ghc-...'). However, to compile such a library we need
>> GHC, so I think that 'string=?' should do.
>
>
> Thinking again about this: In principle a user can install a 'ghc-...'
> library without having GHC in his profile. I'm not sure how practical
> this would be, but I don't see a good reason to not support this by
> not creating the cache.  For this reason I would prefer to keep the
> '(any ...)' variant.

Is the package database cache used by anything other than GHC?  If the
user doesn't have GHC installed, what would consult the GHC package
database cache?

      Mark

Re: [PATCH] profiles: Generate GHC's package database cache.

[Federico Beffa]

On Mon, Apr 6, 2015 at 10:24 AM, Mark H Weaver <mhw@netris.org> wrote:
> Is the package database cache used by anything other than GHC?  If the
> user doesn't have GHC installed, what would consult the GHC package
> database cache?

Maybe a GHC not in Guix? Not, sure if it would work.

Fede

Re: [PATCH] profiles: Generate GHC's package database cache.

[Ludovic Courtès]

Federico Beffa <beffa@ieee.org> skribis:

> On Mon, Apr 6, 2015 at 10:24 AM, Mark H Weaver <mhw@netris.org> wrote:
>> Is the package database cache used by anything other than GHC?  If the
>> user doesn't have GHC installed, what would consult the GHC package
>> database cache?
>
> Maybe a GHC not in Guix? Not, sure if it would work.

Or it could be that GHC and the libs are installed in different profiles
(though I admit it may not be that convenient.)

Either way is fine with me, so please push whatever seems most
appropriate to you.

Thanks!

Ludo’.

Re: [PATCH] profiles: Generate GHC's package database cache.

[Alexandre Héaumé]

[-- Attachment #1: Type: text/plain, Size: 1141 bytes --]

On 6 Apr 2015 21:00, "Ludovic Courtès" <ludo@gnu.org> wrote:
>
> Federico Beffa <beffa@ieee.org> skribis:
>
> > On Mon, Apr 6, 2015 at 10:24 AM, Mark H Weaver <mhw@netris.org> wrote:
> >> Is the package database cache used by anything other than GHC?  If the
> >> user doesn't have GHC installed, what would consult the GHC package
> >> database cache?
> >
> > Maybe a GHC not in Guix? Not, sure if it would work.
>
> Or it could be that GHC and the libs are installed in different profiles
> (though I admit it may not be that convenient.)
>
> Either way is fine with me, so please push whatever seems most
> appropriate to you.
>
> Thanks!
>
> Ludo’.
>

Hello all,

I just stumbled upon <https://github.com/acowley/cabbage>https
<https://github.com/acowley/cabbage>:// <https://github.com/acowley/cabbage>
github.com <https://github.com/acowley/cabbage>/
<https://github.com/acowley/cabbage>acowley
<https://github.com/acowley/cabbage>/cabbage
<https://github.com/acowley/cabbage> which builds nix packages from cabal
files.

Maybe their approach can help you decide how to deal with the cache?

Alex

[-- Attachment #2: Type: text/html, Size: 1609 bytes --]

Re: [PATCH] profiles: Generate GHC's package database cache.

[Ludovic Courtès]

Federico Beffa <beffa@ieee.org> skribis:

> From dfe3b875267731006512b8a9803aaa56f07db12e Mon Sep 17 00:00:00 2001
> From: Federico Beffa <beffa@fbengineering.ch>
> Date: Sat, 4 Apr 2015 22:51:13 +0200
> Subject: [PATCH] profiles: Generate GHC's package database cache.
>
> * guix/profiles.scm (ghc-package-cache-file): New procedure.
>   (profile-derivation): Add 'ghc-package-cache?' keyword argument.  If true
>   (the default), add the result of 'ghc-package-cache-file' to 'inputs'.
> * guix/scripts/package.scm (guix-package)[process-actions]: Pass
>   #:ghc-package-cache? to 'profile-generation'.
> * tests/packages.scm ("--search-paths with pattern"): Likewise.
> * tests/profiles.scm ("profile-derivation"): Likewise.

LG, but...

> On Sun, Apr 5, 2015 at 7:21 AM, Mark H Weaver <mhw@netris.org> wrote:
>> I believe this will require GHC to build *any* non-empty profile.  Since
>> GHC is only available on Intel platforms, this will break profile
>> building on MIPS and ARM.
>>
>> Even on Intel platforms, I don't want to have to install GHC to build
>> profiles that don't contain any Haskell packages, especially since it
>> involves trusting the upstream binaries from GHC.

All good points.

> Good point. Would this check be satisfactory for you?
>
>   ;; Don't depend on GHC when there's nothing to do.
>   (if (any (cut string-prefix? "ghc" <>)
>            (map manifest-entry-name (manifest-entries manifest)))
>       (gexp->derivation "ghc-package-cache" build
>                         #:modules '((guix build utils))
>                         #:local-build? #t)
>       (gexp->derivation "ghc-package-cache" #~(mkdir #$output))))

I think that would do.

Eventually we could think of more sophisticated way to determine whether
to trigger a given “hook” (info-dir, ca-certificates, ghc.)  Perhaps
this ‘ghc-package-cache-file’ procedure could be added as an
‘environment-hook’ field of the ghc package and it would be triggered
iff GHC is installed.  (This approach would work well for GHC, but not
for info-dir or ca-certificates, though.)

>> What if I want to use Hugs instead, or some other Haskell implementation
>> that is capable of being bootstrapped from source code.  Is GHC the only
>> tool that can do this job?
>
> As far as I know, the library database is compiler specific. Here we
> are handling GHC only.

I think this is fine.  A similar hook could be added for Hugs if/when we
see fit and presumably they would not interfere.

Thoughts?

Thanks,
Ludo’.

Re: [PATCH] profiles: Generate GHC's package database cache.

[Federico Beffa]

ludo@gnu.org (Ludovic Courtès) writes:

> Eventually we could think of more sophisticated way to determine whether
> to trigger a given “hook” (info-dir, ca-certificates, ghc.)  Perhaps
> this ‘ghc-package-cache-file’ procedure could be added as an
> ‘environment-hook’ field of the ghc package and it would be triggered
> iff GHC is installed.  (This approach would work well for GHC, but not
> for info-dir or ca-certificates, though.)

On top of the mentioned cases above, there are many programs making use
of caches. Some are:

* GLib's IO modules normally use a cache file 'giomodule.cache'. We set
  the env. var. GIO_EXTRA_MODULES, but it's use is explicitly
  discouraged by the project.

* XDG icons 'icon-theme.cache': I see some clashes in my profile (e.g.,
  gtk and Inkscape).

* TeX 'ls-R' files: it would probably simplify modularization.

* I see python's easy_install site.py clashes in my profile.  These are
  probably harmless, but annoying.

It would definitely be good to establish a solid and general way to run
"hooks" to address this kind of situations.

Regards,
Fede

Re: [PATCH] profiles: Generate GHC's package database cache.

[Ludovic Courtès]

Federico Beffa <beffa@ieee.org> skribis:

> On top of the mentioned cases above, there are many programs making use
> of caches. Some are:
>
> * GLib's IO modules normally use a cache file 'giomodule.cache'. We set
>   the env. var. GIO_EXTRA_MODULES, but it's use is explicitly
>   discouraged by the project.
>
> * XDG icons 'icon-theme.cache': I see some clashes in my profile (e.g.,
>   gtk and Inkscape).
>
> * TeX 'ls-R' files: it would probably simplify modularization.
>
> * I see python's easy_install site.py clashes in my profile.  These are
>   probably harmless, but annoying.
>
> It would definitely be good to establish a solid and general way to run
> "hooks" to address this kind of situations.

Commit aa46a02 generalizes the mechanism a bit.

I think it’s an improvement, but it remains confined to (guix profiles).

Anyway, feedback welcome!

Ludo’.