[PATCH 0/2] native-search-paths for GHC

[PATCH 0/2] native-search-paths for GHC

[ericbavier]

From: Eric Bavier <bavier@member.fsf.org>

The first of these patches lets search-path-as-list function correctly when a
pattern is given and the 'directory file type is specified.

The second adds a native-search-paths field to our ghc package.  It modifies
our haskell-build-system to install a package-specific package database with
each haskell library.  GHC insists on a binary package cache file called
'package.cache' to be in each directory listed in GHC_PACKAGE PATH, so we
uniquely name the package database directories, and use a file-pattern to add
those to GHC_PACKAGE_PATH.

The benefit of this over the current situation is that one gets the benefit of
`guix package --search-paths`.  Currently, after installing ghc packages, the
user needs to know to manually add
~/.guix-profile/lib/ghc-7.8.4/package.conf.d to their GHC_PACKAGE_PATH.  GHC
package recipes would no longer need to propagate runtime dependencies, and
'guix environment' also works nicely out-of-the-box::

$ guix environment --ad-hoc ghc ghc-attoparsec
$ ghc-pkg list
/gnu/store/...-ghc-mtl-2.1.3.1/lib/ghc-7.8.4/ghc-mtl-2.1.3.1.conf.d
   base-4.7.0.2
   ghc-prim-0.3.1.0
   integer-gmp-0.5.1.0
   mtl-2.1.3.1
   rts-1.0
   transformers-0.3.0.0
/gnu/store/...-ghc-regex-base-0.93.2/lib/ghc-7.8.4/ghc-regex-base-0.93.2.conf.d
   array-0.5.0.0
   base-4.7.0.2
   bytestring-0.10.4.0
...
/gnu/store/4vvmngz1w8ccm7v7mk4f4dxk45834464-ghc-attoparsec-0.13.0.0/lib/ghc-7.8.4/ghc-attoparsec-0.13.0.0.conf.d
   array-0.5.0.0
   attoparsec-0.13.0.0
...

Though, as you can see in this example, libraries may be listed more than
once.  As far as I can tell at this point, that is just an aesthetic detail.

Future work might involve filtering build-only library dependencies from the
generated package database.  We could probably also remove the ghc package
database creation during profile generation.

I'd be insterested in hearing others' thoughts on this approach.

Eric Bavier (2):
  utils: Have search-path-as-list pattern search for directories.
  gnu: ghc: Add GHC_PACKAGE_PATH native search path.

 gnu/packages/haskell.scm            |   6 +++
 guix/build/haskell-build-system.scm | 100 +++++++++++++++++++++++++++++-------
 guix/build/utils.scm                |   9 ++--
 guix/search-paths.scm               |   6 ---
 4 files changed, 93 insertions(+), 28 deletions(-)

-- 
2.4.3

Re: [PATCH 0/2] native-search-paths for GHC

[Federico Beffa]

ericbavier@openmailbox.org writes:

> From: Eric Bavier <bavier@member.fsf.org>
>
> The first of these patches lets search-path-as-list function correctly when a
> pattern is given and the 'directory file type is specified.
>
> The second adds a native-search-paths field to our ghc package.  It modifies
> our haskell-build-system to install a package-specific package database with
> each haskell library.  GHC insists on a binary package cache file called
> 'package.cache' to be in each directory listed in GHC_PACKAGE PATH, so we
> uniquely name the package database directories, and use a file-pattern to add
> those to GHC_PACKAGE_PATH.
>
> The benefit of this over the current situation is that one gets the benefit of
> `guix package --search-paths`.  Currently, after installing ghc packages, the
> user needs to know to manually add
> ~/.guix-profile/lib/ghc-7.8.4/package.conf.d to their GHC_PACKAGE_PATH.  GHC
> package recipes would no longer need to propagate runtime dependencies, and
> 'guix environment' also works nicely out-of-the-box::
>
> $ guix environment --ad-hoc ghc ghc-attoparsec
> $ ghc-pkg list
> /gnu/store/...-ghc-mtl-2.1.3.1/lib/ghc-7.8.4/ghc-mtl-2.1.3.1.conf.d
>    base-4.7.0.2
>    ghc-prim-0.3.1.0
>    integer-gmp-0.5.1.0
>    mtl-2.1.3.1
>    rts-1.0
>    transformers-0.3.0.0
> /gnu/store/...-ghc-regex-base-0.93.2/lib/ghc-7.8.4/ghc-regex-base-0.93.2.conf.d
>    array-0.5.0.0
>    base-4.7.0.2
>    bytestring-0.10.4.0
> ...
> /gnu/store/4vvmngz1w8ccm7v7mk4f4dxk45834464-ghc-attoparsec-0.13.0.0/lib/ghc-7.8.4/ghc-attoparsec-0.13.0.0.conf.d
>    array-0.5.0.0
>    attoparsec-0.13.0.0
> ...
>
> Though, as you can see in this example, libraries may be listed more than
> once.  As far as I can tell at this point, that is just an aesthetic detail.
>
> Future work might involve filtering build-only library dependencies from the
> generated package database.  We could probably also remove the ghc package
> database creation during profile generation.
>
> I'd be insterested in hearing others' thoughts on this approach.

Hi Eric,

sounds like a good approach to work around the "package.cache" clash. I
have a couple of questions:

* If I understand correctly, the configuration files of dependencies are
  copied in a unique directory for each package.  Instead of copying
  would a symlink work? (There are literally thousands of packages on
  Hackage and hopefully Guix will get more of them.)

* Some Haskell libraries have a rather large list of dependencies. For
  this reason I can imagine that in some situations GHC_PACKAGE_PATH
  could grow rather long. This thought made me wonder if there is a
  maximum length to the value of environment variables that we could
  possibly hit.

Regards,
Fede

Re: [PATCH 0/2] native-search-paths for GHC

[Eric Bavier]

On 2015-10-07 11:07, Federico Beffa wrote:
> ericbavier@openmailbox.org writes:
> 
>> From: Eric Bavier <bavier@member.fsf.org>
>> 
>> The first of these patches lets search-path-as-list function correctly 
>> when a
>> pattern is given and the 'directory file type is specified.
>> 
>> The second adds a native-search-paths field to our ghc package.  It 
>> modifies
>> our haskell-build-system to install a package-specific package 
>> database with
>> each haskell library.  GHC insists on a binary package cache file 
>> called
>> 'package.cache' to be in each directory listed in GHC_PACKAGE PATH, so 
>> we
>> uniquely name the package database directories, and use a file-pattern 
>> to add
>> those to GHC_PACKAGE_PATH.
>> 
>> The benefit of this over the current situation is that one gets the 
>> benefit of
>> `guix package --search-paths`.  Currently, after installing ghc 
>> packages, the
>> user needs to know to manually add
>> ~/.guix-profile/lib/ghc-7.8.4/package.conf.d to their 
>> GHC_PACKAGE_PATH.  GHC
>> package recipes would no longer need to propagate runtime 
>> dependencies, and
>> 'guix environment' also works nicely out-of-the-box::
>> 
>> $ guix environment --ad-hoc ghc ghc-attoparsec
>> $ ghc-pkg list
>> /gnu/store/...-ghc-mtl-2.1.3.1/lib/ghc-7.8.4/ghc-mtl-2.1.3.1.conf.d
>>    base-4.7.0.2
>>    ghc-prim-0.3.1.0
>>    integer-gmp-0.5.1.0
>>    mtl-2.1.3.1
>>    rts-1.0
>>    transformers-0.3.0.0
>> /gnu/store/...-ghc-regex-base-0.93.2/lib/ghc-7.8.4/ghc-regex-base-0.93.2.conf.d
>>    array-0.5.0.0
>>    base-4.7.0.2
>>    bytestring-0.10.4.0
>> ...
>> /gnu/store/4vvmngz1w8ccm7v7mk4f4dxk45834464-ghc-attoparsec-0.13.0.0/lib/ghc-7.8.4/ghc-attoparsec-0.13.0.0.conf.d
>>    array-0.5.0.0
>>    attoparsec-0.13.0.0
>> ...
>> 
>> Though, as you can see in this example, libraries may be listed more 
>> than
>> once.  As far as I can tell at this point, that is just an aesthetic 
>> detail.
>> 
>> Future work might involve filtering build-only library dependencies 
>> from the
>> generated package database.  We could probably also remove the ghc 
>> package
>> database creation during profile generation.
>> 
>> I'd be interested in hearing others' thoughts on this approach.
> 
> Hi Eric,
> 
> sounds like a good approach to work around the "package.cache" clash. I
> have a couple of questions:
> 
> * If I understand correctly, the configuration files of dependencies 
> are
>   copied in a unique directory for each package.  Instead of copying
>   would a symlink work? (There are literally thousands of packages on
>   Hackage and hopefully Guix will get more of them.)

I don't know if symlinking is allowed across store directories, though 
if it is, that'd be something to try.

> * Some Haskell libraries have a rather large list of dependencies. For
>   this reason I can imagine that in some situations GHC_PACKAGE_PATH
>   could grow rather long.

Not every package would end up with an entry in GHC_PACKAGE_PATH, only 
those that are installed, or declared inputs.  Dependent libraries would 
be found when GHC examines the package databases of those packages.

>   This thought made me wonder if there is a
>   maximum length to the value of environment variables that we could
>   possibly hit.

We've already pushed this quite far.  E.g. the hydra package creates a 
rather sizable PER5LIB path.

-- 
`~Eric

Re: [PATCH 0/2] native-search-paths for GHC

[Ludovic Courtès]

Eric Bavier <ericbavier@openmailbox.org> skribis:

> On 2015-10-07 11:07, Federico Beffa wrote:

[...]

>> * If I understand correctly, the configuration files of dependencies
>> are
>>   copied in a unique directory for each package.  Instead of copying
>>   would a symlink work? (There are literally thousands of packages on
>>   Hackage and hopefully Guix will get more of them.)
>
> I don't know if symlinking is allowed across store directories, though
> if it is, that'd be something to try.

Yes it’s allowed.  That’s what ‘union-build’ (the procedure that builds
profiles) does.

>> * Some Haskell libraries have a rather large list of dependencies. For
>>   this reason I can imagine that in some situations GHC_PACKAGE_PATH
>>   could grow rather long.
>
> Not every package would end up with an entry in GHC_PACKAGE_PATH, only
> those that are installed, or declared inputs.  Dependent libraries
> would be found when GHC examines the package databases of those
> packages.
>
>>   This thought made me wonder if there is a
>>   maximum length to the value of environment variables that we could
>>   possibly hit.
>
> We've already pushed this quite far.  E.g. the hydra package creates a
> rather sizable PER5LIB path.

Nice.  :-)

AFAIK there’s no limit on the size of an environment variable’s value.

However, it’s a good idea to keep search paths small, to reduce the
number of ‘stat’ calls that have to be done before a given file is
found.

Using a symlink forest is a good way to achieve that.

Thanks,
Ludo’.

Re: [PATCH 0/2] native-search-paths for GHC

[Eric Bavier]

Hello Guix,

With all the Haskell packages that have made their way to the list, I 
was wondering if anyone had any further comments on these patches.

`~Eric

On 2015-10-06 16:26, ericbavier@openmailbox.org wrote:
> From: Eric Bavier <bavier@member.fsf.org>
> 
> The first of these patches lets search-path-as-list function correctly 
> when a
> pattern is given and the 'directory file type is specified.
> 
> The second adds a native-search-paths field to our ghc package.  It 
> modifies
> our haskell-build-system to install a package-specific package database 
> with
> each haskell library.  GHC insists on a binary package cache file 
> called
> 'package.cache' to be in each directory listed in GHC_PACKAGE PATH, so 
> we
> uniquely name the package database directories, and use a file-pattern 
> to add
> those to GHC_PACKAGE_PATH.
> 
> The benefit of this over the current situation is that one gets the 
> benefit of
> `guix package --search-paths`.  Currently, after installing ghc 
> packages, the
> user needs to know to manually add
> ~/.guix-profile/lib/ghc-7.8.4/package.conf.d to their GHC_PACKAGE_PATH. 
>  GHC
> package recipes would no longer need to propagate runtime dependencies, 
> and
> 'guix environment' also works nicely out-of-the-box::
> 
> $ guix environment --ad-hoc ghc ghc-attoparsec
> $ ghc-pkg list
> /gnu/store/...-ghc-mtl-2.1.3.1/lib/ghc-7.8.4/ghc-mtl-2.1.3.1.conf.d
>    base-4.7.0.2
>    ghc-prim-0.3.1.0
>    integer-gmp-0.5.1.0
>    mtl-2.1.3.1
>    rts-1.0
>    transformers-0.3.0.0
> /gnu/store/...-ghc-regex-base-0.93.2/lib/ghc-7.8.4/ghc-regex-base-0.93.2.conf.d
>    array-0.5.0.0
>    base-4.7.0.2
>    bytestring-0.10.4.0
> ...
> /gnu/store/4vvmngz1w8ccm7v7mk4f4dxk45834464-ghc-attoparsec-0.13.0.0/lib/ghc-7.8.4/ghc-attoparsec-0.13.0.0.conf.d
>    array-0.5.0.0
>    attoparsec-0.13.0.0
> ...
> 
> Though, as you can see in this example, libraries may be listed more 
> than
> once.  As far as I can tell at this point, that is just an aesthetic 
> detail.
> 
> Future work might involve filtering build-only library dependencies 
> from the
> generated package database.  We could probably also remove the ghc 
> package
> database creation during profile generation.
> 
> I'd be insterested in hearing others' thoughts on this approach.
> 
> Eric Bavier (2):
>   utils: Have search-path-as-list pattern search for directories.
>   gnu: ghc: Add GHC_PACKAGE_PATH native search path.
> 
>  gnu/packages/haskell.scm            |   6 +++
>  guix/build/haskell-build-system.scm | 100 
> +++++++++++++++++++++++++++++-------
>  guix/build/utils.scm                |   9 ++--
>  guix/search-paths.scm               |   6 ---
>  4 files changed, 93 insertions(+), 28 deletions(-)

-- 
`~Eric