GHC packages' inputs leak in guix shell

GHC packages' inputs leak in guix shell

[Saku Laesvuori]

[-- Attachment #1: Type: text/plain, Size: 434 bytes --]

Running `guix shell --pure ghc-esqueleto -D ghc-bytestring-builder`
results in an environment that has the base64-bytestring package (from
ghc-base64-bytestring) visible, even though it is not listed on listed
the command line (ghc-bytestring-builder doesn't dependend on it). It
seems to "leak" from the inputs of ghc-esqueleto which does depend on
ghc-base64-bytestring.

Is this a bug in guix or is it unavoidable for some reason?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: GHC packages' inputs leak in guix shell

[Simon Tournier]

Hi,

On Tue, 15 Aug 2023 at 09:51, Saku Laesvuori <saku@laesvuori.fi> wrote:
> Running `guix shell --pure ghc-esqueleto -D ghc-bytestring-builder`
> results in an environment that has the base64-bytestring package (from
> ghc-base64-bytestring) visible, even though it is not listed on listed
> the command line (ghc-bytestring-builder doesn't dependend on it). It
> seems to "leak" from the inputs of ghc-esqueleto which does depend on
> ghc-base64-bytestring.

What do you mean by “leak”?

Does it happen if you run

    guix shell -C ghc-esqueleto -D ghc-bytestring-builder

?

Cheers,
simon

Re: GHC packages' inputs leak in guix shell

[Saku Laesvuori]

[-- Attachment #1: Type: text/plain, Size: 750 bytes --]

> > Running `guix shell --pure ghc-esqueleto -D ghc-bytestring-builder`
> > results in an environment that has the base64-bytestring package (from
> > ghc-base64-bytestring) visible, even though it is not listed on listed
> > the command line (ghc-bytestring-builder doesn't dependend on it). It
> > seems to "leak" from the inputs of ghc-esqueleto which does depend on
> > ghc-base64-bytestring.
> 
> What do you mean by “leak”?

I would expect packages to keep their (non-propagated) inputs separate
from the environment I use. Here ghc-esqueleto makes it's haskell inputs
visible to the ghc in my environment.

> Does it happen if you run
> 
>     guix shell -C ghc-esqueleto -D ghc-bytestring-builder
> 
> ?

Yes it does.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: GHC packages' inputs leak in guix shell

[Simon Tournier]

Hi,

On Thu, 24 Aug 2023 at 19:16, Saku Laesvuori <saku@laesvuori.fi> wrote:
>> > Running `guix shell --pure ghc-esqueleto -D ghc-bytestring-builder`
>> > results in an environment that has the base64-bytestring package (from
>> > ghc-base64-bytestring) visible, even though it is not listed on listed
>> > the command line (ghc-bytestring-builder doesn't dependend on it). It
>> > seems to "leak" from the inputs of ghc-esqueleto which does depend on
>> > ghc-base64-bytestring.
>> 
>> What do you mean by “leak”?
>
> I would expect packages to keep their (non-propagated) inputs separate
> from the environment I use. Here ghc-esqueleto makes it's haskell inputs
> visible to the ghc in my environment.

Could you be more explicit?

The package ghc-base64-bytestring does not seems being propagated; the
store item ghc-base64-bytestring does not appear in the profile
generated by “guix shell”.  However,

--8<---------------cut here---------------start------------->8---
$ guix shell -C ghc-esqueleto ghc gcc-toolchain -- ghci
GHCi, version 9.2.5: https://www.haskell.org/ghc/  :? for help
ghci> import Data.ByteString.Base64.URL.Lazy
ghci> :t encode
encode
  :: Data.ByteString.Lazy.Internal.ByteString
     -> Data.ByteString.Lazy.Internal.ByteString
--8<---------------cut here---------------end--------------->8---

and instead, you would like:

    <no location info>: error:
        Could not find module `Data.ByteString.Base64.URL.Lazy'
        It is not a module in the current program, or in any known package.

Right?  Well, I do not know if it is possible.  I guess it is because of
this file:

--8<---------------cut here---------------start------------->8---
$ find $(guix build ghc-esqueleto) -type f -print | grep base64
/gnu/store/zqax59v1v537h26g0kypka6klaaahnqf-ghc-esqueleto-3.5.8.1/lib/ghc-9.2.5/ghc-esqueleto-3.5.8.1.conf.d/base64-bytestring-1.2.1.0-CQYLTs5ShsEFl2lwe4hRrI.conf
--8<---------------cut here---------------end--------------->8---

Lars, WDYT?

Cheers,
simon

Re: GHC packages' inputs leak in guix shell

[Saku Laesvuori]

[-- Attachment #1: Type: text/plain, Size: 2688 bytes --]

On Mon, Aug 28, 2023 at 01:40:14PM +0200, Simon Tournier wrote:
> Hi,
> 
> On Thu, 24 Aug 2023 at 19:16, Saku Laesvuori <saku@laesvuori.fi> wrote:
> >> > Running `guix shell --pure ghc-esqueleto -D ghc-bytestring-builder`
> >> > results in an environment that has the base64-bytestring package (from
> >> > ghc-base64-bytestring) visible, even though it is not listed on listed
> >> > the command line (ghc-bytestring-builder doesn't dependend on it). It
> >> > seems to "leak" from the inputs of ghc-esqueleto which does depend on
> >> > ghc-base64-bytestring.
> >> 
> >> What do you mean by “leak”?
> >
> > I would expect packages to keep their (non-propagated) inputs separate
> > from the environment I use. Here ghc-esqueleto makes it's haskell inputs
> > visible to the ghc in my environment.
> 
> Could you be more explicit?
> 
> The package ghc-base64-bytestring does not seems being propagated; the
> store item ghc-base64-bytestring does not appear in the profile
> generated by “guix shell”.

Yes, it is not propagated but it's still visible to ghc, which is not
something I would expect. I'd expect to not be able to import modules
from packages that are not in any of the active profiles.

> However,
> 
> --8<---------------cut here---------------start------------->8---
> $ guix shell -C ghc-esqueleto ghc gcc-toolchain -- ghci
> GHCi, version 9.2.5: https://www.haskell.org/ghc/  :? for help
> ghci> import Data.ByteString.Base64.URL.Lazy
> ghci> :t encode
> encode
>   :: Data.ByteString.Lazy.Internal.ByteString
>      -> Data.ByteString.Lazy.Internal.ByteString
> --8<---------------cut here---------------end--------------->8---
> 
> and instead, you would like:
> 
>     <no location info>: error:
>         Could not find module `Data.ByteString.Base64.URL.Lazy'
>         It is not a module in the current program, or in any known package.
> 
> Right?

Exactly.

> Well, I do not know if it is possible.  I guess it is because of this
> file:
> 
> --8<---------------cut here---------------start------------->8---
> $ find $(guix build ghc-esqueleto) -type f -print | grep base64
> /gnu/store/zqax59v1v537h26g0kypka6klaaahnqf-ghc-esqueleto-3.5.8.1/lib/ghc-9.2.5/ghc-esqueleto-3.5.8.1.conf.d/base64-bytestring-1.2.1.0-CQYLTs5ShsEFl2lwe4hRrI.conf
> --8<---------------cut here---------------end--------------->8---

I think that is most likely the reason. I don't think cabal has this
problem (I haven't actually used cabal much at all so this is just
speculation), so it could be possible to fix. Of course, cabal-install
and guix have different ways of packaging so it might also be
impossible.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: GHC packages' inputs leak in guix shell

[Lars-Dominik Braun]

Hi Simon,

> Right?  Well, I do not know if it is possible.  I guess it is because of
> this file:
> 
> --8<---------------cut here---------------start------------->8---
> $ find $(guix build ghc-esqueleto) -type f -print | grep base64
> /gnu/store/zqax59v1v537h26g0kypka6klaaahnqf-ghc-esqueleto-3.5.8.1/lib/ghc-9.2.5/ghc-esqueleto-3.5.8.1.conf.d/base64-bytestring-1.2.1.0-CQYLTs5ShsEFl2lwe4hRrI.conf
> --8<---------------cut here---------------end--------------->8---
> 
> Lars, WDYT?

your analysis is correct, but due to the nature of how Haskell builds work
we cannot remove that file (because anything depending on ghc-esqueleto
would not build any more) or relocate it to a different output (because
that would cause a cycle).

I can’t check right now, but I’m guessing a plain `cabal install`
would also add base64-bytestring to GHC’s visible packages?

Cheers,
Lars

Re: GHC packages' inputs leak in guix shell

[Saku Laesvuori]

[-- Attachment #1: Type: text/plain, Size: 968 bytes --]

> I can’t check right now, but I’m guessing a plain `cabal install`
> would also add base64-bytestring to GHC’s visible packages?

I tested with cabal-install and it somehow manages to hide the haskell
packages that are installed as dependencies.

```
$ guix shell -CN cabal-install coreutils zlib -D ghc-old-time
## Building old-time failed so I added it's inputs.
## Zlib is also required for building the dependency chain.
$ cabal update
$ env -u GHC_PACKAGE_PATH cabal install --lib esqueleto
$ ghci
ﬦ import Database.Esqueleto.Experimental
-- This works, so the package from cabal-install is visible
ﬦ import Data.ByteString.Base64.URL.Lazy

<no location info>: error:
    Could not load module `Data.ByteString.Base64.URL.Lazy'
    It is a member of the hidden package `base64-bytestring-1.2.1.0'.
    You can run `:set -package base64-bytestring' to expose it.
    (Note: this unloads all the modules in the current scope.)
```

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: GHC packages' inputs leak in guix shell

[Saku Laesvuori]

[-- Attachment #1: Type: text/plain, Size: 4605 bytes --]

> > I can’t check right now, but I’m guessing a plain `cabal install`
> > would also add base64-bytestring to GHC’s visible packages?
> 
> I tested with cabal-install and it somehow manages to hide the haskell
> packages that are installed as dependencies.

Apparently cabal uses ghc environments[1], which are files that define a
list of flags for ghc, to hide all packages except the explicitly
installed ones. Guix could probably also create a file like that for
every profile that contains ghc and/or packages for it. Another way
would be adding a phase to hide all the haskell packages in the
package-db ($out/lib/ghc-9.2.5/ghc-esqueleto-3.5.8.1.conf.d) except for
the package itself.

Creating environment files could maybe cause problems when combining
profiles, so I think hiding dependency packages in a build phase would
be a better solution. I tried this with ghc-esqueleto and it seems to
work (though I'm sure the code isn't particularly clean and it certainly
is slower than I would like).

[1]: https://ghc.gitlab.haskell.org/ghc/doc/users_guide/packages.html#package-environments

```
(define-public ghc-esqueleto
  (package
    (name "ghc-esqueleto")
    (version "3.5.8.1")
    (source (origin
              (method url-fetch)
              (uri (hackage-uri "esqueleto" version))
              (sha256
               (base32
                "0k7h2hbxv14x0kq9w2wi83h0swzlri99ic9rj76540l39yqwjc5v"))))
    (build-system haskell-build-system)
    (properties '((upstream-name . "esqueleto")))
    (inputs (list ghc-aeson
                  ghc-attoparsec
                  ghc-blaze-html
                  ghc-conduit
                  ghc-monad-logger
                  ghc-persistent
                  ghc-resourcet
                  ghc-tagged
                  ghc-unliftio
                  ghc-unordered-containers
                  openssl
                  zlib))
    (native-inputs (list ghc-hspec
                         ghc-hspec-core
                         ghc-mysql
                         ghc-mysql-simple
                         ghc-persistent-mysql
                         ghc-persistent-postgresql
                         ghc-persistent-sqlite
                         ghc-postgresql-simple
                         ghc-quickcheck))
    (arguments
     (list
      #:tests? #f ; Needs a running MySQLd.
      #:phases #~(modify-phases %standard-phases
                   (add-after 'register 'hide-dependencies
                     (begin
                       (use-modules (srfi srfi-1)
                                    (ice-9 popen)
                                    (ice-9 rdelim))
                       (lambda* (#:key name inputs #:allow-other-keys)
                         (let* ((out #$output)
                                (lib (string-append out "/lib"))
                                (haskell  (assoc-ref inputs "haskell"))
                                (name-version (strip-store-file-name haskell))
                                (version (last (string-split name-version #\-)))
                                (conf-dir (string-append lib "/ghc-" version "/" name ".conf.d"))
                                (port (open-input-pipe
                                       (string-append "ghc-pkg list --simple-output "
                                                      "--package-db=" conf-dir)))
                                (pkgs (string-split (read-line port) #\space))
                                (ghc-pkg (lambda (args)
                                           (apply invoke "ghc-pkg"
                                                  (string-append "--package-db=" conf-dir)
                                                  args))))
                           (for-each (lambda (pkg)
                                       (ghc-pkg (list "hide" pkg)))
                                     pkgs)
                           (ghc-pkg (list "expose" (string-drop name 4)))))))))) ; drop "ghc-"
    (home-page "https://github.com/bitemyapp/esqueleto")
    (synopsis "Type-safe embedded domain specific language for SQL queries")
    (description
     "This library provides a type-safe embedded domain specific
language (EDSL) for SQL queries that works with SQL backends as provided by
@code{ghc-persistent}.  Its language closely resembles SQL, so you don't have
to learn new concepts, just new syntax, and it's fairly easy to predict the
generated SQL and optimize it for your backend.")
    (license license:bsd-3)))
```

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#66347: GHC packages inputs leak in guix shell

[Simon Tournier]

Hi,

Consider this:

--8<---------------cut here---------------start------------->8---
$ guix shell -C ghc-esqueleto ghc gcc-toolchain -- ghci
GHCi, version 9.2.5: https://www.haskell.org/ghc/  :? for help
ghci> import Data.ByteString.Base64.URL.Lazy
ghci> :t encode
encode
  :: Data.ByteString.Lazy.Internal.ByteString
     -> Data.ByteString.Lazy.Internal.ByteString
--8<---------------cut here---------------end--------------->8---

The package ghc-base64-bytestring should not be visible and instead, the
user should see:

    <no location info>: error:
        Could not find module `Data.ByteString.Base64.URL.Lazy'
        It is not a module in the current program, or in any known package.

See discussion for more details if needed.

        GHC packages' inputs leak in guix shell
        Saku Laesvuori <saku@laesvuori.fi>
        Tue, 15 Aug 2023 09:51:50 +0300
        id:20230815065150.5joaxyts646mnpex@X-kone
        https://lists.gnu.org/archive/html/help-guix/2023-08
        https://yhetil.org/guix/20230815065150.5joaxyts646mnpex@X-kone


As reported in the discussion above, cabal is not exposing the package
required as dependency.

--8<---------------cut here---------------start------------->8---
$ guix shell -CN cabal-install coreutils zlib -D ghc-old-time
$ cabal update
$ env -u GHC_PACKAGE_PATH cabal install --lib esqueleto
$ ghci
ghci> import Database.Esqueleto.Experimental
ghci> :t encode

<interactive>:1:1: error: Variable not in scope: encode
ghci> import Data.ByteString.Base64.URL.Lazy

<no location info>: error:
    Could not load module `Data.ByteString.Base64.URL.Lazy'
    It is a member of the hidden package `base64-bytestring-1.2.1.0'.
    You can run `:set -package base64-bytestring' to expose it.
    (Note: this unloads all the modules in the current scope.)
--8<---------------cut here---------------end--------------->8---


Cheers,
simon