Re: updating many haskell packages

[Troy Sankey <sankeytms@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 4070 bytes --]

Quoting Federico Beffa (2017-02-18 04:43:51)
> On Fri, Feb 17, 2017 at 6:47 PM, Troy Sankey <sankeytms@gmail.com> wrote:
> > Forgive me if my understanding of build systems in Guix is flawed, but
> > let me explain my idea with more detail:
> >
> > First, make a data-only package called "ghc-all-cabal-files" containing
> > the checkout of a specific commit of the all-cabal-files repository from
> > github.  We can periodically update this package, but there is no
> > traditional "release"---we just keep pulling the HEAD of the hackage
> > branch.  This package would then act as a helper package for the haskell
> > build system---every haskell package should implicitly use this package
> > as input.  Then we can write a post-unpack phase for the
> > haskell-build-system which updates the unpacked .cabal file iff it finds
> > a newer .cabal file in ghc-all-cabal-files (we know how to determine if
> > the cabal file is newer: it will have a higher "x-revision" value, or
> > that key will merely exist).
> >
> > One problem I have not fully solved is the technical debt associated
> > with keeping the proposed ghc-all-cabal-files package up-to-date.  I
> > believe updating it would require all haskell packages to be rebuilt.
> > We could create a build system argument called use-newest-cabal-file to
> > toggle the feature, in which case we would only switch it to #t if we
> > already know the .cabal file to be stale.  Then only a small subset of
> > packages would need to be rebuilt, and there is less technical debt than
> > the current solution which involves monkey-patching every cabal file
> > that needs it.
> 
> My worry with this approach is that every time that a single cabal
> file in 'ghc-all-cabal-files' is updated all packages will fail to
> build (the implicit input will fail). Given that there are several
> thousands of cabal files, I suspect that this could occur quite often.

Can you elaborate on how all packages will fail to build?

Also, my hope is that with the toggle switch 'use-newest-cabal-file' we
do not have to make ghc-all-cabal-files an implicit input for all
haskell packages, only a few.

> Maybe another approach could be: when we update the GHC packages we
> follow a specific Stackage release. If a package tar includes an old
> cabal file, we enable a 'jailbreak' flag such that the build system
> clears version bounds. This of course relies on the fact that Stackage
> verified that this is fine.

What you describe is almost what I mean.  It is a fine alternative
solution, and I would be open to implementing this for the immediate
term.

My approach implies that we use Stackage LTS, but instead of "jailbreak"
use "use-newest-cabal-file" since the newer cabal file often will
include those relaxed version bounds, so the effect is almost the same.
The newer cabal file may also contain other changes unrelated to
dependency version bounds [0] and those changes are implicitly verified
by Stackage LTS which also uses newer cabal files.  In fact, by not
using the updated cabal files, we are essentially diverging from the
Stackage LTS.

Remember, all-cabal-files tracks cabal revisions for every package
version.  If Stackage LTS references an old version of a haskell
package, then we can still find the "latest" cabal file for that old
version of the package in all-cabal-files.

There is one more problem, and it is not a minor problem...

------------------------------- cut here -------------------------------
$ du -hs tmp/all-cabal-files
862M    tmp/all-cabal-files
$ du -hs tmp/all-cabal-files/.git
140M    tmp/all-cabal-files/.git
$ echo '862 - 140' | bc
722
------------------------------- cut here -------------------------------

The filesize of the package output will be 722 MiB.  We could prune it
of all the unrelated versions according to the stackage LTS, but this
starts to get complicated and error-prone...

Troy

[0] https://www.haskell.org/cabal/users-guide/developing-packages.html#build-information

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEE0zLJ6STd4Cp+CgbIgs677ofYV8IFAliohvoACgkQgs677ofY
V8KH4BAAo3Qletdl0B19BuJfXD3RIZy9L3nLOoq4ucrq01YKLyxgmaQF2ZcGjIS8
azdpVlnGlaS+PXgI/LxOYezIxx9JD7+crptvYzQEbU//nKhYG3I00E09QYyOVlDR
lgxKRL2bIVWuNBjKNlCBjXywWOpxYMPRyOjsi/bTTTbjIxZ2tZgzcK4v2/49VmSn
+Udw160t7ZkZ0M9eTo1hMIniyBUggD9U8krmmoJr1V2TYRCYO45cwo9fXzQyjsMt
xVPbFAsMsi8VtlN0ZpM6m0u3DO9kQdqFbpovHi0rgXqlx3ERsx1ZJSIPNIAjeh0L
CmmGh8ygcUcMToBLojourlMT8P001RXXZnSNyDe4SLW6ycm7ohSrUV6LWFcXm1EB
ArCu/3J1LcDORjbpwze20JF51nfS7zSA+qKT8pPcectFPShHx7d/U2o+x8HHfa0V
oOwnaJQaTDKUIv7Kvkmh9kVozwijiNATins7rlZXU9YpDmbjoG+mn3T+vmqxGeAl
1qkex4+XsJ+3pfJR5Ckj6oBayCOf9kQqPJIy/B0qGEi3GSXnG37Wsbf1ke2I4SQ7
xrND4bkBGA5qFg5ZM/4xKqGzX+ur7qE+R1yC5fmrRsUVteZ1Th7XFQMDUt8cQC9w
/jlw55+917X/MwuCZxGvOUOLTPjMXPY6prphpEIV6ub2JjGzb8Q=
=gKvM
-----END PGP SIGNATURE-----