Re: updating many haskell packages

Re: updating many haskell packages

[Federico Beffa]

Troy Sankey <sankeytms@gmail.com> writes:

> I have a WIP branch which contains many haskell package updates and
> additions, and makes haskell-build-system use the latest ghc (8.0.2).
>
> https://github.com/pwnage101/guix/tree/add-gitit
>
> It started as a small project to create a Gitit package (hence the name
> of the branch), but blew up into something larger.  It's hard work, so I
> just wanted to check in with you all to see if this seems like the right
> direction.
>
> As of this writing, I have over 66 packages updated, 16 added, and
> various fixes on existing packages to get everything buildable against
> ghc@8.0.2.  I'm debating whether I should systematically update all the
> haskell packages while I'm at it, or just make sure everything is
> buildable under 8.0.2.  There are over 277 haskell packages in all.
>
> I wonder how I should send these patches after I'm all done.  git
> send-email?  That seems crazy.

I agree, it's a lot of work and guaranteeing package compatibility is
not straightforward.  To simplify things I would like to suggest to
exploit the work done by Stackage (www.stackage.org) and follow the
latest LTS release.

I prepared a 'stackage' importer.  It looks up the list of packages at
https://www.stackage.org/lts (can select which LTS version), extract the
version used there and downloads that version of the package from
hackage (with the hackage importer).

Two points are open:

(i) I still need to finish the updater (to work with 'refresh').

(ii) The latest LTS still uses GHC 8.0.1.  I'm not sure if this is
because of some real problem (GHC 8.0.2 introduces some
incompatibilities) or if it is because changing things takes time.

I will probably be able to share the importer/updater in one week (don't
have much time to work on it during the week).

Regards,
Fede

Re: updating many haskell packages

[Federico Beffa]

On Mon, Feb 6, 2017 at 9:03 AM, Federico Beffa <beffa@ieee.org> wrote:

> (ii) The latest LTS still uses GHC 8.0.1.  I'm not sure if this is
> because of some real problem (GHC 8.0.2 introduces some
> incompatibilities) or if it is because changing things takes time.

Looks like LTS8 is on its way http://lwm.github.io/stackage-8.0.2/

Re: updating many haskell packages

[Troy Sankey]

[-- Attachment #1: Type: text/plain, Size: 778 bytes --]

Quoting Federico Beffa (2017-02-06 03:13:16)
> On Mon, Feb 6, 2017 at 9:03 AM, Federico Beffa <beffa@ieee.org> wrote:
> 
> > (ii) The latest LTS still uses GHC 8.0.1.  I'm not sure if this is
> > because of some real problem (GHC 8.0.2 introduces some
> > incompatibilities) or if it is because changing things takes time.
> 
> Looks like LTS8 is on its way http://lwm.github.io/stackage-8.0.2/

Stackage LTS 8.0 has been released:

https://www.stackage.org/lts-8.0

In the coming days, I'll work to make all our current haskell packages
(and my additions) converge to LTS 8.0.  I may backport the stackage
importer [0] if I need to, but I don't expect that it will be required.

Troy

[0] https://lists.gnu.org/archive/html/guix-devel/2017-02/msg00270.html

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEE0zLJ6STd4Cp+CgbIgs677ofYV8IFAlikpvMACgkQgs677ofY
V8Ih8Q//cCMOccTqnA6cPp6H2B8XKXdx/WzJjcSZWiS3gflrOYrfiMZ65mA4TV0l
nApJXulRtSt3ZdNQVMM2PaQz4RAU8F41T/26rM/fQK55HQmj0a30IVgWlGu+Oi2c
5qoBS27lZ9Hr0SDUsAQTQ1Ljitrr9gRq9NhGz03dgd7myb9czGMMCIkjm/FVQlVA
7XrRO9HMOFb/lVP2RaOadRobXO+92z7+F3H35viceRxs8Oc6QBjNWuMDfnkEXLEX
2px6h0WpMLWl1aZY/pXwYrDpjU8H1nhu+yCf7AkoH7582wfH9X/7vjA7R3Bpe3L7
1KNhO8/RL6SFi73OaUTphwtjK9a/uq6tPCfJh0MXWJhAxg9PoFCAc0lGt3aB6NhL
4VeZQs+hEGUPLjEPG5VhjhCvWRHx4QdLGJcl3ORJFI/FNVeLmg9kyXWSPE6ahOwI
tNABSAG6a1mxMt/uuya+PMxfZgZk7cQ3fh5LWjNie4y7nk9cO6zVR20/xnSzAvQF
2jQDFAjRqmXDgD8vE/eMrTd7v6oZ+sNmCmlUWlU49CZRllDGi3tyPrXFIaFu6eGM
e3sYBcKY3aMfZLMgHOYNNNbcyMGLKqPxnD1uhzCJbUMm0wLfRmD0x7WIRznOY2I/
YHZ5ppkCJBnzESfkONkj9qF2/0ONhizAQHE5zYJ7PYvIx+9bUi8=
=Q4JR
-----END PGP SIGNATURE-----

Re: updating many haskell packages

[Ricardo Wurmus]

Troy Sankey <sankeytms@gmail.com> writes:

> Quoting Federico Beffa (2017-02-06 03:13:16)
>> On Mon, Feb 6, 2017 at 9:03 AM, Federico Beffa <beffa@ieee.org> wrote:
>> 
>> > (ii) The latest LTS still uses GHC 8.0.1.  I'm not sure if this is
>> > because of some real problem (GHC 8.0.2 introduces some
>> > incompatibilities) or if it is because changing things takes time.
>> 
>> Looks like LTS8 is on its way http://lwm.github.io/stackage-8.0.2/
>
> Stackage LTS 8.0 has been released:
>
> https://www.stackage.org/lts-8.0
>
> In the coming days, I'll work to make all our current haskell packages
> (and my additions) converge to LTS 8.0.  I may backport the stackage
> importer [0] if I need to, but I don't expect that it will be required.

Excellent!  Thank you for trying to tackle this!

-- 
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: updating many haskell packages

[Troy Sankey]

[-- Attachment #1: Type: text/plain, Size: 3371 bytes --]

Quoting Federico Beffa (2017-02-06 03:03:34)
> Troy Sankey <sankeytms@gmail.com> writes:
> 
> > I have a WIP branch which contains many haskell package updates and
> > additions, and makes haskell-build-system use the latest ghc (8.0.2).
> >
> > https://github.com/pwnage101/guix/tree/add-gitit
> >
> > It started as a small project to create a Gitit package (hence the name
> > of the branch), but blew up into something larger.  It's hard work, so I
> > just wanted to check in with you all to see if this seems like the right
> > direction.
> >
> > As of this writing, I have over 66 packages updated, 16 added, and
> > various fixes on existing packages to get everything buildable against
> > ghc@8.0.2.  I'm debating whether I should systematically update all the
> > haskell packages while I'm at it, or just make sure everything is
> > buildable under 8.0.2.  There are over 277 haskell packages in all.
> >
> > I wonder how I should send these patches after I'm all done.  git
> > send-email?  That seems crazy.
> 
> I agree, it's a lot of work and guaranteeing package compatibility is
> not straightforward.  To simplify things I would like to suggest to
> exploit the work done by Stackage (www.stackage.org) and follow the
> latest LTS release.

Hm, I should have been following a stackage LTS.  Maybe I wouldn't need
to relax any dependency versions.  I also wonder if I could have avoided
some discrepancies between hackage and released .cabal files regarding
dependency versions (there are quite a few [0]).

> I prepared a 'stackage' importer.  It looks up the list of packages at
> https://www.stackage.org/lts (can select which LTS version), extract the
> version used there and downloads that version of the package from
> hackage (with the hackage importer).
> 
> Two points are open:
> 
> (i) I still need to finish the updater (to work with 'refresh').

TIL guix refresh.  TIL stackage LTS.  I am slow to learn...

> (ii) The latest LTS still uses GHC 8.0.1.  I'm not sure if this is
> because of some real problem (GHC 8.0.2 introduces some
> incompatibilities) or if it is because changing things takes time.
> 
> I will probably be able to share the importer/updater in one week (don't
> have much time to work on it during the week).
> 
> Regards,
> Fede

Awesome :)

Troy

[0]
https://github.com/pwnage101/guix/blob/00cc2021d9f3019c8dc69383ff2038eb54d1ba5a/gnu/packages/haskell.scm#L1375
https://github.com/pwnage101/guix/blob/00cc2021d9f3019c8dc69383ff2038eb54d1ba5a/gnu/packages/haskell.scm#L1404
https://github.com/pwnage101/guix/blob/00cc2021d9f3019c8dc69383ff2038eb54d1ba5a/gnu/packages/haskell.scm#L1537
https://github.com/pwnage101/guix/blob/00cc2021d9f3019c8dc69383ff2038eb54d1ba5a/gnu/packages/haskell.scm#L1594
https://github.com/pwnage101/guix/blob/00cc2021d9f3019c8dc69383ff2038eb54d1ba5a/gnu/packages/haskell.scm#L1944
https://github.com/pwnage101/guix/blob/00cc2021d9f3019c8dc69383ff2038eb54d1ba5a/gnu/packages/haskell.scm#L1987
https://github.com/pwnage101/guix/blob/00cc2021d9f3019c8dc69383ff2038eb54d1ba5a/gnu/packages/haskell.scm#L2018
https://github.com/pwnage101/guix/blob/00cc2021d9f3019c8dc69383ff2038eb54d1ba5a/gnu/packages/haskell.scm#L5902
https://github.com/pwnage101/guix/blob/00cc2021d9f3019c8dc69383ff2038eb54d1ba5a/gnu/packages/haskell.scm#L8793

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEE0zLJ6STd4Cp+CgbIgs677ofYV8IFAliYlNsACgkQgs677ofY
V8IwOhAAn2oBufPtZrrtE3e07nTsqNvDstbhiaeqwhaVdZxyfo9tylyBu3FFX5zR
xC7qKXZzm8XPaRXpx5F6QXjmaNZTTsDRmGeNpn0ieY0LQEOWOXPuBxzdFVGsWSO4
4h1Wmm/jjlVf5WS/qCvS3BGUzlPaZswU/p0xiOn+xMWfFzuIcepd6bI/SZI3tiat
cR1l+maKrdb0TFwzkmw8Rl+K3pEETJ53O7QIboII+uCN/VCczra6It8yCeynbSBK
FpuF4cPsxjonjajZdnjDyElwGxo1vTghGDghA6DDe25UBSIouMhBzc/WOLHrzFYj
U1ALCahsQMj5LmUBJ+8T/5q1nYDXjCxWGT+sWP/wIvWoFMKI0KMNtZ8Q/8EOZaVh
8Aw9JP7XDh9WAESICxY67NVh0RwiuINqLQh3QblJbGQ6/95ylDtpTnBtxSieosKo
ErXqEx7f/OgpLWbU9HWgplagLEuQLpyBGfXF+w6ol4ZINCCH46OM/I4Ujnl7hK0f
3R/7qC3AIl1ROn+DcQJSrkIgOmFnWDoaxDQsmamb/6WnOwL8jC2chfpkjM0fl8Zj
gnh2R3GoqCy3M6JZknW6pr4HSWtLR4znNLdB3ETwm4HC4aFze/QYFyb8dSFIwnd5
uKjw64kq2gj9OTMMrOnk62z6P929RL3MSrN/gk9WBIZ3rKPVMHc=
=Yf8B
-----END PGP SIGNATURE-----

Re: updating many haskell packages

[Federico Beffa]

On Mon, Feb 6, 2017 at 4:23 PM, Troy Sankey <sankeytms@gmail.com> wrote:
> Hm, I should have been following a stackage LTS.  Maybe I wouldn't need
> to relax any dependency versions.  I also wonder if I could have avoided
> some discrepancies between hackage and released .cabal files regarding
> dependency versions (there are quite a few [0]).

As far as I know in the past NIX was aggressively removing upper
version constraints, but this seems to have led to problems and they
changed policy:
http://lists.science.uu.nl/pipermail/nix-dev/2015-January/015608.html

Having a set of compatible packages is one of the key Stackage goals
https://github.com/fpco/stackage/blob/master/MAINTAINERS.md#adding-a-package
If the project takes traction (as it seems) this problem should disappear.

I would consider a discrepancy between a cabal file on Hackage and the
actual cabal file included in a tar archive a bug.  It may be helpful
to report it to the author.

Regards,
Fede

[...]

> [0]
> https://github.com/pwnage101/guix/blob/00cc2021d9f3019c8dc69383ff2038eb54d1ba5a/gnu/packages/haskell.scm#L1375
> https://github.com/pwnage101/guix/blob/00cc2021d9f3019c8dc69383ff2038eb54d1ba5a/gnu/packages/haskell.scm#L1404
> https://github.com/pwnage101/guix/blob/00cc2021d9f3019c8dc69383ff2038eb54d1ba5a/gnu/packages/haskell.scm#L1537
> https://github.com/pwnage101/guix/blob/00cc2021d9f3019c8dc69383ff2038eb54d1ba5a/gnu/packages/haskell.scm#L1594
> https://github.com/pwnage101/guix/blob/00cc2021d9f3019c8dc69383ff2038eb54d1ba5a/gnu/packages/haskell.scm#L1944
> https://github.com/pwnage101/guix/blob/00cc2021d9f3019c8dc69383ff2038eb54d1ba5a/gnu/packages/haskell.scm#L1987
> https://github.com/pwnage101/guix/blob/00cc2021d9f3019c8dc69383ff2038eb54d1ba5a/gnu/packages/haskell.scm#L2018
> https://github.com/pwnage101/guix/blob/00cc2021d9f3019c8dc69383ff2038eb54d1ba5a/gnu/packages/haskell.scm#L5902
> https://github.com/pwnage101/guix/blob/00cc2021d9f3019c8dc69383ff2038eb54d1ba5a/gnu/packages/haskell.scm#L8793

Re: updating many haskell packages

[Troy Sankey]

[-- Attachment #1: Type: text/plain, Size: 606 bytes --]

Quoting Federico Beffa (2017-02-06 15:34:47)
> I would consider a discrepancy between a cabal file on Hackage and the
> actual cabal file included in a tar archive a bug.  It may be helpful
> to report it to the author.

I found this issue on the hackage github:

https://github.com/haskell/hackage-server/issues/503

It looks like it's a feature of hackage and/or cabal to allow multiple
revisions of cabal files per tarball release.  IIUC, the `cabal get`
utility takes care of automatically updating the cabal file to the
latest revision.

I'll have to look more into this later.

Troy

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEE0zLJ6STd4Cp+CgbIgs677ofYV8IFAliY4+gACgkQgs677ofY
V8K1ohAApkHRXd+rPT+U7vJBKph9H8O2tKmZ8hLjxUVjhYRr8k2nlFVX3etrknp5
smL3bm5e3+HfmecmD4XaH9oB+Zmdo+Ff0ZMQACZplfXLuSP0y3BbsTm8rQk5/P49
ncevzKwM0Z87HQ7lA66NKLhYRr3B2cdrXZEjLq9vRPfQ8DaFJZ/jbhY7SdQQsoIM
BT+us7YUFSchN4UMhBJwnKeYzTu4mqIXjbELmogWl62KwxFLJQYaaSGMbgIEbXxe
aqhxEV6yPAcNjHF0kpc0C6hr1uViyXUWByoixVq1tM9rOB9XtC59SgrjbPdRZHpA
mhC2NmauDtROmvCQ4vC4Cao8aBjVu/wxcoladluJYpmcCEZD4fWhkqLwicBGPX6P
lgmf5zb6N3y+FjONDq1/YNn+nyciKws3t/X27jwK38i4jIDcC/XMaZ9tr5MnVM/Y
KgGzu3eFC1u7rpPElDs8U7S6zbBZPxVbOGTPTHt6nTkAKryhexoTqReA9DfwRUzk
agyMADA17E9NbE4td17FA2oQZVNFDUJ5u4eq+Z2My8nOujJpEGGhe7mblYjgxzu+
JVUUhrEKnwKlVyGsyDN9+4kidtollBgNsdZt6BEnWMJS430gdIXie36XwdFFB2+o
zwOTCgDtYIFJNzvoJyaFanz1oq5G7m+DEXRZpqxz3dGVzoEyNRE=
=H0dD
-----END PGP SIGNATURE-----

Re: updating many haskell packages

[Troy Sankey]

[-- Attachment #1: Type: text/plain, Size: 1647 bytes --]

Quoting Troy Sankey (2017-02-06 16:00:29)
> Quoting Federico Beffa (2017-02-06 15:34:47)
> > I would consider a discrepancy between a cabal file on Hackage and the
> > actual cabal file included in a tar archive a bug.  It may be helpful
> > to report it to the author.
> 
> I found this issue on the hackage github:
> 
> https://github.com/haskell/hackage-server/issues/503
> 
> It looks like it's a feature of hackage and/or cabal to allow multiple
> revisions of cabal files per tarball release.  IIUC, the `cabal get`
> utility takes care of automatically updating the cabal file to the
> latest revision.
> 
> I'll have to look more into this later.

Indeed the .cabal files can be revised between releases.  Somehow the
new revisions get pushed to the all-cabal-files repository [0] with an
additional "x-revision" key [1], and Hackage picks up the new cabal
file.  The `cabal` and `stack` utilities substitute the .cabal file from
the release tarball with the revised .cabal file from Hackage if it is
newer.

In theory, anything in the .cabal file could be modified, not just
dependency bounds.  I think guix should use the updated .cabal files for
better consistency with the rest of the haskell ecosystem, and also for
less per-package maintenance cost.

How can guix adapt?  Should we somehow incorporate the all-cabal-files
repository in the haskell build system, adding a post-unpack phase to
substitute the entire .cabal file?

Troy

[0] https://github.com/commercialhaskell/all-cabal-files
[1] https://github.com/commercialhaskell/all-cabal-files/blob/hackage/old-time/1.1.0.3/old-time.cabal#L3

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEE0zLJ6STd4Cp+CgbIgs677ofYV8IFAlikrIsACgkQgs677ofY
V8JDow//VxkiaNLTaZm72oK3d3/YZ+QbXmLlnC4NT1x2YFccMperaiR3JMjlotFD
1t9QWzppy2URRjJ8UcsbcuuVH5OVMpJZ8t4iybpJRlu3eqg/2W89rD/b6FtCIIkT
0ot78TL/HoLlzyY2gkW0/zHIAcMuEYNZvPmdUMteneqF6K8GuViMedxgMZ0TuMRT
QTkR6R7lOfNcYf670BU+giFsKMC1q+zqfEHtu16d+M3p970q6PjcwUFO1XdGoeZ8
BrSsBK5lYIFUBHBT85gZFePAi2hUchb2sPAg8peMkWkFiQbUi+C8yy92iWUUcgT2
lLhkAncty0VV4zToQtyTA+j1nKa/CnzYHSkV/n+hTA/yRXqsR1GZqa7dxnANAPQ6
M406A5K/Ntr6d0cbXRvBjbR0Bpr1kKSDQ5Y8fIsLHFnSk9pPXNqKamF4yBH/ROy5
aj/YOMVZ/PxUR57P4bbR8Ciaxx9KPvJS8UwO5RjszvSEzh+NLYKvC6Xa2OAHq8xs
cFHs71vG3giHpeX9rm4nRPcDyitfCEISOXtZPEQ6hc8RYVur/aqBd1wTgjAG8BXQ
0Alq9dbjnkvfouXZkheWRxx/r4NJx5ypImQ0RAoGpnrKFFGR/LVNckSZgXMQQykR
uMEyFm3AcuejO02yE94UXhqxXauSZwhnyFGe85NezHxZBnsD7ss=
=BsnY
-----END PGP SIGNATURE-----

Re: updating many haskell packages

[Federico Beffa]

On Wed, Feb 15, 2017 at 8:31 PM, Troy Sankey <sankeytms@gmail.com> wrote:
> Quoting Troy Sankey (2017-02-06 16:00:29)
>> Quoting Federico Beffa (2017-02-06 15:34:47)
>> > I would consider a discrepancy between a cabal file on Hackage and the
>> > actual cabal file included in a tar archive a bug.  It may be helpful
>> > to report it to the author.
>>
>> I found this issue on the hackage github:
>>
>> https://github.com/haskell/hackage-server/issues/503
>>
>> It looks like it's a feature of hackage and/or cabal to allow multiple
>> revisions of cabal files per tarball release.  IIUC, the `cabal get`
>> utility takes care of automatically updating the cabal file to the
>> latest revision.
>>
>> I'll have to look more into this later.
>
> Indeed the .cabal files can be revised between releases.  Somehow the
> new revisions get pushed to the all-cabal-files repository [0] with an
> additional "x-revision" key [1], and Hackage picks up the new cabal
> file.  The `cabal` and `stack` utilities substitute the .cabal file from
> the release tarball with the revised .cabal file from Hackage if it is
> newer.
>
> In theory, anything in the .cabal file could be modified, not just
> dependency bounds.  I think guix should use the updated .cabal files for
> better consistency with the rest of the haskell ecosystem, and also for
> less per-package maintenance cost.
>
> How can guix adapt?  Should we somehow incorporate the all-cabal-files
> repository in the haskell build system, adding a post-unpack phase to
> substitute the entire .cabal file?

Packages are build in isolated environments without network.
Therefore it's not possible to access Hackage with a phase of the
build system.  Even if it were possible it would be undesirable
because it would make things non reproducible.

Files not included in the tar can be added as origin inputs to a
package definition (see the testsuite for GHC in the ghc package).
However, they are verified with hashes. Any change to those files will
break the package.

Fede

Re: updating many haskell packages

[Troy Sankey]

[-- Attachment #1: Type: text/plain, Size: 3658 bytes --]

Quoting Federico Beffa (2017-02-17 03:00:04)
> On Wed, Feb 15, 2017 at 8:31 PM, Troy Sankey <sankeytms@gmail.com> wrote:
> > Quoting Troy Sankey (2017-02-06 16:00:29)
> >> Quoting Federico Beffa (2017-02-06 15:34:47)
> >> > I would consider a discrepancy between a cabal file on Hackage and the
> >> > actual cabal file included in a tar archive a bug.  It may be helpful
> >> > to report it to the author.
> >>
> >> I found this issue on the hackage github:
> >>
> >> https://github.com/haskell/hackage-server/issues/503
> >>
> >> It looks like it's a feature of hackage and/or cabal to allow multiple
> >> revisions of cabal files per tarball release.  IIUC, the `cabal get`
> >> utility takes care of automatically updating the cabal file to the
> >> latest revision.
> >>
> >> I'll have to look more into this later.
> >
> > Indeed the .cabal files can be revised between releases.  Somehow the
> > new revisions get pushed to the all-cabal-files repository [0] with an
> > additional "x-revision" key [1], and Hackage picks up the new cabal
> > file.  The `cabal` and `stack` utilities substitute the .cabal file from
> > the release tarball with the revised .cabal file from Hackage if it is
> > newer.
> >
> > In theory, anything in the .cabal file could be modified, not just
> > dependency bounds.  I think guix should use the updated .cabal files for
> > better consistency with the rest of the haskell ecosystem, and also for
> > less per-package maintenance cost.
> >
> > How can guix adapt?  Should we somehow incorporate the all-cabal-files
> > repository in the haskell build system, adding a post-unpack phase to
> > substitute the entire .cabal file?
> 
> Packages are build in isolated environments without network.
> Therefore it's not possible to access Hackage with a phase of the
> build system.  Even if it were possible it would be undesirable
> because it would make things non reproducible.
> 
> Files not included in the tar can be added as origin inputs to a
> package definition (see the testsuite for GHC in the ghc package).
> However, they are verified with hashes. Any change to those files will
> break the package.
> 
> Fede

Forgive me if my understanding of build systems in Guix is flawed, but
let me explain my idea with more detail:

First, make a data-only package called "ghc-all-cabal-files" containing
the checkout of a specific commit of the all-cabal-files repository from
github.  We can periodically update this package, but there is no
traditional "release"---we just keep pulling the HEAD of the hackage
branch.  This package would then act as a helper package for the haskell
build system---every haskell package should implicitly use this package
as input.  Then we can write a post-unpack phase for the
haskell-build-system which updates the unpacked .cabal file iff it finds
a newer .cabal file in ghc-all-cabal-files (we know how to determine if
the cabal file is newer: it will have a higher "x-revision" value, or
that key will merely exist).

One problem I have not fully solved is the technical debt associated
with keeping the proposed ghc-all-cabal-files package up-to-date.  I
believe updating it would require all haskell packages to be rebuilt.
We could create a build system argument called use-newest-cabal-file to
toggle the feature, in which case we would only switch it to #t if we
already know the .cabal file to be stale.  Then only a small subset of
packages would need to be rebuilt, and there is less technical debt than
the current solution which involves monkey-patching every cabal file
that needs it.

Troy

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEE0zLJ6STd4Cp+CgbIgs677ofYV8IFAlinNyEACgkQgs677ofY
V8IZEw/8CaSba8UOqeuDqYIjm2fWbKpvR+6XKqtbh6ENvglAtUkU1bEO0M5NWJX5
8gTqytGy/C50HPDiiHZhv6ruhl/s4J1oRJJgPolNA+PsQHqHTmuRvKeVrbRkLgz0
FTnwKpkFC83lbtbD5w/j66iE604VNLmcRGMN8uhS3TZJara0FaLYZnpEj3/zuHIq
Qid+V4I/z5/gc2/dSlqJ0z/1rwTKLSDjbSfB5Qsfj4D873VeH87XG40XOHhqQypj
hQvFDIS7G0o9ZibA6MVcp2HXM4YqgUQ+4+GTtgMJmJMkoB7kS7DVyUi+N0FibLtV
uFIsbIcY2kPByYhPqXLc32BQA6hF3v8e8kXC+SMYLGE7svoKnegFaLDwuGUlhEhY
RtspzKdRPhOsC8Y7s2UawnhzUGJBwc4sV+s3dEFHf/Jus4LCG5bNkBsqj/lEiHQZ
C7bl4tMt4bVqBzNSfi1AfLUVdhN0b6mXr3u5sqj1voyB+mH3tvv3I+HqaGSoevtr
LY/KDq1lV9/YgKK/FYgFD42Y43Pazcf1J6UEJbs8MWU+5AAlz/4e7VswQT+FkZ6e
EQpUQRxJMERK4cxsZaoQ/ff0Wiuatx1T+nhAP/FmuOkKn9bD0dP2+Ntf18z3AMBm
KOzkot0XjjbU7J34ug1f3t8zN1MSGvvYWCt4ZupwDMUtp4eR8YQ=
=Wysj
-----END PGP SIGNATURE-----

Re: updating many haskell packages

[Federico Beffa]

On Fri, Feb 17, 2017 at 6:47 PM, Troy Sankey <sankeytms@gmail.com> wrote:
> Quoting Federico Beffa (2017-02-17 03:00:04)
>> On Wed, Feb 15, 2017 at 8:31 PM, Troy Sankey <sankeytms@gmail.com> wrote:
>> > Quoting Troy Sankey (2017-02-06 16:00:29)
>> >> Quoting Federico Beffa (2017-02-06 15:34:47)
>> >> > I would consider a discrepancy between a cabal file on Hackage and the
>> >> > actual cabal file included in a tar archive a bug.  It may be helpful
>> >> > to report it to the author.
>> >>
>> >> I found this issue on the hackage github:
>> >>
>> >> https://github.com/haskell/hackage-server/issues/503
>> >>
>> >> It looks like it's a feature of hackage and/or cabal to allow multiple
>> >> revisions of cabal files per tarball release.  IIUC, the `cabal get`
>> >> utility takes care of automatically updating the cabal file to the
>> >> latest revision.
>> >>
>> >> I'll have to look more into this later.
>> >
>> > Indeed the .cabal files can be revised between releases.  Somehow the
>> > new revisions get pushed to the all-cabal-files repository [0] with an
>> > additional "x-revision" key [1], and Hackage picks up the new cabal
>> > file.  The `cabal` and `stack` utilities substitute the .cabal file from
>> > the release tarball with the revised .cabal file from Hackage if it is
>> > newer.
>> >
>> > In theory, anything in the .cabal file could be modified, not just
>> > dependency bounds.  I think guix should use the updated .cabal files for
>> > better consistency with the rest of the haskell ecosystem, and also for
>> > less per-package maintenance cost.
>> >
>> > How can guix adapt?  Should we somehow incorporate the all-cabal-files
>> > repository in the haskell build system, adding a post-unpack phase to
>> > substitute the entire .cabal file?
>>
>> Packages are build in isolated environments without network.
>> Therefore it's not possible to access Hackage with a phase of the
>> build system.  Even if it were possible it would be undesirable
>> because it would make things non reproducible.
>>
>> Files not included in the tar can be added as origin inputs to a
>> package definition (see the testsuite for GHC in the ghc package).
>> However, they are verified with hashes. Any change to those files will
>> break the package.
>>
>> Fede
>
> Forgive me if my understanding of build systems in Guix is flawed, but
> let me explain my idea with more detail:
>
> First, make a data-only package called "ghc-all-cabal-files" containing
> the checkout of a specific commit of the all-cabal-files repository from
> github.  We can periodically update this package, but there is no
> traditional "release"---we just keep pulling the HEAD of the hackage
> branch.  This package would then act as a helper package for the haskell
> build system---every haskell package should implicitly use this package
> as input.  Then we can write a post-unpack phase for the
> haskell-build-system which updates the unpacked .cabal file iff it finds
> a newer .cabal file in ghc-all-cabal-files (we know how to determine if
> the cabal file is newer: it will have a higher "x-revision" value, or
> that key will merely exist).
>
> One problem I have not fully solved is the technical debt associated
> with keeping the proposed ghc-all-cabal-files package up-to-date.  I
> believe updating it would require all haskell packages to be rebuilt.
> We could create a build system argument called use-newest-cabal-file to
> toggle the feature, in which case we would only switch it to #t if we
> already know the .cabal file to be stale.  Then only a small subset of
> packages would need to be rebuilt, and there is less technical debt than
> the current solution which involves monkey-patching every cabal file
> that needs it.

My worry with this approach is that every time that a single cabal
file in 'ghc-all-cabal-files' is updated all packages will fail to
build (the implicit input will fail). Given that there are several
thousands of cabal files, I suspect that this could occur quite often.

Maybe another approach could be: when we update the GHC packages we
follow a specific Stackage release. If a package tar includes an old
cabal file, we enable a 'jailbreak' flag such that the build system
clears version bounds. This of course relies on the fact that Stackage
verified that this is fine.

Regards,
Fede

Re: updating many haskell packages

[Troy Sankey]

[-- Attachment #1: Type: text/plain, Size: 4070 bytes --]

Quoting Federico Beffa (2017-02-18 04:43:51)
> On Fri, Feb 17, 2017 at 6:47 PM, Troy Sankey <sankeytms@gmail.com> wrote:
> > Forgive me if my understanding of build systems in Guix is flawed, but
> > let me explain my idea with more detail:
> >
> > First, make a data-only package called "ghc-all-cabal-files" containing
> > the checkout of a specific commit of the all-cabal-files repository from
> > github.  We can periodically update this package, but there is no
> > traditional "release"---we just keep pulling the HEAD of the hackage
> > branch.  This package would then act as a helper package for the haskell
> > build system---every haskell package should implicitly use this package
> > as input.  Then we can write a post-unpack phase for the
> > haskell-build-system which updates the unpacked .cabal file iff it finds
> > a newer .cabal file in ghc-all-cabal-files (we know how to determine if
> > the cabal file is newer: it will have a higher "x-revision" value, or
> > that key will merely exist).
> >
> > One problem I have not fully solved is the technical debt associated
> > with keeping the proposed ghc-all-cabal-files package up-to-date.  I
> > believe updating it would require all haskell packages to be rebuilt.
> > We could create a build system argument called use-newest-cabal-file to
> > toggle the feature, in which case we would only switch it to #t if we
> > already know the .cabal file to be stale.  Then only a small subset of
> > packages would need to be rebuilt, and there is less technical debt than
> > the current solution which involves monkey-patching every cabal file
> > that needs it.
> 
> My worry with this approach is that every time that a single cabal
> file in 'ghc-all-cabal-files' is updated all packages will fail to
> build (the implicit input will fail). Given that there are several
> thousands of cabal files, I suspect that this could occur quite often.

Can you elaborate on how all packages will fail to build?

Also, my hope is that with the toggle switch 'use-newest-cabal-file' we
do not have to make ghc-all-cabal-files an implicit input for all
haskell packages, only a few.

> Maybe another approach could be: when we update the GHC packages we
> follow a specific Stackage release. If a package tar includes an old
> cabal file, we enable a 'jailbreak' flag such that the build system
> clears version bounds. This of course relies on the fact that Stackage
> verified that this is fine.

What you describe is almost what I mean.  It is a fine alternative
solution, and I would be open to implementing this for the immediate
term.

My approach implies that we use Stackage LTS, but instead of "jailbreak"
use "use-newest-cabal-file" since the newer cabal file often will
include those relaxed version bounds, so the effect is almost the same.
The newer cabal file may also contain other changes unrelated to
dependency version bounds [0] and those changes are implicitly verified
by Stackage LTS which also uses newer cabal files.  In fact, by not
using the updated cabal files, we are essentially diverging from the
Stackage LTS.

Remember, all-cabal-files tracks cabal revisions for every package
version.  If Stackage LTS references an old version of a haskell
package, then we can still find the "latest" cabal file for that old
version of the package in all-cabal-files.

There is one more problem, and it is not a minor problem...

------------------------------- cut here -------------------------------
$ du -hs tmp/all-cabal-files
862M    tmp/all-cabal-files
$ du -hs tmp/all-cabal-files/.git
140M    tmp/all-cabal-files/.git
$ echo '862 - 140' | bc
722
------------------------------- cut here -------------------------------

The filesize of the package output will be 722 MiB.  We could prune it
of all the unrelated versions according to the stackage LTS, but this
starts to get complicated and error-prone...

Troy

[0] https://www.haskell.org/cabal/users-guide/developing-packages.html#build-information

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEE0zLJ6STd4Cp+CgbIgs677ofYV8IFAliohvoACgkQgs677ofY
V8KH4BAAo3Qletdl0B19BuJfXD3RIZy9L3nLOoq4ucrq01YKLyxgmaQF2ZcGjIS8
azdpVlnGlaS+PXgI/LxOYezIxx9JD7+crptvYzQEbU//nKhYG3I00E09QYyOVlDR
lgxKRL2bIVWuNBjKNlCBjXywWOpxYMPRyOjsi/bTTTbjIxZ2tZgzcK4v2/49VmSn
+Udw160t7ZkZ0M9eTo1hMIniyBUggD9U8krmmoJr1V2TYRCYO45cwo9fXzQyjsMt
xVPbFAsMsi8VtlN0ZpM6m0u3DO9kQdqFbpovHi0rgXqlx3ERsx1ZJSIPNIAjeh0L
CmmGh8ygcUcMToBLojourlMT8P001RXXZnSNyDe4SLW6ycm7ohSrUV6LWFcXm1EB
ArCu/3J1LcDORjbpwze20JF51nfS7zSA+qKT8pPcectFPShHx7d/U2o+x8HHfa0V
oOwnaJQaTDKUIv7Kvkmh9kVozwijiNATins7rlZXU9YpDmbjoG+mn3T+vmqxGeAl
1qkex4+XsJ+3pfJR5Ckj6oBayCOf9kQqPJIy/B0qGEi3GSXnG37Wsbf1ke2I4SQ7
xrND4bkBGA5qFg5ZM/4xKqGzX+ur7qE+R1yC5fmrRsUVteZ1Th7XFQMDUt8cQC9w
/jlw55+917X/MwuCZxGvOUOLTPjMXPY6prphpEIV6ub2JjGzb8Q=
=gKvM
-----END PGP SIGNATURE-----

Re: updating many haskell packages

[Federico Beffa]

On Sat, Feb 18, 2017 at 6:40 PM, Troy Sankey <sankeytms@gmail.com> wrote:
> Quoting Federico Beffa (2017-02-18 04:43:51)
>> On Fri, Feb 17, 2017 at 6:47 PM, Troy Sankey <sankeytms@gmail.com> wrote:
>> > Forgive me if my understanding of build systems in Guix is flawed, but
>> > let me explain my idea with more detail:
>> >
>> > First, make a data-only package called "ghc-all-cabal-files" containing
>> > the checkout of a specific commit of the all-cabal-files repository from
>> > github.  We can periodically update this package, but there is no
>> > traditional "release"---we just keep pulling the HEAD of the hackage
>> > branch.  This package would then act as a helper package for the haskell
>> > build system---every haskell package should implicitly use this package
>> > as input.  Then we can write a post-unpack phase for the
>> > haskell-build-system which updates the unpacked .cabal file iff it finds
>> > a newer .cabal file in ghc-all-cabal-files (we know how to determine if
>> > the cabal file is newer: it will have a higher "x-revision" value, or
>> > that key will merely exist).
>> >
>> > One problem I have not fully solved is the technical debt associated
>> > with keeping the proposed ghc-all-cabal-files package up-to-date.  I
>> > believe updating it would require all haskell packages to be rebuilt.
>> > We could create a build system argument called use-newest-cabal-file to
>> > toggle the feature, in which case we would only switch it to #t if we
>> > already know the .cabal file to be stale.  Then only a small subset of
>> > packages would need to be rebuilt, and there is less technical debt than
>> > the current solution which involves monkey-patching every cabal file
>> > that needs it.
>>
>> My worry with this approach is that every time that a single cabal
>> file in 'ghc-all-cabal-files' is updated all packages will fail to
>> build (the implicit input will fail). Given that there are several
>> thousands of cabal files, I suspect that this could occur quite often.
>
> Can you elaborate on how all packages will fail to build?

I was thinking that the 'ghc-all-cabal-files' would come from a
released tar file that would change and therefore the hash would not
match.  However, as you mentioned, the package would really come from
a git checkout and would therefore not become unavailable.

My bad, sorry.

Fede

Re: updating many haskell packages

[Troy Sankey]

[-- Attachment #1: Type: text/plain, Size: 921 bytes --]

Quoting Troy Sankey (2017-02-18 12:40:15)
> There is one more problem, and it is not a minor problem...
> 
> ------------------------------- cut here -------------------------------
> $ du -hs tmp/all-cabal-files
> 862M    tmp/all-cabal-files
> $ du -hs tmp/all-cabal-files/.git
> 140M    tmp/all-cabal-files/.git
> $ echo '862 - 140' | bc
> 722
> ------------------------------- cut here -------------------------------
> 
> The filesize of the package output will be 722 MiB.  We could prune it
> of all the unrelated versions according to the stackage LTS, but this
> starts to get complicated and error-prone...

On second thought, this is not too much of an issue since
ghc-all-cabal-files would only be an implicit *native* input.
Bystanders who install ghc-old-time (or any other package with a revised
cabal file) from a guix substitute will not need to download
ghc-all-cabal-files.

Troy

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEE0zLJ6STd4Cp+CgbIgs677ofYV8IFAlip8FMACgkQgs677ofY
V8IRPRAAiY4vfEgF3KMEvKW7zLtxmmLVbec3Jv3WZ+2UGiCuad7MpzUsCWgvcl/i
DvNuRpC1bKfjcAMQ9vUz0Rgh8yKY9JWNiJMN63opID5SDjK4HcQlwRStCo708FuY
IsIlGwEKFxnMnauJKvff97JPzYjNbLWkTwgdb8hMHMgTCEQ6H6qvU2Mq+yC06WL4
qJwGAGQs99uuMG6k7ed1J2xwfzsZuxW96Sg8Iqa1TODAf1yjfAgH+1haYqa4fiPk
2lLwl7TMmNlOQbO5gk8wegDvoAK4ZYCtwuTi4EFDm+HMYDz1o8mw02vBovod/Sh5
FbOmAxYiXPkbyArYtwPVQwyfEGTXyWyz+8XEgqv8yUvnuXoMoj6/6mJbM0XqZjqJ
1bIF4ecenV4AFaNbs8uzdMr551yLTlkc58XrYwJnq95yWQt6kDH5Mzqbce9BCSuH
t7WFj17Nkb377jJ7hqtp/CA2NZwxaQ3uSanGwk0sO28DWTUCUYPyZLYcTT6yLAvy
57J4OR20hFQS2LTxjd1W32bKVoqZxBfZy7G1Qu+n4EhSzRcGZy5hGjm5G3gTU/lv
0McLJg99o/s4z9kuTWrKoPNp47LFogr8PGQfUAturoC6DUUcthLSvtIwejjZj+tK
O6Yzb/nJIhl/iemPfgjGX9uWszHiLVM2L5PeLXlQ/rlE2pOq3ng=
=fBRr
-----END PGP SIGNATURE-----

updating many haskell packages

[Troy Sankey]

[-- Attachment #1: Type: text/plain, Size: 887 bytes --]

I have a WIP branch which contains many haskell package updates and
additions, and makes haskell-build-system use the latest ghc (8.0.2).

https://github.com/pwnage101/guix/tree/add-gitit

It started as a small project to create a Gitit package (hence the name
of the branch), but blew up into something larger.  It's hard work, so I
just wanted to check in with you all to see if this seems like the right
direction.

As of this writing, I have over 66 packages updated, 16 added, and
various fixes on existing packages to get everything buildable against
ghc@8.0.2.  I'm debating whether I should systematically update all the
haskell packages while I'm at it, or just make sure everything is
buildable under 8.0.2.  There are over 277 haskell packages in all.

I wonder how I should send these patches after I'm all done.  git
send-email?  That seems crazy.

Troy

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEE0zLJ6STd4Cp+CgbIgs677ofYV8IFAliYCEMACgkQgs677ofY
V8KA0Q/+LeWmLqnQRWt04JFtO2v0Vai//JqzESel/Jqs5XmpDltb8R5j3MlbLoLD
88wdrwf3RLlMElYxJtukSap+DWXBDmxZ2u8CkRGxTLXAPTNb/EqcRVRfUaguQ04z
R2nDqeqrm7CpazbiBbpEx3CpSRqYfrGWkAZHZZlNeOqFoSeDT9Tb7TVgj/Z6kmnH
ea9ALO3yYqPdCqqfgsffhixMGGew95ccLG+C9jeMQyLdeszXYkDoRbLV2TRHAt52
myrFdpv1aoDqaVsQs9YIFWalXECsaaFODgw9C1s7BPoTT5pnwb0/SHXKloPPPWom
aX4iIVg9MTuGDDycopfPW0qIpxPcWtZDYYkrUEfpklg+ouVEqCyTPZdcTAt2JBa1
erc35Pk2MjoBGhbSJN30QLqkYYdqGI5lq+dZTfE8kXWcAP1unCn7Z3fvhb9HkA4R
33uScZ56kF/Tm9uBO6eYw0oGt4eVfMdrZfs0AjpDcK3RQ3RtwWH/BQohutHJhbJp
/ARBgWyk9QQA8qHEGdUHjkNkRVi2KJJPqq+Sz380GZBpW50hcGzW12ZRRGVRlhTr
PusvoRtrmYVECcKpNIoNDiM3vdJg2UzYpRKvv36ytMdGumrreBrmRQAvItJFaBW+
3Kyd9Ek7sqlQoUwwy3V+LevRSX7GGHklcq6rqUOjLK4KfcOwbHk=
=As5G
-----END PGP SIGNATURE-----

Re: updating many haskell packages

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 448 bytes --]

On Mon, Feb 06, 2017 at 12:23:23AM -0500, Troy Sankey wrote:
> I wonder how I should send these patches after I'm all done.  git
> send-email?  That seems crazy.

Dozens of patches in their own messages is a little hard to work with as
a reviewer. But, you can put them all in one patch file that is trivial
to apply to my Git tree:

$ git format-patch your-branch..master --stdout > patch-series.patch

And on my end:

$ git am patch-series.patch

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]