From mboxrd@z Thu Jan 1 00:00:00 1970 From: Federico Beffa Subject: Re: updating many haskell packages Date: Sat, 18 Feb 2017 10:43:51 +0100 Message-ID: References: <148639459210.5455.4004512896538726304@what> <148641482905.2874.1459989202091260937@what> <148718709098.32672.6688568200387118544@what> <148735363629.2068.10173874369279597347@what> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:52033) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cf1YY-0001i1-IB for guix-devel@gnu.org; Sat, 18 Feb 2017 04:43:56 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cf1YX-0001L6-8e for guix-devel@gnu.org; Sat, 18 Feb 2017 04:43:54 -0500 Received: from mail-ua0-x22d.google.com ([2607:f8b0:400c:c08::22d]:34663) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1cf1YX-0001Kr-2L for guix-devel@gnu.org; Sat, 18 Feb 2017 04:43:53 -0500 Received: by mail-ua0-x22d.google.com with SMTP id 35so42984628uak.1 for ; Sat, 18 Feb 2017 01:43:52 -0800 (PST) In-Reply-To: <148735363629.2068.10173874369279597347@what> List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Troy Sankey Cc: Guix-devel On Fri, Feb 17, 2017 at 6:47 PM, Troy Sankey wrote: > Quoting Federico Beffa (2017-02-17 03:00:04) >> On Wed, Feb 15, 2017 at 8:31 PM, Troy Sankey wrote: >> > Quoting Troy Sankey (2017-02-06 16:00:29) >> >> Quoting Federico Beffa (2017-02-06 15:34:47) >> >> > I would consider a discrepancy between a cabal file on Hackage and the >> >> > actual cabal file included in a tar archive a bug. It may be helpful >> >> > to report it to the author. >> >> >> >> I found this issue on the hackage github: >> >> >> >> https://github.com/haskell/hackage-server/issues/503 >> >> >> >> It looks like it's a feature of hackage and/or cabal to allow multiple >> >> revisions of cabal files per tarball release. IIUC, the `cabal get` >> >> utility takes care of automatically updating the cabal file to the >> >> latest revision. >> >> >> >> I'll have to look more into this later. >> > >> > Indeed the .cabal files can be revised between releases. Somehow the >> > new revisions get pushed to the all-cabal-files repository [0] with an >> > additional "x-revision" key [1], and Hackage picks up the new cabal >> > file. The `cabal` and `stack` utilities substitute the .cabal file from >> > the release tarball with the revised .cabal file from Hackage if it is >> > newer. >> > >> > In theory, anything in the .cabal file could be modified, not just >> > dependency bounds. I think guix should use the updated .cabal files for >> > better consistency with the rest of the haskell ecosystem, and also for >> > less per-package maintenance cost. >> > >> > How can guix adapt? Should we somehow incorporate the all-cabal-files >> > repository in the haskell build system, adding a post-unpack phase to >> > substitute the entire .cabal file? >> >> Packages are build in isolated environments without network. >> Therefore it's not possible to access Hackage with a phase of the >> build system. Even if it were possible it would be undesirable >> because it would make things non reproducible. >> >> Files not included in the tar can be added as origin inputs to a >> package definition (see the testsuite for GHC in the ghc package). >> However, they are verified with hashes. Any change to those files will >> break the package. >> >> Fede > > Forgive me if my understanding of build systems in Guix is flawed, but > let me explain my idea with more detail: > > First, make a data-only package called "ghc-all-cabal-files" containing > the checkout of a specific commit of the all-cabal-files repository from > github. We can periodically update this package, but there is no > traditional "release"---we just keep pulling the HEAD of the hackage > branch. This package would then act as a helper package for the haskell > build system---every haskell package should implicitly use this package > as input. Then we can write a post-unpack phase for the > haskell-build-system which updates the unpacked .cabal file iff it finds > a newer .cabal file in ghc-all-cabal-files (we know how to determine if > the cabal file is newer: it will have a higher "x-revision" value, or > that key will merely exist). > > One problem I have not fully solved is the technical debt associated > with keeping the proposed ghc-all-cabal-files package up-to-date. I > believe updating it would require all haskell packages to be rebuilt. > We could create a build system argument called use-newest-cabal-file to > toggle the feature, in which case we would only switch it to #t if we > already know the .cabal file to be stale. Then only a small subset of > packages would need to be rebuilt, and there is less technical debt than > the current solution which involves monkey-patching every cabal file > that needs it. My worry with this approach is that every time that a single cabal file in 'ghc-all-cabal-files' is updated all packages will fail to build (the implicit input will fail). Given that there are several thousands of cabal files, I suspect that this could occur quite often. Maybe another approach could be: when we update the GHC packages we follow a specific Stackage release. If a package tar includes an old cabal file, we enable a 'jailbreak' flag such that the build system clears version bounds. This of course relies on the fact that Stackage verified that this is fine. Regards, Fede