on cabal revisions

on cabal revisions

[Robert Vollmert]

Hello all,

I have a question regarding how cabal revisions are handled
for haskell packages. Namely, would it make sense / is it
possible to make the revised cabal file part of the source
field in the package definition?


Summary for non-haskell-experts: A hackage package of
a given version can have metadata revisions that are applied
to its cabal file, but not the source tarball itself.

Consider e.g. the package utf8-string-1.0.1.1,

 http://hackage.haskell.org/package/utf8-string-1.0.1.1

The source tarball includes a file utf8-string.cabal, which
is “revision 0”. The latest revision of utf8-string-1.0.1.1
is revision 3, with cabal file at

 http://hackage.haskell.org/package/utf8-string-1.0.1.1/revision/3.cabal

Typically, such revisions update the version bounds on
dependencies; e.g. a package might not build against the
current guix set of haskell packages at revision 0, but might
build at a higher revision because some restrictive bound has
been lifted.


Currently, haskell-build-system supports cabal revision via
an argument field, e.g.:

   (arguments
    `(#:cabal-revision
      ("3" "02vhj5gykkqa2dyn7s6gn8is1b5fdn9xcqqvlls268g7cpv6rk38")))

This works; I’ve posted a patch

 https://debbugs.gnu.org/cgi/bugreport.cgi?bug=36048

to make guix import hackage aware of this so that it doesn’t
import stale version by default.


However, I was thinking that it might be better to have this
variant cabal file be part of the source field of the package
definition. Is there a nice way to do that?

It feels like this might be hacked in via the patching mechanism,
but that feels dirty. I’d rather patching be generalized to
something that both supports applying a patch (from a local file
or a URL), or copying a file. (Or unpacking another tar ball to
some subdirectory, for that matter.)

Alternatively, could this be achieved through the snippet field?
I couldn’t work out how, and none of the uses of snippet that I
found used any file input.

What I’m imagining is something roughly like this:

   (source
     (origin
       (method url-fetch)
       (uri “https://hackage.haskell.org/package-sources.tar.gz”))
       (sha256 …))
     (origin
       (destination “package.cabal”)
       (method url-fetch)
       (uri “https://hackage.haskell.org/package/1.cabal”)
       (sha256 …)))

probably with some way to specify how the sources should be
combined, by default unpacking over the previous result
sequentially. Would that be possible? A good idea even?


(My reasons why using the source field instead of the argument
field might be nicer:
- all sources in one place
- less special-casing for the haskell build system
- simpler 
- maaaybe a useful abstraction that allows simplifying things
 like patching, too)


What do you think?

Robert

Re: on cabal revisions

[Timothy Sample]

Hi Robert,

I patched the “cabal-revision” stuff into the build system, so I suppose
I should weigh in here.  :)

Robert Vollmert <rob@vllmrt.net> writes:

> Hello all,
>
> I have a question regarding how cabal revisions are handled
> for haskell packages. Namely, would it make sense / is it
> possible to make the revised cabal file part of the source
> field in the package definition?

Yes it makes sense, and yes it is possible.  This is something I wanted
to do before, but I wasn’t sure how.  The only idea I had was to make a
new download method, but that’s a very heavy solution for such a simple
task.

> Summary for non-haskell-experts: A hackage package of
> a given version can have metadata revisions that are applied
> to its cabal file, but not the source tarball itself.
>
> Consider e.g. the package utf8-string-1.0.1.1,
>
>  http://hackage.haskell.org/package/utf8-string-1.0.1.1
>
> The source tarball includes a file utf8-string.cabal, which
> is “revision 0”. The latest revision of utf8-string-1.0.1.1
> is revision 3, with cabal file at
>
>  http://hackage.haskell.org/package/utf8-string-1.0.1.1/revision/3.cabal
>
> Typically, such revisions update the version bounds on
> dependencies; e.g. a package might not build against the
> current guix set of haskell packages at revision 0, but might
> build at a higher revision because some restrictive bound has
> been lifted.
>
>
> Currently, haskell-build-system supports cabal revision via
> an argument field, e.g.:
>
>    (arguments
>     `(#:cabal-revision
>       ("3" "02vhj5gykkqa2dyn7s6gn8is1b5fdn9xcqqvlls268g7cpv6rk38")))
>
> This works; I’ve posted a patch
>
>  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=36048
>
> to make guix import hackage aware of this so that it doesn’t
> import stale version by default.

That’s fantastic, thanks!  I see it hasn’t been reviewed yet – I’ll try
and take a look tomorrow.

> However, I was thinking that it might be better to have this
> variant cabal file be part of the source field of the package
> definition. Is there a nice way to do that?
>
> It feels like this might be hacked in via the patching mechanism,
> but that feels dirty. I’d rather patching be generalized to
> something that both supports applying a patch (from a local file
> or a URL), or copying a file. (Or unpacking another tar ball to
> some subdirectory, for that matter.)
>
> Alternatively, could this be achieved through the snippet field?
> I couldn’t work out how, and none of the uses of snippet that I
> found used any file input.

There is a way to do it through the mighty power of gexps!  Behold:

    (origin
      (method url-fetch)
      (uri (string-append "https://hackage.haskell.org/package/"
                          "tree-diff/tree-diff-0.0.1.tar.gz"))
      (sha256
       (base32
        "049v44c520jy3icxlnrvbdblh3mjmvd7m6qmkzxbzkf02x63xqmz"))
      (snippet
       #~(copy-file
          #+(origin
              (method url-fetch)
              (uri (string-append "https://hackage.haskell.org/package/"
                                  "tree-diff-0.0.1/revision/4.cabal"))
              (sha256
               (base32
                "1rqxxyj6hqllahs11693g855cxz8mgnb490s7j1ksd300i5xgjsp")))
          "tree-diff.cabal")))

(I’m constantly amazed at how useful, composable, and empowering gexps
are.)  This could be cleaned up with some procedures and made just as
nice as the current setup.  What do you think?

> What I’m imagining is something roughly like this:
>
>    (source
>      (origin
>        (method url-fetch)
>        (uri “https://hackage.haskell.org/package-sources.tar.gz”))
>        (sha256 …))
>      (origin
>        (destination “package.cabal”)
>        (method url-fetch)
>        (uri “https://hackage.haskell.org/package/1.cabal”)
>        (sha256 …)))
>
> probably with some way to specify how the sources should be
> combined, by default unpacking over the previous result
> sequentially. Would that be possible? A good idea even?

This makes sense, but I’m not sure it’s a common enough use case to
warrant a specialized interface.  Besides these Cabal revisions, only a
handful of packages (that I’ve seen) need to download extra non-patch
stuff.  Some packages have external test suites, for instance.  Since it
would be mostly just the Haskell packages that would use it, and we
already have one and a half Cabal revision interfaces, I don’t see a
strong need.  That being said, I don’t know the package collection as
well as many others here, so maybe I’ve missed something.

> (My reasons why using the source field instead of the argument
> field might be nicer:
> - all sources in one place
> - less special-casing for the haskell build system
> - simpler 
> - maaaybe a useful abstraction that allows simplifying things
>  like patching, too)
>
>
> What do you think?

I think the first three points you make are good ones!  I’m less sure
about that last one, though.  :)


-- Tim

Re: on cabal revisions

[Robert Vollmert]

Hi Timothy,

thanks for your reply.

> On 12. Jun 2019, at 06:54, Timothy Sample <samplet@ngyro.com> wrote:
> 
> Hi Robert,
> 
> I patched the “cabal-revision” stuff into the build system, so I suppose
> I should weigh in here.  :)
> 
> Robert Vollmert <rob@vllmrt.net> writes:
> 
>> Hello all,
>> 
>> I have a question regarding how cabal revisions are handled
>> for haskell packages. Namely, would it make sense / is it
>> possible to make the revised cabal file part of the source
>> field in the package definition?
> 
> Yes it makes sense, and yes it is possible.  This is something I wanted
> to do before, but I wasn’t sure how.  The only idea I had was to make a
> new download method, but that’s a very heavy solution for such a simple
> task.

[…]

>> Alternatively, could this be achieved through the snippet field?
>> I couldn’t work out how, and none of the uses of snippet that I
>> found used any file input.
> 
> There is a way to do it through the mighty power of gexps!  Behold:
> 
>    (origin
>      (method url-fetch)
>      (uri (string-append "https://hackage.haskell.org/package/"
>                          "tree-diff/tree-diff-0.0.1.tar.gz"))
>      (sha256
>       (base32
>        "049v44c520jy3icxlnrvbdblh3mjmvd7m6qmkzxbzkf02x63xqmz"))
>      (snippet
>       #~(copy-file
>          #+(origin
>              (method url-fetch)
>              (uri (string-append "https://hackage.haskell.org/package/"
>                                  "tree-diff-0.0.1/revision/4.cabal"))
>              (sha256
>               (base32
>                "1rqxxyj6hqllahs11693g855cxz8mgnb490s7j1ksd300i5xgjsp")))
>          "tree-diff.cabal")))
> 
> (I’m constantly amazed at how useful, composable, and empowering gexps
> are.)  This could be cleaned up with some procedures and made just as
> nice as the current setup.  What do you think?

Lovely, thanks. The following works for me:

    (source
     (origin
       (method url-fetch)
       (uri (string-append
             "https://hackage.haskell.org/package/tree-diff/tree-diff-"
             version
             ".tar.gz"))
       (sha256
        (base32
         "049v44c520jy3icxlnrvbdblh3mjmvd7m6qmkzxbzkf02x63xqmz"))
       (snippet
        (snippet-hackage-revision "tree-diff" version "4"
          (base32
            "1rqxxyj6hqllahs11693g855cxz8mgnb490s7j1ksd300i5xgjsp")))))

with

(define* (snippet-hackage-revision hackage-name version revision hash)
  #~(copy-file
     #+(origin
         (method url-fetch)
         (uri (string-append "https://hackage.haskell.org/package/"
                             hackage-name "-" version "/revision/"
                             revision ".cabal"))
         (sha256 hash))
     (string-append #+hackage-name ".cabal")))

Would this be a worthwhile change? What would be a good place (and name)
for snippet-hackage-revision?

One disadvantage that I see is that it moves the revision information
from plain data in the arguments field to a rather opaque code snippet.

>> (My reasons why using the source field instead of the argument
>> field might be nicer:
>> - all sources in one place
>> - less special-casing for the haskell build system
>> - simpler 
>> - maaaybe a useful abstraction that allows simplifying things
>> like patching, too)

I remembered one more: guix refresh. As far as I understand, it works
only on the level of the source field, thus having the revision there
would help make it revision-aware. I’m unsure though whether the
snippet approach actually improves things here — probably the refresh
code would see the resulting gexp and not the raw data.

Cheers
Rob

Re: on cabal revisions

[Timothy Sample]

Hi Robert,

Robert Vollmert <rob@vllmrt.net> writes:

> Hi Timothy,
>
> thanks for your reply.
>
>> On 12. Jun 2019, at 06:54, Timothy Sample <samplet@ngyro.com> wrote:
>> 
>> Hi Robert,
>> 
>> I patched the “cabal-revision” stuff into the build system, so I suppose
>> I should weigh in here.  :)
>> 
>> Robert Vollmert <rob@vllmrt.net> writes:
>> 
>>> Hello all,
>>> 
>>> I have a question regarding how cabal revisions are handled
>>> for haskell packages. Namely, would it make sense / is it
>>> possible to make the revised cabal file part of the source
>>> field in the package definition?
>> 
>> Yes it makes sense, and yes it is possible.  This is something I wanted
>> to do before, but I wasn’t sure how.  The only idea I had was to make a
>> new download method, but that’s a very heavy solution for such a simple
>> task.
>
> […]
>
>>> Alternatively, could this be achieved through the snippet field?
>>> I couldn’t work out how, and none of the uses of snippet that I
>>> found used any file input.
>> 
>> There is a way to do it through the mighty power of gexps!  Behold:
>> 
>>    (origin
>>      (method url-fetch)
>>      (uri (string-append "https://hackage.haskell.org/package/"
>>                          "tree-diff/tree-diff-0.0.1.tar.gz"))
>>      (sha256
>>       (base32
>>        "049v44c520jy3icxlnrvbdblh3mjmvd7m6qmkzxbzkf02x63xqmz"))
>>      (snippet
>>       #~(copy-file
>>          #+(origin
>>              (method url-fetch)
>>              (uri (string-append "https://hackage.haskell.org/package/"
>>                                  "tree-diff-0.0.1/revision/4.cabal"))
>>              (sha256
>>               (base32
>>                "1rqxxyj6hqllahs11693g855cxz8mgnb490s7j1ksd300i5xgjsp")))
>>          "tree-diff.cabal")))
>> 
>> (I’m constantly amazed at how useful, composable, and empowering gexps
>> are.)  This could be cleaned up with some procedures and made just as
>> nice as the current setup.  What do you think?
>
> Lovely, thanks. The following works for me:
>
>     (source
>      (origin
>        (method url-fetch)
>        (uri (string-append
>              "https://hackage.haskell.org/package/tree-diff/tree-diff-"
>              version
>              ".tar.gz"))
>        (sha256
>         (base32
>          "049v44c520jy3icxlnrvbdblh3mjmvd7m6qmkzxbzkf02x63xqmz"))
>        (snippet
>         (snippet-hackage-revision "tree-diff" version "4"
>           (base32
>             "1rqxxyj6hqllahs11693g855cxz8mgnb490s7j1ksd300i5xgjsp")))))
>
> with
>
> (define* (snippet-hackage-revision hackage-name version revision hash)
>   #~(copy-file
>      #+(origin
>          (method url-fetch)
>          (uri (string-append "https://hackage.haskell.org/package/"
>                              hackage-name "-" version "/revision/"
>                              revision ".cabal"))
>          (sha256 hash))
>      (string-append #+hackage-name ".cabal")))
>
> Would this be a worthwhile change? What would be a good place (and name)
> for snippet-hackage-revision?
>
> One disadvantage that I see is that it moves the revision information
> from plain data in the arguments field to a rather opaque code snippet.

True, but we could move the procedure to the build side, and then the
gexp would be simple enough to understand mechanically.  The downside to
this is that we would (probably) have to put something in the “modules”
field in the “origin” record, making it a little more verbose.

To be clear, what I mean is to write

    (snippet
     #~(snippet-hackage-revision "tree-diff" version "4"
         (base32
          "1rqxxyj6hqllahs11693g855cxz8mgnb490s7j1ksd300i5xgjsp")))

and then put “snippet-hackage-revision” somewhere in “(guix build ...)”.
This way, at least in most cases, you could get the Cabal revision if
you had nothing but an “origin” record.  I’m not sure how important this
is, though (see below).

>>> (My reasons why using the source field instead of the argument
>>> field might be nicer:
>>> - all sources in one place
>>> - less special-casing for the haskell build system
>>> - simpler 
>>> - maaaybe a useful abstraction that allows simplifying things
>>> like patching, too)
>
> I remembered one more: guix refresh. As far as I understand, it works
> only on the level of the source field, thus having the revision there
> would help make it revision-aware. I’m unsure though whether the
> snippet approach actually improves things here — probably the refresh
> code would see the resulting gexp and not the raw data.

This is a very good point!  Unfortunately, AFAICT, the refresh mechanism
only works on the level of versions.  It has no support for automated
patches or snippet changes.  It looks like updating the Cabal revision
using the refresh script would require one or two far-reaching changes.
Are there other package repositories that use a release-with-patches
style?  Having another example would make it easier to design at the
right level of abstraction.

Now I think that we should have a good plan for refreshing before making
a bunch of changes to the Cabal revision stuff.  Otherwise, we might
have to change everything again if and when we make refreshing work.

Thoughts?


-- Tim

Re: on cabal revisions

[Robert Vollmert]

On 13. Jun 2019, at 16:25, Timothy Sample <samplet@ngyro.com> wrote:
> 
>> I remembered one more: guix refresh. As far as I understand, it works
>> only on the level of the source field, thus having the revision there
>> would help make it revision-aware. I’m unsure though whether the
>> snippet approach actually improves things here — probably the refresh
>> code would see the resulting gexp and not the raw data.
> 
> This is a very good point!  Unfortunately, AFAICT, the refresh mechanism
> only works on the level of versions.  It has no support for automated
> patches or snippet changes.  It looks like updating the Cabal revision
> using the refresh script would require one or two far-reaching changes.
> Are there other package repositories that use a release-with-patches
> style?  Having another example would make it easier to design at the
> right level of abstraction.
> 
> Now I think that we should have a good plan for refreshing before making
> a bunch of changes to the Cabal revision stuff.  Otherwise, we might
> have to change everything again if and when we make refreshing work.
> 
> Thoughts?

I agree that reworking the revision thing shouldn’t happen first. I’m
happy to understand how it might work for now.

Is “guix refresh” even a good approach for the haskell packages? There’s
a lot of work being done to put consistent sets of haskell packages, e.g.
the whole stackage project. Does Guix mean to replicate this work, or would
it maybe be a good choice to follow stackage sets?

How big is the diff between the current haskell package definitions and
the result of guix hackage import? It would be tempting to consider most
as an output of guix hackage import, and to refresh them by reimporting,
possibly based on a version set pulled from stackage.

Hmm another tangential question: Is there any particular system to the
splitting of gnu/packages/haskell*.scm?

haskell.scm itself is huge — does this have performance implications? I’ve been
wondering whether guile scales well with module size. It appears that circular
imports between package modules are not a problem? Personally I’d like to
see the haskell compiler in a different module from hackage modules. Other
than that, keeping our own categorization seems wasteful, why not one
haskell-hackage.scm that’s sorted alphabetically by package name?

Robert

Re: on cabal revisions

[Timothy Sample]

Hi Robert,

I don’t have a lot of time, so here’s a “hot take” on the points you
bring up.

Robert Vollmert <rob@vllmrt.net> writes:

> On 13. Jun 2019, at 16:25, Timothy Sample <samplet@ngyro.com> wrote:
>> 
>>> I remembered one more: guix refresh. As far as I understand, it works
>>> only on the level of the source field, thus having the revision there
>>> would help make it revision-aware. I’m unsure though whether the
>>> snippet approach actually improves things here — probably the refresh
>>> code would see the resulting gexp and not the raw data.
>> 
>> This is a very good point!  Unfortunately, AFAICT, the refresh mechanism
>> only works on the level of versions.  It has no support for automated
>> patches or snippet changes.  It looks like updating the Cabal revision
>> using the refresh script would require one or two far-reaching changes.
>> Are there other package repositories that use a release-with-patches
>> style?  Having another example would make it easier to design at the
>> right level of abstraction.
>> 
>> Now I think that we should have a good plan for refreshing before making
>> a bunch of changes to the Cabal revision stuff.  Otherwise, we might
>> have to change everything again if and when we make refreshing work.
>> 
>> Thoughts?
>
> I agree that reworking the revision thing shouldn’t happen first. I’m
> happy to understand how it might work for now.
>
> Is “guix refresh” even a good approach for the haskell packages? There’s
> a lot of work being done to put consistent sets of haskell packages, e.g.
> the whole stackage project. Does Guix mean to replicate this work, or would
> it maybe be a good choice to follow stackage sets?

We mostly do this already.  There is a Stackage importer, and a little
while ago I updated most of our Haskell packages to their LTS 12
versions.

> How big is the diff between the current haskell package definitions and
> the result of guix hackage import? It would be tempting to consider most
> as an output of guix hackage import, and to refresh them by reimporting,
> possibly based on a version set pulled from stackage.

IIRC, Nix automatically imports all Haskell packages from Hackage (I’m a
little fuzzy on the details, though).

Our refresh script knows about Stackage, so you can use it to keep
packages up to date with Stackage.  One of the main issues with
automatic package maintenance in Guix is that we have some
unconventional, non-technical and semi-technical requirements on our
packages.  We need to follow the FSDG (Free Software Distribution
Guidelines); we try and make sure that everything has a useful synopsis
and description; and we try to make sure everything is bootstrappable
and reproducible.  I say “unconventional” above because there are often
upstream issues with these things that need to be fixed manually.

The reason I bring these up is that we would have to develop some
mechanism to maintain documentation changes, extra license information,
and any other changes we make.  Basically, we can’t “just” reimport
every time.  I also worry that trying to move in that direction would
result in less care taken to maintain our own standards of quality,
since the tools would put pressure on us to just use whatever we get
from upstream.

This is just me thinking out loud, of course.

> Hmm another tangential question: Is there any particular system to the
> splitting of gnu/packages/haskell*.scm?

Not beyond the obvious one (that is, crypto packages go in “crypto”).

> haskell.scm itself is huge — does this have performance implications?

Yes.  It is an ongoing problem with Guix, and one of the reasons we
split up the files in the first place.  If you want to know more, there
have been many discussions on this list.

> I’ve been
> wondering whether guile scales well with module size. It appears that circular
> imports between package modules are not a problem? Personally I’d like to
> see the haskell compiler in a different module from hackage modules.

We are moving towards this in general.  The “guile.scm” and
“guile-xyz.scm” split is a recent example of this.

> Other
> than that, keeping our own categorization seems wasteful, why not one
> haskell-hackage.scm that’s sorted alphabetically by package name?

The subcategories (e.g., “web”) are mostly for performance reasons,
AIUI.  I’m not sure why packages are not alphabetical.  I don’t think
there is any formal organization to the packages within modules.

Hope that helps!


-- Tim

Re: on cabal revisions

[Ricardo Wurmus]

Timothy Sample <samplet@ngyro.com> writes:

>> Is “guix refresh” even a good approach for the haskell packages? There’s
>> a lot of work being done to put consistent sets of haskell packages, e.g.
>> the whole stackage project. Does Guix mean to replicate this work, or would
>> it maybe be a good choice to follow stackage sets?
>
> We mostly do this already.  There is a Stackage importer, and a little
> while ago I updated most of our Haskell packages to their LTS 12
> versions.

At the beginning we just had all the latest packages from Hackage and it
took considerable time and effort to upgrade them all to a consistent
LTS set.

This makes the review of updates to ghc-* packages a little slower for
me because these packages really ought to be upgraded together so that
they closely resemble whichever LTS version we’re following.

(This is what I’m doing for Bioconductor and CRAN packages and it would
be great if we could regularly sync our Haskell packages in the same
manner.)

--
Ricardo

haskell package organization (Re: on cabal revisions)

[Robert Vollmert]

On 14. Jun 2019, at 22:24, Ricardo Wurmus <rekado@elephly.net> wrote:
> Timothy Sample <samplet@ngyro.com> writes:
>> 
>> We mostly do this already.  There is a Stackage importer, and a little
>> while ago I updated most of our Haskell packages to their LTS 12
>> versions.
> 
> At the beginning we just had all the latest packages from Hackage and it
> took considerable time and effort to upgrade them all to a consistent
> LTS set.
> 
> This makes the review of updates to ghc-* packages a little slower for
> me because these packages really ought to be upgraded together so that
> they closely resemble whichever LTS version we’re following.

[ This is really in reply to multiple threads of this discussion. Also
  it’s based on my limited understanding of issues — please correct my
  misunderstandings. E.g. I don’t quite understand how guile works around
  circular imports. Finally, for my motivation: I’m packaging a few
  Haskell programs that pull in quite a few new libraries, and would
  rather use that opportunity to make matters better not worse. ]

So what do you all think of some organization like the following:

gnu/packages/haskell.scm:
  Haskell compilers (ghc), and their dependencies if any. Shouldn’t import
  any of the the other haskell modules.

gnu/packages/haskell-stackage-*.scm:
  Various Haskell library modules (ghc-*) that are consistent with some
  stackage set (LTS 12 currently).

gnu/packages/haskell-other.scm:
  Various Haskell library modules that are either on a non-LTS version,
  or not on stackage at all. This might include alternate versions of
  stackage modules, under a different name.

gnu/packages/haskell-tools.scm:
  Haskell executables that are Haskell specific. E.g. hlint,
  hspec-discover, ….

gnu/packages/*.scm:
  Programs that happen to be built using the Haskell build system. E.g.
  things that are currently in haskell-apps.scm: darcs, git-annex.

The idea would be that the stackage updates would be clearly targeted
to those haskell-stackage files.

My personal preference for the haskell-stackage-*.scm files would be to
simply split them by name prefix, e.g.

  haskell-stackage-abc.scm, haskell-stackage-def.scm

so that they are of manageable size. (I think the current categorization
into haskell-web, haskell-crypto etc. causes quite a bit of cognitive
overhead for little benefit: Where should I file this oauth package?
Where do I import it from?)

Robert

Re: on cabal revisions

[Ricardo Wurmus]

Timothy Sample <samplet@ngyro.com> writes:

>> I’ve been
>> wondering whether guile scales well with module size. It appears that circular
>> imports between package modules are not a problem? Personally I’d like to
>> see the haskell compiler in a different module from hackage modules.
>
> We are moving towards this in general.  The “guile.scm” and
> “guile-xyz.scm” split is a recent example of this.

Separating the compiler packages from the libraries and applications
written in the language is a goal.  We need the work to be done for at
least Java and for Haskell.

--
Ricardo

reproducibility and bootstrapping in mid 2019 (was Re: on cabal revisions)

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 4239 bytes --]

Hello Timothy and Robert,

(a little OT re cabal import, but...)

sorry if I repeat something already said on this list... and sorry for a
*personal* rant :-)

Timothy Sample <samplet@ngyro.com> writes:

[...]

> IIRC, Nix automatically imports all Haskell packages from Hackage (I’m a
> little fuzzy on the details, though).

https://github.com/NixOS/nixpkgs/blob/release-19.03/pkgs/development/haskell-modules/hackage-packages.nix

--8<---------------cut here---------------start------------->8---
/* hackage-packages.nix is an auto-generated file -- DO NOT EDIT! */
--8<---------------cut here---------------end--------------->8---

Please do not this in Guix.

I don't have details about auto imported Haskell packages
reproducibility in Nix, I just remember other *historical* approaces
with Javascript packages in a fine 2015 analisys by Christopher Lemmer
Webber

https://web.archive.org/web/20180528141816/http:/dustycloud.org/blog/javascript-packaging-dystopia/

«Unfortunately, Nix just downloads the prebuilt binary and installs
that»

Today Nix ships this jquery related packages:
https://nixos.org/nixos/packages.html#jquery

and AFAIU jquery (python37Packages.xstatic-jquery,
haskellPackages.js-jquery) are still static prebuilt binaries

Are they counted as reproducible by this kind of checks:
https://r13y.com/ ?

[...]

> One of the main issues with automatic package maintenance in Guix is
> that we have some unconventional, non-technical and semi-technical
> requirements on our packages.  We need to follow the FSDG (Free
> Software Distribution Guidelines); we try and make sure that
> everything has a useful synopsis and description; and we try to make
> sure everything is bootstrappable and reproducible.  I say
> “unconventional” above because there are often upstream issues with
> these things that need to be fixed manually.

Re. reproducibility, unfortunately it's **not** a shared goal between
_packaging systems_ developers, so for example we have a good pile of
npm packaged code that is _still_ a *nightmare* from the reproducibility
POV

Hey developers: please come and work with Guix on a reproducible way to
build and distribute software; Guix have all it's needed, please stop
reinventing the *square* wheel.

Citing Christopher article above:

«And let's face it, "fuck it, I'm out" seems to be the mantra of web
application packaging these days. Our deployment and build setups have
gotten so complicated that I doubt anyone really has a decent
understanding of what is going on, really.»

AFAIU this is _still_ the sad situation today with web applications
development, probably 99% of web developers/deployers in the world are
"solving" this with Docker app bundles containing piles of misterious
layers of mixed reproducible and static binaries downloaded _somewhere_,
cryptominers included

There are succesful projects out there that _proudly_ declare the *only*
officially supported distribution method is their Docker
bundle... World: **we have a problem**

https://web.archive.org/web/20180528121826/https://www.vitavonni.de/blog/201503/2015031201-the-sad-state-of-sysadmin-in-the-age-of-containers.html
«Feels like downloading Windows shareware in the 90s to me.»

I don't know the reproducibility situation with Stackage or Hackage
defined packages (and web applications), I just hope it's better than
this.

Re. boostrapping, unfortunately we still have an unbootstrappable
Haskell toolchain
https://web.archive.org/web/20190615075205/http://www.joachim-breitner.de/blog/748-Thoughts_on_bootstrapping_GHC
https://elephly.net/posts/2017-01-09-bootstrapping-haskell-part-1.html

...but that's the same with some jvm languages
http://bootstrappable.org/projects/jvm-languages.html

Anyway at least "the plan" to work it out is in good shape... the world
just needs to invest much more resurces on this serious infrastrutcure
problems, but it's too... distracted :-D

Last but not least: a huge pile of deep gratitude to all the people
around the world working to solve this sad sad situation!

Happy Guix! Gio'.

[...]

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: on cabal revisions

[Ricardo Wurmus]

Hi,

>> What I’m imagining is something roughly like this:
>>
>>    (source
>>      (origin
>>        (method url-fetch)
>>        (uri “https://hackage.haskell.org/package-sources.tar.gz”))
>>        (sha256 …))
>>      (origin
>>        (destination “package.cabal”)
>>        (method url-fetch)
>>        (uri “https://hackage.haskell.org/package/1.cabal”)
>>        (sha256 …)))
>>
>> probably with some way to specify how the sources should be
>> combined, by default unpacking over the previous result
>> sequentially. Would that be possible? A good idea even?
>
> This makes sense, but I’m not sure it’s a common enough use case to
> warrant a specialized interface.  Besides these Cabal revisions, only a
> handful of packages (that I’ve seen) need to download extra non-patch
> stuff.

One notorious class of packages that could benefit from a general
mechanism like this: the TeX Live packages.  They consist of various
subsets of a big SVN repository.  Currently, we pick an arbitrary
directory of the SVN repository as the main source and add all the other
locations as native inputs.

I wanted to add a procedure that accepts multiple locations in the SVN
repository, creates a union, and computes the single hash of the union.
This would make the texlive-* packages a lot simpler.

--
Ricardo