Input needed: Plan for packaging scala

Input needed: Plan for packaging scala

[Katherine Cox-Buday]

Hi all,

I'm planning to have a go at packaging scala. This is my first time
packaging software of any kind, let alone for Guix, so guidance is
appreciated!

I'm brand-new to scala, but it appears that building it from source
involves circular dependencies. Here's the situation:

- scala 2.12.1 is the latest version. To build it from source, a tool
  called "sbt" (scala build tool) is required.
- sbt 0.13 is the latest version. To build it from source requires a
  prior version of sbt.
- I was able to find older versions[1] of sbt which can be built with
  only scala.
- scala 2.9.2 appears to be the last version that can be built with ant.

So, to untangle this knot to achieve reproducibility, I was planning on
first packaging scala 2.9.2, then the latest version of sbt that could
be built with scala, then sbt 0.13 (which would be useful in its own
right as a package), and then finally scala 2.12.1.

The build for scala 2.9.2 attempts to pull some libraries from a remote
source, so I'm assuming I'll need to also package those?

For a new contributor, this is a pile of work, but I don't mind too much
since my need drives me, and I'll be learning more about Guix in the
process :) I do, however, want to confirm that this is a sane plan
before I set out on this journey. It would be easier if I could just do
a binary package to bootstrap the toolchain and back-fill when I have
more experience, but I wasn't sure if that is acceptable.

[1] - https://github.com/sbt/sbt-zero-seven

-- 
Katherine

Re: Input needed: Plan for packaging scala

[Danny Milosavljevic]

Hi,

On Fri, 24 Feb 2017 08:39:24 -0600
Katherine Cox-Buday <cox.katherine.e@gmail.com> wrote:

> So, to untangle this knot to achieve reproducibility, I was planning on
> first packaging scala 2.9.2, then the latest version of sbt that could
> be built with scala, then sbt 0.13 (which would be useful in its own
> right as a package), and then finally scala 2.12.1.

It seems you already did the hard work of finding out how to bootstrap Scala. (I think that writing the package definitions is the easy part. Finding the Scala versions that can be compiled by Java and then compile the correct newer Scala version using it is the hard part)

> The build for scala 2.9.2 attempts to pull some libraries from a remote
> source, so I'm assuming I'll need to also package those?

Yes please.

> I do, however, want to confirm that this is a sane plan
> before I set out on this journey. 

Sounds good.

> It would be easier if I could just do
> a binary package to bootstrap the toolchain and back-fill when I have
> more experience, but I wasn't sure if that is acceptable.

It's acceptable. You should make sure to eventually do a reproducable build - but it's OK to start with a binary package as a builder for the time being.

The question is whether it's actually easier that way. Guix uses non-standard locations (for good reasons) - so a random executable you download won't run because it won't find any libraries. The situation is not that bad if it's in Java, though - you can always specify the classpath when invoking the VM.

P.S. For storing custom package definitions, do you use GUIX_PACKAGE_PATH ? I didn't when I started out using Guix - but I should have. It's easiest to get started writing your own package definitions that way.

Re: Input needed: Plan for packaging scala

[Katherine Cox-Buday]

Danny Milosavljevic <dannym@scratchpost.org> writes:

> Hi,
>
> On Fri, 24 Feb 2017 08:39:24 -0600
> Katherine Cox-Buday <cox.katherine.e@gmail.com> wrote:
>
>> So, to untangle this knot to achieve reproducibility, I was planning
>> on
>> first packaging scala 2.9.2, then the latest version of sbt that
>> could
>> be built with scala, then sbt 0.13 (which would be useful in its own
>> right as a package), and then finally scala 2.12.1.
>
> It seems you already did the hard work of finding out how to bootstrap
> Scala. (I think that writing the package definitions is the easy
> part. Finding the Scala versions that can be compiled by Java and then
> compile the correct newer Scala version using it is the hard part)

Looks like not quite! After speaking to some friendly scala community
members, it looks[1] like the bootstrapping process is much more
harrowing. The last version of scala which only used Java to compile was
pre 2.0 (for reference the 2.0 commit appears to be here[2]). So we'd be
looking at a chain of scala packages numbering possibly in the hundreds
and going back 10+ years.

>> The build for scala 2.9.2 attempts to pull some libraries from a
>> remote
>> source, so I'm assuming I'll need to also package those?
>
> Yes please.
>
>> I do, however, want to confirm that this is a sane plan
>> before I set out on this journey. 
>
> Sounds good.
>
>> It would be easier if I could just do
>> a binary package to bootstrap the toolchain and back-fill when I
>> have
>> more experience, but I wasn't sure if that is acceptable.
>
> It's acceptable. You should make sure to eventually do a reproducable
> build - but it's OK to start with a binary package as a builder for
> the time being.

For the aforementioned reasons, I'm going to start by just packaging the
source of sbt v0.13, but also the pre-built binary. This should help us
at least bootstrap scala in a sane way, if not immediately
sbt. Hopefully this still sounds OK?

> The question is whether it's actually easier that way. Guix uses
> non-standard locations (for good reasons) - so a random executable you
> download won't run because it won't find any libraries. The situation
> is not that bad if it's in Java, though - you can always specify the
> classpath when invoking the VM.
>
> P.S. For storing custom package definitions, do you use
> GUIX_PACKAGE_PATH ? I didn't when I started out using Guix - but I
> should have. It's easiest to get started writing your own package
> definitions that way.

I'm not, thanks for pointing that out!

[1] - https://issues.scala-lang.org/browse/SI-10172?focusedCommentId=76415&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-76415
[2] - https://github.com/scala/legacy-svn-scala/commit/d5f3f603228cb3a0cae1536dba9990a061fcded7

-- 
Katherine

Re: Input needed: Plan for packaging scala

[Ricardo Wurmus]

Hi Katherine,

it’s great that you want to take on Scala!

I’ve done a bit of work in the past to get some Java things packaged and
had to stop when I realised that nobody in the Java world seems to build
dependencies from source.  This makes it very hard for us to construct
proper dependency graphs for Java packages.

This is the primary reason why we only have a handful of Java packages
in Guix.  We don’t want to make an exception for Java and accept
pre-built binaries just to increase the number of packages.  (Luckily
for us, Java build artifacts are often quite portable, so there’s not as
much urgency to make it work with Guix as there is for applications
written in other languages.)

The same applies to build systems other than ant.  Maven cannot be built
without Maven, and its dependencies have dependency cycles.  I’ve been
using the ant-build-system’s feature to generate a default build.xml to
build some of these dependencies that would usually require Maven to
build them.  I’m not currently working on Java packaging (though I still
have a couple of patches that are almost ready).

>> It seems you already did the hard work of finding out how to bootstrap
>> Scala. (I think that writing the package definitions is the easy
>> part. Finding the Scala versions that can be compiled by Java and then
>> compile the correct newer Scala version using it is the hard part)
>
> Looks like not quite! After speaking to some friendly scala community
> members, it looks[1] like the bootstrapping process is much more
> harrowing. The last version of scala which only used Java to compile was
> pre 2.0 (for reference the 2.0 commit appears to be here[2]). So we'd be
> looking at a chain of scala packages numbering possibly in the hundreds
> and going back 10+ years.

Oof!  Looking at the latest build.xml in the legacy-svn-scala repository
the build requires a couple of jars that we don’t have in Guix.  These
(and their dependencies, recursively) would have to be built from source
first.  This alone would be a very demanding task.

If we were able to build a version of the Scala compilers and libraries
that does not require sbt it is unclear whether this would be a viable
path towards a modern version of the language.  Sadly, these
bootstrapping problems are very common, which prompted some of us to
start the Bootstrappable Builds project:

    http://boostrappable.org (temporarily offline)

> For the aforementioned reasons, I'm going to start by just packaging the
> source of sbt v0.13, but also the pre-built binary. This should help us
> at least bootstrap scala in a sane way, if not immediately
> sbt. Hopefully this still sounds OK?

sbt (even at version 0.13) is written in Scala, so maybe it would be
better to package a version of the Scala libraries and an implementation
of scalac first.  Or does sbt provide the complete toolchain?

Are you aware of any other implementation of the Scala toolchain that we
might be able to use for bootstrapping purposes?

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: Input needed: Plan for packaging scala

[Pjotr Prins]

On Sat, Feb 25, 2017 at 11:51:57AM -0600, Katherine Cox-Buday wrote:
> Looks like not quite! After speaking to some friendly scala community
> members, it looks[1] like the bootstrapping process is much more
> harrowing. The last version of scala which only used Java to compile was
> pre 2.0 (for reference the 2.0 commit appears to be here[2]). So we'd be
> looking at a chain of scala packages numbering possibly in the hundreds
> and going back 10+ years.

Maybe we can skip quite a few - it is expected that a Scala compiler
will compile with much older versions of Scala. But it is forensic
work and it is sad that the project itself does not care. I know, for
example, that the D community is aware and wants to make sure there is
a sane bootstrap path to support different backends.

The good news that if the compiler compiles and tests pass you should
be good - one advantage of strong typing. So you could automate
testing compilers and going forward in time from the earliest
versions. It would be an interesting exercise.

> >> The build for scala 2.9.2 attempts to pull some libraries from a
> >> remote
> >> source, so I'm assuming I'll need to also package those?
> >
> > Yes please.
> >
> >> I do, however, want to confirm that this is a sane plan
> >> before I set out on this journey. 
> >
> > Sounds good.
> >
> >> It would be easier if I could just do
> >> a binary package to bootstrap the toolchain and back-fill when I
> >> have
> >> more experience, but I wasn't sure if that is acceptable.
> >
> > It's acceptable. You should make sure to eventually do a reproducable
> > build - but it's OK to start with a binary package as a builder for
> > the time being.

Yes. And if Guix does not accept it on master you can provide an
alternative package repository and publishing host. This is possible
today and with the planned 'guix channnels' feature it will get even
easier for users to use that.

  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=22629

> For the aforementioned reasons, I'm going to start by just packaging the
> source of sbt v0.13, but also the pre-built binary. This should help us
> at least bootstrap scala in a sane way, if not immediately
> sbt. Hopefully this still sounds OK?
> 
> > The question is whether it's actually easier that way. Guix uses
> > non-standard locations (for good reasons) - so a random executable you
> > download won't run because it won't find any libraries. The situation
> > is not that bad if it's in Java, though - you can always specify the
> > classpath when invoking the VM.

sbt is much like Java ant and Ruby bundler that it pulls in
dependencies as binary blobs. Circumventing that behaviour is possible
by providing local jar files - so sbt can find them - but they have to
be in place. I.e., they have been built earlier. To be honest, I never
liked sbt because there was too much 'magic' going on. Only good thing
about sbt is that the definitions are not XML but plain Scala ;). What
should really be a simple Makefile has become a system to deal with
development dependencies. A job much better left to a proper
deployment system such as Guix. That is why I wrote about Ruby being
in that boat and getting rid of rvm, rbenv and bundler:

  https://github.com/pjotrp/guix-notes/blob/master/RUBY.org#getting-rid-of-rvm-rbenv-and-bundler

Same goes for Python virtualenv etc. Essentially these tools are
solving the wrong problem. But then, they were not using Guix! Poor
man's dependency resolution.

In Guix we do not use Ruby bundler. The ruby-build-system is based on
Ruby gem to build gem's locally - that is possible because the gems
contain the full ruby source of the gem. For sbt may eventually need
something similar. Take a look at

  ./guix/build-system/ruby.scm 
  ./guix/build/ruby-build-system.scm 

Don't worry about this too much now. I remember we wrote the
ruby-build-system fairly quickly once you decide what solution to
pursue. Key is to decide where to get the sources of Scala projects,
compile them as packages using the scala-build-system and tell sbt to
use these without trying to pull in jars.

Ricardo and friends have achieved this for other JVM tools.

Pj.

Re: Input needed: Plan for packaging scala

[Pjotr Prins]

On Sun, Feb 26, 2017 at 12:08:37AM +0100, Ricardo Wurmus wrote:
> I’ve done a bit of work in the past to get some Java things packaged and
> had to stop when I realised that nobody in the Java world seems to build
> dependencies from source.  This makes it very hard for us to construct
> proper dependency graphs for Java packages.
> 
> This is the primary reason why we only have a handful of Java packages
> in Guix.  We don’t want to make an exception for Java and accept
> pre-built binaries just to increase the number of packages.  (Luckily
> for us, Java build artifacts are often quite portable, so there’s not as
> much urgency to make it work with Guix as there is for applications
> written in other languages.)

You mean that users can easily deploy JVM packages (jars and wars) on
top of on existing Guix JVM? 

> The same applies to build systems other than ant.  Maven cannot be built
> without Maven, and its dependencies have dependency cycles.  I’ve been
> using the ant-build-system’s feature to generate a default build.xml to
> build some of these dependencies that would usually require Maven to
> build them.  I’m not currently working on Java packaging (though I still
> have a couple of patches that are almost ready).

(...)

> Are you aware of any other implementation of the Scala toolchain that we
> might be able to use for bootstrapping purposes?

sbt is pretty much it. The advantage and disadvantage of a relatively
young language. Maybe sbt can be taught to generate ant or Makefiles.
It probably is possible with some Scala hacking.

It appears to me that we should have a discussion here. If we were to
provide support for JVM build tools, such as Maven and sbt, we could
move forward providing proper packages for almost all projects on the
JVM (10Ks)

Why don't we accept these particular build tools as binary bootstraps
for now? It will be a limited set of true FOSS tools that are very
visible with their source code. Also, unlike other binaries, JVM byte
code is reversible - so we can actually get the source code even if it
is not commented and copyrighted and that can be compared to the
original source code. I am sure that when enough people start using
this maven and sbt functionality there will be volunteers who will
look into removing those binaries again. It is just a matter of time.

Guix is not exactly pure anyway - we bootstrap build binaries for
pragmatic reasons.  Having users now deploy their own jars on top of
Guix appears to be the lesser solution to me - we are forcing them to
use insecure software! I think the real problem now is that we have
few people interested in the JVM. But, do realize that the reason
could be that we do not properly support it. I am sure there would be
great interest in JVM support - the JVM's dependency hell is just as
large as any.

Nix, as far as I remember, simply installs jars. At least that way the
depency graph is under control. We could do better by only
bootstrapping the 'impossible' dependencies as jars and continue
building source from there. It may look lame to purists to bootstrap
the build tools, but for those who are most critical I ask you to fix
the problem instead. In the end it all software and fixable. But, if
no one is actively working on Maven, I suggest to use a jar for the
time being. 

Q: who here wants to work on this eyesore and challenge called Maven?
   So far Ricardo and Roel have worked on it and given up.

Time for some heroics. Or take the pill.

Pj.

Re: Input needed: Plan for packaging scala

[Ricardo Wurmus]

Pjotr Prins <pjotr.public12@thebird.nl> writes:

> On Sun, Feb 26, 2017 at 12:08:37AM +0100, Ricardo Wurmus wrote:
>> This is the primary reason why we only have a handful of Java packages
>> in Guix.  We don’t want to make an exception for Java and accept
>> pre-built binaries just to increase the number of packages.  (Luckily
>> for us, Java build artifacts are often quite portable, so there’s not as
>> much urgency to make it work with Guix as there is for applications
>> written in other languages.)
>
> You mean that users can easily deploy JVM packages (jars and wars) on
> top of on existing Guix JVM?

I don’t know about wars, but the jars we worked with at the MDC worked
just fine on top of the JVM provided by the icedtea packages.

> Why don't we accept these particular build tools as binary bootstraps
> for now?

The reason is that we cannot make one decision for all tools.  A big
chunk of the work is to figure out *which* of the dependencies can be
considered an “impossible” dependency.  Having a binary for Maven alone
wouldn’t be very helpful.  The primary purpose of Maven is to download
required binaries from a central site.  It doesn’t help us *build* any
packages from source.  When we provide Maven it has to be with a
maven-build-system that takes all Java inputs and arranges them in the
layout that a local Maven repository containing all these inputs would
have.  Maven on its own would not be very useful.

In fact, I’ve been packaging libraries that officially require Maven to
build them, but I used plain ant with the default build.xml the
ant-build-system generates.

> Guix is not exactly pure anyway - we bootstrap build binaries for
> pragmatic reasons.

And we try very hard to reduce the number of bootstrap binaries.  I
don’t think it would be a good idea to open the flood gates and accept
binaries for Java tools — because that’s not an exceptional case;
downloading binaries is in the fabric of the Java culture.

We need a few more libraries packaged the hard way, which would make the
next steps a lot easier.  My recommendation is to continue following the
bootstrap path of Maven and its dependencies and see where that leads
us.

> But, if no one is actively working on Maven, I suggest to use a jar
> for the time being.

I’ll get back to it soon.  It’s in my own interest to reduce the number
of unpushed patches I have on my machines :)

> Q: who here wants to work on this eyesore and challenge called Maven?
>    So far Ricardo and Roel have worked on it and given up.

(Hartmut also worked on it and I’m still working on some of his
abandoned patches in a branch.)

I would be very happy to collaborate with others who are interested in
moving Java on Guix forward.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: Input needed: Plan for packaging scala

[Ricardo Wurmus]

Hi Katherine,

>> It seems you already did the hard work of finding out how to bootstrap
>> Scala. (I think that writing the package definitions is the easy
>> part. Finding the Scala versions that can be compiled by Java and then
>> compile the correct newer Scala version using it is the hard part)
>
> Looks like not quite! After speaking to some friendly scala community
> members, it looks[1] like the bootstrapping process is much more
> harrowing. The last version of scala which only used Java to compile was
> pre 2.0 (for reference the 2.0 commit appears to be here[2]). So we'd be
> looking at a chain of scala packages numbering possibly in the hundreds
> and going back 10+ years.

I just found out about http://www.scala-native.org.  It’s a project to
build a stand-alone Scala compiler on top of LLVM.  Version 0.1 has just
been released.

Maybe this could be used to build the build tools needed to build an
initial version of Scala on top of the JVM?  Sounds better to me than
having to go back in time for a fragile bootstrap.

What do you think?

-- 
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net