some questions about GUIX

some questions about GUIX

[Sam Halliday]

[-- Attachment #1: Type: text/plain, Size: 4656 bytes --]

Dear GUIX-SD,

I am interested in this initiative because it is the GNU distribution
and also because there is the possibility to address some major
technical problems I have with current GNU / Linux distributions.

I have several questions, and I would appreciate it if you could attempt
to answer some of them.


* Integration with existing software distribution managers

Modern languages (I use this term loosely) have developed their own
cross-platform distribution mechanisms. It feels like a terrible waste
of effort for GNU / Linux distributions to repackage such artefacts
instead of relying on the well maintained industry standards.

For example, on the JVM, Maven Central is the central point for hosting
source code, binaries and documentations for individual libraries, with
GPG signatures, checksums and license information. Most commercial
organisations host their on mirror of Maven Central, giving them full
control over which artefacts are approved for use internally.

Developers of Java / Scala / etc use tools (maven, ivy, etc) to manage
downloading local filesystem caches of the exact artefacts that they
need (and their transitive dependencies).

However, distributions such as Debian and Redhat repackage all files and
make a complete mess of it. It would be far superior if the OS package
manager simply had a Maven backend that populated a system-wide local
filesystem maven repository. Then Java applications supported by guix
could follow a basic "java application" template, and there would be no
need for the OS package manager to track each individual transitive
dependency. Installation of Java applications would then be trivial.

I don't want to single out Java, because this is true of a large variety
of other languages: Haskell (cabal), Go (built-in), Python, Perl, etc.
One could even argue that Emacs ELPA falls into this category also, and
that a locally hosted MELPA would make sense for locked-down machines.

The result is that I tend to ignore any java, scala, haskell or go
application or library from apt, and use the preferred language-specific
package manager to install and manage that application. It would be
really nice if this was all integrated into guix such that I had one
package management interface for system wide installations.

How are you planning on handling these more modern languages that manage
their own dependencies?


* Docker image

Most GNU / Linux distributions have uploaded a base image of their OS to
hub.docker.com will you be creating and uploading a similar image? I'd
love to try one out.

I use docker to maintain a consistent build environment (across many
heterogenous devices) for many of my free software projects and it is
incredibly well suited to this task.

I strongly recommend docker as a way to build artefacts. With the
immergence of lightweight CI build tools such as
http://github.com/drone/drone/ it would be an incredible way to boost
your build farm! Indeed, you wouldn't need to ask for hardware donations
and could instead ask for docker worker donations (users open up an SSL
connection to their hardware and you use it from an orchestrated drone).
It also makes it a lot easier for users to test their packaging scripts
before submitting their changes to your central repository.


* Issue tracker / comm channels

I see that you are using debbugs, savannah and email mailing lists as
the main communication channels. This is a very traditional approach to
managing a free software community. Along with a large number of other
people, I have found that a single, integrated, web based approach is
far easier to engage with as a user, for example github (free as in
beer) or gitlab (free as in freedom, but less complete). The "pull
request" concept makes it exceptionally easy to make contributions, and
easy for admins to review and accept those proposed changes.

Will you be continuing to use debbugs, savannah and mailing lists going
forward or would you consider moving to a modern community management
system like gitlab?


* Custom / full install images

Debian and co use an antiquated concept of CD/DVD isos for
installations. The vast majority of modern users want to put either a
full or a custom (where they have selected all the packages)
installation image onto a USB. It is not always possible to use network
installation from a minimal image due to firewall rules or connectivity
issues.

Something I've wanted for a long time would be the ability to create an
installation image on a USB, and fill the remainder of the USB with a
VFAT partition. This is remarkably hard to achieve with gparted and a
fixed size installation image.



[-- Attachment #2.1: Type: text/plain, Size: 27 bytes --]


-- 
Best regards,
Sam

[-- Attachment #2.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 180 bytes --]

Re: some questions about GUIX

[Ludovic Courtès]

Hi,

Sam Halliday <sam.halliday@gmail.com> skribis:

> * Integration with existing software distribution managers

[...]

> How are you planning on handling these more modern languages that manage
> their own dependencies?

Currently, we import those languages’ dependency trees into the Guix
dependency tree, and so some additional QA (make sure tests pass,
provide adequate licensing info, descriptions and synopses, etc.)  We
have ‘guix import’ to simplify the initial import, and ‘guix refresh’ to
simplify subsequent updates:

  http://www.gnu.org/software/guix/manual/html_node/Invoking-guix-import.html
  http://www.gnu.org/software/guix/manual/html_node/Invoking-guix-refresh.html

So far it works well, but we have only small subsets of Hackage, CPAN,
etc.

It’s not clear to me how well that will scale, especially with large
repositories with a lot of churn such as Hackage.

There’s always the possibility to do something fully automatic.
However, to me, part of the job of distro developers is to do some
“editorial work” and QA as described above.

> * Docker image
>
> Most GNU / Linux distributions have uploaded a base image of their OS to
> hub.docker.com will you be creating and uploading a similar image? I'd
> love to try one out.

I don’t think there’ll be an “official” presence there, but everyone is
welcome to do it.

> I use docker to maintain a consistent build environment (across many
> heterogenous devices) for many of my free software projects and it is
> incredibly well suited to this task.
>
> I strongly recommend docker as a way to build artefacts. With the
> immergence of lightweight CI build tools such as
> http://github.com/drone/drone/ it would be an incredible way to boost
> your build farm! Indeed, you wouldn't need to ask for hardware donations
> and could instead ask for docker worker donations (users open up an SSL
> connection to their hardware and you use it from an orchestrated drone).
> It also makes it a lot easier for users to test their packaging scripts
> before submitting their changes to your central repository.

Guix’s build daemon uses containers to perform isolated builds:

  http://www.gnu.org/software/guix/manual/html_node/Features.html

So at the lower level, it’s comparable to Docker.  However, by
integrating package management and container provisioning, Guix
inherently provides more fine grain control over what goes into
containers.  Containers are also provisioned from pure declarations, as
opposed to Dockerfiles which describe sequences of commands to obtain
the desired container state.

On that topic, see also the new container features in Guix:

  https://savannah.gnu.org/forum/forum.php?forum_id=8386

> * Issue tracker / comm channels

[...]

> Will you be continuing to use debbugs, savannah and mailing lists going
> forward or would you consider moving to a modern community management
> system like gitlab?

I hear the appeal of GitLab and the like.  However, as was recently
discussed on guix-devel, while I think we must find ways to improve our
workflows (for instance, tracking patches is becoming tricky), I don’t
see us moving to one of those web-based approaches for several reasons:

  https://lists.gnu.org/archive/html/guix-devel/2015-12/msg00429.html

> * Custom / full install images
>
> Debian and co use an antiquated concept of CD/DVD isos for
> installations. The vast majority of modern users want to put either a
> full or a custom (where they have selected all the packages)
> installation image onto a USB. It is not always possible to use network
> installation from a minimal image due to firewall rules or connectivity
> issues.
>
> Something I've wanted for a long time would be the ability to create an
> installation image on a USB, and fill the remainder of the USB with a
> VFAT partition. This is remarkably hard to achieve with gparted and a
> fixed size installation image.

It’s easy to create custom system image from whatever GuixSD declaration
you want and then ‘guix system disk-image’:

  https://www.gnu.org/software/guix/manual/html_node/Invoking-guix-system.html

In fact, the system on the installation image is a regular declaration:

  https://www.gnu.org/software/guix/manual/html_node/System-Installation.html#Building-the-Installation-Image
  http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/system/install.scm#n325

The basics are in place to build customized images.  What’s missing is
convenient command-line interfaces to build specific variants.  For
example, building a live system for a USB image where /home is mounted
from a second partition of the USB disk can easily be done, but not from
the command-line.

Your contribution is welcome to help identify and fix those issues!  :-)

Thanks for your thoughtful questions.

Ludo’.

Re: some questions about GUIX

[Sam Halliday]

[-- Attachment #1: Type: text/plain, Size: 17 bytes --]

Thanks Ludovic,


[-- Attachment #2.1: Type: text/plain, Size: 2492 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:
>> * Integration with existing software distribution managers
>>
>> How are you planning on handling these more modern languages that
>> manage their own dependencies?
>
> Currently, we import those languages’ dependency trees into the Guix
> dependency tree, and so some additional QA (make sure tests pass,
> provide adequate licensing info, descriptions and synopses, etc.)

OK, I'm not entirely sure what that means for JVM / Maven Central
applications but it sounds like you're doing something sane.

The important thing for JVM applications is that each jar doesn't end up
getting tracked as a separate entity, because that just makes it
infeasible and painful to package anything through the official
channels.

The main problem I want to avoid is the situation where it can take
longer to package a small application than it does to write it.


>> * Docker image
>
> I don’t think there’ll be an “official” presence there, but everyone is
> welcome to do it.

OK, I'll watch out for one appearing.


> Guix’s build daemon uses containers to perform isolated builds:
>
>   http://www.gnu.org/software/guix/manual/html_node/Features.html

Interesting. I wonder if you wouldn't benefit from a docker / drone
network, just as a distribution mechanism for your own build farm. It
would be a shame to expend effort on that since it is somewhat something
of a solved problem (and purely a DevOps matter, not a user concern).


>> * Issue tracker / comm channels
>>
>> Will you be continuing to use debbugs, savannah and mailing lists going
>> forward or would you consider moving to a modern community management
>> system like gitlab?
>
> I hear the appeal of GitLab and the like.  However, as was recently
> discussed on guix-devel, while I think we must find ways to improve our
> workflows (for instance, tracking patches is becoming tricky), I don’t
> see us moving to one of those web-based approaches for several reasons:
>
>   https://lists.gnu.org/archive/html/guix-devel/2015-12/msg00429.html

I've never used GitLab, but I understand that it is free software. The
thread above seems to suggest that it is proprietary.


Perhaps a challenge as large as GUIX itself, but I'd love to see
something from GNU to genuinely compete with GitHub on a technical level
and gitlab seems like the closest thing available in the free software
world.


-- 
Best regards,
Sam

[-- Attachment #2.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 180 bytes --]

Re: some questions about GUIX

[Ricardo Wurmus]

Sam Halliday <sam.halliday@gmail.com> writes:

> Ludovic Courtès <ludo@gnu.org> writes:
>>> * Integration with existing software distribution managers
>>>
>>> How are you planning on handling these more modern languages that
>>> manage their own dependencies?
>>
>> Currently, we import those languages’ dependency trees into the Guix
>> dependency tree, and so some additional QA (make sure tests pass,
>> provide adequate licensing info, descriptions and synopses, etc.)
>
> OK, I'm not entirely sure what that means for JVM / Maven Central
> applications but it sounds like you're doing something sane.
>
> The important thing for JVM applications is that each jar doesn't end up
> getting tracked as a separate entity, because that just makes it
> infeasible and painful to package anything through the official
> channels.

I’m not sure I understand what you mean here.  I have been packaging a
couple of Java things and the reliance on prebuilt jars and Maven causes
quite a few problems.  This is the single most important reason why
there isn’t much of Java to be found in Guix yet.

Building a full non-trivial Java application from source without
resorting to some black box jars along the way is very difficult.  I’m
still working on slowly packaging the dependencies of log4j, one jar at
a time, ... and I even forgot why I’m working on log4j because the
dependency graph for an arbitrary Java package is overwhelmingly large.

(We have a minimal Ant, though, and I just finished abstracting an
ant-build-system, so packaging is already becoming a little easier and
considerably faster.)

We are building library for library as individual packages in Guix.  We
certainly won’t bundle prebuilt jars from Maven if it can be avoided at
all.

> The main problem I want to avoid is the situation where it can take
> longer to package a small application than it does to write it.

This depends a lot on the development culture fostered by the language
community.  For R packages, for example, packaging is almost trivial.
With the CRAN importer I created Guix package expressions from (almost)
all packages on CRAN within minutes.  Now I just have to look through
the stash for packages that I want to submit to Guix upstream, build
them and that’s it.

For Java, however, it is definitely true (for now) that packaging an
application can certainly take longer than writing one from scratch in a
different language.  This is largely due to the fact that Java
programmes are rarely ever built from source (recursively) as Maven
conveniently takes care of downloading binary artifacts.

I suppose we could have a Maven importer in due time that converts the
information contained in pom.xml files to a Guix package expression.  We
won’t have to be using Maven itself.  This would make packaging Java
libraries and applications about as easy as it is now to package R
packages from Bioconductor or CRAN.

>> Guix’s build daemon uses containers to perform isolated builds:
>>
>>   http://www.gnu.org/software/guix/manual/html_node/Features.html
>
> Interesting. I wonder if you wouldn't benefit from a docker / drone
> network, just as a distribution mechanism for your own build farm. It
> would be a shame to expend effort on that since it is somewhat something
> of a solved problem (and purely a DevOps matter, not a user concern).

Work is under way to distribute build artifacts over GNUnet.  Currently
it is already possible to share build results over HTTP.  Ideally,
package building is a distributed effort.  (We aren’t there yet.)

>>> * Issue tracker / comm channels
>>>
>>> Will you be continuing to use debbugs, savannah and mailing lists going
>>> forward or would you consider moving to a modern community management
>>> system like gitlab?
>>
>> I hear the appeal of GitLab and the like.  However, as was recently
>> discussed on guix-devel, while I think we must find ways to improve our
>> workflows (for instance, tracking patches is becoming tricky), I don’t
>> see us moving to one of those web-based approaches for several reasons:
>>
>>   https://lists.gnu.org/archive/html/guix-devel/2015-12/msg00429.html
>
> I've never used GitLab, but I understand that it is free software. The
> thread above seems to suggest that it is proprietary.

There are two variants AFAIU; the hosted GitLab service uses the
proprietary version.

Although I personally started contributing to free software projects
regularly only at the time when Github came around (before that it was
quite awkward for me to send patches via email), I have come to really
appreciate the mail-based workflow we have for GNU projects.

Savannah certainly isn’t pretty (and I only used the web interface once
or twice to get the info I needed to get started), and I find debbugs
somewhat hard to use, but I think using mailing lists for discussing
patches (but not to keep track of them) is vastly superiour to the
workflow imposed by Github and similar systems.

~~ Ricardo

Re: some questions about GUIX

[Leo Famulari]

On Tue, Dec 29, 2015 at 08:11:55PM +0100, Ricardo Wurmus wrote:
> 
> Sam Halliday <sam.halliday@gmail.com> writes:
> 
> > Ludovic Courtès <ludo@gnu.org> writes:
> >>> * Issue tracker / comm channels
> >>>
> >>> Will you be continuing to use debbugs, savannah and mailing lists going
> >>> forward or would you consider moving to a modern community management
> >>> system like gitlab?
> >>
> >> I hear the appeal of GitLab and the like.  However, as was recently
> >> discussed on guix-devel, while I think we must find ways to improve our
> >> workflows (for instance, tracking patches is becoming tricky), I don’t
> >> see us moving to one of those web-based approaches for several reasons:
> >>
> >>   https://lists.gnu.org/archive/html/guix-devel/2015-12/msg00429.html
> >
> > I've never used GitLab, but I understand that it is free software. The
> > thread above seems to suggest that it is proprietary.
> 
> There are two variants AFAIU; the hosted GitLab service uses the
> proprietary version.
> 
> Although I personally started contributing to free software projects
> regularly only at the time when Github came around (before that it was
> quite awkward for me to send patches via email), I have come to really
> appreciate the mail-based workflow we have for GNU projects.
> 
> Savannah certainly isn’t pretty (and I only used the web interface once
> or twice to get the info I needed to get started), and I find debbugs
> somewhat hard to use, but I think using mailing lists for discussing
> patches (but not to keep track of them) is vastly superiour to the
> workflow imposed by Github and similar systems.

I also have come to prefer the email workflow over web-based systems
like Github and Gitlab.

For one thing, I automatically have a complete copy of the project's
development discussions (since I joined) on my workstation, because I do
IMAP sync to have local emails. This means that the "human" part of the
development process is distributed, in addition to the source code part
(with git). Github is useless when I'm offline.

It integrates very simply with git-send-email and git-am (for applying
patches to my work tree). I find it much faster than clicking around
github.com. Sure, it takes a little longer to learn than Github, but so
does writing software versus just using it.

I do like Github et al for sending one-off patches or bug reports to
projects that I am not going to be spending a lot of time on. I would
like to see a system that hooks into the mailing list and provides a
"dual-interface" where both email and webpages are equal citizens, so
that drive-bys have a chance of contributing without signing up for yet
another mailing list (the true backcronym of YAML? ;)

Tracking patches is more cumbersome with the current email system than
Github but, on the other hand, I think that the patch's author should be
paying enough attention to be able to keep track of it themselves [1].
If I have a local branch that isn't merged, I should know why. It's
either incomplete or waiting for review, and if it's been waiting for
review for a while, then I can send a reminder to the list.  And if
somebody else's mail is sitting in my guix-inbox for a while, I know
why. This method works for me; I don't know if it works for other
people.

[1] This first half of these archived LKML messages describe what I'm
talking about:
http://yarchive.net/comp/linux/patches.html

Re: some questions about GUIX

[Ludovic Courtès]

Ricardo Wurmus <rekado@elephly.net> skribis:

> Sam Halliday <sam.halliday@gmail.com> writes:
>
>> Ludovic Courtès <ludo@gnu.org> writes:

[...]

>>> Guix’s build daemon uses containers to perform isolated builds:
>>>
>>>   http://www.gnu.org/software/guix/manual/html_node/Features.html
>>
>> Interesting. I wonder if you wouldn't benefit from a docker / drone
>> network, just as a distribution mechanism for your own build farm. It
>> would be a shame to expend effort on that since it is somewhat something
>> of a solved problem (and purely a DevOps matter, not a user concern).
>
> Work is under way to distribute build artifacts over GNUnet.  Currently
> it is already possible to share build results over HTTP.  Ideally,
> package building is a distributed effort.  (We aren’t there yet.)

There’s also the offloading mechanism, which may be closer to what Sam
was mentioning:

  https://www.gnu.org/software/guix/manual/html_node/Daemon-Offload-Setup.html

>>>> * Issue tracker / comm channels
>>>>
>>>> Will you be continuing to use debbugs, savannah and mailing lists going
>>>> forward or would you consider moving to a modern community management
>>>> system like gitlab?
>>>
>>> I hear the appeal of GitLab and the like.  However, as was recently
>>> discussed on guix-devel, while I think we must find ways to improve our
>>> workflows (for instance, tracking patches is becoming tricky), I don’t
>>> see us moving to one of those web-based approaches for several reasons:
>>>
>>>   https://lists.gnu.org/archive/html/guix-devel/2015-12/msg00429.html
>>
>> I've never used GitLab, but I understand that it is free software. The
>> thread above seems to suggest that it is proprietary.
>
> There are two variants AFAIU; the hosted GitLab service uses the
> proprietary version.

Right, gitlab.com runs “GitLab EE”, which is proprietary.

Thanks,
Ludo’.

Re: some questions about GUIX

[Sam Halliday]

[-- Attachment #1: Type: text/plain, Size: 17 bytes --]

Thanks Ricardo,


[-- Attachment #2.1: Type: text/plain, Size: 2723 bytes --]

Ricardo Wurmus writes:
> I’m not sure I understand what you mean here. I have been packaging a
> couple of Java things and the reliance on prebuilt jars and Maven
> causes quite a few problems. This is the single most important reason
> why there isn’t much of Java to be found in Guix yet.

If the GUIX-SD goal is to build every package from source, then I can
see why you're doing it this way. It is possible to achieve this noble
goal, but you are embarking on an absolutely *gigantic* mission.


> Building a full non-trivial Java application from source without
> resorting to some black box jars along the way is very difficult. I’m
> still working on slowly packaging the dependencies of log4j, one jar
> at a time, ... and I even forgot why I’m working on log4j because the
> dependency graph for an arbitrary Java package is overwhelmingly
> large.

I'm not sure I would refer to Maven Central binaries as "black box". The
jars on Maven Central are digitally signed by the distributors and the
source jars are available beside them, with meta data such as license
and homepages available in the pom file.

It should not be difficult to set up Maven and Ivy to only use a
GUIX-hosted repository (many organisations do this), or a local
repository, but it will involve an incredible amount of effort to
actually *rebuild* everything, as you are discovering.

What practical benefit does rebuilding on the GUIX farm actually bring?
Is there a claim that the binary build can be traced directly to a
"trusted" machine and source code?


In the Scala community, there is an attempt to rebuild a range of free
software projects as part of the continuous integration of the scala
compiler itself:

  https://github.com/scala/community-builds

It may be worthwhile collaborating with the authors in order to avoid
duplicating efforts.


> We are building library for library as individual packages in Guix. We
> certainly won’t bundle prebuilt jars from Maven if it can be avoided
> at all.

As a Java / Scala developer, I consider the maven / ivy dependency
tracking to be the environment where I define the exact and definitive
versions of dependencies that I require and I would not be happy
depending on the GUIX-SD jars. Although, I would not object to using
GUIX-SD jars as a user, if the installation was easier than self-managed
and I did not experience and bugs as a result of the re-packaging.

I would be very interested if you had a way of integrating with Maven
Central artefacts in order to install applications, as it would make it
significantly easier for me, as a developer, to produce packages for
GUIX-SD.


-- 
Best regards,
Sam

[-- Attachment #2.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 180 bytes --]

Re: some questions about GUIX

[Sam Halliday]

[-- Attachment #1: Type: text/plain, Size: 77 bytes --]

Hi Ricardo,

I have a few more questions about your proposed jar packaging.


[-- Attachment #2.1: Type: text/plain, Size: 1055 bytes --]

Ricardo Wurmus writes:
> We are building library for library as individual packages in Guix. We
> certainly won’t bundle prebuilt jars from Maven if it can be avoided
> at all.

Does this mean that you have a GUIX package for every jar? If so, can
you have multiple versions of the same jar installed at the same time?
Support for multiple versions of a library will be necessary as it is
not always a simple case of bumping the version to use a library: many
libraries introduce breaking changes at both source and binary level.

Will you be using the same version names as the official upstream
binaries? I strongly recommend against doing this. The convention in
corporate environments is that rebuilds of jars incur a postfix to their
version. E.g. a rebuild of guava 18.0 (even with no changes to the
sources) would be 18.0-guix1. Of course, there is no way for you to know
that jars are not being loaded by name at runtime through the
classloader, so you introduce further opportunity for bugs here.


-- 
Best regards,
Sam

[-- Attachment #2.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 180 bytes --]

Re: some questions about GUIX

[Leo Famulari]

On Wed, Dec 30, 2015 at 01:36:12PM +0000, Sam Halliday wrote:
> Hi Ricardo,
> 
> I have a few more questions about your proposed jar packaging.
> 

> Ricardo Wurmus writes:
> > We are building library for library as individual packages in Guix. We
> > certainly won’t bundle prebuilt jars from Maven if it can be avoided
> > at all.
> 
> Does this mean that you have a GUIX package for every jar? If so, can
> you have multiple versions of the same jar installed at the same time?
> Support for multiple versions of a library will be necessary as it is
> not always a simple case of bumping the version to use a library: many
> libraries introduce breaking changes at both source and binary level.

Yes, we can have multiple versions of every library installed at the
same time, and users of those libraries are linked to a specific library
at build time.

Our goal is to build everything from source. We even build documentation
such as manpages and PDFs from source wherever possible.

And if upstream software distributors do not clearly specify which
versions of dependent libraries are compatible, instead choosing to
bundle binary artifacts of those libraries without providing
instructions for rebuilding from source, I'd call that a security risk
and affront to users' freedom.

> 
> Will you be using the same version names as the official upstream
> binaries? I strongly recommend against doing this. The convention in
> corporate environments is that rebuilds of jars incur a postfix to their
> version. E.g. a rebuild of guava 18.0 (even with no changes to the
> sources) would be 18.0-guix1. Of course, there is no way for you to know
> that jars are not being loaded by name at runtime through the
> classloader, so you introduce further opportunity for bugs here.

I'm not a Java programmer so I can't get very deep into the specifics. I
have tried to package some Java software, though.

I can say that we are actively working towards being able to
reproducibly build all the software we package, and so "postfixing" the
version for a new build of the same source is unnecessary once we have
established that a particular code base can be built reproducibly. In
fact, in Guix, trying to rebuild the same source code will only start an
actual compilation if the GNU store [0] does not include the memoized
output of the last build.

Postfixing the binary name sounds like a last-ditch attempt to keep
track of binary artifacts that have no clear provenance, using build
systems and distribution methods that do not empower [1] downstream
users to build software from source.

If you find Guix interesting but somewhat confusing, I recommend you
spend some time learning more about it, and please keep asking
questions! You may find Guix a breath of fresh air after wondering if
"guava-18.0-guix1" included that one special platform-specific bug fix
or tweak, or if it was "guava-18.0-guix2" ;)

Several of us read this blog post [2] on the state of Java packaging
recently. It echoed my experiences trying to package Java software and
it clearly explains the potential negative consequences of the current
methods, and it says it all better than I can.

[0]
https://www.gnu.org/software/guix/manual/html_node/The-Store.html

[1]
In some cases it is so difficult that it may as well be closed source.
Free software (or open-source, as the license may be) in name only, if
you ask me.

[2]
http://www.vitavonni.de/blog/201504/2015042601-big-data-toolchains-are-a-security-risk.html

> 
> 
> -- 
> Best regards,
> Sam

Re: some questions about GUIX

[Leo Famulari]

On Wed, Dec 30, 2015 at 09:02:53PM -0500, Leo Famulari wrote:
> On Wed, Dec 30, 2015 at 01:36:12PM +0000, Sam Halliday wrote:
> > Hi Ricardo,
> > 
> > I have a few more questions about your proposed jar packaging.
> > 
> 
> > Ricardo Wurmus writes:
> > > We are building library for library as individual packages in Guix. We
> > > certainly won’t bundle prebuilt jars from Maven if it can be avoided
> > > at all.
> > 
> > Does this mean that you have a GUIX package for every jar? If so, can
> > you have multiple versions of the same jar installed at the same time?
> > Support for multiple versions of a library will be necessary as it is
> > not always a simple case of bumping the version to use a library: many
> > libraries introduce breaking changes at both source and binary level.
> 
> Yes, we can have multiple versions of every library installed at the
> same time, and users of those libraries are linked to a specific library
> at build time.
> 
> Our goal is to build everything from source. We even build documentation
> such as manpages and PDFs from source wherever possible.
> 
> And if upstream software distributors do not clearly specify which
> versions of dependent libraries are compatible, instead choosing to
> bundle binary artifacts of those libraries without providing
> instructions for rebuilding from source, I'd call that a security risk
> and affront to users' freedom.
> 
> > 
> > Will you be using the same version names as the official upstream
> > binaries? I strongly recommend against doing this. The convention in
> > corporate environments is that rebuilds of jars incur a postfix to their
> > version. E.g. a rebuild of guava 18.0 (even with no changes to the
> > sources) would be 18.0-guix1. Of course, there is no way for you to know
> > that jars are not being loaded by name at runtime through the
> > classloader, so you introduce further opportunity for bugs here.
> 
> I'm not a Java programmer so I can't get very deep into the specifics. I
> have tried to package some Java software, though.
> 
> I can say that we are actively working towards being able to
> reproducibly build all the software we package, and so "postfixing" the
> version for a new build of the same source is unnecessary once we have
> established that a particular code base can be built reproducibly. In
> fact, in Guix, trying to rebuild the same source code will only start an
> actual compilation if the GNU store [0] does not include the memoized
> output of the last build.
> 
> Postfixing the binary name sounds like a last-ditch attempt to keep
> track of binary artifacts that have no clear provenance, using build
> systems and distribution methods that do not empower [1] downstream
> users to build software from source.
> 
> If you find Guix interesting but somewhat confusing, I recommend you
> spend some time learning more about it, and please keep asking
> questions! You may find Guix a breath of fresh air after wondering if
> "guava-18.0-guix1" included that one special platform-specific bug fix
> or tweak, or if it was "guava-18.0-guix2" ;)

I have to make another point about this. I work with a lot of people who
are not computer programmers. Most of their work is collaborating on
"office" type documents and slideshows. They are constantly wondering if
the right document is "reportDONE", "reportDONEFINAL", "report-FINAL2", etc.
It makes me so sad, knowing the tools we have for source code management
like git and mercurial, and how these tools have empowered us to work so
much more effectively and efficiently than before.

Hopefully you see the analogue ;) I promise you we aren't totally crazy.

> 
> Several of us read this blog post [2] on the state of Java packaging
> recently. It echoed my experiences trying to package Java software and
> it clearly explains the potential negative consequences of the current
> methods, and it says it all better than I can.
> 
> [0]
> https://www.gnu.org/software/guix/manual/html_node/The-Store.html
> 
> [1]
> In some cases it is so difficult that it may as well be closed source.
> Free software (or open-source, as the license may be) in name only, if
> you ask me.
> 
> [2]
> http://www.vitavonni.de/blog/201504/2015042601-big-data-toolchains-are-a-security-risk.html
> 
> > 
> > 
> > -- 
> > Best regards,
> > Sam
> 
> 
> 
> 

Re: some questions about GUIX

[Ricardo Wurmus]

Sam Halliday <sam.halliday@gmail.com> writes:

> Does this mean that you have a GUIX package for every jar?

Some packages may provide more than one jar archive.  This depends on
what the build system prescribes.  For every *project* there will be at
least one jar, though.

> If so, can
> you have multiple versions of the same jar installed at the same time?

Yes!  This is a core feature enabled by functional package management.
We treat packages as if they were pure functions in that the exact same
set of inputs (source tarballs, compilers, libraries, and their own
inputs, and the inputs of those inputs, etc...) produces the exact same
output.  Instead of computing the output from scratch each time we want
to use the value (e.g. as an input to another package or in a profile)
we can cache or memoise it.  We do this by storing the unique output
artifact for a given bag of inputs in /gnu/store; outputs are prefixed
with a unique hash that is computed from all inputs.

Installing a package then is little more than the act of linking
individual items from the store into a profile.  We can store any
package output in /gnu/store because they all have their very own
directory; they don’t overlap.  Thus we can install/link different
versions into different profiles.
 
> Support for multiple versions of a library will be necessary as it is
> not always a simple case of bumping the version to use a library: many
> libraries introduce breaking changes at both source and binary level.

Correct.  We don’t have a problem with this as we never install anything
into a “global” namespace (such as /usr, /lib, or whatever).

> Will you be using the same version names as the official upstream
> binaries? I strongly recommend against doing this. The convention in
> corporate environments is that rebuilds of jars incur a postfix to their
> version. E.g. a rebuild of guava 18.0 (even with no changes to the
> sources) would be 18.0-guix1. Of course, there is no way for you to know
> that jars are not being loaded by name at runtime through the
> classloader, so you introduce further opportunity for bugs here.

FWIW Debian builds Java libraries from sources, too, and they do not
rename the resulting jars.

There is really no need to do that in Guix anyway, because installing
these jars is not going to affect global state — they are not installed
in well-known global directories, so they cannot be “picked up” by
accident.

There is, of course, one potential issue with building Java stuff on
Guix that I haven’t assessed yet.  I don’t yet know if there’s something
in the Java world that’s similar to the RUNPATH feature in native
applications, or if visibility of jars is determined exclusively by the
clunky CLASSPATH environment variable instead.  If there is no way to
“link” jars explicitly by their full path, then dependencies need to be
propagated and applications may need to be wrapped in an appropriate
CLASSPATH, which is somewhat inelegant, but not a real problem.  (This
is done for Python, R, Perl, and Ruby applications in Guix already,
because these languages control module visibility primarily through
environment variables.)

Re: some questions about GUIX

[Ricardo Wurmus]

Sam Halliday <sam.halliday@gmail.com> writes:

> Thanks Ricardo,
>
> Ricardo Wurmus writes:
>> I’m not sure I understand what you mean here. I have been packaging a
>> couple of Java things and the reliance on prebuilt jars and Maven
>> causes quite a few problems. This is the single most important reason
>> why there isn’t much of Java to be found in Guix yet.
>
> If the GUIX-SD goal is to build every package from source, then I can
> see why you're doing it this way. It is possible to achieve this noble
> goal, but you are embarking on an absolutely *gigantic* mission.

It can be quite difficult to do this because many people seem to think
that it’s good enough to be able to build something with a binary
version of the very same thing.

For example: gradle needs gradle and groovy to be built, groovy needs
gradle and groovy to be built.  So you are encouraged to take their
binaries and use those to build binaries — that you already have, so
what is the point of building new binaries again...?

Whenever we have to bootstrap language environments we try hard to find
ways that do not require trust to be placed in binary blobs.

>> Building a full non-trivial Java application from source without
>> resorting to some black box jars along the way is very difficult. I’m
>> still working on slowly packaging the dependencies of log4j, one jar
>> at a time, ... and I even forgot why I’m working on log4j because the
>> dependency graph for an arbitrary Java package is overwhelmingly
>> large.
>
> I'm not sure I would refer to Maven Central binaries as "black box". The
> jars on Maven Central are digitally signed by the distributors and the
> source jars are available beside them, with meta data such as license
> and homepages available in the pom file.

We have the sources, so we should be able to build the binaries.  If we
cannot do that then the sources are really just unreliable/untrustworthy
documentation.  There is no clear relationship between published sources
and published binaries.  Users have no way of making this connection.
They have to trust that these binaries correspond to the published
sources.  We shouldn’t *have* to trust.

> It should not be difficult to set up Maven and Ivy to only use a
> GUIX-hosted repository (many organisations do this), or a local
> repository, but it will involve an incredible amount of effort to
> actually *rebuild* everything, as you are discovering.

You can continue to use Maven or any other repository.  But for Guix
packages we will just download binary artifacts.  We don’t do this for
any other language either (e.g. we don’t just download Ruby gems from
rubygems.org).  If people really want to use those artifacts they can
just continue to use “mvn” or “gem”.  Guix won’t stop them from doing
this.  But like any other GNU distribution the packages we provide will
not use pre-built binary artifacts.

~~ Ricardo

Re: some questions about GUIX

[Sam Halliday]

[-- Attachment #1: Type: text/plain, Size: 4068 bytes --]

Leo Famulari writes:
> Our goal is to build everything from source.

OK, that helps me to understand why you are doing things the way you are
doing it.

Indeed, this means that your efforts are incompatible with my original
hope: that guix would make packaging of Java applications significantly
easier.

As a Java / Scala developer, I avoid all repackaged jars unless they
contain a bug or security fix. I don't see why I should trust a package
manager (and their machines), who are self-confessed non-experts, more
than the original author of that library, who has GPG signed their
artefacts and made their source code available.


> I'm not a Java programmer so I can't get very deep into the specifics. I
> have tried to package some Java software, though.

In that case, I strongly encourage you and other GUIX contributors to
halt all Java packaging efforts immediately.

Please dedicate time to scope out what you're planning to do: have
discussions with experienced Java developers. Find a common ground so
that the jar artefact packaging is compatible with developer needs.

It is most important that this is done properly from the beginning,
because the longer you go down this route, the more time you will have
ultimately wasted, and the harder it will be to dig yourself out of it.

From what I've seen, you're solving a problem at the expense of
excluding everybody who would use it.

One thing I can see that would be compatible would be if you rebuilt
everything as Maven artefacts, using a postfix to the versions such as
-guix, and when packages are installed they get placed into exactly the
location one would expect a maven artefact to go.

This would allow the "GUIX build" world to co-exist with the developer's
"Maven Central" world. Build tool plugins may then arise that would
transparently swap between the two at the developer's request, thereby
greatly simplifying your repackaging effort.


> Postfixing the binary name sounds like a last-ditch attempt to keep
> track of binary artifacts that have no clear provenance

It would be GUIX demonstrating that the artefacts are not the same
artefacts that Java developers expect to receive from Maven Central,
(GPG signed by the author). It doesn't hurt you to do it this way, but
it will be appreciated by all Java developers.


> Several of us read this blog post [2] on the state of Java packaging
> recently. It echoed my experiences trying to package Java software and
> it clearly explains the potential negative consequences of the current
> methods, and it says it all better than I can.
>
> [2]
> http://www.vitavonni.de/blog/201504/2015042601-big-data-toolchains-are-a-security-risk.html

I agree that Maven Central needs a distributed verification mechanism.
It was designed for an era where trusting the developer's artefacts
(through GPG) was enough. (The article appears to be confused on this
matter, and fails to recognise that Maven Central artefacts are, in
fact, GPG signed, checksummed and transmitted securely over HTTPS)



I am unsubscribing from this mailing list now as I only came on to ask
these questions, and I feel I have received an answer already, thank you
all for answering!

I will leave by encouraging GUIX to move towards a more user-focused
infrastructure than mailing lists, as it is extremely difficult for
people like myself who want to dip in and out to get involved.
Therefore, I do not feel it is the right time for me to move to GUIX
just yet. A clone of github, meeting all the FSF's ethical hosting
guidelines, is absolutely essential, and I hope it is a strategic goal
of GNU.

If GUIX are planning to dedicate a significant amount of effort to
rebuilding the entire Java universe from scratch, I would prefer that
you not support Java at all (except by providing OpenJDK builds).
Instead, put that effort into building a github-like community portal.
Once GUIX proves the viability of reproducible builds, Java will catch
up.


-- 
Best regards,
Sam

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 180 bytes --]

Re: some questions about GUIX

[Ludovic Courtès]

Sam Halliday <sam.halliday@gmail.com> skribis:

> As a Java / Scala developer, I avoid all repackaged jars unless they
> contain a bug or security fix. I don't see why I should trust a package
> manager (and their machines), who are self-confessed non-experts, more
> than the original author of that library, who has GPG signed their
> artefacts and made their source code available.

Jars are binaries.  What Guix seeks to offer is a way for users to make
sure that the binaries they run correspond to the source code they think
it corresponds to.  See:

  https://savannah.gnu.org/forum/forum.php?forum_id=8407

The practice of installing binaries built by a third-party without
having any way to verify that those binaries are authentic is inherently
unsafe.  It’s also a serious attack on user freedom if users aren’t able
to rebuild software by themselves.

We set out to address that.  The cost may be, indeed, that it’ll be some
time before a large set of Java/Scala packages can be imported in Guix.

Thanks for your feedback,
Ludo’.