Docker and singularity containers

Docker and singularity containers

[Pjotr Prins]

I know this is not a priority right now for maintainers, but I think
it would be really useful if we provide ready-made containers for
certain packages. This will allow people who have the infrastructure
to run containers to start using reproducible Guix builds. With that,
I think there will be an increased interest in our technology.

Maybe it is something to discuss at FOSDEM, but I wonder what it would
take to have the build farm automatically provide Docker containers
for marked packages and offer them through the web interface.

Unlike substitutes we only need to hold the last version. People can
provide their own infrastructure to retain/share them.

Pj.

Re: Docker and singularity containers

[Ludovic Courtès]

Hello,

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> I know this is not a priority right now for maintainers, but I think
> it would be really useful if we provide ready-made containers for
> certain packages. This will allow people who have the infrastructure
> to run containers to start using reproducible Guix builds. With that,
> I think there will be an increased interest in our technology.

FWIW at work people a colleague of mine got into Guix with the goal of
being able to run their software on a different cluster that has
Singularity but not Guix.

People interested in this might use Guix on their laptop just to build
Singularity images and view it as a tool to build bundles (which is of
course unsatisfactory but it’s a way to get started!).

> Maybe it is something to discuss at FOSDEM, but I wonder what it would
> take to have the build farm automatically provide Docker containers
> for marked packages and offer them through the web interface.

A while back we fantasized about the possibility of having a web service
that would produce “packs” of Guix packages on demand.  That may allow
us to be #1 on HN for a couple of hours ;-), but it would also be quite
resource-hungry (and the service itself needs to be written to begin
with.)  Such a service would have to use its own resources IMO, not
those of the build farm, which is already busy enough.

Ludo’.

Re: Docker and singularity containers

[Ricardo Wurmus]

Ludovic Courtès <ludovic.courtes@inria.fr> writes:

> A while back we fantasized about the possibility of having a web service
> that would produce “packs” of Guix packages on demand.  That may allow
> us to be #1 on HN for a couple of hours ;-), but it would also be quite
> resource-hungry (and the service itself needs to be written to begin
> with.)  Such a service would have to use its own resources IMO, not
> those of the build farm, which is already busy enough.

My research group was interested in a service just like that, actually.
After having connected the external storage directly to the
berlin.guixsd.org we have a new server that could be used solely for
this purpose.

But even this may not be enough for an on-demand service.  It might,
however, be enough for a service that regularly builds packs for certain
common environments from packages the build farm has already built.

(This would be done with the squashfs backend, which is way faster than
the Docker backend.)

--
Ricardo

Re: Docker and singularity containers

[Pjotr Prins]

On Wed, Sep 12, 2018 at 07:43:42PM +0200, Ricardo Wurmus wrote:
> 
> Ludovic Courtès <ludovic.courtes@inria.fr> writes:
> 
> > A while back we fantasized about the possibility of having a web service
> > that would produce “packs” of Guix packages on demand.  That may allow
> > us to be #1 on HN for a couple of hours ;-), but it would also be quite
> > resource-hungry (and the service itself needs to be written to begin
> > with.)  Such a service would have to use its own resources IMO, not
> > those of the build farm, which is already busy enough.
> 
> My research group was interested in a service just like that, actually.
> After having connected the external storage directly to the
> berlin.guixsd.org we have a new server that could be used solely for
> this purpose.

Demand there is for sure! And these infrastructures are only
expanding.

> But even this may not be enough for an on-demand service.  It might,
> however, be enough for a service that regularly builds packs for certain
> common environments from packages the build farm has already built.
> 
> (This would be done with the squashfs backend, which is way faster than
> the Docker backend.)

Also my original comment said to have a limited number of packages to
expose. I mean command line tools make sense and dedicated
environments. But we don't need an image for every R, Ruby and Python
package.

We should just compile a meta list of those. And if someone requests
one we add it. 


Pj.

Re: Docker and singularity containers

[Ricardo Wurmus]

Hi Pjotr,

>> But even this may not be enough for an on-demand service.  It might,
>> however, be enough for a service that regularly builds packs for certain
>> common environments from packages the build farm has already built.
>>
>> (This would be done with the squashfs backend, which is way faster than
>> the Docker backend.)
>
> Also my original comment said to have a limited number of packages to
> expose. I mean command line tools make sense and dedicated
> environments. But we don't need an image for every R, Ruby and Python
> package.

Yes, I agree, though it may be useful to have an image containing a
comprehensive environment for bioinformatics analysis with R, containing
commonly used bioconductor packages; or an image with the full
scientific Python stack.

> We should just compile a meta list of those. And if someone requests
> one we add it.

Right.  In the meantime we may want to discuss how to implement this.
Could it just be done as a specification for the Guix CI?  Since this
should not be stealing resources from the build farm and thus should run
on a separate system, how can we make sure that it only builds packs
from packages for which we already have substitutes on the build farm?

I would gladly host a prototype of a system like this.

--
Ricardo

Re: Docker and singularity containers

[Pjotr Prins]

On Wed, Sep 12, 2018 at 10:59:33PM +0200, Ricardo Wurmus wrote:
> 
> Hi Pjotr,
> 
> >> But even this may not be enough for an on-demand service.  It might,
> >> however, be enough for a service that regularly builds packs for certain
> >> common environments from packages the build farm has already built.
> >>
> >> (This would be done with the squashfs backend, which is way faster than
> >> the Docker backend.)
> >
> > Also my original comment said to have a limited number of packages to
> > expose. I mean command line tools make sense and dedicated
> > environments. But we don't need an image for every R, Ruby and Python
> > package.

> Yes, I agree, though it may be useful to have an image containing a
> comprehensive environment for bioinformatics analysis with R, containing
> commonly used bioconductor packages; or an image with the full
> scientific Python stack.

An R.sumo would be a dream. That is coolness in the extreme. Same for
Python and Ruby.

BTW I notice Rstudio is missing in Guix. Any reason for that? 

We may be going down the Jupyter route, so it may not matter that much.

> > We should just compile a meta list of those. And if someone requests
> > one we add it.
> 
> Right.  In the meantime we may want to discuss how to implement this.
> Could it just be done as a specification for the Guix CI?  Since this
> should not be stealing resources from the build farm and thus should run
> on a separate system, how can we make sure that it only builds packs
> from packages for which we already have substitutes on the build farm?
> 
> I would gladly host a prototype of a system like this.

Excellent. We may also be able to free up resources. I'll follow up on
this.

Pj.

Re: Docker and singularity containers

[Ludovic Courtès]

Hello!

Ricardo Wurmus <rekado@elephly.net> skribis:

> Right.  In the meantime we may want to discuss how to implement this.
> Could it just be done as a specification for the Guix CI?  Since this
> should not be stealing resources from the build farm and thus should run
> on a separate system, how can we make sure that it only builds packs
> from packages for which we already have substitutes on the build farm?

This could be a regular Guix CI spec where each job would have the
result of ‘squashfs-image’ as its derivation.

That machine could use berlin as its substitute server.  In the
(hopefully unlikely) case that a substitute is missing for one of the
packages of interests, it would build it, but I don’t think that’s much
of a problem.

Sounds like a good way to get started with this!

Thanks,
Ludo’.

Re: Docker and singularity containers

[Ricardo Wurmus]

Pjotr Prins <pjotr.public12@thebird.nl> writes:

> BTW I notice Rstudio is missing in Guix. Any reason for that?

It’s here:

    https://github.com/BIMSBbioinfo/guix-bimsb/blob/master/bimsb/packages/staging.scm#L95

Problems are the use of GWT, which I haven’t been able to build from
source yet.

Rstudio Desktop is here (but the build is currently broken):

    https://github.com/BIMSBbioinfo/guix-bimsb/blob/master/bimsb/packages/staging.scm#L250

I don’t know if Rstudio Desktop actually needs GWT.

--
Ricardo

Re: Docker and singularity containers

[zimoun]

Dear,

I am interested by Docker images for specific tasks (scientific
python, R.sumo, etc.).
Around me, people are pulling a debian Docker image then feed it with
some specific dockerfile to produce a R environment ready to use.
And I trying to do that for some bioconductor packages about Flow
Cytometry data analysis.

Well, why not push to DockerHub time to time.
I agree that will attract people if Guix provide an official docker
image reachable from DockerHub.
For example, there is an official image of neuroscience Debian packages:
https://hub.docker.com/_/neurodebian

When searching about Guix, it is not so meaningful:
https://hub.docker.com/search?q=guix&type=image

Therefore:
 - which tasks and which packages?
 - how?
even if the how is already discussed.


Could we discuss that at the before-FOSDEM days.


What do you think?


All the best,
simon

Re: Docker and singularity containers

[Ricardo Wurmus]

Hi simon,

> Well, why not push to DockerHub time to time.
> I agree that will attract people if Guix provide an official docker
> image reachable from DockerHub.
[…]
> Therefore:
>  - which tasks and which packages?

I don’t know.  We could have one with R + all CRAN and bioconductor
packages that we currently offer.

>  - how?

That’s easier: with “guix pack”.  We would create a Cuirass job to build
a pack regularly.  We’d need to add support for hooks to publish the
generated pack on DockerHub, though we could just as well host them
ourselves.

Would you like to give this a try?  A good first step is to list the
steps that need to be performed.  A first implementation might even be
done with a cron job.  I could add this to berlin.guixsd.org when you’re
ready.

--
Ricardo

Re: Docker and singularity containers

[Mike Gerwitz]

[-- Attachment #1: Type: text/plain, Size: 850 bytes --]

On Sat, Jan 05, 2019 at 17:14:31 +0100, Ricardo Wurmus wrote:
> That’s easier: with “guix pack”.  We would create a Cuirass job to build
> a pack regularly.  We’d need to add support for hooks to publish the
> generated pack on DockerHub, though we could just as well host them
> ourselves.

Last I checked, an account couldn't be registered on DockerHub without
running non-free JS.  Here's a thread on emacs-devel where I raised this
issue:

  https://lists.gnu.org/archive/html/emacs-devel/2017-02/msg00114.html

But I haven't verified this since.  After an account is registered,
though, their APIs can be used without having to use DockerHub's web
interface.

-- 
Mike Gerwitz
Free Software Hacker+Activist | GNU Maintainer & Volunteer
GPG: D6E9 B930 028A 6C38 F43B  2388 FEF6 3574 5E6F 6D05
https://mikegerwitz.com

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: Docker and singularity containers

[zimoun]

Dear Mike,

Thank you to raise this concern here.
I was aware of this thread and I do not fully agree with the arguments. :-)
I am doing 3 comments.


First, from my point of view, we need to distinguish between the
"puller" and the "pusher".
And correct me if I am wrong, but one does not need an account to pull
a Docker from DockerHub. Therefore, I do not see the issue from the
puller side.
Because if we apply the argument, do the GPL licensed softwares on
GitHub respect the freedom of the user?

The issue is about the "pusher" i.e. the GNU Guix project. And yes,
the GNU Guix has to accept to run non-free softwares to be able to
push on DockerHub. Is it acceptable?


Hence, my second comment is about the _how_ to distribute. Currently,
there is no free alternative to publish Docker image; even if docker
provides a mechanism to pull from elsewhere than DockerHub.
It is an issue about money and man power. It will be a pity to not
spread enough free software political ideas because the movement lacks
resources. And it is not about be hypocritical, I guess.

If I may, I quote the paper from the Guix maintainers---correct me if
I am wrong.
https://arxiv.org/abs/1506.02822 last paragraph from section 5
<<
Proprietary software.
GNU Guix does not provide proprietary software packages.
Unfortunately, proprietary software is still relatively common in HPC,
be it linear algebra libraries or GPU support. Yet, we see it as a
strength more than a limitation. Often, these “black boxes” inherently
limit reproducibility—how is one going to reproduce a software
environment without permission to run the software in the first place?
What if the software depends on the ability to “call home” to function
at all? More importantly, we view reproducible software environments
and reproducible science as a tool towards improved and shared
knowledge; developers who deny the freedom to study and modify their
code work against this goal.
>>>

Here, my personal opinion. Today, people think that the Science crisis
about reproducibility will be tackled by Docker and containers. On one
hand, I am here because I think it is wrong and it is not the path to
go. On the other hand, I need to pragmatic: people in labs have built
infrastructures using Docker or equivalent; they wont be convinced
easily to switch and so I think I want to ease the switch in playing
directly in their ground.


Last, I do not understand how to apply the argument against pushing to
DockerHub to the Windows port of Emacs.
Somehow, GNU has to run non-free softwares to provide this port. At
least to launch some tests.

My personal opinion is that it is good. Because this spreads the
message about freedom, this helps people to be aware of the movement,
this should be a first step in liberating users.


Thank you if you have comments and/or if you have arguments that
explain me where it is wrong.


All the best,
simon

Re: Docker and singularity containers

[swedebugia]

[-- Attachment #1.1: Type: text/html, Size: 4717 bytes --]

[-- Attachment #1.2: Type: text/plain, Size: 3955 bytes --]

zimoun <zimon.toutoune@gmail.com> skrev: (6 januari 2019 13:09:33 CET)
>Dear Mike,
>
>Thank you to raise this concern here.
>I was aware of this thread and I do not fully agree with the arguments.
>:-)
>I am doing 3 comments.
>
>
>First, from my point of view, we need to distinguish between the
>"puller" and the "pusher".
>And correct me if I am wrong, but one does not need an account to pull
>a Docker from DockerHub. Therefore, I do not see the issue from the
>puller side.
>Because if we apply the argument, do the GPL licensed softwares on
>GitHub respect the freedom of the user?
>
>The issue is about the "pusher" i.e. the GNU Guix project. And yes,
>the GNU Guix has to accept to run non-free softwares to be able to
>push on DockerHub. Is it acceptable?
>
>
>Hence, my second comment is about the _how_ to distribute. Currently,
>there is no free alternative to publish Docker image; even if docker
>provides a mechanism to pull from elsewhere than DockerHub.
>It is an issue about money and man power. It will be a pity to not
>spread enough free software political ideas because the movement lacks
>resources. And it is not about be hypocritical, I guess.
>
>If I may, I quote the paper from the Guix maintainers---correct me if
>I am wrong.
>https://arxiv.org/abs/1506.02822 last paragraph from section 5
><<
>Proprietary software.
>GNU Guix does not provide proprietary software packages.
>Unfortunately, proprietary software is still relatively common in HPC,
>be it linear algebra libraries or GPU support. Yet, we see it as a
>strength more than a limitation. Often, these “black boxes” inherently
>limit reproducibility—how is one going to reproduce a software
>environment without permission to run the software in the first place?
>What if the software depends on the ability to “call home” to function
>at all? More importantly, we view reproducible software environments
>and reproducible science as a tool towards improved and shared
>knowledge; developers who deny the freedom to study and modify their
>code work against this goal.
>>>>
>
>Here, my personal opinion. Today, people think that the Science crisis
>about reproducibility will be tackled by Docker and containers. On one
>hand, I am here because I think it is wrong and it is not the path to
>go. On the other hand, I need to pragmatic: people in labs have built
>infrastructures using Docker or equivalent; they wont be convinced
>easily to switch and so I think I want to ease the switch in playing
>directly in their ground.
>
>
>Last, I do not understand how to apply the argument against pushing to
>DockerHub to the Windows port of Emacs.
>Somehow, GNU has to run non-free softwares to provide this port. At
>least to launch some tests.
>
>My personal opinion is that it is good. Because this spreads the
>message about freedom, this helps people to be aware of the movement,
>this should be a first step in liberating users.
>
>
>Thank you if you have comments and/or if you have arguments that
>explain me where it is wrong.
>
>
>All the best,
>simon

+1 
I never used docker. 

It would probably have helped me when I set up my now broken accounting software (odoo 9) before I knew about dolibarr and before guix had containers. (It broke some time ago when I upgraded arch :/ Shared libraries are a stability risk and pain in the long run)

According to the Swedish state I have to make sure my accounting software is working 7 years after the end of each year.
This is to my surprise really hard with software like odoo. I would have upgraded long ago would it have been straightforward by the developers (odooization as i call it is a problem, they make money on making odoo a fast changing maze that has really poor documentation, cryptic errors and no free upgrade scripts necessitating a whole industry of paid supporters.)
-- 
Sent from my p≡p for Android.

[-- Attachment #2: pEpkey.asc --]
[-- Type: application/pgp-keys, Size: 3825 bytes --]

Re: Docker and singularity containers

[swedebugia]

[-- Attachment #1.1: Type: text/html, Size: 4101 bytes --]

[-- Attachment #1.2: Type: text/plain, Size: 3359 bytes --]

zimoun <zimon.toutoune@gmail.com> skrev: (6 januari 2019 13:09:33 CET)
>Dear Mike,
>
>Thank you to raise this concern here.
>I was aware of this thread and I do not fully agree with the arguments.
>:-)
>I am doing 3 comments.
>
>
>First, from my point of view, we need to distinguish between the
>"puller" and the "pusher".
>And correct me if I am wrong, but one does not need an account to pull
>a Docker from DockerHub. Therefore, I do not see the issue from the
>puller side.
>Because if we apply the argument, do the GPL licensed softwares on
>GitHub respect the freedom of the user?
>
>The issue is about the "pusher" i.e. the GNU Guix project. And yes,
>the GNU Guix has to accept to run non-free softwares to be able to
>push on DockerHub. Is it acceptable?
>
>
>Hence, my second comment is about the _how_ to distribute. Currently,
>there is no free alternative to publish Docker image; even if docker
>provides a mechanism to pull from elsewhere than DockerHub.
>It is an issue about money and man power. It will be a pity to not
>spread enough free software political ideas because the movement lacks
>resources. And it is not about be hypocritical, I guess.
>
>If I may, I quote the paper from the Guix maintainers---correct me if
>I am wrong.
>https://arxiv.org/abs/1506.02822 last paragraph from section 5
><<
>Proprietary software.
>GNU Guix does not provide proprietary software packages.
>Unfortunately, proprietary software is still relatively common in HPC,
>be it linear algebra libraries or GPU support. Yet, we see it as a
>strength more than a limitation. Often, these “black boxes” inherently
>limit reproducibility—how is one going to reproduce a software
>environment without permission to run the software in the first place?
>What if the software depends on the ability to “call home” to function
>at all? More importantly, we view reproducible software environments
>and reproducible science as a tool towards improved and shared
>knowledge; developers who deny the freedom to study and modify their
>code work against this goal.
>>>>
>
>Here, my personal opinion. Today, people think that the Science crisis
>about reproducibility will be tackled by Docker and containers. On one
>hand, I am here because I think it is wrong and it is not the path to
>go. On the other hand, I need to pragmatic: people in labs have built
>infrastructures using Docker or equivalent; they wont be convinced
>easily to switch and so I think I want to ease the switch in playing
>directly in their ground.
>
>
>Last, I do not understand how to apply the argument against pushing to
>DockerHub to the Windows port of Emacs.
>Somehow, GNU has to run non-free softwares to provide this port. At
>least to launch some tests.
>
>My personal opinion is that it is good. Because this spreads the
>message about freedom, this helps people to be aware of the movement,
>this should be a first step in liberating users.
>
>
>Thank you if you have comments and/or if you have arguments that
>explain me where it is wrong.
>
>
>All the best,
>simon

Is it time to make a guixhub full of tips how to migrate and perhaps a few references to people/consultants offering to help?
It could host all kinds of guix packs and their recipes too. 
-- 
Sent from my p≡p for Android.

[-- Attachment #2: pEpkey.asc --]
[-- Type: application/pgp-keys, Size: 3825 bytes --]

Re: Docker and singularity containers

[zimoun]

Dear Ricardo,

> I don’t know.  We could have one with R + all CRAN and bioconductor
> packages that we currently offer.

Ok, it seems a good start.


> >  - how?
>
> That’s easier: with “guix pack”.  We would create a Cuirass job to build
> a pack regularly.  We’d need to add support for hooks to publish the
> generated pack on DockerHub, though we could just as well host them
> ourselves.

Aside the DockerHub issue, ok.

And let fix the DockerHub ethical issue later. :-)
I mean let start by build Docker images with Cuirass, then we will see
how/where to publish them.

Just to fix the idea, the simple Docker image containing
Emacs+Guile+Geiser (the scheme SDK ;-) is about 300MB with '.tar.gz'
and 1GB with only '.tar'


>
> Would you like to give this a try?  A good first step is to list the
> steps that need to be performed.  A first implementation might even be
> done with a cron job.  I could add this to berlin.guixsd.org when you’re
> ready.

I am giving a try.


Thank you.


All the best,
simon