unofficial mirror of guix-devel@gnu.org 
 help / color / Atom feed
* Reproducible Research Hackathon: Friday, July 3rd
@ 2020-06-30 12:40 simon tournier
  2020-07-01  4:57 ` Pjotr Prins
  2020-07-03 17:10 ` zimoun
  0 siblings, 2 replies; 12+ messages in thread
From: simon tournier @ 2020-06-30 12:40 UTC (permalink / raw)
  To: guix-devel, guix-hpc

Hello,

We are pleased to announce a Hackathon about Reproducible Science.

We propose to collectively tackle some issues on *Friday, July 3rd*:

 - identify stumbling blocks in using Guix to write end-to-end
pipelines,
 - document how to achieve this,
 - feed the Guix-Past channel by other old packages,
 - provide guix.scm for some of the ReScience C submissions.

Feel free to contact us at guix-hpc@gnu.org if you would like to hack
with us.

We will meet at *9:30 CEST* on the #guix-hpc channel of
irc.freenode.net.  You can use this web client to reach us:

  http://guix.gnu.org/contact/irc/

(tweaking the channel name).

Read more details at:

  https://hpc.guix.info/blog/2020/06/reproducible-research-hackathon

Hope to see you on Friday.


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Reproducible Research Hackathon: Friday, July 3rd
  2020-06-30 12:40 Reproducible Research Hackathon: Friday, July 3rd simon tournier
@ 2020-07-01  4:57 ` Pjotr Prins
  2020-07-01  9:10   ` zimoun
  2020-07-03 17:10 ` zimoun
  1 sibling, 1 reply; 12+ messages in thread
From: Pjotr Prins @ 2020-07-01  4:57 UTC (permalink / raw)
  To: simon tournier; +Cc: guix-devel, guix-hpc

We at UTHSC are game. We are struggling with an old version of the
GeneNetwork web service 1.x which is now running on a 10 year old
CentOS.  The backup time machine server died recently.

GenNetwork1 depends on Python 2.4(!) with modules that have not been
updated this century, and an older version of Apache with mod_python,
amongst other things. We would like to use the guix time-machine
feature to run older versions on demand in containers, also for the
recent GeneNetwork2 version which runs on a modern Guix stack. When we
get it to work I would like to push the older packages in Guix-Past.

Bit much for one day, but let's see where it ends.

There is no point in updating the older GeneNetwork1 instances, I mean
the code. The point really is that it is a reproducible version tied
to older scientific publications that people still can explore. 

Note that GeneNetwork requires a largisch MySQL/MariaDB (170GB)
which is also a snapshot in time. We have 5+ snapshots of that
database that go with 5+ versions of the code. We want to run them all
under Guix so we no longer have to care about the underlying Linux
distro. 

With the newer GeneNetwork2 stack we are thinking of making use of
Guix timemachine to run snapshots, firing containers up on demand.
These will have to be tied with a certain version of the database, so
we have to force time points maintained outside the package Guix git
repo. The databases can be ready to run - listening on a different
port for each version. Nice challenge and it will make a great story
when it works.

Pj.

On Tue, Jun 30, 2020 at 02:40:12PM +0200, simon tournier wrote:
> Hello,
> 
> We are pleased to announce a Hackathon about Reproducible Science.
> 
> We propose to collectively tackle some issues on *Friday, July 3rd*:
> 
>  - identify stumbling blocks in using Guix to write end-to-end
> pipelines,
>  - document how to achieve this,
>  - feed the Guix-Past channel by other old packages,
>  - provide guix.scm for some of the ReScience C submissions.
> 
> Feel free to contact us at guix-hpc@gnu.org if you would like to hack
> with us.
> 
> We will meet at *9:30 CEST* on the #guix-hpc channel of
> irc.freenode.net.  You can use this web client to reach us:
> 
>   http://guix.gnu.org/contact/irc/
> 
> (tweaking the channel name).
> 
> Read more details at:
> 
>   https://hpc.guix.info/blog/2020/06/reproducible-research-hackathon
> 
> Hope to see you on Friday.


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Reproducible Research Hackathon: Friday, July 3rd
  2020-07-01  4:57 ` Pjotr Prins
@ 2020-07-01  9:10   ` zimoun
  2020-07-01  9:21     ` Efraim Flashner
  0 siblings, 1 reply; 12+ messages in thread
From: zimoun @ 2020-07-01  9:10 UTC (permalink / raw)
  To: Pjotr Prins; +Cc: guix-devel, guix-hpc

Hi Pjotr,

Thank you for sharing your plans.

On Tue, 30 Jun 2020 at 23:57, Pjotr Prins <pjotr.public12@thebird.nl> wrote:

> GenNetwork1 depends on Python 2.4(!) with modules that have not been
> updated this century, and an older version of Apache with mod_python,
> amongst other things. We would like to use the guix time-machine
> feature to run older versions on demand in containers, also for the
> recent GeneNetwork2 version which runs on a modern Guix stack. When we
> get it to work I would like to push the older packages in Guix-Past.

I guess that the recent commit [1] by Efraim is part of this effort. :-)

By container, what do you have in mind:
    guix time-machine --channels=file.scm \

 -- environment --container
or 
 -- pack -f 
or
 -- system *-image

?

1: https://gitlab.inria.fr/guix-hpc/guix-past/-/commit/f9fe4d8ac2706834fcff477db8f5421de537d78e


> Note that GeneNetwork requires a largisch MySQL/MariaDB (170GB)
> which is also a snapshot in time. We have 5+ snapshots of that
> database that go with 5+ versions of the code. We want to run them all
> under Guix so we no longer have to care about the underlying Linux
> distro.

I agree that Guix does not have (yet!) good management of large data
set.  The /gnu/store is not designed for that, if I have correctly
understood.  Even we have already discussed Content-Addressable Storage
(CAS) on gwl-devel@gnu.org, AFAIR, we have not ended up with a good
plan.  Well, it will not be fixed on Friday. :-)

But for sure, it is something to keep in mind.


So, see you on Friday. :-)

Cheers,
simon


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Reproducible Research Hackathon: Friday, July 3rd
  2020-07-01  9:10   ` zimoun
@ 2020-07-01  9:21     ` Efraim Flashner
  0 siblings, 0 replies; 12+ messages in thread
From: Efraim Flashner @ 2020-07-01  9:21 UTC (permalink / raw)
  To: zimoun; +Cc: guix-devel, guix-hpc

[-- Attachment #1: Type: text/plain, Size: 2484 bytes --]

On Wed, Jul 01, 2020 at 11:10:23AM +0200, zimoun wrote:
> Hi Pjotr,
> 
> Thank you for sharing your plans.
> 
> On Tue, 30 Jun 2020 at 23:57, Pjotr Prins <pjotr.public12@thebird.nl> wrote:
> 
> > GenNetwork1 depends on Python 2.4(!) with modules that have not been
> > updated this century, and an older version of Apache with mod_python,
> > amongst other things. We would like to use the guix time-machine
> > feature to run older versions on demand in containers, also for the
> > recent GeneNetwork2 version which runs on a modern Guix stack. When we
> > get it to work I would like to push the older packages in Guix-Past.
> 
> I guess that the recent commit [1] by Efraim is part of this effort. :-)

Part of the effort :) I have a bunch more older packages that I'll
probably push to guix-past as time goes on. Right now I have a
no-longer-working octave-3.4.3 with some deps, apache-2.2 and possibly
php-5.6. They're possibly ready to push as-is, but I want to make sure
they actually work for some task before adding them.

> 
> By container, what do you have in mind:
>     guix time-machine --channels=file.scm \
> 
>  -- environment --container
> or 
>  -- pack -f 
> or
>  -- system *-image
> 
> ?

I think we're looking at 'guix system container' so it can be a
"standard" guix service using them (as much as custom guix services are
standard).

> 
> 1: https://gitlab.inria.fr/guix-hpc/guix-past/-/commit/f9fe4d8ac2706834fcff477db8f5421de537d78e
> 
> 
> > Note that GeneNetwork requires a largisch MySQL/MariaDB (170GB)
> > which is also a snapshot in time. We have 5+ snapshots of that
> > database that go with 5+ versions of the code. We want to run them all
> > under Guix so we no longer have to care about the underlying Linux
> > distro.
> 
> I agree that Guix does not have (yet!) good management of large data
> set.  The /gnu/store is not designed for that, if I have correctly
> understood.  Even we have already discussed Content-Addressable Storage
> (CAS) on gwl-devel@gnu.org, AFAIR, we have not ended up with a good
> plan.  Well, it will not be fixed on Friday. :-)
> 
> But for sure, it is something to keep in mind.
> 
> 
> So, see you on Friday. :-)
> 
> Cheers,
> simon
> 

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Reproducible Research Hackathon: Friday, July 3rd
  2020-06-30 12:40 Reproducible Research Hackathon: Friday, July 3rd simon tournier
  2020-07-01  4:57 ` Pjotr Prins
@ 2020-07-03 17:10 ` zimoun
  2020-07-03 17:36   ` Ricardo Wurmus
                     ` (2 more replies)
  1 sibling, 3 replies; 12+ messages in thread
From: zimoun @ 2020-07-03 17:10 UTC (permalink / raw)
  To: simon tournier, guix-devel, guix-hpc

Dear,

Thank you all for joining!


A PAD is still open at:

  https://mensuel.framapad.org/p/guix-hackathon-9hlj?lang=en

Please add whatever you have been up.  Or drop an email.

We are interested to hear your feedback.  Especially about what pass,
what fail and what you have learnt, if you enjoyed the experience, or on
the contrary if you not, what could be improved for the next round.


Personally, I spent the day with the good ol' Fortran and codes using
the norm 77.  And I have enjoyed the flexibility of Guix: navigate
between the versions, running in container, etc..  For example,

  guix time-machine -C channels.scm -- build -L . foo
  guix time-machine -C channels.scm \
      -- environment --container -L . foo --ad-hoc bar@x.y
  [env] ./configure --with-bar=xx && make

And Guix helps a lot to expose "hidden" dependencies, i.e., not always
explicitly mentioned in paper's instructions. Well, something that I
learnt is you need some resource at hand, at least download or build the
substitutes in advance.

The "g77" issue [1] is not tackled but some points have been raised, so
maybe soonish. :-)

1: https://github.com/ReScience/submissions/issues/41

On the Guix side, a missing feature is to search back in history, i.e.,
somehow improves [2] (which starts 2019-01-01) and be able to find the
commit (or range of commits) which provided foo@x.y and bar@a.b

2: https://data.guix.gnu.org/repository/1/branch/master/package/boost


Well, thank you all again!  And hope to see you again on #guix-hpc. :-)

Cheers,
simon


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Reproducible Research Hackathon: Friday, July 3rd
  2020-07-03 17:10 ` zimoun
@ 2020-07-03 17:36   ` Ricardo Wurmus
  2020-07-03 17:56   ` Paul Garlick
  2020-07-04  9:51   ` Konrad Hinsen
  2 siblings, 0 replies; 12+ messages in thread
From: Ricardo Wurmus @ 2020-07-03 17:36 UTC (permalink / raw)
  To: zimoun; +Cc: guix-devel, simon tournier, guix-hpc


zimoun <zimon.toutoune@gmail.com> writes:

> On the Guix side, a missing feature is to search back in history, i.e.,
> somehow improves [2] (which starts 2019-01-01) and be able to find the
> commit (or range of commits) which provided foo@x.y and bar@a.b
>
> 2: https://data.guix.gnu.org/repository/1/branch/master/package/boost

I have been using things like “git log | grep -B10 "boost: Update"” to
find the commit updating a package, and then “git show
caffebabe…:gnu/packages/boost.scm” to view the file at the time of that
commit.

-- 
Ricardo


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Reproducible Research Hackathon: Friday, July 3rd
  2020-07-03 17:10 ` zimoun
  2020-07-03 17:36   ` Ricardo Wurmus
@ 2020-07-03 17:56   ` Paul Garlick
  2020-07-03 18:19     ` zimoun
  2020-07-03 21:17     ` Ricardo Wurmus
  2020-07-04  9:51   ` Konrad Hinsen
  2 siblings, 2 replies; 12+ messages in thread
From: Paul Garlick @ 2020-07-03 17:56 UTC (permalink / raw)
  To: zimoun, simon tournier, guix-devel, guix-hpc

Hi Simon,

Thank you for organising the day.

> We are interested to hear your feedback.  Especially about what pass,
> what fail and what you have learnt, if you enjoyed the experience, or on
> the contrary if you not, what could be improved for the next round.

One outstanding puzzle for me is to figure out how to test a package
definition in a new channel.  For example, I have cloned the guix-past
repository and started to add a new package.  However, what is the best
workflow to attempt a build from the channel?  Do you add the channel
to the git checkout of guix?

Best regards,

Paul.



^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Reproducible Research Hackathon: Friday, July 3rd
  2020-07-03 17:56   ` Paul Garlick
@ 2020-07-03 18:19     ` zimoun
  2020-07-07 17:52       ` Paul Garlick
  2020-07-03 21:17     ` Ricardo Wurmus
  1 sibling, 1 reply; 12+ messages in thread
From: zimoun @ 2020-07-03 18:19 UTC (permalink / raw)
  To: Paul Garlick, simon tournier, guix-devel, guix-hpc

Dear Pauk,

On Fri, 03 Jul 2020 at 18:56, Paul Garlick <pgarlick@tourbillion-technology.com> wrote:

> One outstanding puzzle for me is to figure out how to test a package
> definition in a new channel.  For example, I have cloned the guix-past
> repository and started to add a new package.  However, what is the best
> workflow to attempt a build from the channel?  Do you add the channel
> to the git checkout of guix?

I do not know if it is the best but I do:

  guix build -L path/to/clone the-package
  guix install -L path/to/clone the-package -p /tmp/test
  /tmp/test/bin/the-package


Otherwise, I put in a channels.scm file something like:

--8<---------------cut here---------------start------------->8---
(list
 (channel
    (name 'kikoo)
    (url "file:////home/simon/path/to/clone")
    (branch "master"))
 (channel
  (name 'guix) ; avoid to recompute heavy derivations and build modules
  (url "https://git.savannah.gnu.org/git/guix.git")
  (commit "000c7a0f70248ccf9dc774ef23da3e26d142c610"))
--8<---------------cut here---------------end--------------->8---

where commit is something I already have in /gnu/store, then:

  guix pull -C channels.scm -p /tmp/loc
  /tmp/loc/bin/guix install foo -p /tmp/test

but it is less convenient.


All the best,
simon


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Reproducible Research Hackathon: Friday, July 3rd
  2020-07-03 17:56   ` Paul Garlick
  2020-07-03 18:19     ` zimoun
@ 2020-07-03 21:17     ` Ricardo Wurmus
  2020-07-07 17:44       ` Paul Garlick
  1 sibling, 1 reply; 12+ messages in thread
From: Ricardo Wurmus @ 2020-07-03 21:17 UTC (permalink / raw)
  To: Paul Garlick; +Cc: guix-devel, guix-hpc, simon tournier


Paul Garlick <pgarlick@tourbillion-technology.com> writes:

> One outstanding puzzle for me is to figure out how to test a package
> definition in a new channel.  For example, I have cloned the guix-past
> repository and started to add a new package.  However, what is the best
> workflow to attempt a build from the channel?  Do you add the channel
> to the git checkout of guix?

I did this from the checkout of “guix-past”:

    GUIX_PACKAGE_PATH=$PWD guix build my-package

-- 
Ricardo


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Reproducible Research Hackathon: Friday, July 3rd
  2020-07-03 17:10 ` zimoun
  2020-07-03 17:36   ` Ricardo Wurmus
  2020-07-03 17:56   ` Paul Garlick
@ 2020-07-04  9:51   ` Konrad Hinsen
  2 siblings, 0 replies; 12+ messages in thread
From: Konrad Hinsen @ 2020-07-04  9:51 UTC (permalink / raw)
  To: zimoun, simon tournier, guix-devel, guix-hpc

Hi Simon et al.,

> We are interested to hear your feedback.  Especially about what pass,
> what fail and what you have learnt, if you enjoyed the experience, or on
> the contrary if you not, what could be improved for the next round.

For me this was the occasion to finally start playing with a channel
(guix-past), which has been on my agenda for a while. It was much easier
with people around to answer practical questions that would take a long
time to resolve by going through the documentation.

It was also a good occasion for thinking about how to package artifacts
of historical interest (i.e. recording release dates as metadata).

In terms of packaging techniques, there wasn't much new in it for me.
Python 2.4 is handled very much like Python 2.7, and since I installed
all this stuff by hand many times back in 2008, it felt almost familiar.

There is an interesting issue left for me to explore, which is why
Python 2.4 compiled with today's gcc has a bug that it definitely didn't
have back then. It's probably related to the many intentional
ambiguities in the C language standard, check out John Regehr's blog
(https://blog.regehr.org/) for interesting examples. That could become a
selling point for Guix because we have the option of compiling old
software with old compilers, which is hard with other infrastructures.

Cheers,
  Konrad


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Reproducible Research Hackathon: Friday, July 3rd
  2020-07-03 21:17     ` Ricardo Wurmus
@ 2020-07-07 17:44       ` Paul Garlick
  0 siblings, 0 replies; 12+ messages in thread
From: Paul Garlick @ 2020-07-07 17:44 UTC (permalink / raw)
  To: Ricardo Wurmus; +Cc: guix-devel, guix-hpc, simon tournier

Hi Ricardo,

Many thanks for this tip.

> I did this from the checkout of “guix-past”:
> 
>     GUIX_PACKAGE_PATH=$PWD guix build my-package
> 

First, I changed directory to the 'modules' directory in the guix-past
checkout.  Then   the 'guix build' command can follow the
subdirectories and find the package definitions.

Best regards,

Paul.



^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Reproducible Research Hackathon: Friday, July 3rd
  2020-07-03 18:19     ` zimoun
@ 2020-07-07 17:52       ` Paul Garlick
  0 siblings, 0 replies; 12+ messages in thread
From: Paul Garlick @ 2020-07-07 17:52 UTC (permalink / raw)
  To: zimoun, simon tournier, guix-devel, guix-hpc

Hi Simon,

On Fri, 2020-07-03 at 20:19 +0200, zimoun wrote:
> 
> I do not know if it is the best but I do:
> 
>   guix build -L path/to/clone the-package
>   guix install -L path/to/clone the-package -p /tmp/test
>   /tmp/test/bin/the-package

This works for me if I use the command:

$guix build -L /path/to/guix-past-checkout/modules the-package

Many thanks for the tip.

Best regards,

Paul.



^ permalink raw reply	[flat|nested] 12+ messages in thread

end of thread, back to index

Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-06-30 12:40 Reproducible Research Hackathon: Friday, July 3rd simon tournier
2020-07-01  4:57 ` Pjotr Prins
2020-07-01  9:10   ` zimoun
2020-07-01  9:21     ` Efraim Flashner
2020-07-03 17:10 ` zimoun
2020-07-03 17:36   ` Ricardo Wurmus
2020-07-03 17:56   ` Paul Garlick
2020-07-03 18:19     ` zimoun
2020-07-07 17:52       ` Paul Garlick
2020-07-03 21:17     ` Ricardo Wurmus
2020-07-07 17:44       ` Paul Garlick
2020-07-04  9:51   ` Konrad Hinsen

unofficial mirror of guix-devel@gnu.org 

Archives are clonable:
	git clone --mirror https://yhetil.org/guix-devel/0 guix-devel/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 guix-devel guix-devel/ https://yhetil.org/guix-devel \
		guix-devel@gnu.org
	public-inbox-index guix-devel

Example config snippet for mirrors

Newsgroups are available over NNTP:
	nntp://news.yhetil.org/yhetil.gnu.guix.devel
	nntp://news.gmane.io/gmane.comp.gnu.guix.devel


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git