Towards reproducibly Jupyter notebooks with Guix-Jupyter

Towards reproducibly Jupyter notebooks with Guix-Jupyter

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 679 bytes --]

Hello Guix!

I’m happy to announce the first release of Guix-Jupyter!

  https://hpc.guix.info/blog/2019/10/towards-reproducible-jupyter-notebooks/

Guix-Jupyter is a Jupyter “kernel” that is able to interpret annotations
describing the environment in which notebook cells will be executed.
The end goal is to be able to regard notebooks as pure functions.

The code is here:

  https://gitlab.inria.fr/guix-hpc/guix-kernel

I presented it minutes ago at JCAD, a conference gathering French HPC
practitioners, and where many talks happen to talk about Jupyter.  :-)

  https://jcad2019.sciencesconf.org/resource/page/id/6

Feedback welcome!

Ludo’.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[zimoun]

Hi Ludo and Pierre-Antoine,

Congrats for these piece of work!
To me, it is very funny that all the folks find Jupyter amazing when
it is a worse version of Org-mode (troll troll) ;-)
One example of many others, the Notebook are not human readable (I
mean, I am not fluent in JSON reading.)
Well, the hype will always appear to me weird.
Whatever!

I will give a try.
Just two random thoughts.

Is the UI ;;guix aligned with the CLI UI ($ guix)?
For example ;;guix environment R <- r-foo
instead of guix environment --ad-hoc r r-foo --pure (or something like that)
Or ;;guix pin <commit>
instead of guix pull --commit=<commit> (or something like that)
It should be confusing.

Why not use the same syntax for ;;guix than for $ guix?


People enjoy Jupyter Notebook because they are "portable".
Some friends use them as Numercial Lab in class. And the students can
replay the lab at home, tweak some cells and see what happens. Or they
sometimes provide a Notebook backbone with some cells missing and the
students have to complete them as homework.
So the "pip install" and "conda" way is easy because it works (almost)
out-the-box in these cases. On Windows, Mac.
Yes, we are not talking about Reproducible Science in this case. But
one ring to rull them all. :-)

Why not add a kind of ;;guix pack at the end of the Notebook to
generate a container image, for example Docker.
Then the student can just run this Docker wherever they wants.


Thank you for all this work!

All the best,
simon

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[Ludovic Courtès]

Hello!

(And hi! Pierre-Antoine.  :-))

zimoun <zimon.toutoune@gmail.com> skribis:

> Congrats for these piece of work!
> To me, it is very funny that all the folks find Jupyter amazing when
> it is a worse version of Org-mode (troll troll) ;-)
> One example of many others, the Notebook are not human readable (I
> mean, I am not fluent in JSON reading.)
> Well, the hype will always appear to me weird.

I’m very much into hype so I won’t comment.  :-)

> Is the UI ;;guix aligned with the CLI UI ($ guix)?
> For example ;;guix environment R <- r-foo
> instead of guix environment --ad-hoc r r-foo --pure (or something like that)
> Or ;;guix pin <commit>
> instead of guix pull --commit=<commit> (or something like that)
> It should be confusing.

I think it’s fine to have a simplified interface.  After all, it’s not a
CLI, and it has a more focused goal.  Also, I was thinking that
eventually we might be able to use Jupyter widgets here and there
instead of text, though I’m not sure what that could look like.

> Why not add a kind of ;;guix pack at the end of the Notebook to
> generate a container image, for example Docker.
> Then the student can just run this Docker wherever they wants.

That’s a very good idea!  We’d need to think about what that entails,
but it sounds like a great feature for the 0.2.0.  :-)

Thanks for your feedback and ideas!

Ludo’.

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[Konrad Hinsen]

Hi Ludo and Simon,

Ludovic Courtès <ludovic.courtes@inria.fr> writes:

> I’m happy to announce the first release of Guix-Jupyter!

This looks very good, even though I will probably have to rework my
reproducible-research-tutorial-with-guix tutorial now ;-)

I haven't looked at this yet in any detail, but I wonder how you deal
with Jupyter pluging (e.g. widgets). For reproducibility, they ought to
be taken from the notebook's specific environment, rather than from the
environment from which Jupyter and the Guix kernel are taken. But this
looks difficult to do, given Jupyter's messy architecture.


zimoun <zimon.toutoune@gmail.com> writes:

> To me, it is very funny that all the folks find Jupyter amazing when
> it is a worse version of Org-mode (troll troll) ;-)

Yes and no. Org-mode doesn't have anything close to Jupyter's widgets.
But otherwise, I agree.

It would be nice in fact to adapt the ideas behind Guix-Jupyter (and
perhaps parts of the code) to Org-mode. Some integration with Emacs will
be necessary to tell Org-mode to start Python etc. from the Guix
environment.

> Is the UI ;;guix aligned with the CLI UI ($ guix)?

I actually wonder which way the alignment should be done - the current
guix CLI is a bit of a mess already.

Cheers,
  Konrad.

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[Ludovic Courtès]

Hi Konrad!

Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

> This looks very good, even though I will probably have to rework my
> reproducible-research-tutorial-with-guix tutorial now ;-)

Heheh.  :-)

> I haven't looked at this yet in any detail, but I wonder how you deal
> with Jupyter pluging (e.g. widgets). For reproducibility, they ought to
> be taken from the notebook's specific environment, rather than from the
> environment from which Jupyter and the Guix kernel are taken. But this
> looks difficult to do, given Jupyter's messy architecture.

It’s an area that needs more testing, as they say.  ;-)

That reminds me of an interesting issue regarding
bitwise-reproducibility that was raised on the Reproducible Builds
mailing list:

  https://lists.reproducible-builds.org/pipermail/rb-general/2019-September/001657.html

> It would be nice in fact to adapt the ideas behind Guix-Jupyter (and
> perhaps parts of the code) to Org-mode. Some integration with Emacs will
> be necessary to tell Org-mode to start Python etc. from the Guix
> environment.

I’d love to see that happen!  I thought perhaps we could trick Alex Kost
or Pierre Neidhardt to hack on that, let’s see.  :-)

Thanks for your feedback,
Ludo’.

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[Konrad Hinsen]

Hi Ludo,

> That reminds me of an interesting issue regarding
> bitwise-reproducibility that was raised on the Reproducible Builds
> mailing list:
>
>   https://lists.reproducible-builds.org/pipermail/rb-general/2019-September/001657.html

We ran into this problem as well in the Reproducible Research MOOC. It's
hard to test equality for notebooks. But we went for solution a) because
our main goal is to teach good practices for writers of future
notebooks, rather than patch bad practices of the past with an
additional layer of build tools.

>> It would be nice in fact to adapt the ideas behind Guix-Jupyter (and
>> perhaps parts of the code) to Org-mode. Some integration with Emacs will
>> be necessary to tell Org-mode to start Python etc. from the Guix
>> environment.
>
> I’d love to see that happen!  I thought perhaps we could trick Alex Kost
> or Pierre Neidhardt to hack on that, let’s see.  :-)

I might also do it myself because I suspect it would be less work to
write the required Emacs package than to explain how to live without it.
Tutorial-driven development :-)

What would be the best way to run code in a specific environment created
for recorded channels? The obvious approach would be

   guix pull -C channels.scm -p /tmp/temp-profile
   /tmp/temp-profile/bin/guix environment –-pure -m manifest.scm –-
   python script.py
   rm -rf /tmp/temp-profile

but doing that properly involves the usual messy precautions for dealing
with temporary directories, and it's probably expensive to do the "guix
pull" repeatedly for the same channel file just because the temporary
profile gets deleted immediately.

This looks like a use case for "guix inferior", but is that already
stable enough to be talked about in public?

Konrad.

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[Ludovic Courtès]

Hi Konrad,

Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

>> That reminds me of an interesting issue regarding
>> bitwise-reproducibility that was raised on the Reproducible Builds
>> mailing list:
>>
>>   https://lists.reproducible-builds.org/pipermail/rb-general/2019-September/001657.html
>
> We ran into this problem as well in the Reproducible Research MOOC. It's
> hard to test equality for notebooks. But we went for solution a) because
> our main goal is to teach good practices for writers of future
> notebooks, rather than patch bad practices of the past with an
> additional layer of build tools.

Yeah.

(Though printing addresses in a REPL isn’t “bad practice” IMO, it’s just
that it doesn’t mesh well with the intended use of notebooks.)

>>> It would be nice in fact to adapt the ideas behind Guix-Jupyter (and
>>> perhaps parts of the code) to Org-mode. Some integration with Emacs will
>>> be necessary to tell Org-mode to start Python etc. from the Guix
>>> environment.
>>
>> I’d love to see that happen!  I thought perhaps we could trick Alex Kost
>> or Pierre Neidhardt to hack on that, let’s see.  :-)
>
> I might also do it myself because I suspect it would be less work to
> write the required Emacs package than to explain how to live without it.
> Tutorial-driven development :-)

Would be sweet!  :-)

> What would be the best way to run code in a specific environment created
> for recorded channels? The obvious approach would be
>
>    guix pull -C channels.scm -p /tmp/temp-profile
>    /tmp/temp-profile/bin/guix environment –-pure -m manifest.scm –-
>    python script.py
>    rm -rf /tmp/temp-profile
>
> but doing that properly involves the usual messy precautions for dealing
> with temporary directories, and it's probably expensive to do the "guix
> pull" repeatedly for the same channel file just because the temporary
> profile gets deleted immediately.

At the API level, there’s ‘inferior-for-channels’ which does that +
registers a GC root + maintains a cache so that the second time you use
a given instance of Guix it’s immediately available.

> This looks like a use case for "guix inferior", but is that already
> stable enough to be talked about in public?

It’s still documented as “subject to change”.  :-)

  https://guix.gnu.org/manual/devel/en/html_node/Inferiors.html

Now, I think you can safely use it in your code, I don’t foresee any
significant change at this point.

As to whether you’d want to talk about it in the MOOC for instance, you
could, but maybe with a word of warning because the MOOC may be around
for a long time.

Thanks,
Ludo’.

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[Konrad Hinsen]

Hi Ludo,

> (Though printing addresses in a REPL isn’t “bad practice” IMO, it’s just
> that it doesn’t mesh well with the intended use of notebooks.)

And that's exactly the difference between a REPL and a reproducible
document. Unfortunately, Jupyter tries to be both and thus never
explains the differences clearly.

> At the API level, there’s ‘inferior-for-channels’ which does that +
> registers a GC root + maintains a cache so that the second time you use
> a given instance of Guix it’s immediately available.

Just what I need...

> Now, I think you can safely use it in your code, I don’t foresee any
> significant change at this point.
>
> As to whether you’d want to talk about it in the MOOC for instance, you

Certainly not. I just want to be sure that the functionality won't go
away. If the API changes a bit, that can be fixed.

I'll start playing with this...

Cheers,
  Konrad.

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[Ludovic Courtès]

Howdy,

Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

>> (Though printing addresses in a REPL isn’t “bad practice” IMO, it’s just
>> that it doesn’t mesh well with the intended use of notebooks.)
>
> And that's exactly the difference between a REPL and a reproducible
> document. Unfortunately, Jupyter tries to be both and thus never
> explains the differences clearly.

Yeah.

>> At the API level, there’s ‘inferior-for-channels’ which does that +
>> registers a GC root + maintains a cache so that the second time you use
>> a given instance of Guix it’s immediately available.
>
> Just what I need...

Awesome, let us know how it goes!

Ludo’.

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[Konrad Hinsen]

Hi Ludo,

>>> At the API level, there’s ‘inferior-for-channels’ which does that +
>>> registers a GC root + maintains a cache so that the second time you use
>>> a given instance of Guix it’s immediately available.
>>
>> Just what I need...
>
> Awesome, let us know how it goes!

Not so well...

If I run "guix environment" in the inferior, it's the inferior that
forks and runs the specified command in the newly created environment.
But the inferior doesn't have access to the tty, so this is bound to
fail for running interactive commands.

Unless there is some Linux magic that I am not aware of for transferring
the tty temporarily to another process, the fork would thus have to
happen in the main Guix process rather than in the inferior. But any
scheme I can come up with for creating a profile in the inferior and
using it in the main process ends up generating the same problem that I
am trying to solve: the need for a temporary reference (profile or GC
root) that must later be removed.

Cheers,
  Konrad.

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[Ludovic Courtès]

Hi,

Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

>>>> At the API level, there’s ‘inferior-for-channels’ which does that +
>>>> registers a GC root + maintains a cache so that the second time you use
>>>> a given instance of Guix it’s immediately available.
>>>
>>> Just what I need...
>>
>> Awesome, let us know how it goes!
>
> Not so well...

Did you define an environment along the lines of the example at
<https://guix.gnu.org/manual/devel/en/html_node/Inferiors.html>?

If so, this should behave in exactly the same way as a “regular” ‘guix
environment’, so I’m not sure where the issues regarding access to the
tty come from?

Depending on the use case, another option is to not use inferiors at all
and simply do:

  guix pull --commit=XYZ -p ./my-old-guix
  ./my-old-guix/bin/guix environment --ad-hoc whatevern

But maybe I’m missing something, let me know.  :-)

Thanks,
Ludo’.

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[Konrad Hinsen]

Hi Ludo,

> Did you define an environment along the lines of the example at
> <https://guix.gnu.org/manual/devel/en/html_node/Inferiors.html>?

No. I did look at this approach briefly, but it looks difficult at
least, if not impossible.

The situation I need to address is re-creating an environment from
- a channel file (from "guix describe")
- a manifest file (typically hand written)

This should not require editing the manifest file, and it should work
for any manifest file, even if it included package transformations or
literal package definitions (yes, I do that, shame on me, I ought to set
up my own channel instead).

A manifest definition using only lookups by package name and version
would be easy to translate into a manifest containing instances of
inferior-package, but I don't see how to deal with the more exotic
cases. I'd have to construct corresponding instances of inferior-package
myself, which is perhaps possible, but not obvious.

> Depending on the use case, another option is to not use inferiors at all
> and simply do:
>
>   guix pull --commit=XYZ -p ./my-old-guix
>   ./my-old-guix/bin/guix environment --ad-hoc whatevern
>
> But maybe I’m missing something, let me know.  :-)

That was my starting point. It's OK for manual execution, but if I want
an Org-Mode file to this automatically, I need to generate a temporary
file in place of ./my-old-guix with all the associated hassles: check
that it doesn't exist before, and clean up in the end even in the case
of errors. Maybe I'd even have to worry about two processes accessing
the same temporary file and do some locking.

Moreover, I may end up having a performance problem with "guix pull"
being run again for every little script in my Org-Mode file. The big
difference between Jupyter and Org-Mode is that Jupyter has only one
kernel process per notebook, whereas Org-Mode by default has one process
per code block.

My dream would be to have something like

  guix environment -C channels.scm -m manifest.scm

with reasonable caching of the pulled channels. That's in fact what I
was trying to implement.

Cheers,
  Konrad.

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[Ludovic Courtès]

Hi Konrad,

Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

>> Depending on the use case, another option is to not use inferiors at all
>> and simply do:
>>
>>   guix pull --commit=XYZ -p ./my-old-guix
>>   ./my-old-guix/bin/guix environment --ad-hoc whatevern
>>
>> But maybe I’m missing something, let me know.  :-)
>
> That was my starting point. It's OK for manual execution, but if I want
> an Org-Mode file to this automatically, I need to generate a temporary
> file in place of ./my-old-guix with all the associated hassles: check
> that it doesn't exist before, and clean up in the end even in the case
> of errors. Maybe I'd even have to worry about two processes accessing
> the same temporary file and do some locking.

I see.  In a way one could argue that it’s not Guix’ problem, but OTOH
it’s clearly a problem that Guix doesn’t make it more convenient.

> I ended up finding an approach that does work, though it relies on
> undocumented properties:
>
>    (use-modules (guix channels)
>                 (guix inferior))
>
>    (define channels
>      (list (channel
>             (name 'guix)
>             (url "https://git.savannah.gnu.org/git/guix.git")
>             (commit
>              "44881cad93801de9462d469500d582af79b99959"))))
>
>    (define inferior
>      (inferior-for-channels channels))
>
>    (define inferior-executable
>      (inferior-eval
>       '(car (program-arguments))
>       inferior))
>
>    (define inferior-command-line
>      (cons inferior-executable (cdr (program-arguments))))
>
>    (apply system* inferior-command-line)
>
> This script uses the inferior mechanism for creating and caching an
> older guix, but then calls its executable directly.
>
> Of course this idea could be implemented in a better way, by factoring
> out the caching mechanism from inferior-for-channels and providing a
> command-line interface. Something like
>
>    guix time-machine channels.scm command args ...
>
> Does this sound reasonable? I volunteer for implementing this, but I'd
> like to have an expert opinion first.

That sounds reasonable to me: it’s a simple and easily-understood UI
(easier than the ‘guix pull -p’ dance!), and the implementation should
be easy (thus the maintenance cost is very low).

Looking forward to a patch!  :-)

Thank you,
Ludo’.

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[Konrad Hinsen]

Hi Ludo,

> I see.  In a way one could argue that it’s not Guix’ problem, but OTOH
> it’s clearly a problem that Guix doesn’t make it more convenient.

I don't know much about Guix' problem, not being a software
psychologist, but in the meantime I am trying to solve a
problem that some Guix users have :-)

> That sounds reasonable to me: it’s a simple and easily-understood UI
> (easier than the ‘guix pull -p’ dance!), and the implementation should
> be easy (thus the maintenance cost is very low).

Indeed.

> Looking forward to a patch!  :-)

If all goes well, next week !

Cheers,
  Konrad.

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[Konrad Hinsen]

Konrad Hinsen <konrad.hinsen@fastmail.net> writes:

>> Looking forward to a patch!  :-)
>
> If all goes well, next week !

Done:

   https://debbugs.gnu.org/cgi/bugreport.cgi?bug=37978

Cheers,
  Konrad.

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 121 bytes --]

I had a quick glance at the patch: It's awesome!
Thanks a lot, Konrad!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[Pjotr Prins]

Dear Konrad,

I don't thing we have an R kernel yet for Jupyter. How hard would it
be to add that? I have been using the Python version in class. It is
great :). And you don't really need Jupyterhub for a class.

Pj.

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[zimoun]

Hi Pjotr,

On Thu, 27 Feb 2020 at 15:33, Pjotr Prins <pjotr.public12@thebird.nl> wrote:

> I don't thing we have an R kernel yet for Jupyter. How hard would it
> be to add that? I have been using the Python version in class. It is
> great :). And you don't really need Jupyterhub for a class.

Is ' r-irkernel' not what you want?

All the best,
simon

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[Pjotr Prins]

On Thu, Feb 27, 2020 at 03:36:37PM +0100, zimoun wrote:
> Hi Pjotr,
> 
> On Thu, 27 Feb 2020 at 15:33, Pjotr Prins <pjotr.public12@thebird.nl> wrote:
> 
> > I don't thing we have an R kernel yet for Jupyter. How hard would it
> > be to add that? I have been using the Python version in class. It is
> > great :). And you don't really need Jupyterhub for a class.
> 
> Is ' r-irkernel' not what you want?

Indeed. Thank you :). Why not call it something with jupyter in the name?

Pj.

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[zimoun]

On Thu, 27 Feb 2020 at 19:11, Pjotr Prins <pjotr.public12@thebird.nl> wrote:
>
> On Thu, Feb 27, 2020 at 03:36:37PM +0100, zimoun wrote:
> > Hi Pjotr,
> >
> > On Thu, 27 Feb 2020 at 15:33, Pjotr Prins <pjotr.public12@thebird.nl> wrote:
> >
> > > I don't thing we have an R kernel yet for Jupyter. How hard would it
> > > be to add that? I have been using the Python version in class. It is
> > > great :). And you don't really need Jupyterhub for a class.
> >
> > Is ' r-irkernel' not what you want?
>
> Indeed. Thank you :). Why not call it something with jupyter in the name?

Because other kernels are named: python-ipykernel or ruby-iruby.
Another: python-pari-jupyter.
And another other: jupyter-guile-kernel.
Well, there is no real convention yet, if I understand correctly.

A good move should to start a gnu/packages/jupyter.scm to put all the
Jupyter related machinery inside... I do not know.


All the best,
simon

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[Vincent Legoll]

hello

On Thu, Feb 27, 2020 at 7:20 PM zimoun <zimon.toutoune@gmail.com> wrote:
> Because other kernels are named: python-ipykernel or ruby-iruby.
> Another: python-pari-jupyter.
> And another other: jupyter-guile-kernel.
> Well, there is no real convention yet, if I understand correctly.
>
> A good move should to start a gnu/packages/jupyter.scm to put all the
> Jupyter related machinery inside... I do not know.

Maybe we can help them show up properly in guix search results, like
adding what could be missing in description.

-- 
Vincent Legoll

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[zimoun]

On Thu, 27 Feb 2020 at 19:43, Vincent Legoll <vincent.legoll@gmail.com> wrote:
>
> hello
>
> On Thu, Feb 27, 2020 at 7:20 PM zimoun <zimon.toutoune@gmail.com> wrote:
> > Because other kernels are named: python-ipykernel or ruby-iruby.
> > Another: python-pari-jupyter.
> > And another other: jupyter-guile-kernel.
> > Well, there is no real convention yet, if I understand correctly.
> >
> > A good move should to start a gnu/packages/jupyter.scm to put all the
> > Jupyter related machinery inside... I do not know.
>
> Maybe we can help them show up properly in guix search results, like
> adding what could be missing in description.

Hum? I am not convinced that something is missing in the
description... I mean "guix search jupyter kernel | less" does already
the job, IMHO.
What should be missing is some convention in the naming scheme; and it
seems easier to force a convention when all the same related packages
live in the same file.


All the best,
simon

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[Ricardo Wurmus]

zimoun <zimon.toutoune@gmail.com> writes:

> On Thu, 27 Feb 2020 at 19:43, Vincent Legoll <vincent.legoll@gmail.com> wrote:
>>
>> hello
>>
>> On Thu, Feb 27, 2020 at 7:20 PM zimoun <zimon.toutoune@gmail.com> wrote:
>> > Because other kernels are named: python-ipykernel or ruby-iruby.
>> > Another: python-pari-jupyter.
>> > And another other: jupyter-guile-kernel.
>> > Well, there is no real convention yet, if I understand correctly.
>> >
>> > A good move should to start a gnu/packages/jupyter.scm to put all the
>> > Jupyter related machinery inside... I do not know.
>>
>> Maybe we can help them show up properly in guix search results, like
>> adding what could be missing in description.
>
> Hum? I am not convinced that something is missing in the
> description... I mean "guix search jupyter kernel | less" does already
> the job, IMHO.
> What should be missing is some convention in the naming scheme; and it
> seems easier to force a convention when all the same related packages
> live in the same file.

The name r-irkernel follows the R package convention; a similar reason
applies to python-ipykernel and ruby-iruby.

-- 
Ricardo

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[zimoun]

On Thu, 27 Feb 2020 at 20:39, Ricardo Wurmus <rekado@elephly.net> wrote:
> zimoun <zimon.toutoune@gmail.com> writes:
> > On Thu, 27 Feb 2020 at 19:43, Vincent Legoll <vincent.legoll@gmail.com> wrote:
> >> On Thu, Feb 27, 2020 at 7:20 PM zimoun <zimon.toutoune@gmail.com> wrote:

> >> > Because other kernels are named: python-ipykernel or ruby-iruby.
> >> > Another: python-pari-jupyter.
> >> > And another other: jupyter-guile-kernel.
> >> > Well, there is no real convention yet, if I understand correctly.
> >> >
> >> > A good move should to start a gnu/packages/jupyter.scm to put all the
> >> > Jupyter related machinery inside... I do not know.
> >>
> >> Maybe we can help them show up properly in guix search results, like
> >> adding what could be missing in description.
> >
> > Hum? I am not convinced that something is missing in the
> > description... I mean "guix search jupyter kernel | less" does already
> > the job, IMHO.
> > What should be missing is some convention in the naming scheme; and it
> > seems easier to force a convention when all the same related packages
> > live in the same file.
>
> The name r-irkernel follows the R package convention; a similar reason
> applies to python-ipykernel and ruby-iruby.

Yes, I know. :-)
Well, "it seems easier to force a convention when all the same related
packages live in the same file" so the current convention is the
cran.scm, ruby.scm or python.scm one. And I was suggesting to move the
packages related to Jupyter to its own module using the convention
'jupyter-' convention, e.g., 'jupyter-irkernel'. And even a more
uniformized naming scheme between all the kernels. As I said, I do not
know if it is a good idea or not. :-)


Cheers,
simon

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[Konrad Hinsen]

Hi Ludo,

> Did you define an environment along the lines of the example at
> <https://guix.gnu.org/manual/devel/en/html_node/Inferiors.html>?
>
> If so, this should behave in exactly the same way as a “regular” ‘guix
> environment’, so I’m not sure where the issues regarding access to the
> tty come from?

I ended up finding an approach that does work, though it relies on
undocumented properties:

   (use-modules (guix channels)
                (guix inferior))

   (define channels
     (list (channel
            (name 'guix)
            (url "https://git.savannah.gnu.org/git/guix.git")
            (commit
             "44881cad93801de9462d469500d582af79b99959"))))

   (define inferior
     (inferior-for-channels channels))

   (define inferior-executable
     (inferior-eval
      '(car (program-arguments))
      inferior))

   (define inferior-command-line
     (cons inferior-executable (cdr (program-arguments))))

   (apply system* inferior-command-line)

This script uses the inferior mechanism for creating and caching an
older guix, but then calls its executable directly.

Of course this idea could be implemented in a better way, by factoring
out the caching mechanism from inferior-for-channels and providing a
command-line interface. Something like

   guix time-machine channels.scm command args ...

Does this sound reasonable? I volunteer for implementing this, but I'd
like to have an expert opinion first.

Cheers,
  Konrad.

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[zimoun]

Hi Konrad,

I am not sure to understand everything, so my questions are:

 - Do you consider binaries coming from multiple commits and/or
multiple channels?
 - Why splits the channel file and the manifest file? I am thinking to
improve the DSL of the manifest file.


All the best,
simon

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[Konrad Hinsen]

Hi Simon,

> I am not sure to understand everything, so my questions are:
>
>  - Do you consider binaries coming from multiple commits and/or
> multiple channels?

No, at least not explicitly. My goal is reproducing computations from
the past, so I need to re-animate old manifest files. These could of
course contain references to inferior-packages, so they could be
multi-commit, but this is not my focus.

>  - Why splits the channel file and the manifest file? I am thinking to
> improve the DSL of the manifest file.

I have thought about this as well, but I am not convinced this is a good
idea, for two reasons.

First, the manifest file is hand-written, the channel file is produced
by "guix describe". How would a combined file be created? And how would
it be updated? The two main kinds of update are 1) updating the software
and 2) changing the components of the environment. Today each part is
covered by one file, so there is no conflict. With a single file, would
"guix pull" have an option to patch in a new commit number?

Second, combining both informations would likely require a restriction
of what is allowed in a manifest file. For example, only allow package
specifications by name and version, without transformations. Otherwise
the implementation would be quite complicated. Imagine someone applying
a transformation using today's guix to a package from last year's guix
that didn't have a feature used in the transformation.

What a combined file would make sense for is saving a profile to a
manifest file. As far as I know there is no support for that at the
moment, but it could be useful.

Cheers,
  Konrad.

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[zimoun]

On Wed, 23 Oct 2019 at 14:15, Konrad Hinsen <konrad.hinsen@fastmail.net> wrote:
>
> Hi Simon,
>
> > I am not sure to understand everything, so my questions are:
> >
> >  - Do you consider binaries coming from multiple commits and/or
> > multiple channels?
>
> No, at least not explicitly. My goal is reproducing computations from
> the past, so I need to re-animate old manifest files. These could of
> course contain references to inferior-packages, so they could be
> multi-commit, but this is not my focus.

But does it fit with the current practice?
I mean, from my experience, I work by steps. Say I use package foo
coming from the commit C1 and I complete the first step. Then, weeks
later, I need the package bar and it is not provided by the commit C1,
so I "guix pull" and the second step is completed with the commit C2.
However, the package foo has been grafted (or update) between C1 and
C2. Therefore, it is becoming hard to run the 2 steps in one-go.

I do not know if it is a way to go: allow profile/environment
populated with binaries from different commits (without speaking about
possible clash.)

Back to my example, all the computations done with the commit C1 have
to be re-done with all the binaries coming from the commit C2---in
order to have only one commit. Or the redo implies the 2 steps---when
it is 2, it seems doable however if it is more it won't.

What do you think?


> >  - Why splits the channel file and the manifest file? I am thinking to
> > improve the DSL of the manifest file.
>
> I have thought about this as well, but I am not convinced this is a good
> idea, for two reasons.
>
> First, the manifest file is hand-written, the channel file is produced
> by "guix describe". How would a combined file be created? And how would
> it be updated? The two main kinds of update are 1) updating the software
> and 2) changing the components of the environment. Today each part is
> covered by one file, so there is no conflict. With a single file, would
> "guix pull" have an option to patch in a new commit number?

This information is already available in one location:
<profile>/manifest, I guess.


> What a combined file would make sense for is saving a profile to a
> manifest file. As far as I know there is no support for that at the
> moment, but it could be useful.

I agree. :-)

Pierre wrote an example to bridge the profile to the manifest.
See https://ambrevar.xyz/guix-advance/index.html
section Generate a manifest  (bottom)

All the best,
simon

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[Konrad Hinsen]

zimoun <zimon.toutoune@gmail.com> writes:

>> No, at least not explicitly. My goal is reproducing computations from
>> the past, so I need to re-animate old manifest files. These could of
>> course contain references to inferior-packages, so they could be
>> multi-commit, but this is not my focus.
>
> But does it fit with the current practice?

With mine, yes :-)

Installing packages one by one into my profile is something I only do
for quick tests. All my profiles are defined by manifest files under
version control, and occasionally updated as a whole.

If there were a straightforward way to save profiles to a manifest
file, I might be less strict about this procedure, though I don't find
it constraining in practice. It fits well with my older habit of having
all configuration files under version control.

> I do not know if it is a way to go: allow profile/environment
> populated with binaries from different commits (without speaking about
> possible clash.)

I'd say it's doable, but so far not implemented. Restoring such an
environment is likely to be expensive, as you have to download multiple
Guix commits and then maybe compile dozens of versions of old compilers
etc.

Assuming that one day in the near future, all scientists use Guix and
routinely publish their manifest files, I'd expect reviewers to ask
authors to clean up their manifest files for the benefit of everyone
else. So I adopt this habit now before I am forced to ;-)

> This information is already available in one location:
> <profile>/manifest, I guess.

Indeed, but it's an undocumented feature for now. The format is
undocumented and not accepted by "guix package" or "guix environment".

> Pierre wrote an example to bridge the profile to the manifest.
> See https://ambrevar.xyz/guix-advance/index.html
> section Generate a manifest  (bottom)

Interesting, but it records only one commit, for the current Guix.
If you have older non-updated packages, this will go unnoticed.
I see this mainly as a tool for migrating from historical-accident
profiles to managed profiles, but that's perhaps just my own bias.

Cheers,
  Konrad

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 447 bytes --]

Indeed.

Konrad's patch / idea is brilliant in my opinion.  Looking forward to
seeing it merged!

In the meantime I've played with Guix + Emacs and wrapped Guix CLI in
hopefully a more convenient way that makes it easy to track the channel
specifications for various manifests, to reproduce them, etc.

See https://gitlab.com/emacs-guix/emacs-guix/issues/13, maybe that'll
inspire you :)

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[Konrad Hinsen]

Hi Pierre,

> In the meantime I've played with Guix + Emacs and wrapped Guix CLI in
> hopefully a more convenient way that makes it easy to track the channel
> specifications for various manifests, to reproduce them, etc.
>
> See https://gitlab.com/emacs-guix/emacs-guix/issues/13, maybe that'll
> inspire you :)

That looks like a layer on top of the Guix commands which does
profile/manifes management based on a few conventions. Nice!

Cheers,
  Konrad.

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 445 bytes --]

Hi Konrad,

Konrad Hinsen <konrad.hinsen@fastmail.net> writes:

> That looks like a layer on top of the Guix commands which does
> profile/manifes management based on a few conventions. Nice!

Exactly.  It's very usable, but of course having it embedded in the CLI
would be much better, since it would be even more usable (e.g. caching
of the Guix checkout) + it would benefit everyone.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[Roel Janssen]

On Thu, 2019-10-10 at 10:21 +0200, Ludovic Courtès wrote:
> Hello Guix!
> 
> I’m happy to announce the first release of Guix-Jupyter!
> 
>   
> https://hpc.guix.info/blog/2019/10/towards-reproducible-jupyter-notebooks/
> 
> Guix-Jupyter is a Jupyter “kernel” that is able to interpret
> annotations
> describing the environment in which notebook cells will be executed.
> The end goal is to be able to regard notebooks as pure functions.
> 
> The code is here:
> 
>   https://gitlab.inria.fr/guix-hpc/guix-kernel
> 
> I presented it minutes ago at JCAD, a conference gathering French HPC
> practitioners, and where many talks happen to talk about Jupyter.  :-
> )
> 
>   https://jcad2019.sciencesconf.org/resource/page/id/6
> 
> Feedback welcome!
> 
> Ludo’.

The animated GIFs are really useful!  If I understand this correctly,
the Guix Jupyter kernel allows one to use multiple (completely
distinct) environments in a single Notebook.  So, mix Python, R and
Scheme in a single notebook.  That's pretty neat!

Kind regards,
Roel Janssen

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[Ludovic Courtès]

Howdy!

Roel Janssen <roel@gnu.org> skribis:

> The animated GIFs are really useful!  If I understand this correctly,
> the Guix Jupyter kernel allows one to use multiple (completely
> distinct) environments in a single Notebook.  So, mix Python, R and
> Scheme in a single notebook.  That's pretty neat!

Yes, I think it’s pretty fun.  :-)

In practice, I guess most users just need a single kernel, but there are
probably cases where mixing kernels can be useful.

Thanks,
Ludo’.

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[Ludovic Courtès]

Roel Janssen <roel@gnu.org> skribis:

> The animated GIFs are really useful!  If I understand this correctly,
> the Guix Jupyter kernel allows one to use multiple (completely
> distinct) environments in a single Notebook.  So, mix Python, R and
> Scheme in a single notebook.  That's pretty neat!

I’ve just learned that <https://github.com/twosigma/beakerx> supports
“polyglot” notebooks as well, though I don’t know how it works.

Ludo’.