Guile 3 migration plan

Guile 3 migration plan

[Ludovic Courtès]

Hello Guix!

Today is Guile 3 day, a day to celebrate, woohoo!  \o/

How will Guix migrate to Guile 3?  That’s a threefold process:

  1. We must first migrate packages.  I propose to rename “guile-next”
     to “guile” today, such that “guix install guile” will now install
     Guile 3 (no mass rebuild involved: the “default” Guile remains
     2.2.)

     The downside is that if you run “guix install guile guile-json”,
     you’ll get Guile 3 with Guile-JSON for 2.2; instead, you have to
     explicitly select “guile3.0-json”.

     I propose renaming all the “guile3.0-” packages to “guile-”, and
     all the “guile-” packages to “guile2.2-” as a second step, which
     will address this issue.  This could be done within a few days or
     weeks.

  2. We must be able to run Guix itself on Guile 3.  This is already
     possible if you add “3.0” to the ‘GUILE_PKG’ invocation in
     ‘configure.ac’, but there are still a few test failures that I’ve
     been working on.  Almost all of them relate to the use of ‘@@’ and
     ‘mock’, which do not work with Guile 3 “declarative modules”.

     In many cases, it’s an opportunity to clean up our tests; see for
     example 4aea90b1876179aab8d603a42533a6bdf97ccd3c and the preceding
     commit.  It should be possible to be done real soon!

     Once we’re done, we can change (guix self) so that ‘guix pull’
     pulls Guix on Guile 3.

  3. Change ‘default-guile’ to point to Guile 3 in ‘core-updates’, and
     switch all the packages that consume .go files coming from
     ‘with-imported-modules’ to Guile 3, in particular ‘shepherd’ and
     ‘mcron’.

Thoughts?

You can celebrate Guile 3 day by helping out any of these!  Let’s
synchronize here and on IRC!  :-)

Ludo.’

Re: Guile 3 migration plan

[zimoun]

Hi,

On Thu, 16 Jan 2020 at 11:08, Ludovic Courtès <ludo@gnu.org> wrote:

>   1. We must first migrate packages.  I propose to rename “guile-next”
>      to “guile” today, such that “guix install guile” will now install
>      Guile 3 (no mass rebuild involved: the “default” Guile remains
>      2.2.)
>
>      The downside is that if you run “guix install guile guile-json”,
>      you’ll get Guile 3 with Guile-JSON for 2.2; instead, you have to
>      explicitly select “guile3.0-json”.
>
>      I propose renaming all the “guile3.0-” packages to “guile-”, and
>      all the “guile-” packages to “guile2.2-” as a second step, which
>      will address this issue.  This could be done within a few days or
>      weeks.

It is not related to only the migration to Guile 3 but I feel
something is lacking about switching between the underlying "compiler"
used by the build system.

I agree that it is combinatorial and so not possible to support all.
However, I would like to easily switch between Guile 2 and Guile 3,
between R 3.6 and R 3.5, between GCC 8 and GCC 9, etc. building the
same definition of the package using another "builder" than the
default one.

For example, today there is the 'package-with-python2' function to use
Python 2 instead of Python 3 for the same python-package. There is
also 'package-with-explicit-inputs' to help to switch. Or the
'inherit' mechanism. But it does not appear to me easy to finely tune
the building toolchain.

Well, does it make sense to expose "parameters" as the version of the
compiler/interpreter of the build systems?


All the best,
simon

Re: Guile 3 migration plan

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 2704 bytes --]

On Thu, Jan 16, 2020 at 11:08:15AM +0100, Ludovic Courtès wrote:
> Hello Guix!
> 
> Today is Guile 3 day, a day to celebrate, woohoo!  \o/
> 
> How will Guix migrate to Guile 3?  That’s a threefold process:
> 
>   1. We must first migrate packages.  I propose to rename “guile-next”
>      to “guile” today, such that “guix install guile” will now install
>      Guile 3 (no mass rebuild involved: the “default” Guile remains
>      2.2.)
> 
>      The downside is that if you run “guix install guile guile-json”,
>      you’ll get Guile 3 with Guile-JSON for 2.2; instead, you have to
>      explicitly select “guile3.0-json”.

I feel like these should move together. Imagine if we had python
libraries that only worked with python2 but called them python-foo
anyway. Having consistent naming between the packages is important.

> 
>      I propose renaming all the “guile3.0-” packages to “guile-”, and
>      all the “guile-” packages to “guile2.2-” as a second step, which
>      will address this issue.  This could be done within a few days or
>      weeks.

It seems to me we could switch the guile packages to guile3.0 and leave
guix at guile2.2. In any case, we should create a guix-with-guile3
package to encourage people to try to fix bugs and try it out.

> 
>   2. We must be able to run Guix itself on Guile 3.  This is already
>      possible if you add “3.0” to the ‘GUILE_PKG’ invocation in
>      ‘configure.ac’, but there are still a few test failures that I’ve
>      been working on.  Almost all of them relate to the use of ‘@@’ and
>      ‘mock’, which do not work with Guile 3 “declarative modules”.
> 
>      In many cases, it’s an opportunity to clean up our tests; see for
>      example 4aea90b1876179aab8d603a42533a6bdf97ccd3c and the preceding
>      commit.  It should be possible to be done real soon!

I'm going to miss '@@'

> 
>      Once we’re done, we can change (guix self) so that ‘guix pull’
>      pulls Guix on Guile 3.
> 
>   3. Change ‘default-guile’ to point to Guile 3 in ‘core-updates’, and
>      switch all the packages that consume .go files coming from
>      ‘with-imported-modules’ to Guile 3, in particular ‘shepherd’ and
>      ‘mcron’.
> 
> Thoughts?
> 
> You can celebrate Guile 3 day by helping out any of these!  Let’s
> synchronize here and on IRC!  :-)
> 
> Ludo.’
> 

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Guile 3 migration plan

[Ludovic Courtès]

Hello!

Efraim Flashner <efraim@flashner.co.il> skribis:

> On Thu, Jan 16, 2020 at 11:08:15AM +0100, Ludovic Courtès wrote:
>> Hello Guix!
>> 
>> Today is Guile 3 day, a day to celebrate, woohoo!  \o/
>> 
>> How will Guix migrate to Guile 3?  That’s a threefold process:
>> 
>>   1. We must first migrate packages.  I propose to rename “guile-next”
>>      to “guile” today, such that “guix install guile” will now install
>>      Guile 3 (no mass rebuild involved: the “default” Guile remains
>>      2.2.)
>> 
>>      The downside is that if you run “guix install guile guile-json”,
>>      you’ll get Guile 3 with Guile-JSON for 2.2; instead, you have to
>>      explicitly select “guile3.0-json”.
>
> I feel like these should move together. Imagine if we had python
> libraries that only worked with python2 but called them python-foo
> anyway. Having consistent naming between the packages is important.

Note that it means that a number of “guile-” packages (those that do not
work yet with Guile 3) will disappear or be temporarily superseded by a
“guile2.2-” package, which again will not work with Guile 3.

So it’s not a great solution either; people may still have to explicitly
select guile@2 or something.

WDYT?

Now, whatever we choose, we can add an entry to ‘etc/news.scm’ to make
the transition smoother.

>>      I propose renaming all the “guile3.0-” packages to “guile-”, and
>>      all the “guile-” packages to “guile2.2-” as a second step, which
>>      will address this issue.  This could be done within a few days or
>>      weeks.
>
> It seems to me we could switch the guile packages to guile3.0 and leave
> guix at guile2.2. In any case, we should create a guix-with-guile3
> package to encourage people to try to fix bugs and try it out.

Sure.

>>   2. We must be able to run Guix itself on Guile 3.  This is already
>>      possible if you add “3.0” to the ‘GUILE_PKG’ invocation in
>>      ‘configure.ac’, but there are still a few test failures that I’ve
>>      been working on.  Almost all of them relate to the use of ‘@@’ and
>>      ‘mock’, which do not work with Guile 3 “declarative modules”.
>> 
>>      In many cases, it’s an opportunity to clean up our tests; see for
>>      example 4aea90b1876179aab8d603a42533a6bdf97ccd3c and the preceding
>>      commit.  It should be possible to be done real soon!
>
> I'm going to miss '@@'

It’s still there, it just doesn’t give you what you want in some cases.

Anyhow, happy Guile 3 day!  :-)

Ludo’.

Re: Guile 3 migration plan

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 777 bytes --]

On 2020-01-16, Ludovic Courtès wrote:
> Today is Guile 3 day, a day to celebrate, woohoo!  \o/

Yay!

> How will Guix migrate to Guile 3?  That’s a threefold process:
...
>   2. We must be able to run Guix itself on Guile 3.  This is already
>      possible if you add “3.0” to the ‘GUILE_PKG’ invocation in
>      ‘configure.ac’, but there are still a few test failures that I’ve
>      been working on.
...
>      Once we’re done, we can change (guix self) so that ‘guix pull’
>      pulls Guix on Guile 3.

Will it still be buildable from source with only guile 2.2? I have no
idea when Debian will have guile 3 available, for example, and was
hoping to be able to start the process of uploading ... "soon".

live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Guile 3 migration plan

[Ludovic Courtès]

Hi!

Vagrant Cascadian <vagrant@debian.org> skribis:

> On 2020-01-16, Ludovic Courtès wrote:
>> Today is Guile 3 day, a day to celebrate, woohoo!  \o/
>
> Yay!
>
>> How will Guix migrate to Guile 3?  That’s a threefold process:
> ...
>>   2. We must be able to run Guix itself on Guile 3.  This is already
>>      possible if you add “3.0” to the ‘GUILE_PKG’ invocation in
>>      ‘configure.ac’, but there are still a few test failures that I’ve
>>      been working on.
> ...
>>      Once we’re done, we can change (guix self) so that ‘guix pull’
>>      pulls Guix on Guile 3.
>
> Will it still be buildable from source with only guile 2.2? I have no
> idea when Debian will have guile 3 available, for example, and was
> hoping to be able to start the process of uploading ... "soon".

For some time (a few months at least?) we’ll make sure Guix can be built
from source with both 2.2 and 3.0.

Ludo’.

Re: Guile 3 migration plan

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 1152 bytes --]

Hello Guix!

Ludovic Courtès <ludo@gnu.org> skribis:

>   2. We must be able to run Guix itself on Guile 3.  This is already
>      possible if you add “3.0” to the ‘GUILE_PKG’ invocation in
>      ‘configure.ac’, but there are still a few test failures that I’ve
>      been working on.  Almost all of them relate to the use of ‘@@’ and
>      ‘mock’, which do not work with Guile 3 “declarative modules”.
>
>      In many cases, it’s an opportunity to clean up our tests; see for
>      example 4aea90b1876179aab8d603a42533a6bdf97ccd3c and the preceding
>      commit.  It should be possible to be done real soon!

Good news everyone!  It builds and all the tests pass since
7b2a47a702b7393cd968640079f8703c932d1405, and commit
da7651806102d637253cb9f5677b96d6a178fc05 adds a “guile3.0-guix” package!

>      Once we’re done, we can change (guix self) so that ‘guix pull’
>      pulls Guix on Guile 3.

The attached patch does that, I tested it with “make as-derivation”.
I’ll do some more testing but unless I’m overlooking something, we can
apply it anytime now.

Ludo’.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 2321 bytes --]

diff --git a/guix/self.scm b/guix/self.scm
index 207e80d842..f883f14a2c 100644
--- a/guix/self.scm
+++ b/guix/self.scm
@@ -49,13 +49,13 @@
   (let ((ref (lambda (module variable)
                (module-ref (resolve-interface module) variable))))
     (match-lambda
-      ("guile"      (ref '(gnu packages commencement) 'guile-final))
-      ("guile-json" (ref '(gnu packages guile) 'guile-json-3))
-      ("guile-ssh"  (ref '(gnu packages ssh)   'guile-ssh))
-      ("guile-git"  (ref '(gnu packages guile) 'guile-git))
-      ("guile-sqlite3" (ref '(gnu packages guile) 'guile-sqlite3))
-      ("guile-gcrypt"  (ref '(gnu packages gnupg) 'guile-gcrypt))
-      ("gnutls"     (ref '(gnu packages tls) 'gnutls))
+      ("guile"      (ref '(gnu packages guile) 'guile-3.0))
+      ("guile-json" (ref '(gnu packages guile) 'guile3.0-json))
+      ("guile-ssh"  (ref '(gnu packages ssh)   'guile3.0-ssh))
+      ("guile-git"  (ref '(gnu packages guile) 'guile3.0-git))
+      ("guile-sqlite3" (ref '(gnu packages guile) 'guile3.0-sqlite3))
+      ("guile-gcrypt"  (ref '(gnu packages gnupg) 'guile3.0-gcrypt))
+      ("gnutls"     (ref '(gnu packages tls) 'guile3.0-gnutls))
       ("zlib"       (ref '(gnu packages compression) 'zlib))
       ("lzlib"      (ref '(gnu packages compression) 'lzlib))
       ("gzip"       (ref '(gnu packages compression) 'gzip))
@@ -1121,9 +1121,9 @@ is not supported."
         version))
 
   (define guile
-    ;; When PULL-VERSION >= 1, produce a self-contained Guix and use Guile 2.2
-    ;; unconditionally.
-    (default-guile))
+    ;; When PULL-VERSION >= 1, produce a self-contained Guix and use the
+    ;; current Guile unconditionally.
+    (specification->package "guile"))
 
   (when (and (< pull-version 1)
              (not (string=? (package-version guile) guile-version)))
@@ -1142,7 +1142,7 @@ is not supported."
                                                      (shorten version))
                                #:pull-version pull-version
                                #:guile-version (if (>= pull-version 1)
-                                                   "2.2" guile-version)
+                                                   "3.0" guile-version)
                                #:guile-for-build guile)))
       (if guix
           (lower-object guix)

Re: Guile 3 migration plan

[Ludovic Courtès]

Hi,

Ludovic Courtès <ludo@gnu.org> skribis:

>>      Once we’re done, we can change (guix self) so that ‘guix pull’
>>      pulls Guix on Guile 3.
>
> The attached patch does that, I tested it with “make as-derivation”.
> I’ll do some more testing but unless I’m overlooking something, we can
> apply it anytime now.

The showstopper is that currently “guile3.0-guix” doesn’t build on all
arches:

  https://ci.guix.gnu.org/search?query=guile3.0-guix

On AArch64, it’s test failures (that needs to be investigated, but I
suspect it’s nothing serious.)

However, on ARMv7, Guile 3.0 segfaults as it’s building:

  https://ci.guix.gnu.org/log/8b8c0hxvm9qa5kff168vdr3943cc2s61-guile-next-3.0.0

That needs more work!

Ludo’.

Re: Guile 3 migration plan

[Ludovic Courtès]

Hello,

Ludovic Courtès <ludo@gnu.org> skribis:

> However, on ARMv7, Guile 3.0 segfaults as it’s building:
>
>   https://ci.guix.gnu.org/log/8b8c0hxvm9qa5kff168vdr3943cc2s61-guile-next-3.0.0

It seems that the regression happened between 2.9.4 and 2.9.9¹:

  https://ci.guix.gnu.org/build/1687242/details

I’ve reported it here:

  https://issues.guix.gnu.org/issue/39208

Ludo’.

¹ libguile 2.9.5 to 2.9.8 included would fail to build on ARMv7:
  <https://issues.guix.gnu.org/issue/38890>.

Re: Guile 3 migration plan

[Ludovic Courtès]

Ludovic Courtès <ludo@gnu.org> skribis:

> Ludovic Courtès <ludo@gnu.org> skribis:
>
>> However, on ARMv7, Guile 3.0 segfaults as it’s building:
>>
>>   https://ci.guix.gnu.org/log/8b8c0hxvm9qa5kff168vdr3943cc2s61-guile-next-3.0.0
>
> It seems that the regression happened between 2.9.4 and 2.9.9¹:
>
>   https://ci.guix.gnu.org/build/1687242/details
>
> I’ve reported it here:
>
>   https://issues.guix.gnu.org/issue/39208

Commit 6e6aab0b6af0bf60b2c7276fa9ee6ca6da9aef88 disables JIT on
armhf-linux to work around this issue.

From a quick test on a machine that wasn’t quite idle, so take with
quite a bit of salt, the non-JIT ‘guile3.0-guix’ is ~6% *slower* than
‘guix’ on armhf-linux (wall-clock time of “guix build libreoffice -nd”).

Could someone check what they get?

That’s a bummer but it’s understandable because the instruction set of
the VM in Guile 3 is larger and lower-level:

  https://wingolog.org/archives/2018/01/17/instruction-explosion-in-guile

Regardless, I would go ahead and move (guix self) to Guile 3 now.

Thoughts?

Thanks,
Ludo’.

Re: Guile 3 migration plan

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> Regardless, I would go ahead and move (guix self) to Guile 3 now.

Sounds good to me!

-- 
Ricardo

Re: Guile 3 migration plan

[Ludovic Courtès]

Hi!

Ricardo Wurmus <rekado@elephly.net> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Regardless, I would go ahead and move (guix self) to Guile 3 now.
>
> Sounds good to me!

Done in 039cb8e6b11928639915eba40a9ead1bed1fef5f.

There’s a build failure on aarch64 that I’m tracking down:

  http://ci.guix.info/build/2184296/details

Let’s see…

Ludo’.

Re: Guile 3 migration plan

[Ludovic Courtès]

Hello!

Yesterday I published a post on this topic:

  https://guix.gnu.org/blog/2020/guile-3-and-guix/

Enjoy!  :-)

Ludo’.

Re: Guile 3 migration plan

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> Yesterday I published a post on this topic:
>
>   https://guix.gnu.org/blog/2020/guile-3-and-guix/

Very nice!  It’s impressive how you managed to give structure to
something that initially seems like “just a few changes here and
there”.  Thanks for writing this!

Two minor things I noticed:

* “we will switch to Guile_3” <— the underscore should be replaced with
  a non-breaking space.

* The comma in “Long live, Guile 3!” should be removed.

--
Ricardo

Re: Guile 3 migration plan

[Ludovic Courtès]

Hi,

Ricardo Wurmus <rekado@elephly.net> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Yesterday I published a post on this topic:
>>
>>   https://guix.gnu.org/blog/2020/guile-3-and-guix/
>
> Very nice!  It’s impressive how you managed to give structure to
> something that initially seems like “just a few changes here and
> there”.  Thanks for writing this!

Heheh, thank you.

> Two minor things I noticed:

Oops, fixed!

Ludo’.