Building Guix with Guile 2.1

Building Guix with Guile 2.1

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 1102 bytes --]

Hello!

Nalaginrut reported that Guix fails to build with Guile 2.2, which was a
bit of a shame, hence commits e465d9e19087ab150f7e31f21c09e4a147b93b36
and 9d126aa2b504bb9fad536eac186805ff623e96be.

Now, the hack in build-aux/compile-all.scm doesn’t quite work with 2.2,
and it was already quite fragile.

So I think we need a different approach.  A more robust approach that
comes to mind would be to look at our module dependency graph (with help
from (guix modules)), run N Guile processes (instead of threads), and
have each of them build a subset of the graph.  It would essentially
require us to reimplement a subset of Make (or guix-daemon).

I’ve tried generating makefile rules from module dependencies.  That’s
pretty cool, but compilation of the package modules remains damn slow
(2nd patch attached).

What do people think?  Anyone has a better idea?

The attached patch allows us to build the modules sequentially but in
topologically-sorted order, which means we can avoid the ugly
load-modules-first workaround.  Still, it’s a bit too slow.

Ludo’.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 3010 bytes --]

diff --git a/build-aux/compile-all.scm b/build-aux/compile-all.scm
index 7c937a0..3703187 100644
--- a/build-aux/compile-all.scm
+++ b/build-aux/compile-all.scm
@@ -21,6 +21,7 @@
              (system base message)
              (ice-9 match)
              (ice-9 threads)
+             (guix modules)
              (guix build utils))
 
 (define warnings
@@ -55,19 +56,15 @@
     (map string->symbol
          (string-split module-path #\/))))
 
-;;; To work around <http://bugs.gnu.org/15602> (FIXME), we want to load all
-;;; files to be compiled first.  We do this via resolve-interface so that the
-;;; top-level of each file (module) is only executed once.
-(define (load-module-file file)
-  (let ((module (file->module file)))
-    (format #t "  LOAD     ~a~%" module)
-    (resolve-interface module)))
+(define (module->file module)
+  (string-append srcdir "/"
+                 (string-join (map symbol->string module) "/")
+                 ".scm"))
 
-(define (compile-file* file output-mutex)
+(define (compile-file* file)
   (let ((go (scm->go file)))
-    (with-mutex output-mutex
-      (format #t "  GUILEC   ~a~%" go)
-      (force-output))
+    (format #t "  GUILEC   ~a~%" go)
+    (force-output)
     (mkdir-p (dirname go))
     (with-fluids ((*current-warning-prefix* ""))
       (with-target host
@@ -78,12 +75,12 @@
 
 (match (command-line)
   ((_ . files)
-   (let ((files (filter file-needs-compilation? files)))
-     (for-each load-module-file files)
-     (let ((mutex (make-mutex)))
-       (par-for-each (lambda (file)
-                       (compile-file* file mutex))
-                     files)))))
+   (for-each (lambda (module)
+               (let ((file (module->file module)))
+                 (when (file-needs-compilation? file)
+                   (compile-file* file))))
+             (source-module-closure
+              (map file->module files)))))
 
 ;;; Local Variables:
 ;;; eval: (put 'with-target 'scheme-indent-function 1)
diff --git a/guix/modules.scm b/guix/modules.scm
index 24f613f..837d150 100644
--- a/guix/modules.scm
+++ b/guix/modules.scm
@@ -118,14 +118,19 @@ module names.  Only modules that match SELECT? are considered."
              (visited  (set)))
     (match modules
       (()
-       (reverse result))
+       (values result visited))
       ((module rest ...)
        (cond ((set-contains? visited module)
               (loop rest result visited))
              ((select? module)
-              (loop (append (dependencies module) rest)
-                    (cons module result)
-                    (set-insert module visited)))
+              (call-with-values
+                  (lambda ()
+                    (loop rest result
+                          (set-insert module visited)))
+                (lambda (result visited)
+                  (loop (dependencies module)
+                        (cons module result)
+                        visited))))
              (else
               (loop rest result visited)))))))
 

[-- Attachment #3: the 2nd patch --]
[-- Type: text/x-patch, Size: 7119 bytes --]

From 9621aa00b4378036e906f58dc40e431748ba9790 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
Date: Wed, 21 Sep 2016 23:45:07 +0900
Subject: [PATCH] build: Generate makefile rules from .scm files.

* guix/modules.scm (source-module-dependencies): Export.
* build-aux/generate-make-rules.scm: New file.
* Makefile.am (EXTRA_DIST): Add it.
(AM_V_GUILEC, AM_V_GUILEC_, AM_V_GUILEC_0)
(GUILD_COMPILE_FLAGS): New variables.
(.scm.go): New rule.
(%.go, make-go): Remove.
(build-aux/module-rules.mk): New rule.  Add "-include" statement.
(CLEANFILES): Add this file.
---
 Makefile.am                       | 45 +++++++++++++++++++------
 build-aux/generate-make-rules.scm | 71 +++++++++++++++++++++++++++++++++++++++
 guix/modules.scm                  |  3 +-
 3 files changed, 108 insertions(+), 11 deletions(-)
 create mode 100644 build-aux/generate-make-rules.scm

diff --git a/Makefile.am b/Makefile.am
index f9fe141..6720773 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -355,6 +355,7 @@ EXTRA_DIST =						\
   .dir-locals.el					\
   build-aux/build-self.scm				\
   build-aux/compile-all.scm				\
+  build-aux/generate-make-rules.scm			\
   build-aux/hydra/evaluate.scm				\
   build-aux/hydra/gnu-system.scm			\
   build-aux/hydra/demo-os.scm				\
@@ -390,26 +391,50 @@ CLEANFILES =					\
   $(GOBJECTS)					\
   $(SCM_TESTS:tests/%.scm=%.log)
 
+AM_V_GUILEC = $(AM_V_GUILEC_$(V))
+AM_V_GUILEC_ = $(AM_V_GUILEC_$(AM_DEFAULT_VERBOSITY))
+AM_V_GUILEC_0 = @echo "  GUILEC" $@;
+
+# Flags passed to 'guild compile'.
+GUILD_COMPILE_FLAGS =				\
+  -Wformat -Wunbound-variable -Warity-mismatch
+
 # Unset 'GUILE_LOAD_COMPILED_PATH' altogether while compiling.  Otherwise, if
 # $GUILE_LOAD_COMPILED_PATH contains $(moduledir), we may find .go files in
 # there that are newer than the local .scm files (for instance because the
 # user ran 'make install' recently).  When that happens, we end up loading
 # those previously-installed .go files, which may be stale, thereby breaking
-# the whole thing.  Likewise, set 'XDG_CACHE_HOME' to avoid loading possibly
-# stale files from ~/.cache/guile/ccache.
-%.go: make-go ; @:
-make-go: $(MODULES) guix/config.scm guix/tests.scm
-	$(AM_V_at)echo "Compiling Scheme modules..." ;			\
+# the whole thing.
+#
+# XXX: Use the C locale for when Guile lacks
+# <http://git.sv.gnu.org/cgit/guile.git/commit/?h=stable-2.0&id=e2c6bf3866d1186c60bacfbd4fe5037087ee5e3f>.
+.scm.go:
+	$(AM_V_GUILEC)$(MKDIR_P) `dirname "$@"` ;			\
+	$(AM_V_P) && out=1 || out=- ;					\
 	unset GUILE_LOAD_COMPILED_PATH ;				\
-	XDG_CACHE_HOME=/nowhere						\
-	host=$(host) srcdir="$(top_srcdir)"				\
+	LC_ALL=C							\
 	$(top_builddir)/pre-inst-env					\
-	$(GUILE) -L "$(top_builddir)" -L "$(top_srcdir)"		\
-	  --no-auto-compile 						\
-	  -s "$(top_srcdir)"/build-aux/compile-all.scm $^
+	$(GUILD) compile -L "$(top_builddir)" -L "$(top_srcdir)"	\
+	  $(GUILD_COMPILE_FLAGS) --target="$(host)"			\
+	  -o "$@" "$<" >&$$out
+
 
 SUFFIXES = .go
 
+# Dependency among Scheme modules.
+-include build-aux/module-rules.mk
+
+build-aux/module-rules.mk: $(MODULES)
+	$(AM_V_GEN)$(MKDIR_P) `dirname "$@"` ;			\
+	unset GUILE_LOAD_COMPILED_PATH ;			\
+	LC_ALL=C srcdir="$(srcdir)"				\
+	$(top_builddir)/pre-inst-env				\
+	  $(GUILE) --no-auto-compile				\
+	  build-aux/generate-make-rules.scm $^ > "$@.tmp"
+	mv "$@.tmp" "$@"
+
+CLEANFILES += build-aux/module-rules.mk
+
 # Make sure source files are installed first, so that the mtime of
 # installed compiled files is greater than that of installed source
 # files.  See
diff --git a/build-aux/generate-make-rules.scm b/build-aux/generate-make-rules.scm
new file mode 100644
index 0000000..c38b364
--- /dev/null
+++ b/build-aux/generate-make-rules.scm
@@ -0,0 +1,71 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2016 Ludovic Courtès <ludo@gnu.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(use-modules (guix modules)
+             (guix utils)
+             (srfi srfi-26)
+             (ice-9 match))
+
+(define (relevant-module? module)
+  (match module
+    (('guix _ ...) #t)
+    (('gnu 'packages _ ...) #f)
+    (('gnu _ ...) #t)
+    (_ #f)))
+
+(define srcdir (or (getenv "srcdir")"."))
+
+(define (relative-file file)
+  (if (string-prefix? (string-append srcdir "/") file)
+      (string-drop file (+ 1 (string-length srcdir)))
+      file))
+
+(define (file->module file)
+  (let* ((relative (relative-file file))
+         (module-path (string-drop-right relative 4)))
+    (map string->symbol
+         (string-split module-path #\/))))
+
+(define (module->file module)
+  (string-append srcdir "/"
+                 (string-join (map symbol->string module) "/")))
+
+(define (display-module-rule module dependencies port)
+  (format port "~a.go: ~a.scm~{ ~a~}~%"
+          (module->file module)
+          (module->file module)
+          (map (compose (cut string-append <> ".go")
+                        module->file) dependencies)))
+
+(match (command-line)
+  ((command . files)
+   (format #t "# Automatically generated.           -*- read-only -*-\n")
+   (for-each (lambda (file)
+               (match (file->module file)
+                 (('gnu 'packages _ ...)
+                  ;; We don't want to rebuild package modules too often,
+                  ;; especially since it's rarely useful.  Thus, generate a
+                  ;; simple static rule.
+                  (format #t "~a.go: ~a guix/packages.go~%"
+                          (file-sans-extension file) file))
+                 (module
+                  (display-module-rule module
+                                       (filter relevant-module?
+                                               (source-module-dependencies module))
+                                       (current-output-port)))))
+             files)))
diff --git a/guix/modules.scm b/guix/modules.scm
index 24f613f..d78faf2 100644
--- a/guix/modules.scm
+++ b/guix/modules.scm
@@ -21,7 +21,8 @@
   #:use-module (guix sets)
   #:use-module (srfi srfi-26)
   #:use-module (ice-9 match)
-  #:export (source-module-closure
+  #:export (source-module-dependencies
+            source-module-closure
             live-module-closure
             guix-module-name?))
 
-- 
2.10.0

Re: Building Guix with Guile 2.1

[Andy Wingo]

On Wed 21 Sep 2016 10:23, ludo@gnu.org (Ludovic Courtès) writes:

> Hello!
>
> Nalaginrut reported that Guix fails to build with Guile 2.2, which was a
> bit of a shame, hence commits e465d9e19087ab150f7e31f21c09e4a147b93b36
> and 9d126aa2b504bb9fad536eac186805ff623e96be.
>
> Now, the hack in build-aux/compile-all.scm doesn’t quite work with 2.2,
> and it was already quite fragile.

Which hack?  You mean loading modules before compiling?  I guess you
should load the compiler modules too FWIW, perhaps that's an issue.

If there is a nice change we can do to make module-loading thread-safe,
let's think about that :)  It's probably the biggest thread-safety
problem we have in Guile and now would be the right time to fix it.

Andy

Re: Building Guix with Guile 2.1

[Taylan Ulrich Bayırlı/Kammer]

Andy Wingo <wingo@igalia.com> writes:

> On Wed 21 Sep 2016 10:23, ludo@gnu.org (Ludovic Courtès) writes:
>
>> Hello!
>>
>> Nalaginrut reported that Guix fails to build with Guile 2.2, which was a
>> bit of a shame, hence commits e465d9e19087ab150f7e31f21c09e4a147b93b36
>> and 9d126aa2b504bb9fad536eac186805ff623e96be.
>>
>> Now, the hack in build-aux/compile-all.scm doesn’t quite work with 2.2,
>> and it was already quite fragile.
>
> Which hack?  You mean loading modules before compiling?  I guess you
> should load the compiler modules too FWIW, perhaps that's an issue.
>
> If there is a nice change we can do to make module-loading thread-safe,
> let's think about that :)  It's probably the biggest thread-safety
> problem we have in Guile and now would be the right time to fix it.

It would be neat to make module loading thread-safe indeed!  Not sure if
this issue is related to that though...  Currently the .go compilation
phase of 'make' errors like:

> [... snip ...]
>   LOAD     (guix build-system haskell)
>   LOAD     (guix build-system perl)
>   LOAD     (guix build-system python)
>   LOAD     (guix build-system waf)
> Backtrace:
>           18 (primitive-load "/home/taylan/src/guix/./build-aux/comp?")
> In ice-9/eval.scm:
>     608:8 17 (_ #(#(#(#(#(#<directory (guile-user) 26?> ?) ?) ?) ?) ?))
> In ice-9/boot-9.scm:
>    262:13 16 (for-each1 ("guix/build-system/waf.scm" "guix/build-?" ?))
>   2788:17 15 (resolve-interface (guix build-system waf) #:select _ # ?)
>   2713:10 14 (_ (guix build-system waf) _ _ #:ensure _)
>   2989:16 13 (try-module-autoload _ _)
>    2325:4 12 (save-module-excursion #<procedure 2fb5390 at ice-9/boo?>)
>   3009:22 11 (_)
> In unknown file:
>           10 (primitive-load-path "guix/build-system/waf" #<procedur?>)
> In ice-9/eval.scm:
>    710:20  9 (primitive-eval (define-module (guix build-system #) # ?))
> In ice-9/psyntax.scm:
>   1209:36  8 (expand-top-sequence ((define-module (guix # waf) # ?)) ?)
>   1156:24  7 (parse _ (("placeholder" placeholder)) ((top) #(# # ?)) ?)
>    279:10  6 (parse _ (("placeholder" placeholder)) (()) _ c&e (eval) ?)
> In ice-9/eval.scm:
>    293:34  5 (_ #<module (#{ g141635}#) 4809000>)
> In ice-9/boot-9.scm:
>   2849:10  4 (define-module* _ #:filename _ #:pure _ #:version _ # _ ?)
>   2801:10  3 (resolve-interface (guix build-system python) #:select _ ?)
>    262:13  2 (for-each1 (default-python default-python2))
>   2806:38  1 (_ _)
> In unknown file:
>            0 (scm-error misc-error #f "~A" ("no binding `default-p?") ?)
>
> ERROR: In procedure scm-error:
> ERROR: no binding `default-python' in module (guix build-system python)

That's in the phase where the modules are all loaded by calling
'resolve-interface' on their names, which is *not* done in parallel.
To quote compile-all.scm:

> [... snip ...]
>
> ;;; To work around <http://bugs.gnu.org/15602> (FIXME), we want to load all
> ;;; files to be compiled first.  We do this via resolve-interface so that the
> ;;; top-level of each file (module) is only executed once.
> (define (load-module-file file)
>   (let ((module (file->module file)))
>     (format #t "  LOAD     ~a~%" module)
>     (resolve-interface module)))
>
> [... snip ...]
>
> (match (command-line)
>   ((_ . files)
>    (let ((files (filter file-needs-compilation? files)))
>      (for-each load-module-file files)
>      (let ((mutex (make-mutex)))
>        (par-for-each (lambda (file)
>                        (compile-file* file mutex))
>                      files)))))

I ran guile via ./pre-inst-env and executed (resolve-interface '(guix
build-system waf)) manually and actually got a similar / the same error:

> [... snip ...]
> ;;; WARNING: compilation of /home/taylan/src/guix/guix/build-system/waf.scm failed:
> ;;; ERROR: no binding `default-python' in module (guix build-system python)
> ERROR: In procedure scm-error:
> ERROR: no binding `default-python' in module (guix build-system python)

Looking into (guix build-system waf), it contains:

> ...
>   #:use-module ((guix build-system python)
>                 #:select (default-python default-python2))
> ...

but (guix build-system python) does not export default-python.

Apparently Guile 2.0 allows this (to select private bindings), but 2.2
does not.

Reading the documentation for #:select I can't really tell which
behavior is intended.

Taylan

Re: Building Guix with Guile 2.1

[Taylan Ulrich Bayırlı/Kammer]

By the way, for testing guix with guile 2.2 easily:

> $ cat ~/tmp/guix-with-guile-next.scm
> (use-modules
>  (guix packages)
>  (guix build utils)
>  (gnu packages guile)
>  (gnu packages package-management))
>
> (package
>   (inherit guix)
>   (inputs (alist-replace "guile"
>                          (list guile-next)
>                          (package-inputs guix))))
>
> $ guix environment --pure -l ~/tmp/guix-with-guile-next.scm
> $ autoreconf ...

Taylan

Re: Building Guix with Guile 2.1

[Taylan Ulrich Bayırlı/Kammer]

[-- Attachment #1: Type: text/plain, Size: 1444 bytes --]

ludo@gnu.org (Ludovic Courtès) writes:

> Hello!
>
> Nalaginrut reported that Guix fails to build with Guile 2.2, which was a
> bit of a shame, hence commits e465d9e19087ab150f7e31f21c09e4a147b93b36
> and 9d126aa2b504bb9fad536eac186805ff623e96be.

With the attached quick-and-dirty patch, 'make' runs to completion.

I think we can keep the (compile 'dummy) hack.  That leaves two issues
which may be solved in a cleaner manner than in this patch:

- The (define foo (@@ (bar) foo)) parts.
- Making %tty-gid public.  (The above didn't work for this one...)

Both fixes may become unnecessary if Guile 2.2 goes back to allowing
#:select to import private bindings.  Otherwise, recommendations for
cleaner solutions welcome.

Of course, the actual parallelization hack is still fragile, though not
more so than in 2.0.  We still want to fix it eventually, if Guile 2.2
doesn't make module autoloading thread safe yet.  (More precisely,
referencing and thus instantiating modules previously "registered" via
the autoload mechanism, if I understand correctly.)

By the way, compile time seems to increase greatly with 2.2, to the
point I wondered if it's really compiling in parallel, but it does seem
to as evidenced by top(1).  Maybe package modules could be compiled with
certain optims turned off, since they mostly just consist of package
object definitions and not procedures whose performance would matter.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Fix-compilation-on-Guile-2.2.patch --]
[-- Type: text/x-diff, Size: 2685 bytes --]

From 819f07476e8f67acb86b90eab2ea03b7a3d17e33 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Taylan=20Ulrich=20Bay=C4=B1rl=C4=B1/Kammer?=
 <taylanbayirli@gmail.com>
Date: Wed, 21 Sep 2016 22:30:39 +0200
Subject: [PATCH] Fix compilation on Guile 2.2.

---
 build-aux/compile-all.scm   | 1 +
 gnu/system/file-systems.scm | 2 +-
 gnu/system/shadow.scm       | 3 +--
 guix/build-system/waf.scm   | 5 +++--
 4 files changed, 6 insertions(+), 5 deletions(-)

diff --git a/build-aux/compile-all.scm b/build-aux/compile-all.scm
index 7c937a0..577e6bc 100644
--- a/build-aux/compile-all.scm
+++ b/build-aux/compile-all.scm
@@ -80,6 +80,7 @@
   ((_ . files)
    (let ((files (filter file-needs-compilation? files)))
      (for-each load-module-file files)
+     (compile 'dummy)        ;make sure compilation related modules are loaded
      (let ((mutex (make-mutex)))
        (par-for-each (lambda (file)
                        (compile-file* file mutex))
diff --git a/gnu/system/file-systems.scm b/gnu/system/file-systems.scm
index 0dc472e..2e59e48 100644
--- a/gnu/system/file-systems.scm
+++ b/gnu/system/file-systems.scm
@@ -158,7 +158,7 @@ TARGET in the other system."
     (type "binfmt_misc")
     (check? #f)))
 
-(define %tty-gid
+(define-public %tty-gid
   ;; ID of the 'tty' group.  Allocate it statically to make it easy to refer
   ;; to it from here and from the 'tty' group definitions.
   996)
diff --git a/gnu/system/shadow.scm b/gnu/system/shadow.scm
index cfdcf5e..b873021 100644
--- a/gnu/system/shadow.scm
+++ b/gnu/system/shadow.scm
@@ -24,8 +24,7 @@
   #:use-module (guix sets)
   #:use-module (guix ui)
   #:use-module (gnu services)
-  #:use-module ((gnu system file-systems)
-                #:select (%tty-gid))
+  #:use-module (gnu system file-systems)
   #:use-module ((gnu packages admin)
                 #:select (shadow))
   #:use-module (gnu packages bash)
diff --git a/guix/build-system/waf.scm b/guix/build-system/waf.scm
index 044d2a0..ccdf05c 100644
--- a/guix/build-system/waf.scm
+++ b/guix/build-system/waf.scm
@@ -24,14 +24,15 @@
   #:use-module (guix search-paths)
   #:use-module (guix build-system)
   #:use-module (guix build-system gnu)
-  #:use-module ((guix build-system python)
-                #:select (default-python default-python2))
   #:use-module (ice-9 match)
   #:use-module (srfi srfi-26)
   #:export (%waf-build-system-modules
             waf-build
             waf-build-system))
 
+(define default-python (@@ (guix build-system python) default-python))
+(define default-python2 (@@ (guix build-system python) default-python2))
+
 ;; Commentary:
 ;;
 ;; Standard build procedure for applications using 'waf'.  This is very
-- 
2.10.0

Re: Building Guix with Guile 2.1

[Andy Wingo]

On Wed 21 Sep 2016 23:01, taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:

> With the attached quick-and-dirty patch, 'make' runs to completion.

LGTM.

> I think we can keep the (compile 'dummy) hack.  That leaves two issues
> which may be solved in a cleaner manner than in this patch:
>
> - The (define foo (@@ (bar) foo)) parts.
> - Making %tty-gid public.  (The above didn't work for this one...)
>
> Both fixes may become unnecessary if Guile 2.2 goes back to allowing
> #:select to import private bindings.  Otherwise, recommendations for
> cleaner solutions welcome.

I think allowing access to private bindings via #:select was simply an
error and is unlikely to be reinstated.

> By the way, compile time seems to increase greatly with 2.2, to the
> point I wondered if it's really compiling in parallel, but it does seem
> to as evidenced by top(1).  Maybe package modules could be compiled with
> certain optims turned off, since they mostly just consist of package
> object definitions and not procedures whose performance would matter.

How much?

I think turning off most optimizations for the packages is a good idea.
There is not a nice way to do this however.  What `guild compile -O1
...` does is:

  http://git.savannah.gnu.org/gitweb/?p=guile.git;a=blob;f=module/scripts/compile.scm;h=939fb2564ec344de5f4a531b2041383730262d4f;hb=HEAD#l55

The default is -O2.

Andy

Re: Building Guix with Guile 2.1

[Taylan Ulrich Bayırlı/Kammer]

Andy Wingo <wingo@igalia.com> writes:

> On Wed 21 Sep 2016 23:01, taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:
>
>> By the way, compile time seems to increase greatly with 2.2, to the
>> point I wondered if it's really compiling in parallel, but it does seem
>> to as evidenced by top(1).  Maybe package modules could be compiled with
>> certain optims turned off, since they mostly just consist of package
>> object definitions and not procedures whose performance would matter.
>
> How much?

Running make, then make clean-go, then 'time make', we get:

Guile 2.0:
real	2m46.405s
user	6m39.044s
sys	0m2.140s

Guile 2.2:
real	31m44.433s
user	84m32.060s
sys	0m10.880s

Does that look too extreme?  Maybe there's another issue.

> I think turning off most optimizations for the packages is a good idea.
> There is not a nice way to do this however.  What `guild compile -O1
> ...` does is:
>
>   http://git.savannah.gnu.org/gitweb/?p=guile.git;a=blob;f=module/scripts/compile.scm;h=939fb2564ec344de5f4a531b2041383730262d4f;hb=HEAD#l55
>
> The default is -O2.

Thanks for the pointer.  I tried incorporating that the following way;
tell me if there's a mistake:

(The middle hunk is unaltered code copied from scripts/compile.scm.)

--- snip ---
@@ -19,6 +19,8 @@
 
 (use-modules (system base target)
              (system base message)
+             (language tree-il optimize)
+             (language cps optimize)
              (ice-9 match)
              (ice-9 threads)
              (guix build utils))
@@ -63,6 +65,19 @@
     (format #t "  LOAD     ~a~%" module)
     (resolve-interface module)))
 
+(define (available-optimizations)
+  (append (tree-il-default-optimization-options)
+          (cps-default-optimization-options)))
+
+(define (optimizations-for-level level)
+  (let lp ((options (available-optimizations)))
+    (match options
+      (() '())
+      ((#:partial-eval? val . options)
+       (cons* #:partial-eval? (> level 0) (lp options)))
+      ((kw val . options)
+       (cons* kw (> level 1) (lp options))))))
+
 (define (compile-file* file output-mutex)
   (let ((go (scm->go file)))
     (with-mutex output-mutex
@@ -74,7 +89,8 @@
         (lambda ()
           (compile-file file
                         #:output-file go
-                        #:opts `(#:warnings ,warnings)))))))
+                        #:opts `(#:warnings ,warnings
+                                 @,(optimizations-for-level 1))))))))
 
 (match (command-line)
   ((_ . files)
--- snip ---

Using optim level 1, compilation takes the same amount of time *and* I
get a segfault at the end.  When re-running make, it finishes by
compiling only gnu/packages/python.go (indicating that all other .go
files were compiled successfully on the first run), and this time
succeeds without a segfault.

Using optim level 0, it seems to hang at gnu/packages/shells.go.  (More
precisely, I aborted after a total of 118 minutes, most of which was
spent waiting for shells.go to finish.)

So much for today; I'll see that I get a backtrace from the segfault
later.

Taylan

Re: Building Guix with Guile 2.1

[Ludovic Courtès]

Hi!

Andy Wingo <wingo@igalia.com> skribis:

> On Wed 21 Sep 2016 10:23, ludo@gnu.org (Ludovic Courtès) writes:
>
>> Hello!
>>
>> Nalaginrut reported that Guix fails to build with Guile 2.2, which was a
>> bit of a shame, hence commits e465d9e19087ab150f7e31f21c09e4a147b93b36
>> and 9d126aa2b504bb9fad536eac186805ff623e96be.
>>
>> Now, the hack in build-aux/compile-all.scm doesn’t quite work with 2.2,
>> and it was already quite fragile.
>
> Which hack?  You mean loading modules before compiling?  I guess you
> should load the compiler modules too FWIW, perhaps that's an issue.
>
> If there is a nice change we can do to make module-loading thread-safe,
> let's think about that :)  It's probably the biggest thread-safety
> problem we have in Guile and now would be the right time to fix it.

I agree this is the right thing to do.

My guess is that this would have to be approached in C, to avoid the fat
mutex overhead.  But hmm, I dunno exactly what this entails!

Ludo’.

Re: Building Guix with Guile 2.1

[Ludovic Courtès]

taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:

> I think we can keep the (compile 'dummy) hack.  That leaves two issues
> which may be solved in a cleaner manner than in this patch:
>
> - The (define foo (@@ (bar) foo)) parts.
> - Making %tty-gid public.  (The above didn't work for this one...)

Fixed in 3c185b24f593c982aeb33996324fa6878c6ed21b, thanks for reporting
it!

Ludo’.

Re: Building Guix with Guile 2.1

[Taylan Ulrich Bayırlı/Kammer]

[-- Attachment #1: Type: text/plain, Size: 695 bytes --]

ludo@gnu.org (Ludovic Courtès) writes:

> taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:
>
>> I think we can keep the (compile 'dummy) hack.  That leaves two issues
>> which may be solved in a cleaner manner than in this patch:
>>
>> - The (define foo (@@ (bar) foo)) parts.
>> - Making %tty-gid public.  (The above didn't work for this one...)
>
> Fixed in 3c185b24f593c982aeb33996324fa6878c6ed21b, thanks for reporting
> it!

Thanks for the fixes. :-)

I haven't made much progress regarding the -O1 setting to speed up
building, but for the time being here's the patch that would make the
build process work at all on 2.2, including for 'guix pull'.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-build-Improve-Guile-2.2-compatibility.patch --]
[-- Type: text/x-diff, Size: 1633 bytes --]

From 91d8e2f426f442de094166d724e91cc24e6cbd96 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Taylan=20Ulrich=20Bay=C4=B1rl=C4=B1/Kammer?=
 <taylanbayirli@gmail.com>
Date: Tue, 27 Sep 2016 22:34:06 +0200
Subject: [PATCH] build: Improve Guile 2.2 compatibility.

* build-aux/compile-all.scm (compile-file*): Ensure loading of
  compilation related modules before going parallel.
* guix/build/pull.scm (build-guix): Ditto.
---
 build-aux/compile-all.scm | 3 +++
 guix/build/pull.scm       | 3 +++
 2 files changed, 6 insertions(+)

diff --git a/build-aux/compile-all.scm b/build-aux/compile-all.scm
index 7c937a0..46b3817 100644
--- a/build-aux/compile-all.scm
+++ b/build-aux/compile-all.scm
@@ -81,6 +81,9 @@
    (let ((files (filter file-needs-compilation? files)))
      (for-each load-module-file files)
      (let ((mutex (make-mutex)))
+       ;; Make sure compilation related modules are loaded before starting to
+       ;; compile files in parallel.
+       (compile #f)
        (par-for-each (lambda (file)
                        (compile-file* file mutex))
                      files)))))
diff --git a/guix/build/pull.scm b/guix/build/pull.scm
index ccf1868..871bf6f 100644
--- a/guix/build/pull.scm
+++ b/guix/build/pull.scm
@@ -125,6 +125,9 @@ containing the source code.  Write any debugging output to DEBUG-PORT."
       (newline)
       (let ((mutex (make-mutex))
             (completed 0))
+        ;; Make sure compilation related modules are loaded before starting to
+        ;; compile files in parallel.
+        (compile #f)
         (par-for-each
          (lambda (file)
            (with-mutex mutex
-- 
2.10.0

Re: Building Guix with Guile 2.1

[Andy Wingo]

On Wed 28 Sep 2016 16:56, taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:

> I haven't made much progress regarding the -O1 setting to speed up
> building, but for the time being here's the patch that would make the
> build process work at all on 2.2, including for 'guix pull'.

Patch LGTM.

Re: Building Guix with Guile 2.1

[Taylan Ulrich Bayırlı/Kammer]

Andy Wingo <wingo@igalia.com> writes:

> On Wed 28 Sep 2016 16:56, taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:
>
>> I haven't made much progress regarding the -O1 setting to speed up
>> building, but for the time being here's the patch that would make the
>> build process work at all on 2.2, including for 'guix pull'.
>
> Patch LGTM.

Thanks!  Pushed.

Taylan

Re: Building Guix with Guile 2.1

[Ludovic Courtès]

Hi!

taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:

> Andy Wingo <wingo@igalia.com> writes:
>
>> On Wed 21 Sep 2016 23:01, taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:
>>
>>> By the way, compile time seems to increase greatly with 2.2, to the
>>> point I wondered if it's really compiling in parallel, but it does seem
>>> to as evidenced by top(1).  Maybe package modules could be compiled with
>>> certain optims turned off, since they mostly just consist of package
>>> object definitions and not procedures whose performance would matter.
>>
>> How much?
>
> Running make, then make clean-go, then 'time make', we get:
>
> Guile 2.0:
> real	2m46.405s
> user	6m39.044s
> sys	0m2.140s
>
> Guile 2.2:
> real	31m44.433s
> user	84m32.060s
> sys	0m10.880s

[...]

> Using optim level 1, compilation takes the same amount of time *and* I
> get a segfault at the end.  When re-running make, it finishes by
> compiling only gnu/packages/python.go (indicating that all other .go
> files were compiled successfully on the first run), and this time
> succeeds without a segfault.
>
> Using optim level 0, it seems to hang at gnu/packages/shells.go.  (More
> precisely, I aborted after a total of 118 minutes, most of which was
> spent waiting for shells.go to finish.)

This sounds a bit concerning.  Any idea what’s going on?

I think we should make sure we have a way to build with 2.2 in at most
the same amount of time as with 2.0.  (Easier said than done!  ;-))

Ludo’.