From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
To: "Ludovic Courtès" <ludo@gnu.org>
Cc: guix-devel@gnu.org
Subject: Re: [PATCH] build: Speed up .go compilation.
Date: Mon, 11 Jan 2016 22:47:49 +0100 [thread overview]
Message-ID: <874mejwzy2.fsf@T420.taylan> (raw)
In-Reply-To: <8737u3g72u.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Mon, 11 Jan 2016 22:05:45 +0100")
[-- Attachment #1: Type: text/plain, Size: 1548 bytes --]
ludo@gnu.org (Ludovic Courtès) writes:
> Mathieu Lirzin <mthl@gnu.org> skribis:
>
>>>>> # Unset 'GUILE_LOAD_COMPILED_PATH' altogether while compiling. Otherwise, if
>>>>> # $GUILE_LOAD_COMPILED_PATH contains $(moduledir), we may find .go files in
>>>>> # there that are newer than the local .scm files (for instance because the
>>>>> @@ -358,14 +346,16 @@ GUILD_COMPILE_FLAGS = \
>>>>> #
>>>>> # XXX: Use the C locale for when Guile lacks
>>>>> # <http://git.sv.gnu.org/cgit/guile.git/commit/?h=stable-2.0&id=e2c6bf3866d1186c60bacfbd4fe5037087ee5e3f>.
>>>> ^^^
>>>>
>>>>> -.scm.go:
>>>>> - $(AM_V_GUILEC)$(MKDIR_P) `dirname "$@"` ; \
>>>>> +%.go: make-go ; @:
>>>>> +make-go: $(MODULES) guix/config.scm guix/tests.scm
>>>>> + @echo "Compiling Scheme modules..." ; \
>>>>> unset GUILE_LOAD_COMPILED_PATH ; \
>>>>> LC_ALL=C \
>>>> ^^^
>>>>
>>>> This is present because (scripts compile) from "old" Guile doesn't do it
>>>> automatically. What about copying the code from the link above in
>>>> compile-all.scm and removing this from Makefile.am ?
>>>
>>> I should be using the whole (catch ...) expression, right? Done, thanks
>>> for the heads up.
>>
>> Yes I suppose. Maybe Ludo can confirm?
>
> It’s unnecessary to even call ‘setlocale’ in compile-all.scm because we
> don’t rely on anything locale-specific. So there’s no problem.
>
> The LC_ALL=C line can also be removed from Makefile.am.
In light of this and the other changes, here's a new patch:
[-- Attachment #2: 0001-build-Speed-up-.go-compilation.patch --]
[-- Type: text/x-diff, Size: 6581 bytes --]
From 21cb57ce693467faee3ed2dfd48d7676f7d58fd5 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Taylan=20Ulrich=20Bay=C4=B1rl=C4=B1/Kammer?=
<taylanbayirli@gmail.com>
Date: Thu, 5 Nov 2015 23:42:45 +0100
Subject: [PATCH] build: Speed up .go compilation.
* build-aux/compile-all.scm: New file.
* Makefile.am (EXTRA_DIST): Add it.
(%.go, make-go): New rules.
---
Makefile.am | 32 ++++++------------
build-aux/compile-all.scm | 82 +++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 91 insertions(+), 23 deletions(-)
create mode 100644 build-aux/compile-all.scm
diff --git a/Makefile.am b/Makefile.am
index 760caed..dd98564 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -152,11 +152,6 @@ endif BUILD_DAEMON_OFFLOAD
# Internal module with test suite support.
dist_noinst_DATA = guix/tests.scm
-# Because of the autoload hack in (guix build download), we must build it
-# first to avoid errors on systems where (gnutls) is unavailable.
-guix/scripts/download.go: guix/build/download.go
-guix/download.go: guix/build/download.go
-
# Linux-Libre configurations.
KCONFIGS = \
gnu/packages/linux-libre-i686.conf \
@@ -302,6 +297,7 @@ EXTRA_DIST = \
CODE-OF-CONDUCT \
.dir-locals.el \
build-aux/build-self.scm \
+ build-aux/compile-all.scm \
build-aux/hydra/gnu-system.scm \
build-aux/hydra/demo-os.scm \
build-aux/hydra/guix.scm \
@@ -341,31 +337,21 @@ CLEANFILES = \
$(GOBJECTS) \
$(SCM_TESTS:tests/%.scm=%.log)
-AM_V_GUILEC = $(AM_V_GUILEC_$(V))
-AM_V_GUILEC_ = $(AM_V_GUILEC_$(AM_DEFAULT_VERBOSITY))
-AM_V_GUILEC_0 = @echo " GUILEC" $@;
-
-# Flags passed to 'guild compile'.
-GUILD_COMPILE_FLAGS = \
- -Wformat -Wunbound-variable -Warity-mismatch
-
# Unset 'GUILE_LOAD_COMPILED_PATH' altogether while compiling. Otherwise, if
# $GUILE_LOAD_COMPILED_PATH contains $(moduledir), we may find .go files in
# there that are newer than the local .scm files (for instance because the
# user ran 'make install' recently). When that happens, we end up loading
# those previously-installed .go files, which may be stale, thereby breaking
# the whole thing.
-#
-# XXX: Use the C locale for when Guile lacks
-# <http://git.sv.gnu.org/cgit/guile.git/commit/?h=stable-2.0&id=e2c6bf3866d1186c60bacfbd4fe5037087ee5e3f>.
-.scm.go:
- $(AM_V_GUILEC)$(MKDIR_P) `dirname "$@"` ; \
+%.go: make-go ; @:
+make-go: $(MODULES) guix/config.scm guix/tests.scm
+ $(AM_V_at)echo "Compiling Scheme modules..." ; \
unset GUILE_LOAD_COMPILED_PATH ; \
- LC_ALL=C \
+ host=$(host) srcdir="$(top_srcdir)" \
$(top_builddir)/pre-inst-env \
- $(GUILD) compile -L "$(top_builddir)" -L "$(top_srcdir)" \
- $(GUILD_COMPILE_FLAGS) --target="$(host)" \
- -o "$@" "$<"
+ $(GUILE) -L "$(top_builddir)" -L "$(top_srcdir)" \
+ --no-auto-compile \
+ -s "$(top_srcdir)"/build-aux/compile-all.scm $^
SUFFIXES = .go
@@ -457,6 +443,6 @@ assert-final-inputs-self-contained:
$(top_builddir)/pre-inst-env "$(GUILE)" \
"$(top_srcdir)/build-aux/check-final-inputs-self-contained.scm"
-.PHONY: sync-descriptions gen-ChangeLog gen-AUTHORS clean-go
+.PHONY: sync-descriptions gen-ChangeLog gen-AUTHORS clean-go make-go
.PHONY: assert-no-store-file-names assert-binaries-available
.PHONY: assert-final-inputs-self-contained
diff --git a/build-aux/compile-all.scm b/build-aux/compile-all.scm
new file mode 100644
index 0000000..e0877db
--- /dev/null
+++ b/build-aux/compile-all.scm
@@ -0,0 +1,82 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2016 Taylan Ulrich Bayırlı/Kammer <taylanbayirli@gmail.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(use-modules (system base target)
+ (ice-9 match)
+ (ice-9 threads)
+ (guix build utils))
+
+(define compile-options '(format unbound-variable arity-mismatch))
+
+(define host (getenv "host"))
+
+(define srcdir (getenv "srcdir"))
+
+(define (relative-file file)
+ (if (string-prefix? (string-append srcdir "/") file)
+ (string-drop file (+ 1 (string-length srcdir)))
+ file))
+
+(define (file-mtime<? f1 f2)
+ (< (stat:mtime (stat f1))
+ (stat:mtime (stat f2))))
+
+(define (scm->go file)
+ (let* ((relative (relative-file file))
+ (without-extension (string-drop-right relative 4)))
+ (string-append without-extension ".go")))
+
+(define (file-needs-compilation? file)
+ (let ((go (scm->go file)))
+ (or (not (file-exists? go))
+ (file-mtime<? go file))))
+
+(define (file->module file)
+ (let* ((relative (relative-file file))
+ (module-path (string-drop-right relative 4)))
+ (map string->symbol
+ (string-split module-path #\/))))
+
+;;; To work around <http://bugs.gnu.org/15602> (FIXME), we want to load all
+;;; files to be compiled first. We do this via resolve-interface so that the
+;;; top-level of each file (module) is only executed once.
+(define (load-module-file file)
+ (let ((module (file->module file)))
+ (format #t " LOAD ~a~%" module)
+ (resolve-interface module)))
+
+(define (compile-file* file output-mutex)
+ (let ((go (scm->go file)))
+ (with-mutex output-mutex
+ (format #t " GUILEC ~a~%" go)
+ (force-output))
+ (mkdir-p (dirname go))
+ (with-target host
+ (lambda ()
+ (compile-file file
+ #:output-file go
+ #:opts compile-options)))))
+
+(match (command-line)
+ ((_ . files)
+ (let ((files (filter file-needs-compilation? files)))
+ (for-each load-module-file files)
+ (let ((mutex (make-mutex)))
+ (par-for-each (lambda (file)
+ (compile-file* file mutex))
+ files)))))
--
2.6.3
next prev parent reply other threads:[~2016-01-11 21:47 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-11-12 16:41 [PATCH] Makefile: Speed up .go compilation Taylan Ulrich Bayırlı/Kammer
2016-01-08 11:48 ` [PATCH] build: " Taylan Ulrich Bayırlı/Kammer
2016-01-08 17:06 ` Ludovic Courtès
2016-01-09 19:38 ` Taylan Ulrich Bayırlı/Kammer
2016-01-09 21:59 ` Ludovic Courtès
2016-01-10 10:24 ` Taylan Ulrich Bayırlı/Kammer
2016-01-10 17:01 ` Mathieu Lirzin
2016-01-10 20:46 ` Taylan Ulrich Bayırlı/Kammer
2016-01-11 21:16 ` Ludovic Courtès
2016-01-10 13:34 ` Taylan Ulrich Bayırlı/Kammer
2016-01-10 15:11 ` Taylan Ulrich Bayırlı/Kammer
2016-01-10 17:27 ` Mathieu Lirzin
2016-01-10 20:52 ` Taylan Ulrich Bayırlı/Kammer
2016-01-10 21:18 ` Mathieu Lirzin
2016-01-11 21:05 ` Ludovic Courtès
2016-01-11 21:47 ` Taylan Ulrich Bayırlı/Kammer [this message]
2016-01-10 16:47 ` Mark H Weaver
2016-01-10 20:33 ` Taylan Ulrich Bayırlı/Kammer
2016-01-11 21:14 ` Ludovic Courtès
2016-01-11 21:56 ` Taylan Ulrich Bayırlı/Kammer
2016-01-14 14:02 ` Ludovic Courtès
2016-01-17 20:16 ` Ludovic Courtès
2016-01-18 8:05 ` Taylan Ulrich Bayırlı/Kammer
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://guix.gnu.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=874mejwzy2.fsf@T420.taylan \
--to=taylanbayirli@gmail.com \
--cc=guix-devel@gnu.org \
--cc=ludo@gnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).