From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Andy Wingo Newsgroups: gmane.lisp.guile.user,gmane.comp.gnu.lilypond.devel Subject: Re: Generating compiled scm (.go) files as part of LilyPond build Date: Wed, 20 Jul 2011 19:38:25 +0200 Message-ID: <87vcuwj1im.fsf@pobox.com> References: <4CF14302.1020502@hulin.org.uk> <4CF6BC51.3090909@hulin.org.uk> <4D1C7061.1000701@hulin.org.uk> <4D433723.50501@hulin.org.uk> <4E25843A.2000803@hulin.org.uk> <8762myjqee.fsf@pobox.com> <4E260E62.40009@hulin.org.uk> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Trace: dough.gmane.org 1311229504 30099 80.91.229.12 (21 Jul 2011 06:25:04 GMT) X-Complaints-To: usenet@dough.gmane.org NNTP-Posting-Date: Thu, 21 Jul 2011 06:25:04 +0000 (UTC) Cc: guile-user@gnu.org, "lilypond-devel@gnu.org" To: Ian Hulin Original-X-From: guile-user-bounces+guile-user=m.gmane.org@gnu.org Thu Jul 21 08:24:58 2011 Return-path: Envelope-to: guile-user@m.gmane.org Original-Received: from lists.gnu.org ([140.186.70.17]) by lo.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1Qjmgr-0003FG-Oq for guile-user@m.gmane.org; Thu, 21 Jul 2011 08:24:58 +0200 Original-Received: from localhost ([::1]:44856 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Qjdtu-00061p-AW for guile-user@m.gmane.org; Wed, 20 Jul 2011 17:01:50 -0400 Original-Received: from eggs.gnu.org ([140.186.70.92]:56841) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1QjajK-0001Vw-Ut for guile-user@gnu.org; Wed, 20 Jul 2011 13:38:48 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1QjajF-0007fB-5P for guile-user@gnu.org; Wed, 20 Jul 2011 13:38:42 -0400 Original-Received: from a-pb-sasl-sd.pobox.com ([74.115.168.62]:47271 helo=sasl.smtp.pobox.com) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1QjajE-0007dj-TH; Wed, 20 Jul 2011 13:38:37 -0400 Original-Received: from sasl.smtp.pobox.com (unknown [127.0.0.1]) by a-pb-sasl-sd.pobox.com (Postfix) with ESMTP id B57226DFC; Wed, 20 Jul 2011 13:38:29 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=pobox.com; h=from:to:cc :subject:references:date:in-reply-to:message-id:mime-version :content-type:content-transfer-encoding; s=sasl; bh=s9LAEYuItgEX g3cePetLmyMzlWM=; b=VrZiRPGE1baSo73vf7hVKO7+k/NNQ3fBeuPtCVjewFTN MYHB2CMzy55+1KxtousFcYiKfs5QSyPjT5x+touXWFYJvxUFuIXcLVigWN8xRop1 /3gLHZ09BlnFkIBOJ7oYmeADk1zEINr/zFNMQmYzwc80HVOBmcvDs8atrf+9BJI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=pobox.com; h=from:to:cc :subject:references:date:in-reply-to:message-id:mime-version :content-type:content-transfer-encoding; q=dns; s=sasl; b=tUke8U Dk634B+wm20ZR2hQmKWWVBwjlOHa5I2THlnC+wuBIQ+EZy34rMBNakAZt4H/c5D6 MRkGNDlLElW+VT58sVEDO7//rFAjyOJkXHe4mASYpmrr7juOEBYo0YVVCI5MX6C6 VqPRH2656wIuYP7r4D3AvFE52ysH45Z9q2alU= Original-Received: from a-pb-sasl-sd.pobox.com (unknown [127.0.0.1]) by a-pb-sasl-sd.pobox.com (Postfix) with ESMTP id AA3716DFB; Wed, 20 Jul 2011 13:38:29 -0400 (EDT) Original-Received: from badger (unknown [90.164.198.39]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by a-pb-sasl-sd.pobox.com (Postfix) with ESMTPSA id E2F6B6DFA; Wed, 20 Jul 2011 13:38:28 -0400 (EDT) In-Reply-To: <4E260E62.40009@hulin.org.uk> (Ian Hulin's message of "Wed, 20 Jul 2011 00:08:18 +0100") User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.3 (gnu/linux) X-Pobox-Relay-ID: 147307D0-B2F7-11E0-AE1A-B797DE995924-02397024!a-pb-sasl-sd.pobox.com X-detected-operating-system: by eggs.gnu.org: Solaris 10 (beta) X-Received-From: 74.115.168.62 X-BeenThere: guile-user@gnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: General Guile related discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-user-bounces+guile-user=m.gmane.org@gnu.org Original-Sender: guile-user-bounces+guile-user=m.gmane.org@gnu.org Xref: news.gmane.org gmane.lisp.guile.user:8690 gmane.comp.gnu.lilypond.devel:38062 Greets, > On Tue 19 Jul 2011 15:28:41 BST, Andy Wingo wrote: >> On Tue 19 Jul 2011 15:18, Ian Hulin writes: >>=20 >>> It may boil down to a matter of taste, but I find double and >>> triple extensions on a filename intrinsically nasty. I've normally >>> come across them before on Windows systems where a filename such as >>> thing.htm.exe usually means there's malware or a trojan or suchlike >>> on your system. See also comments below. >>=20 >> Consider it an opaque key into a cache. We could change in the >> future to default to the SHA1 of the whole path. That does have >> some advantages regarding flattening the directory hierarchy, >> resulting in fewer stat operations. >>=20 > Sorry, but how does this benefit LilyPond as a user of the Guile API? > I'm used to viewing file extensions as an indication of the format of > the file content. In our simple view scheme sources are *.scm, and > compiled scheme files are *.go. Not sure I understand here; LilyPond would benefit from such a scheme due to having various operations be faster, and by continuing to not have to worry about the form of these file names. > Guile documentation implies something similar (Chap 4 Environment > Variable sections on GUILE_AUTO_COMPILE and GUILE_LOAD_COMPILED_PATH) > > "If a compiled (=E2=80=98.go=E2=80=99) file corresponding to a =E2=80=98.= scm=E2=80=99 file is not found > or is not newer than the =E2=80=98.scm=E2=80=99 file, the =E2=80=98.scm= =E2=80=99 file will be compiled > on the fly, and the > resulting =E2=80=98.go=E2=80=99 file stored away. An advisory note will b= e printed on > the console." > > "GUILE_LOAD_COMPILED_PATH > This variable may be used to augment the path that is searched for > compiled Scheme files (=E2=80=98.go=E2=80=99 files) when loading" > > Neither of these sections leads the reader to expect an actual file name > generated to be > ;;; compiled > /long/path/name/testing.scm.go > ^^^^^^^ I think that we need to be clear here, that Guile really cannot commit to the form of the names of auto-compiled files. That is not part of Guile's API. The reason for a simple string-append is that it is very possible to compile a file named /long/path/name/testing *and also* a file named /long/path/name/testing.scm. Appending a suffix is a simple and effective means to ensuring a unique cache key. >>> ian@nanny-ogg:~/src/Guile/guile-2.0.2$ meta/guile -L $PWD >>=20 >> This will set XDG_CACHE_HOME=3D${top_builddir}/cache. >>=20 > The Guile Reference Manual Chapter 4 p 38 specifically says this > defaults to $HOME/cache Not when running uninstalled, with meta/guile. Many things are different in that case. >>> The problem is knowing where the cache is. For Lilypond, we need to >>> have a common root directory off of which we can hang the compiled >>> files in scm/out. >>=20 >> Why do you care? (Honest question.) >>=20 > (Honest answer) > Because if Lilypond is run up with the Guile default of --auto-compile > it will check for and generate compiled files in a different place to > where we want to have produced them at build-time. Also once for a > project like LilyPond these files would need to be in a shared directory > on the file system. Relying on the default compiled file spec would have > a set of cached files somewhere on each user's cache in their home > directory. I don't understand why this isn't working for you. I think we need to be very specific. Let's look at am/guilec from Guile itself. It has: # -*- makefile -*- GOBJECTS =3D $(SOURCES:%.scm=3D%.go) GUILE_WARNINGS =3D -Wunbound-variable -Warity-mismatch -Wformat moddir =3D $(pkgdatadir)/$(GUILE_EFFECTIVE_VERSION)/$(modpath) nobase_mod_DATA =3D $(SOURCES) $(NOCOMP_SOURCES) ccachedir =3D $(pkglibdir)/$(GUILE_EFFECTIVE_VERSION)/ccache/$(modpath) nobase_ccache_DATA =3D $(GOBJECTS) EXTRA_DIST =3D $(SOURCES) $(NOCOMP_SOURCES) ETAGS_ARGS =3D $(SOURCES) $(NOCOMP_SOURCES) CLEANFILES =3D $(GOBJECTS) # Make sure source files are installed first, so that the mtime of # installed compiled files is greater than that of installed source # files. See # # for details. guile_install_go_files =3D install-nobase_ccacheDATA $(guile_install_go_files): install-nobase_modDATA AM_V_GUILEC =3D $(AM_V_GUILEC_$(V)) AM_V_GUILEC_ =3D $(AM_V_GUILEC_$(AM_DEFAULT_VERBOSITY)) AM_V_GUILEC_0 =3D @echo " GUILEC" $@; SUFFIXES =3D .scm .go .scm.go: $(AM_V_GUILEC)GUILE_AUTO_COMPILE=3D0 \ $(top_builddir)/meta/uninstalled-env \ guild compile $(GUILE_WARNINGS) -o "$@" "$<" Then module/Makefile.am just has to: include $(top_srcdir)/am/guilec # We're at the root of the module hierarchy. modpath =3D SOURCES =3D \ ice-9/psyntax-pp.scm \ ice-9/boot-9.scm \ ... That will result in .go files getting compiled for all Scheme files. They are compiled in an "uninstalled env", which adds the srcdir to the GUILE_LOAD_PATH, and the builddir to the GUILE_LOAD_COMPILED_PATH, while things are being compiled. The bit turning off auto-compilation isn't needed for packages outside of Guile itself; Guile only needs it because its compiler is written in Scheme. The .go files are installed to the correct place. You would probably use $(datadir)/guile/site/$(modpath) instead. The .go files would get installed to $(libdir)/guile/$(GUILE_EFFECTIVE_VERSION)/ccache/$(modpath). They are shared among users. They will be seen as fresh; users will not cause auto-compilation, except for Scheme executable scripts, which, since they are not searched for in a path, don't have a place for their compiled files to go. But that is a minor concern compared to what you're working with right now. >> To me there are two cases: >>=20 >> 1) You compile the files yourself. You either ensure that the .go=20 >> files end up in the right places, or adjust=20 >> GUILE_LOAD_COMPILED_PATH. >>=20 > But GUILE_LOAD_COMPILED_PATH setting won't prevent the vanilla > compiled-file-name from checking in the wrong place for at compile time. If Guile loads a file from the load path, and finds an up-to-date .go file in the GUILE_LOAD_COMPILED_PATH, it will not look in the fallback path. > OK we can brew our own expectation of where the file will be, and > provide our own ly:load and intelligence to pick up the compiled file, > but we have to do this in conjunction with a > scm_putenv("GUILE_AUTO_COMPILE=3D0") call in our initialization code. You certainly might need to do this, in the end. It's a possibility. Hopefully not, though. Does this clarify things for you? Regards, Andy --=20 http://wingolog.org/