From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Thien-Thi Nguyen Newsgroups: gmane.lisp.guile.user Subject: Re: Modules Date: Sun, 30 Jan 2011 11:13:56 +0100 Message-ID: <87vd164s2j.fsf@ambire.localdomain> References: <20101208124502.5f25b64d@halmanfloyd> <20110129131315.506f1e8c@halmanfloyd> <87tygrcnbf.fsf@ossau.uklinux.net> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Trace: dough.gmane.org 1296382791 4618 80.91.229.12 (30 Jan 2011 10:19:51 GMT) X-Complaints-To: usenet@dough.gmane.org NNTP-Posting-Date: Sun, 30 Jan 2011 10:19:51 +0000 (UTC) To: guile-user@gnu.org Original-X-From: guile-user-bounces+guile-user=m.gmane.org@gnu.org Sun Jan 30 11:19:46 2011 Return-path: Envelope-to: guile-user@m.gmane.org Original-Received: from lists.gnu.org ([199.232.76.165]) by lo.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1PjUNl-00010z-3d for guile-user@m.gmane.org; Sun, 30 Jan 2011 11:19:45 +0100 Original-Received: from localhost ([127.0.0.1]:47558 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1PjUNk-0001EX-E6 for guile-user@m.gmane.org; Sun, 30 Jan 2011 05:19:44 -0500 Original-Received: from [140.186.70.92] (port=52819 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1PjUNg-0001ES-2A for guile-user@gnu.org; Sun, 30 Jan 2011 05:19:41 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1PjUNe-0002Vy-Kw for guile-user@gnu.org; Sun, 30 Jan 2011 05:19:39 -0500 Original-Received: from smtp208.alice.it ([82.57.200.104]:44310) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1PjUNe-0002Vq-3E for guile-user@gnu.org; Sun, 30 Jan 2011 05:19:38 -0500 Original-Received: from ambire.localdomain (79.0.68.227) by smtp208.alice.it (8.5.124.08) id 4C1A271610A468E2 for guile-user@gnu.org; Sun, 30 Jan 2011 11:19:36 +0100 Original-Received: from ttn by ambire.localdomain with local (Exim 4.69) (envelope-from ) id 1PjUI8-00011J-G7 for guile-user@gnu.org; Sun, 30 Jan 2011 11:13:56 +0100 In-Reply-To: <87tygrcnbf.fsf@ossau.uklinux.net> (Neil Jerram's message of "Sat, 29 Jan 2011 23:17:08 +0000") User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.0.50 (gnu/linux) X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 82.57.200.104 X-BeenThere: guile-user@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: General Guile related discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Original-Sender: guile-user-bounces+guile-user=m.gmane.org@gnu.org Errors-To: guile-user-bounces+guile-user=m.gmane.org@gnu.org Xref: news.gmane.org gmane.lisp.guile.user:8402 Archived-At: () Neil Jerram () Sat, 29 Jan 2011 23:17:08 +0000 If the modules are installed, that's true. What if they are not? [...] For scripts that use uninstalled modules, then, some kind of solution is needed; ideally one that works for both 1.8 and 1.9/2.0, allows the code needed to live in a single common file, rather than duplicated at the top of each script; and continues to work if the script+module tree as a whole is moved to a different place in the filesystem. This is what "module catalogs" from Guile 1.4.x provides. [verbose explanation follows, skip to end for summary] Basically, you associate with each element in =E2=80=98%load-path=E2=80=99 a sub-association between module name (list of symbols) and implementation resolution method. For Guile 1.4.x, there are two types of modules supported: - scheme source (text) - shared object library Of course the implementation must provide the appropriate interface (as defined by the support in =E2=80=98resolve-module=E2=80=99) f= or loading; this system cannot be used for arbitrary scheme source or shared object libraries. At load-time, the interpreter consults these catalogs preferentially, falling back to old-style filesystem groping on lookup failure. A module (of any form) need not include its location information. To see how this helps, i use two small scripts: st: #!/bin/sh exec strace -f -e open "$@" 2>&1 | grep -v o.such.file sta: #!/bin/sh exec strace -f -e open "$@" 2>&1 Here is a run of loading (database postgres-table) [scheme code] from the Guile-PG build tree. Note that it in turn requires (database postgres) [shared object library]. Also note =E2=80=98-L .=E2=80= =99 which is explained further down. $ cd ~/build/guile-pg/.b $ st guile -L . -c '(use-modules (database postgres-table))' open("/home/ttn/local/lib/libguile.so.9", O_RDONLY) =3D 3 open("/home/ttn/local/lib/libltdl.so.7", O_RDONLY) =3D 3 open("/etc/ld.so.cache", O_RDONLY) =3D 3 open("/lib/i686/cmov/libm.so.6", O_RDONLY) =3D 3 open("/lib/i686/cmov/libdl.so.2", O_RDONLY) =3D 3 open("/lib/i686/cmov/libc.so.6", O_RDONLY) =3D 3 open("/home/ttn/local/lib/guile/site/.module-catalog", O_RDONLY) =3D 3 open("/home/ttn/local/share/guile/site/.module-catalog", O_RDONLY) =3D 3 open("/home/ttn/local/lib/guile/1.4.1.122/.module-catalog", O_RDONLY) =3D= 3 open("/home/ttn/local/lib/guile/site/init.scm", O_RDONLY) =3D 3 open("./.module-catalog", O_RDONLY) =3D 3 open("/home/ttn/build/guile-pg/src/postgres-table.scm", O_RDONLY) =3D 3 open("/home/ttn/local/lib/guile/1.4.1.122/ice-9/common-list.scm", O_RDONL= Y) =3D 4 open("/home/ttn/build/guile-pg/.b/src/.libs/postgres.so.0.0.0", O_RDONLY)= =3D 4 open("/home/ttn/local/lib/libpq.so.3", O_RDONLY) =3D 4 open("/etc/ld.so.cache", O_RDONLY) =3D 4 open("/lib/i686/cmov/libcrypt.so.1", O_RDONLY) =3D 4 open("/lib/i686/cmov/libresolv.so.2", O_RDONLY) =3D 4 open("/lib/i686/cmov/libnsl.so.1", O_RDONLY) =3D 4 open("/home/ttn/build/guile-pg/src/postgres-types.scm", O_RDONLY) =3D 4 open("/home/ttn/build/guile-pg/src/postgres-col-defs.scm", O_RDONLY) =3D 4 open("/home/ttn/build/guile-pg/src/postgres-qcons.scm", O_RDONLY) =3D 4 open("/home/ttn/build/guile-pg/src/postgres-resx.scm", O_RDONLY) =3D 4 If i use =E2=80=98sta=E2=80=99, there are many "no such file" messages, but= NOT with the modules. Strictly speaking, open(2) savings proves improvement only for shared object libraries, so you'll have to trust me (or confirm by looking at the source code) that the scheme source modules are treated analogously (i.e., no probing). $ cd ~/build/guile-pg/.b $ st guile -L . -c '(use-modules (database postgres-table))' > A $ sta guile -L . -c '(use-modules (database postgres-table))' > B $ diff -u A B | uniq --- A 2011-01-30 10:41:25.000000000 +0100 +++ B 2011-01-30 10:41:32.000000000 +0100 @@ -1,8 +1,29 @@ +open("/home/ttn/local/lib/tls/i686/sse2/cmov/libguile.so.9", O_RDONLY) = =3D -1 ENOENT (No such file or directory) +open("/home/ttn/local/lib/tls/i686/sse2/libguile.so.9", O_RDONLY) =3D -1= ENOENT (No such file or directory) +open("/home/ttn/local/lib/tls/i686/cmov/libguile.so.9", O_RDONLY) =3D -1= ENOENT (No such file or directory) +open("/home/ttn/local/lib/tls/i686/libguile.so.9", O_RDONLY) =3D -1 ENOE= NT (No such file or directory) +open("/home/ttn/local/lib/tls/sse2/cmov/libguile.so.9", O_RDONLY) =3D -1= ENOENT (No such file or directory) +open("/home/ttn/local/lib/tls/sse2/libguile.so.9", O_RDONLY) =3D -1 ENOE= NT (No such file or directory) +open("/home/ttn/local/lib/tls/cmov/libguile.so.9", O_RDONLY) =3D -1 ENOE= NT (No such file or directory) +open("/home/ttn/local/lib/tls/libguile.so.9", O_RDONLY) =3D -1 ENOENT (N= o such file or directory) +open("/home/ttn/local/lib/i686/sse2/cmov/libguile.so.9", O_RDONLY) =3D -= 1 ENOENT (No such file or directory) +open("/home/ttn/local/lib/i686/sse2/libguile.so.9", O_RDONLY) =3D -1 ENO= ENT (No such file or directory) +open("/home/ttn/local/lib/i686/cmov/libguile.so.9", O_RDONLY) =3D -1 ENO= ENT (No such file or directory) +open("/home/ttn/local/lib/i686/libguile.so.9", O_RDONLY) =3D -1 ENOENT (= No such file or directory) +open("/home/ttn/local/lib/sse2/cmov/libguile.so.9", O_RDONLY) =3D -1 ENO= ENT (No such file or directory) +open("/home/ttn/local/lib/sse2/libguile.so.9", O_RDONLY) =3D -1 ENOENT (= No such file or directory) +open("/home/ttn/local/lib/cmov/libguile.so.9", O_RDONLY) =3D -1 ENOENT (= No such file or directory) open("/home/ttn/local/lib/libguile.so.9", O_RDONLY) =3D 3 open("/home/ttn/local/lib/libltdl.so.7", O_RDONLY) =3D 3 +open("/home/ttn/local/lib/libm.so.6", O_RDONLY) =3D -1 ENOENT (No such f= ile or directory) open("/etc/ld.so.cache", O_RDONLY) =3D 3 open("/lib/i686/cmov/libm.so.6", O_RDONLY) =3D 3 +open("/home/ttn/local/lib/libdl.so.2", O_RDONLY) =3D -1 ENOENT (No such = file or directory) open("/lib/i686/cmov/libdl.so.2", O_RDONLY) =3D 3 +open("/home/ttn/local/lib/libc.so.6", O_RDONLY) =3D -1 ENOENT (No such f= ile or directory) open("/lib/i686/cmov/libc.so.6", O_RDONLY) =3D 3 open("/home/ttn/local/lib/guile/site/.module-catalog", O_RDONLY) =3D 3 open("/home/ttn/local/share/guile/site/.module-catalog", O_RDONLY) =3D 3 @@ -13,9 +34,21 @@ open("/home/ttn/local/lib/guile/1.4.1.122/ice-9/common-list.scm", O_RDON= LY) =3D 4 open("/home/ttn/build/guile-pg/.b/src/.libs/postgres.so.0.0.0", O_RDONLY= ) =3D 4 open("/home/ttn/local/lib/libpq.so.3", O_RDONLY) =3D 4 +open("/home/ttn/local/lib/libcrypt.so.1", O_RDONLY) =3D -1 ENOENT (No su= ch file or directory) open("/etc/ld.so.cache", O_RDONLY) =3D 4 open("/lib/i686/cmov/libcrypt.so.1", O_RDONLY) =3D 4 +open("/home/ttn/local/lib/libresolv.so.2", O_RDONLY) =3D -1 ENOENT (No s= uch file or directory) open("/lib/i686/cmov/libresolv.so.2", O_RDONLY) =3D 4 +open("/home/ttn/local/lib/libnsl.so.1", O_RDONLY) =3D -1 ENOENT (No such= file or directory) open("/lib/i686/cmov/libnsl.so.1", O_RDONLY) =3D 4 open("/home/ttn/build/guile-pg/src/postgres-types.scm", O_RDONLY) =3D 4 open("/home/ttn/build/guile-pg/src/postgres-col-defs.scm", O_RDONLY) =3D= 4 The =E2=80=98-L .=E2=80=99 means that ~/build/guile-pg/.b/.module-catalog is consulted: ;;; .module-catalog ;;; generated 2010-12-18 20:35:15 UTC -- do not edit! =20=20 ( ((database postgres-resdisp) . "/home/ttn/build/guile-pg/src/postgres-re= sdisp.scm") ((database postgres-col-defs) . "/home/ttn/build/guile-pg/src/postgres-c= ol-defs.scm") ((database postgres-table) . "/home/ttn/build/guile-pg/src/postgres-tabl= e.scm") ((database postgres-meta) . "/home/ttn/build/guile-pg/src/postgres-meta.= scm") ((database postgres-gxrepl) . "/home/ttn/build/guile-pg/src/postgres-gxr= epl.scm") ((database postgres) scm_init_module "scm_init_database_postgres_module"= () . "/home/ttn/build/guile-pg/.b/src/.libs/postgres.so.0.0.0") ((database postgres-types) . "/home/ttn/build/guile-pg/src/postgres-type= s.scm") ((database postgres-qcons) . "/home/ttn/build/guile-pg/src/postgres-qcon= s.scm") ((database postgres-resx) . "/home/ttn/build/guile-pg/src/postgres-resx.= scm") ) =20=20 ;;; .module-catalog ends here This (version 1) format is not very compact. For version 2, we add a magic number, factor the root and all intervening parent directories, provide other meta info in a "header" section, and include other meta info (exports list being the most interesting) in the body. In some sense version 2 is like /etc/ld.so.cache. I imagine Guile 2.x could adopt module catalogs, extending to handle also compiled scheme (.go) and the (future) JIT files as well, but perhaps it's too late. I wonder how module catalogs would fit w/ name canonicalization, which IIUC is specific to compiled scheme. In sum, build-time caching plus run-time indirection as a means to facile module relocatability (plus some performance gain).