build problems

build problems

[Ken Raeburn]

I tried doing a build on my Mac (running 10.6.2) this morning, and ran into some problems.

#1: c-tokenize.lex declares yyget_leng() as returning int, but the flex template defines it as returning yy_size_t (which is size_t, a.k.a. unsigned long), so c-tokenize.c doesn't compile.  Changing the declaration in c-tokenize.lex to return size_t works for me, but the file hasn't been changed in some time, so if "int" was working for other platforms, changing it could be a problem.

We don't actually reference yyget_leng elsewhere explicitly; can we just get rid of the declaration?

#2: I build outside the source tree, with "../relative/path/to/configure --prefix=whatever ...", and make-texinfo.scm was failing for me.  After some time chasing things down with the debugger (thanks Andy for reminding me to recompile to get debug info) and the moral equivalent of debugging printfs, I found that load-module in boot-9.scm tries to interpret a relative path as relative to the filename for the current-load-port, if there is one.  Since the currently loading file was "../../../doc/ref/make-texinfo.scm" and the supplied filename (given on the command line) was "../../../doc/ref/standard-library.scm", load-module jammed the two prefixes together and came up with a pathname pointing off to nowhere.

Changing Makefile.am to remove the "$(srcdir)" prefix works around this, though I can't help but think there may be other potential cases where a relative path *not* coded directly in the loading module itself may be intended to be relative to the current directory and not the loading module's directory.

#3: Okay, I finally got to "make check", and two getaddrinfo tests failed, which Ludo and Andy were talking about on IRC.  I think these are bugs in the tests.  It appears that one ("wrong service name") is looking for EAI_SERVICE, when I think EAI_NONAME is the correct error to be returned in that case, or at least an acceptable error.  In the other ("port 80"), if I read it right, it's looking for AI_ADDRCONFIG to be set in the returned flags?  I don't think that's guaranteed at all.

#4: Looking at the doc for getaddrinfo, AI_ADDRCONFIG is described as returning "only addresses configured on the local system."  I think that should be "only address families...."  Clearly, if you're looking up the address of some server, you don't want your query to return only your own addresses.

Patches coming later, maybe, after work, if no one else gets to them first....

Ken

Re: build problems

[Ludovic Courtès]

Hello,

Ken Raeburn <raeburn@raeburn.org> writes:

> #3: Okay, I finally got to "make check", and two getaddrinfo tests
> failed, which Ludo and Andy were talking about on IRC.  I think these
> are bugs in the tests.  It appears that one ("wrong service name") is
> looking for EAI_SERVICE, when I think EAI_NONAME is the correct error
> to be returned in that case, or at least an acceptable error.  In the
> other ("port 80"), if I read it right, it's looking for AI_ADDRCONFIG
> to be set in the returned flags?  I don't think that's guaranteed at
> all.
>
> #4: Looking at the doc for getaddrinfo, AI_ADDRCONFIG is described as
> returning "only addresses configured on the local system."  I think
> that should be "only address families...."  Clearly, if you're looking
> up the address of some server, you don't want your query to return
> only your own addresses.

I committed a fix for these, thanks for your help on IRC!

  http://git.sv.gnu.org/cgit/guile.git/commit/?id=25bc75c4316497c95b1c3fc17f1678ac47d32041

Ludo’.

Re: build problems

[Ken Raeburn]

On Feb 15, 2010, at 08:41, Ken Raeburn wrote:
> #1: c-tokenize.lex declares yyget_leng() as returning int, but the flex template defines it as returning yy_size_t (which is size_t, a.k.a. unsigned long), so c-tokenize.c doesn't compile.  Changing the declaration in c-tokenize.lex to return size_t works for me, but the file hasn't been changed in some time, so if "int" was working for other platforms, changing it could be a problem.
> 
> We don't actually reference yyget_leng elsewhere explicitly; can we just get rid of the declaration?

This patch has been working fine for me on my Mac; we may be able to delete more of the declarations, depending what other lex implementations do, but I haven't explored that yet and the others aren't causing problems.  (And my Debian box hasn't yet got all the various libraries Guile now depends on, so I still haven't tested that configuration, but flex has the same version number as on my Mac.)

    Don't declare yyget_leng.

diff --git a/libguile/c-tokenize.lex b/libguile/c-tokenize.lex
index 938a5d2..f801526 100644
--- a/libguile/c-tokenize.lex
+++ b/libguile/c-tokenize.lex
@@ -29,7 +29,6 @@ int yylex(void);
 int yyget_lineno (void);
 FILE *yyget_in (void);
 FILE *yyget_out (void);
-int yyget_leng (void);
 char *yyget_text (void);
 void yyset_lineno (int line_number);
 void yyset_in (FILE * in_str);


> #2: I build outside the source tree, with "../relative/path/to/configure --prefix=whatever ...", and make-texinfo.scm was failing for me.  After some time chasing things down with the debugger (thanks Andy for reminding me to recompile to get debug info) and the moral equivalent of debugging printfs, I found that load-module in boot-9.scm tries to interpret a relative path as relative to the filename for the current-load-port, if there is one.  Since the currently loading file was "../../../doc/ref/make-texinfo.scm" and the supplied filename (given on the command line) was "../../../doc/ref/standard-library.scm", load-module jammed the two prefixes together and came up with a pathname pointing off to nowhere.
> 
> Changing Makefile.am to remove the "$(srcdir)" prefix works around this, though I can't help but think there may be other potential cases where a relative path *not* coded directly in the loading module itself may be intended to be relative to the current directory and not the loading module's directory.

Assuming the current 'load-module' behavior is correct -- or not so obviously incorrect that we're going to change it right now -- this patch takes care of this build problem for me.

    Path for snarf_doc input is treated relative to make-texinfo.scm file.

diff --git a/doc/ref/Makefile.am b/doc/ref/Makefile.am
index a587343..1da5cde 100644
--- a/doc/ref/Makefile.am
+++ b/doc/ref/Makefile.am
@@ -150,7 +150,7 @@ include standard-library.am
 $(snarf_doc).texi: $(standard_library_scm_files)
        GUILE_AUTO_COMPILE=0                            \
        "$(preinstguile)" "$(srcdir)/make-texinfo.scm"  \
-         "$(srcdir)/$(snarf_doc).scm" > "$@.tmp"
+         "$(snarf_doc).scm" > "$@.tmp"
        mv "$@.tmp" "$@"
 
 CLEANFILES += $(snarf_doc).texi

Okay to check these in?

Ken

Re: build problems

[Ludovic Courtès]

Hi Ken,

Ken Raeburn <raeburn@raeburn.org> writes:

> On Feb 15, 2010, at 08:41, Ken Raeburn wrote:
>> #1: c-tokenize.lex declares yyget_leng() as returning int, but the flex template defines it as returning yy_size_t (which is size_t, a.k.a. unsigned long), so c-tokenize.c doesn't compile.  Changing the declaration in c-tokenize.lex to return size_t works for me, but the file hasn't been changed in some time, so if "int" was working for other platforms, changing it could be a problem.
>> 
>> We don't actually reference yyget_leng elsewhere explicitly; can we just get rid of the declaration?
>
> This patch has been working fine for me on my Mac; we may be able to delete more of the declarations, depending what other lex implementations do

Keep in mind that ‘c-tokenize.c’ is part of the distribution, so we
shouldn’t worry much about what Flex implementations do, as long as the
file we ship compiles fine.

I don’t have any problem with Flex 2.5.35.  Which version do you use?

>> #2: [...] I found that load-module in boot-9.scm tries to interpret a
>> relative path as relative to the filename for the current-load-port,
>> if there is one.  Since the currently loading file was
>> "../../../doc/ref/make-texinfo.scm" and the supplied filename (given
>> on the command line) was "../../../doc/ref/standard-library.scm",
>> load-module jammed the two prefixes together and came up with a
>> pathname pointing off to nowhere.

[...]

>     Path for snarf_doc input is treated relative to make-texinfo.scm file.
>
> diff --git a/doc/ref/Makefile.am b/doc/ref/Makefile.am
> index a587343..1da5cde 100644
> --- a/doc/ref/Makefile.am
> +++ b/doc/ref/Makefile.am
> @@ -150,7 +150,7 @@ include standard-library.am
>  $(snarf_doc).texi: $(standard_library_scm_files)
>         GUILE_AUTO_COMPILE=0                            \
>         "$(preinstguile)" "$(srcdir)/make-texinfo.scm"  \
> -         "$(srcdir)/$(snarf_doc).scm" > "$@.tmp"
> +         "$(snarf_doc).scm" > "$@.tmp"

This one seems wrong to me since $(snarf_doc).scm is in $(srcdir), not
$(builddir).

Instead I suggest changing $(srcdir) to $(abs_srcdir), which should
solve the problem.

Can you confirm and commit?

Thanks!

Ludo’.

Re: build problems

[Ken Raeburn]

On Mar 2, 2010, at 05:23, Ludovic Courtès wrote:
>>> We don't actually reference yyget_leng elsewhere explicitly; can we just get rid of the declaration?
>> 
>> This patch has been working fine for me on my Mac; we may be able to delete more of the declarations, depending what other lex implementations do
> 
> Keep in mind that ‘c-tokenize.c’ is part of the distribution, so we
> shouldn’t worry much about what Flex implementations do, as long as the
> file we ship compiles fine.

True for most people (if they don't do things that cause timestamps to randomly get updated, like using a version control system or copy program that doesn't preserve timestamps), though that just pushes the compatibility issue over to whatever lex/flex version the person building the distribution has....  But, as long as they try it out before putting it up for download, we should catch any problems fairly easily.

> I don’t have any problem with Flex 2.5.35.  Which version do you use?

Mac OS X includes flex 2.5.35, though I can't assert that Apple hasn't tweaked it any.  I tried this little test case:

> % cat q.lex
> %%
> "hello" { return HELLO; }
> %%
> % flex q.lex
> % grep yyget_leng lex.yy.c 
> yy_size_t yyget_leng (void );
> yy_size_t yyget_leng  (void)
> % 

And yy_size_t is defined to be size_t.
Does it not insert such declarations for you on your system?

>> -         "$(srcdir)/$(snarf_doc).scm" > "$@.tmp"
>> +         "$(snarf_doc).scm" > "$@.tmp"
> 
> This one seems wrong to me since $(snarf_doc).scm is in $(srcdir), not
> $(builddir).

Yes, though changing load-module to interpret paths relative to the working directory rather than the currently loading module looks wrong too.

> 
> Instead I suggest changing $(srcdir) to $(abs_srcdir), which should
> solve the problem.

Thus sidestepping the whole question of how a non-absolute pathname argument to make-texinfo.scm should be interpreted....  Perhaps it should complain if you give a non-absolute name, if we're not going to settle the question?

> Can you confirm and commit?

Sure -- done, and done.  And with the right upstream branch name this time. :-)

But... a new issue -- with SCM_DEBUG=1, my build fails:

GUILE_AUTO_COMPILE=0					\
	../meta/uninstalled-env			\
	guile-tools compile -Wunbound-variable -Warity-mismatch -o "language/glil/decompile-assembly.go" "../../module/language/glil/decompile-assembly.scm"
Non-pair accessed with SCM_C[AD]R: `#<unknown-type (0x13d . 0x101a239a0) @ 0x101a23960>'
make[2]: *** [language/glil/decompile-assembly.go] Abort trap (core dumped)

(gdb) bt
#0  0x00007fff81777fe6 in kill ()
#1  0x00000001000cfb5f in scm_error_pair_access (non_pair=0x10195e230) at ../../libguile/pairs.c:64
#2  0x0000000100022275 in scm_i_dowinds (to=0x101e125e0, delta=1, turn_func=0, data=0x0) at ../../libguile/dynwind.c:309
#3  0x0000000100022390 in scm_dowinds (to=<value temporarily unavailable, due to optimizations>, delta=<value temporarily unavailable, due to optimizations>) at ../../libguile/dynwind.c:228
#4  0x000000010001a79e in scm_c_abort (vm=0x1006f3040, tag=0x101a239a0, n=5, argv=0x7fff5fbfd3a0, cookie=50450) at ../../libguile/control.c:230
#5  0x00000001001128ce in vm_abort (vm=<value temporarily unavailable, due to optimizations>, n=<value temporarily unavailable, due to optimizations>, vm_cookie=<value temporarily unavailable, due to optimizations>) at ../../libguile/vm.c:229
#6  0x000000010011a857 in vm_debug_engine (vm=0x1006f3040, program=0x101083520, argv=0x101001970, nargs=1) at vm-i-system.c:1546
[....]

I note that 0x13d & 0x7f is 61 or scm_tc7_prompt...

Ken

Re: build problems

[Andy Wingo]

On Thu 04 Mar 2010 09:25, Ken Raeburn <raeburn@raeburn.org> writes:

> (gdb) bt
> #0  0x00007fff81777fe6 in kill ()
> #1 0x00000001000cfb5f in scm_error_pair_access (non_pair=0x10195e230) at
> ../../libguile/pairs.c:64
> #2 0x0000000100022275 in scm_i_dowinds (to=0x101e125e0, delta=1,
> turn_func=0, data=0x0) at ../../libguile/dynwind.c:309
> [....]
>
> I note that 0x13d & 0x7f is 61 or scm_tc7_prompt...

Good catch, thanks! Will fix.

Andy
-- 
http://wingolog.org/

Re: build problems

[Ludovic Courtès]

Hi,

Ken Raeburn <raeburn@raeburn.org> writes:

> On Mar 2, 2010, at 05:23, Ludovic Courtès wrote:
>>>> We don't actually reference yyget_leng elsewhere explicitly; can we just get rid of the declaration?
>>> 
>>> This patch has been working fine for me on my Mac; we may be able to delete more of the declarations, depending what other lex implementations do
>> 
>> Keep in mind that ‘c-tokenize.c’ is part of the distribution, so we
>> shouldn’t worry much about what Flex implementations do, as long as the
>> file we ship compiles fine.
>
> True for most people (if they don't do things that cause timestamps to randomly get updated, like using a version control system or copy program that doesn't preserve timestamps), though that just pushes the compatibility issue over to whatever lex/flex version the person building the distribution has....  But, as long as they try it out before putting it up for download, we should catch any problems fairly easily.

I have this:

--8<---------------cut here---------------start------------->8---
$ grep yyget_leng libguile/c-tokenize.c
int yyget_leng (void);
int yyget_leng (void );
int yyget_leng  (void)

$ grep yyget_leng libguile/c-tokenize.lex
int yyget_leng (void);
--8<---------------cut here---------------end--------------->8---

>>> -         "$(srcdir)/$(snarf_doc).scm" > "$@.tmp"
>>> +         "$(snarf_doc).scm" > "$@.tmp"
>> 
>> This one seems wrong to me since $(snarf_doc).scm is in $(srcdir), not
>> $(builddir).
>
> Yes, though changing load-module to interpret paths relative to the
> working directory rather than the currently loading module looks wrong
> too.

Think about it:

  $ cd src/guile

  $ make -C module module/ice-9/q.go
  make: Entering directory `/home/ludo/src/guile/module'
  make: *** No rule to make target `module/ice-9/q.go'.  Stop.
  make: Leaving directory `/home/ludo/src/guile/module'

  $ make -C module ice-9/q.go
  make: Entering directory `/home/ludo/src/guile/module'
  make: `ice-9/q.go' is up to date.
  make: Leaving directory `/home/ludo/src/guile/module'

  $ ls ice-9/q.scm
  ls: cannot access ice-9/q.scm: No such file or directory

  $ ls module/ice-9/q.scm
  module/ice-9/q.scm

Here it’s really an instance of the same problem: the naming context is
lacking from the relative path.  The only sane way to fix this kind of
issue is by passing a file descriptor instead of a path, where the file
descriptor is obtained by resolving a path (relative or absolute) on the
caller’s side.

On Unix passing absolute paths sort-of works around the problem because
they provide enough naming context, provided the caller and callee see
the same file system hierarchy and have the same permissions.

Now, another interesting (but orthogonal) issue is what to do with
‘load’ when it’s passed *literal* relative paths, as in ‘(load
"../foo.scm")’.  It’s an open issue:
http://lists.gnu.org/archive/html/bug-guile/2009-11/msg00001.html .

>> Can you confirm and commit?
>
> Sure -- done, and done.

Thanks!

Ludo’.

Re: build problems

[Ken Raeburn]

On Mar 4, 2010, at 05:52, Ludovic Courtès wrote:
> I have this:
> 
> --8<---------------cut here---------------start------------->8---
> $ grep yyget_leng libguile/c-tokenize.c
> int yyget_leng (void);
> int yyget_leng (void );
> int yyget_leng  (void)

Iiiiinteresting... looks like Apple's not using the 2.5.35 code as released.
Though, in the flex CVS repository, this code has been changed to use yy_size_t, so it would probably become a problem with the next release anyways.

>> Yes, though changing load-module to interpret paths relative to the
>> working directory rather than the currently loading module looks wrong
>> too.
> 
> Think about it:
> 
>  $ cd src/guile
> 
>  $ make -C module module/ice-9/q.go
>  make: Entering directory `/home/ludo/src/guile/module'
>  make: *** No rule to make target `module/ice-9/q.go'.  Stop.
>  make: Leaving directory `/home/ludo/src/guile/module'
> 
>  $ make -C module ice-9/q.go
>  make: Entering directory `/home/ludo/src/guile/module'
>  make: `ice-9/q.go' is up to date.
>  make: Leaving directory `/home/ludo/src/guile/module'
> 
>  $ ls ice-9/q.scm
>  ls: cannot access ice-9/q.scm: No such file or directory
> 
>  $ ls module/ice-9/q.scm
>  module/ice-9/q.scm
> 
> Here it’s really an instance of the same problem: the naming context is
> lacking from the relative path.  The only sane way to fix this kind of
> issue is by passing a file descriptor instead of a path, where the file
> descriptor is obtained by resolving a path (relative or absolute) on the
> caller’s side.

I think being explicit about how a relative path is interpreted is often adequate, if occasionally messy.  Here, it's relative to the '-C' directory, which in turn is relative to the current directory.  You don't usually solve this by using "make -C module /path/to/my/guile/tree/module/ice-9/q.go", even if you ignore the need to match target strings for getting dependencies from the makefile (since targets "/path/to/foo.go" and "foo.go" aren't treated as the same).  But, I'm familiar with "make -C", and the behavior to expect from "make-texinfo.scm" isn't immediately obvious to me. :-)

The "ln" program highlights the issue too, if you look at "ln" vs "ln -s" for creating links that aren't in the directory where ln is being run.  For "ln" the name is interpreted relative to the current directory, and absolute names work just as well; for "ln -s" the string isn't interpreted by ln but will later be interpreted by the file system, relative to its place in the file system, and absolute paths may not work if you need the directory tree to be relocatable.  Which reminds me...

Hard-coding uses of $(abs_srcdir) nails down a build tree to a specific location in the file system.  If I do something like "mv Desktop/guile devel/guile", then, without this change to use $(abs_srcdir), the directory references in debug info in any previously compiled objects might be wrong if the compiler converts them to absolute paths, but otherwise everything would be okay except for one use of $(abs_top_srcdir) in examples/Makefile, which usually I'm ignoring anyways :-); now, even "make clean" isn't enough, I'll need to re-run config.status (or configure) to make doc/ref/Makefile work again if I ever need to rebuild standard-library.texi.  This could be occasionally annoying.

Ken

Re: build problems

[Ludovic Courtès]

Hey,

Ken Raeburn <raeburn@raeburn.org> writes:

> Hard-coding uses of $(abs_srcdir) nails down a build tree to a
> specific location in the file system.

That’s often the case anyway so I wouldn’t worry.

Thanks,
Ludo’.

Re: build problems

[Ludovic Courtès]

Hello!

Ken Raeburn <raeburn@raeburn.org> writes:

> But... a new issue -- with SCM_DEBUG=1, my build fails:
>
> GUILE_AUTO_COMPILE=0					\
> 	../meta/uninstalled-env			\
> 	guile-tools compile -Wunbound-variable -Warity-mismatch -o "language/glil/decompile-assembly.go" "../../module/language/glil/decompile-assembly.scm"
> Non-pair accessed with SCM_C[AD]R: `#<unknown-type (0x13d . 0x101a239a0) @ 0x101a23960>'
> make[2]: *** [language/glil/decompile-assembly.go] Abort trap (core dumped)
>
> (gdb) bt
> #0  0x00007fff81777fe6 in kill ()
> #1  0x00000001000cfb5f in scm_error_pair_access (non_pair=0x10195e230) at ../../libguile/pairs.c:64

I was quite disappointed to see that ‘SCM_DEBUG’ is yet another cpp
variable, different from ‘GUILE_DEBUG’ and ‘SCM_I_GSC_GUILE_DEBUG’,
which are enabled by ‘--enable-guile-debug’.

Anyway, I added a ‘SCM_DEBUG=1’ build to Hydra, as it seems to be
useful:

  http://hydra.nixos.org/jobset/gnu/guile-master

Thanks,
Ludo’.