autocompilation support in master

autocompilation support in master

[Andy Wingo]

Hey folks,

I pushed support for automatic compilation in master. That is, whenever
load-from-path sees that it should compile a file, and autocompilation
is enabled, it compiles the file then and there.

I went ahead and enabled this feature, to see exactly what kinds of bugs
this will cause.

There's a lot of trickiness to this, but I think we all have instincts
on how this should work. I'll write docs soon, but in the meantime, I
would like for folks to ignore the implementation, and just use Guile
for a while, and see if what it does sounds right to you. It prints some
status info on the console when autocompiling.

Of course, after the gut reaction, I would appreciate implementation
criticisms as well :)

Regards,

Andy
-- 
http://wingolog.org/

Re: autocompilation support in master

[Neil Jerram]

Andy Wingo <wingo@pobox.com> writes:

> There's a lot of trickiness to this, but I think we all have instincts
> on how this should work. I'll write docs soon, but in the meantime, I
> would like for folks to ignore the implementation, and just use Guile
> for a while, and see if what it does sounds right to you. It prints some
> status info on the console when autocompiling.

Unfortunately I'm not getting through `make' yet:

...
GUILE_AUTO_COMPILE=0 ../meta/uninstalled-env guile-tools compile -o "language/ecmascript/spec.go" "language/ecmascript/spec.scm"
ERROR: Stack overflow

But if I switch to a shell and repeat with --debug, no problem:

neil@arudy:~/SW/Guile/git/module$ GUILE_AUTO_COMPILE=0 GUILE_FLAGS=--debug ../meta/uninstalled-env guile-tools compile -o "language/ecmascript/spec.go" "language/ecmascript/spec.scm"
wrote `language/ecmascript/spec.go'

And no problem if I now remove the --debug again:

neil@arudy:~/SW/Guile/git/module$ GUILE_AUTO_COMPILE=0 ../meta/uninstalled-env guile-tools compile -o "language/ecmascript/spec.go" "language/ecmascript/spec.scm"
wrote `language/ecmascript/spec.go'

Or even if I delete language/ecmascript/spec.go and retry (so it's not
connected with spec.scm already having been compiled):

neil@arudy:~/SW/Guile/git/module$ rm language/ecmascript/spec.go 
neil@arudy:~/SW/Guile/git/module$ GUILE_AUTO_COMPILE=0 ../meta/uninstalled-env guile-tools compile -o "language/ecmascript/spec.go" "language/ecmascript/spec.scm"
wrote `language/ecmascript/spec.go'

What's the explanation for that?

        Neil

Re: autocompilation support in master

[Neil Jerram]

Neil Jerram <neil@ossau.uklinux.net> writes:

> ...
> GUILE_AUTO_COMPILE=0 ../meta/uninstalled-env guile-tools compile -o "language/ecmascript/spec.go" "language/ecmascript/spec.scm"
> ERROR: Stack overflow
>
> But if I switch to a shell and repeat with --debug, no problem:
>
> neil@arudy:~/SW/Guile/git/module$ GUILE_AUTO_COMPILE=0 GUILE_FLAGS=--debug ../meta/uninstalled-env guile-tools compile -o "language/ecmascript/spec.go" "language/ecmascript/spec.scm"
> wrote `language/ecmascript/spec.go'
>
> And no problem if I now remove the --debug again:
>
> neil@arudy:~/SW/Guile/git/module$ GUILE_AUTO_COMPILE=0 ../meta/uninstalled-env guile-tools compile -o "language/ecmascript/spec.go" "language/ecmascript/spec.scm"
> wrote `language/ecmascript/spec.go'
>
> Or even if I delete language/ecmascript/spec.go and retry (so it's not
> connected with spec.scm already having been compiled):
>
> neil@arudy:~/SW/Guile/git/module$ rm language/ecmascript/spec.go 
> neil@arudy:~/SW/Guile/git/module$ GUILE_AUTO_COMPILE=0 ../meta/uninstalled-env guile-tools compile -o "language/ecmascript/spec.go" "language/ecmascript/spec.scm"
> wrote `language/ecmascript/spec.go'

One more data point.  If I now try `make' again, it fails a bit later
with compile-ghil:

...
Making all in module
make[2]: Entering directory `/home/neil/SW/Guile/git/module'
GUILE_AUTO_COMPILE=0 ../meta/uninstalled-env guile-tools compile -o "language/ecmascript/impl.go" "language/ecmascript/impl.scm"
wrote `language/ecmascript/impl.go'
GUILE_AUTO_COMPILE=0 ../meta/uninstalled-env guile-tools compile -o "language/ecmascript/base.go" "language/ecmascript/base.scm"
wrote `language/ecmascript/base.go'
GUILE_AUTO_COMPILE=0 ../meta/uninstalled-env guile-tools compile -o "language/ecmascript/function.go" "language/ecmascript/function.scm"
wrote `language/ecmascript/function.go'
GUILE_AUTO_COMPILE=0 ../meta/uninstalled-env guile-tools compile -o "language/ecmascript/array.go" "language/ecmascript/array.scm"
wrote `language/ecmascript/array.go'
GUILE_AUTO_COMPILE=0 ../meta/uninstalled-env guile-tools compile -o "language/ecmascript/compile-ghil.go" "language/ecmascript/compile-ghil.scm"
ERROR: Stack overflow

       Neil

Re: autocompilation support in master

[Ludovic Courtès]

Hey,

Andy Wingo <wingo@pobox.com> writes:

> I pushed support for automatic compilation in master. That is, whenever
> load-from-path sees that it should compile a file, and autocompilation
> is enabled, it compiles the file then and there.

Cool!

Incidentally, I just did a `pull', and compiling the compiler appears to
be much slower than in my recollections (pre-syncase merge, I think).
Is it just an impression?

Thanks,
Ludo'.

Re: autocompilation support in master

[Andy Wingo]

On Sat 06 Jun 2009 15:19, ludo@gnu.org (Ludovic Courtès) writes:

> Incidentally, I just did a `pull', and compiling the compiler appears to
> be much slower than in my recollections (pre-syncase merge, I think).
> Is it just an impression?

Nope, not just an impression, it's true. Reasons discussed in the last
message in the "srfi-18 and the vm" thread.

A
-- 
http://wingolog.org/

Re: autocompilation support in master

[Neil Jerram]

Neil Jerram <neil@ossau.uklinux.net> writes:

> Unfortunately I'm not getting through `make' yet:
>
> ...
> GUILE_AUTO_COMPILE=0 ../meta/uninstalled-env guile-tools compile -o "language/ecmascript/spec.go" "language/ecmascript/spec.scm"
> ERROR: Stack overflow

This is still happening for me after pulling the latest master today.
Any ideas?

Regards,
        Neil

Re: autocompilation support in master

[Mark H Weaver]

On Tue, Jun 09, 2009 at 12:24:48AM +0100, Neil Jerram wrote:
> > GUILE_AUTO_COMPILE=0 ../meta/uninstalled-env guile-tools compile -o "language/ecmascript/spec.go" "language/ecmascript/spec.scm"
> > ERROR: Stack overflow
> 
> This is still happening for me after pulling the latest master today.

I have the same problem on Debian Lenny/x86, and am able to work
around it by increasing the stack size resource limit.  On my system,
"ulimit -s 16384" (within bash) before running "make" does the trick.
Unfortunately, this is not a portable solution.

    Mark

Re: autocompilation support in master

[Andy Wingo]

On Tue 09 Jun 2009 02:52, Mark H Weaver <mhw@netris.org> writes:

> On Tue, Jun 09, 2009 at 12:24:48AM +0100, Neil Jerram wrote:
>> > GUILE_AUTO_COMPILE=0 ../meta/uninstalled-env guile-tools compile -o "language/ecmascript/spec.go" "language/ecmascript/spec.scm"
>> > ERROR: Stack overflow
>> 
>> This is still happening for me after pulling the latest master today.
>
> I have the same problem on Debian Lenny/x86, and am able to work
> around it by increasing the stack size resource limit.  On my system,
> "ulimit -s 16384" (within bash) before running "make" does the trick.
> Unfortunately, this is not a portable solution.

It's a strange thing, and I don't see it on my x86-32 laptop running
Fedora. But I've heard reports of this. A backtrace at the time of stack
overflow would be helpful.

Apologies for ongoing badness,

Andy
-- 
http://wingolog.org/

Re: autocompilation support in master

[dsmich]

---- Mark H Weaver <mhw@netris.org> wrote: 
> On Tue, Jun 09, 2009 at 12:24:48AM +0100, Neil Jerram wrote:
> > > GUILE_AUTO_COMPILE=0 ../meta/uninstalled-env guile-tools compile -o "language/ecmascript/spec.go" "language/ecmascript/spec.scm"
> > > ERROR: Stack overflow
> > 
> > This is still happening for me after pulling the latest master today.
> 
> I have the same problem on Debian Lenny/x86, and am able to work
> around it by increasing the stack size resource limit.  On my system,
> "ulimit -s 16384" (within bash) before running "make" does the trick.
> Unfortunately, this is not a portable solution.

Yes, I've experienced this too.  Works for me when I bump up my stack to 9000 from 8192.

I've also found that's it's compiler dependent.  Gcc 4.4 works I think, while 4.3 fails.  (Not sure about that, as I'm not at that machine right now).

-Dale




> 
>     Mark
> 
> 

Re: autocompilation support in master

[Mark H Weaver]

On Tue, Jun 09, 2009 at 09:27:37AM +0200, Andy Wingo wrote:
> It's a strange thing, and I don't see it on my x86-32 laptop running
> Fedora. But I've heard reports of this. A backtrace at the time of stack
> overflow would be helpful.

Strangely, the stack overflow doesn't happen when I run the compile
command (as echoed by make) directly from the command line.  I only
see it happen when compiling via make.

To generate the backtrace, I added the following lines near the top of
guile-tools.  Is there a better way?

  (debug-enable 'debug)
  (debug-enable 'backtrace)
  (debug-set! depth 100)
  (write (debug-options-interface))
  (newline)

The resulting backtrace follows.  This is git master from a few days
ago, commit 12798872ff39e27dbcf90675c3d3554ae27df750.

      Mark


GUILE_AUTO_COMPILE=0 ../meta/uninstalled-env guile-tools compile -o "language/ecmascript/spec.go" "language/ecmascript/spec.scm"
(show-file-name #t stack 40000 debug backtrace depth 100 maxdepth 1000 frames 3 indent 10 width 79 procnames cheap)
Backtrace:
In ice-9/psyntax-pp.scm:
  20: 271  [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
   5: 272  [# core-form # # ...]
In unknown file:
   ?: 273* [map #<program 4060aa30 at ice-9/psyntax-pp.scm:4:2061 (x415)> ((# . #))]
In ice-9/psyntax-pp.scm:
  24: 274* [# # # # ...]
In ice-9/psyntax-pp.scm:
  20: 275  [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
   5: 276  [# core-form # # ...]
In unknown file:
   ?: 277* [map #<program 405fa110 at ice-9/psyntax-pp.scm:4:2061 (x415)> ((# . #))]
In ice-9/psyntax-pp.scm:
  24: 278* [# # # # ...]
In ice-9/psyntax-pp.scm:
  20: 279  [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
   5: 280  [# core-form # # ...]
In unknown file:
   ?: 281* [map #<program 405fee40 at ice-9/psyntax-pp.scm:4:2061 (x415)> ((# . #))]
In ice-9/psyntax-pp.scm:
  24: 282* [# # # # ...]
In ice-9/psyntax-pp.scm:
  20: 283  [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
   5: 284  [# core-form # # ...]
In unknown file:
   ?: 285* [map #<program 405fb0e0 at ice-9/psyntax-pp.scm:4:2061 (x415)> ((# . #))]
In ice-9/psyntax-pp.scm:
  24: 286* [# # # # ...]
In ice-9/psyntax-pp.scm:
  20: 287  [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
   5: 288  [# core-form # # ...]
In unknown file:
   ?: 289* [map #<program 405fc9f0 at ice-9/psyntax-pp.scm:4:2061 (x415)> ((# . #))]
In ice-9/psyntax-pp.scm:
  24: 290* [# # # # ...]
In ice-9/psyntax-pp.scm:
  20: 291  [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
   5: 292  [# core-form # # ...]
In unknown file:
   ?: 293* [map #<program 40608020 at ice-9/psyntax-pp.scm:4:2061 (x415)> ((# . #))]
In ice-9/psyntax-pp.scm:
  24: 294* [# # # # ...]
In ice-9/psyntax-pp.scm:
  20: 295  [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
   5: 296  [# core-form # # ...]
In unknown file:
   ?: 297* [map #<program 40540cf0 at ice-9/psyntax-pp.scm:4:2061 (x415)> ((# . #))]
In ice-9/psyntax-pp.scm:
  24: 298* [# # # # ...]
In ice-9/psyntax-pp.scm:
  20: 299  [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
   5: 300  [# core-form # # ...]
In unknown file:
   ?: 301* [map #<program 4060a130 at ice-9/psyntax-pp.scm:4:2061 (x415)> ((# . #))]
In ice-9/psyntax-pp.scm:
  24: 302* [# # # # ...]
In ice-9/psyntax-pp.scm:
  20: 303  [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
   5: 304  [# core-form # # ...]
In unknown file:
   ?: 305* [map #<program 406020c0 at ice-9/psyntax-pp.scm:4:2061 (x415)> ((# . #))]
In ice-9/psyntax-pp.scm:
  24: 306* [# # # # ...]
In ice-9/psyntax-pp.scm:
  20: 307  [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
   5: 308  [# core-form # # ...]
In unknown file:
   ?: 309* [map #<program 405f6ed0 at ice-9/psyntax-pp.scm:4:2061 (x415)> ((# . #))]
In ice-9/psyntax-pp.scm:
  24: 310* [# # # # ...]
In ice-9/psyntax-pp.scm:
  20: 311  [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
   5: 312  [# core-form # # ...]
In unknown file:
   ?: 313* [map #<program 4060cf40 at ice-9/psyntax-pp.scm:4:2061 (x415)> ((# . #))]
In ice-9/psyntax-pp.scm:
  24: 314* [# # # # ...]
In ice-9/psyntax-pp.scm:
  20: 315  [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
   5: 316  [# core-form # # ...]
In unknown file:
   ?: 317* [map #<program 40605410 at ice-9/psyntax-pp.scm:4:2061 (x415)> ((# . #))]
In ice-9/psyntax-pp.scm:
  24: 318* [# # # # ...]
In ice-9/psyntax-pp.scm:
  20: 319  [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
   5: 320  [# core-form # # ...]
In unknown file:
   ?: 321* [map #<program 405fac30 at ice-9/psyntax-pp.scm:4:2061 (x415)> ((# . #))]
In ice-9/psyntax-pp.scm:
  24: 322* [# # # # ...]
In ice-9/psyntax-pp.scm:
  20: 323  [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
   5: 324  [# core-form # # ...]
In unknown file:
   ?: 325* [map #<program 4053e7f0 at ice-9/psyntax-pp.scm:4:2061 (x415)> ((# . #))]
In ice-9/psyntax-pp.scm:
  24: 326* [# # # # ...]
In ice-9/psyntax-pp.scm:
  20: 327  [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
   5: 328  [# core-form # # ...]
In unknown file:
   ?: 329* [map #<program 40607af0 at ice-9/psyntax-pp.scm:4:2061 (x415)> ((# . #))]
In ice-9/psyntax-pp.scm:
  24: 330* [# # # # ...]
In ice-9/psyntax-pp.scm:
  20: 331  [chi-let1039 (# # #) (# # # # ...) (()) ...]
In unknown file:
   ?: 332* [map #<program 406074d0 at ice-9/psyntax-pp.scm:19:3699 (x1053)> (#)]
In ice-9/psyntax-pp.scm:
  20: 333* [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
   5: 334  [# core-form # # ...]
In unknown file:
   ?: 335* [map #<program 4060ab10 at ice-9/psyntax-pp.scm:4:2061 (x415)> (#)]
In ice-9/psyntax-pp.scm:
  24: 336* [# # # # ...]
In ice-9/psyntax-pp.scm:
  20: 337  [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
   5: 338  [# core-form # # ...]
In unknown file:
   ?: 339* [map #<program 4060d010 at ice-9/psyntax-pp.scm:4:2061 (x415)> (#)]
In ice-9/psyntax-pp.scm:
  24: 340* [# # # # ...]
In ice-9/psyntax-pp.scm:
  24: 341  [# # # # ...]
In ice-9/psyntax-pp.scm:
  20: 342  [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
   5: 343  [# core-form # # ...]
In unknown file:
   ?: 344* [map #<program 4053d830 at ice-9/psyntax-pp.scm:4:2061 (x415)> (#)]
In ice-9/psyntax-pp.scm:
  20: 345* [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
   5: 346  [# core-form # # ...]
In unknown file:
   ?: 347* [map #<program 4053a290 at ice-9/psyntax-pp.scm:4:2061 (x415)> (#)]
In ice-9/psyntax-pp.scm:
  24: 348* [# # # # ...]
In ice-9/psyntax-pp.scm:
  20: 349  [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
   5: 350  [# core-form # # ...]
In unknown file:
   ?: 351* [map #<program 405452f0 at ice-9/psyntax-pp.scm:4:2061 (x415)> (#)]
In ice-9/psyntax-pp.scm:
  20: 352* [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
   5: 353  [# core-form # # ...]
In unknown file:
   ?: 354* [map #<program 405438e0 at ice-9/psyntax-pp.scm:4:2061 (x415)> (#)]
In ice-9/psyntax-pp.scm:
  24: 355* [# # # # ...]
In ice-9/psyntax-pp.scm:
  20: 356  [chi-let1039 (# () #) (# # # # ...) (() # # # ...) ...]
In ice-9/psyntax-pp.scm:
   5: 357  [# global-call # # ...]
In unknown file:
   ?: 358* [map #<program 4053f8e0 at ice-9/psyntax-pp.scm:4:2061 (x415)> ((# . #))]
In ice-9/psyntax-pp.scm:
   6: 359* [# comp #]
In unknown file:
   ?: 360* [map #<program 405f8900 at ice-9/psyntax-pp.scm:5:820 (e452)> (# e)]
In ice-9/psyntax-pp.scm:
   6: 361* [# # #]
In unknown file:
   ?: 362* [map #<program 405ffc20 at ice-9/psyntax-pp.scm:5:820 (e452)> (# # #)]
In ice-9/psyntax-pp.scm:
   6: 363* [# # #]
In unknown file:
   ?: 364* [map #<program 405ff6a0 at ice-9/psyntax-pp.scm:5:820 (e452)> (# # #)]
In ice-9/psyntax-pp.scm:
   6: 365* [chi150 (# #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
   8: 366  [syntax-type148 (# #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
   7: 367  [syntax-type148 quote (# # # # ...) (# shift # # ...) ...]
In ice-9/psyntax-pp.scm:
  16: 368  [lookup111 quote (# # # # ...) (hygiene guile)]
In ice-9/psyntax-pp.scm:
  18: 369  [get-global-definition-hook79 quote (hygiene guile)]
In ice-9/boot-9.scm:
1952: 370  [# #]

ERROR: Stack overflow
make: *** [language/ecmascript/spec.go] Error 1

Re: autocompilation support in master

[Andy Wingo]

On Tue 09 Jun 2009 20:47, Mark H Weaver <mhw@netris.org> writes:

> On Tue, Jun 09, 2009 at 09:27:37AM +0200, Andy Wingo wrote:
>> It's a strange thing, and I don't see it on my x86-32 laptop running
>> Fedora. But I've heard reports of this. A backtrace at the time of stack
>> overflow would be helpful.
>
> Strangely, the stack overflow doesn't happen when I run the compile
> command (as echoed by make) directly from the command line.  I only
> see it happen when compiling via make.

Hm, that is indeed odd. Also odd is that here, I only get errors
building this if I set my ulimit below 128 *kilo*bytes.

gcc (GCC) 4.4.0 20090506 (Red Hat 4.4.0-4)

> To generate the backtrace, I added the following lines near the top of
> guile-tools.  Is there a better way?
>
>   (debug-enable 'debug)
>   (debug-enable 'backtrace)
>   (debug-set! depth 100)
>   (write (debug-options-interface))
>   (newline)

No, this is the best way we have currently. As I understand it, the
debugging evaluator is slower than the normal one -- but at this point,
who cares? With the VM we are already faster and we keep backtraces. We
should have backtraces on, always.

Does anyone have objections to that? If not, I'll make that change
within the week.

> GUILE_AUTO_COMPILE=0 ../meta/uninstalled-env guile-tools compile -o "language/ecmascript/spec.go" "language/ecmascript/spec.scm"
> (show-file-name #t stack 40000 debug backtrace depth 100 maxdepth 1000 frames 3 indent 10 width 79 procnames cheap)
> Backtrace:
> In ice-9/psyntax-pp.scm:
>   20: 271  [chi-let1039 (# # #) (# # # # ...) (()) ...]
> In ice-9/psyntax-pp.scm:
>    5: 272  [# core-form # # ...]
> In unknown file:
>    ?: 273* [map #<program 4060aa30 at ice-9/psyntax-pp.scm:4:2061
> (x415)> ((# . #))]

Hmmmmmmm. I'll poke some options. Thanks for the BT.

Andy
-- 
http://wingolog.org/

Re: autocompilation support in master

[Mark H Weaver]

Andy Wingo <wingo@pobox.com> wrote:
> BTW I pushed something that might affect
> the stack overflow issue, can you give that a try? I have one report of
> it working where it didn't use to work.

It still overflows the stack on my system, but since you changed the
order of compilation, it now fails on a different file.  This with
commit 9ea12179fffffa8e1ba12cde4a10c35504a80012.  I also removed the
(debug-enable 'debug), so the backtrace looks a little different.

      Mark


GUILE_AUTO_COMPILE=0 ../meta/uninstalled-env guile-tools compile -o "language/ecmascript/compile-ghil.go" "language/ecmascript/compile-ghil.scm"
Backtrace:
In ice-9/psyntax-pp.scm:
5842:  44* [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
 535:  45* [# () # ()]
In ice-9/psyntax-pp.scm:
 782:  46  [# core-form # # ...]
In ice-9/psyntax-pp.scm:
5842:  47* [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
 535:  48* [# () # ()]
In ice-9/psyntax-pp.scm:
 782:  49  [# core-form # # ...]
In ice-9/psyntax-pp.scm:
5842:  50* [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
 535:  51* [# () # ()]
In ice-9/psyntax-pp.scm:
 782:  52  [# core-form # # ...]
In ice-9/psyntax-pp.scm:
5842:  53* [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
 535:  54* [# () # ()]
In ice-9/psyntax-pp.scm:
 782:  55  [# core-form # # ...]
In ice-9/psyntax-pp.scm:
5842:  56* [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
 535:  57* [# () # ()]
In ice-9/psyntax-pp.scm:
 782:  58  [# core-form # # ...]
In ice-9/psyntax-pp.scm:
5842:  59* [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
 535:  60* [# () # ()]
In ice-9/psyntax-pp.scm:
 782:  61  [# core-form # # ...]
In ice-9/psyntax-pp.scm:
5842:  62* [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
 535:  63* [# () # ()]
In ice-9/psyntax-pp.scm:
 782:  64  [# core-form # # ...]
In ice-9/psyntax-pp.scm:
5842:  65* [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
 535:  66* [# () # ()]
In ice-9/psyntax-pp.scm:
 782:  67  [# core-form # # ...]
In ice-9/psyntax-pp.scm:
5842:  68* [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
 535:  69* [# () # ()]
In ice-9/psyntax-pp.scm:
 782:  70  [# core-form # # ...]
In ice-9/psyntax-pp.scm:
5842:  71* [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
 535:  72* [# () # ()]
In ice-9/psyntax-pp.scm:
 782:  73  [# core-form # # ...]
In ice-9/psyntax-pp.scm:
5842:  74* [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
 535:  75* [# () # ()]
In ice-9/psyntax-pp.scm:
 782:  76  [# core-form # # ...]
In ice-9/psyntax-pp.scm:
5842:  77* [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
 535:  78* [# () # ()]
In ice-9/psyntax-pp.scm:
 782:  79  [# core-form # # ...]
In ice-9/psyntax-pp.scm:
5842:  80* [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
 535:  81* [# () # ()]
In ice-9/psyntax-pp.scm:
 782:  82  [# core-form # # ...]
In ice-9/psyntax-pp.scm:
5842:  83* [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
 535:  84* [# () # ()]
In ice-9/psyntax-pp.scm:
 782:  85  [# core-form # # ...]
In ice-9/psyntax-pp.scm:
5842:  86* [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
 535:  87* [# () # ()]
In ice-9/psyntax-pp.scm:
 782:  88  [# core-form # # ...]
In ice-9/psyntax-pp.scm:
5842:  89* [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
 535:  90* [# () # ()]
In ice-9/psyntax-pp.scm:
 782:  91  [# core-form # # ...]
In ice-9/psyntax-pp.scm:
5842:  92* [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
 535:  93* [# () # ()]
In ice-9/psyntax-pp.scm:
 782:  94  [# core-form # # ...]
In ice-9/psyntax-pp.scm:
5842:  95* [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
 535:  96* [# () # ()]
In ice-9/psyntax-pp.scm:
 782:  97  [# core-form # # ...]
In ice-9/psyntax-pp.scm:
5842:  98* [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
 535:  99* [# () # ()]
In ice-9/psyntax-pp.scm:
 782: 100  [# core-form # # ...]
In ice-9/psyntax-pp.scm:
5842: 101* [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
 535: 102* [# () # ()]
In ice-9/psyntax-pp.scm:
 782: 103  [# core-form # # ...]
In ice-9/psyntax-pp.scm:
5842: 104* [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
 535: 105* [# () # ()]
In ice-9/psyntax-pp.scm:
 782: 106  [# core-form # # ...]
In ice-9/psyntax-pp.scm:
5842: 107* [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
 535: 108* [# () # ()]
In ice-9/psyntax-pp.scm:
 782: 109  [# core-form # # ...]
In ice-9/psyntax-pp.scm:
5842: 110* [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
 535: 111* [# () # ()]
In ice-9/psyntax-pp.scm:
 782: 112  [# core-form # # ...]
In ice-9/psyntax-pp.scm:
5842: 113* [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
 535: 114* [# () # ()]
In ice-9/psyntax-pp.scm:
 782: 115  [# core-form # # ...]
In ice-9/psyntax-pp.scm:
5842: 116* [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
 535: 117* [# () # ()]
In ice-9/psyntax-pp.scm:
 782: 118  [# core-form # # ...]
In ice-9/psyntax-pp.scm:
5842: 119* [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
 535: 120* [# () # ()]
In ice-9/psyntax-pp.scm:
 782: 121  [# core-form # # ...]
In ice-9/psyntax-pp.scm:
5842: 122* [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
 535: 123* [# () # ()]
In ice-9/psyntax-pp.scm:
 782: 124  [# core-form # # ...]
In ice-9/psyntax-pp.scm:
5842: 125* [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
 535: 126* [# () # ()]
In ice-9/psyntax-pp.scm:
 782: 127  [# core-form # # ...]
In ice-9/psyntax-pp.scm:
5842: 128* [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
 535: 129* [# () # ()]
In ice-9/psyntax-pp.scm:
 782: 130  [# core-form # # ...]
In ice-9/psyntax-pp.scm:
5842: 131* [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
 535: 132* [# () # ()]
In ice-9/psyntax-pp.scm:
 782: 133  [# core-form # # ...]
In ice-9/psyntax-pp.scm:
5842: 134* [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
 535: 135* [# () # ()]
In ice-9/psyntax-pp.scm:
 782: 136  [# core-form # # ...]
In ice-9/psyntax-pp.scm:
5842: 137* [chi-let1039 (# # #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
1121: 138* [chi150 (# () #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
1422: 139  [syntax-type148 (# () #) (# # # # ...) (()) ...]
In ice-9/psyntax-pp.scm:
1378: 140  [syntax-type148 lambda (# # # # ...) (# # shift # ...) ...]
In ice-9/psyntax-pp.scm:
4942: 141  [lookup111 lambda (# # # # ...) (hygiene system base pmatch)]
In ice-9/psyntax-pp.scm:
5294: 142  [get-global-definition-hook79 lambda (hygiene system base pmatch)]
In ice-9/boot-9.scm:
1956: 143  [# #]

ERROR: Stack overflow

Re: autocompilation support in master

[Mark H Weaver]

Earlier, I wrote:
> Strangely, the stack overflow doesn't happen when I run the compile
> command (as echoed by make) directly from the command line.  I only
> see it happen when compiling via make.

I just noticed something.  Look at the result of (debug-options) in
the first backtrace I sent:

> (show-file-name #t stack 40000 debug backtrace depth 100 maxdepth 1000 frames 3 indent 10 width 79 procnames cheap)

Notice the "stack 40000", which means that the stack limit (as far as
guile is concerned) is only 40000 words, i.e. 160 kilobytes.  Other
times, I see much larger numbers there, which are close to the actual
stack resource limit (as set by setrlimit), and in those cases the
stack doesn't overflow.

I'm not able to reliably reproduce these problems, but it seems that
the problem is not in the compiler, but in the code that initializes
guile's internal notion of its stack limit.

Indeed, I'm able to compile the ecmascript code with a 512 kilobyte
stack, as long as guile doesn't get confused about its stack limit.

     Mark

Re: autocompilation support in master

[Mark H Weaver]

I wrote:
> > (show-file-name #t stack 40000 debug backtrace depth 100 maxdepth 1000 frames 3 indent 10 width 79 procnames cheap)
> 
> Notice the "stack 40000", which means that the stack limit (as far as
> guile is concerned) is only 40000 words, i.e. 160 kilobytes.  Other
> times, I see much larger numbers there, which are close to the actual
> stack resource limit (as set by setrlimit), and in those cases the
> stack doesn't overflow.

It turns out that 40000 is the initial value of guile's internal stack
limit, as part of the definition of scm_debug_opts[] in eval.c.  That
value is supposed to be overwritten by init_stack_limit() in debug.c,
but it will be left unchanged if getrlimit() fails, or if both the
hard and soft stack limits are set to infinity.

I suggest increasing the initial value of 40000 to something more
reasonable, perhaps 128000 words (which is approximately what my
system needs to successfully compile the ecmascript code), and
printing an error message if getrlimit fails.

     Mark

Re: autocompilation support in master

[Mark H Weaver]

Apologies for sending so many emails in so short a time, but I've put
together the final pieces of this puzzle.

GNU Make 3.81, the version in Debian lenny, sets the stack soft limit
to match the hard limit "so that alloca does not fail".  On my system,
the default stack hard limit is infinite, so Make sets the soft limit
to be infinite as well.

When both the hard and soft limits are infinite, Guile's
init_stack_limit() function in debug.c leaves the internal stack limit
at the initial value of 40000 words, i.e. 160 kilobytes, as set in
eval.c.

On my system, I need somewhere between 384-512 kilobytes to compile
the ecmascript code.  512k succeeds, and 384k fails.

That's all.  Time for sleep now :)

    Mark


> I wrote:
> > > (show-file-name #t stack 40000 debug backtrace depth 100 maxdepth 1000 frames 3 indent 10 width 79 procnames cheap)
> > 
> > Notice the "stack 40000", which means that the stack limit (as far as
> > guile is concerned) is only 40000 words, i.e. 160 kilobytes.  Other
> > times, I see much larger numbers there, which are close to the actual
> > stack resource limit (as set by setrlimit), and in those cases the
> > stack doesn't overflow.
> 
> It turns out that 40000 is the initial value of guile's internal stack
> limit, as part of the definition of scm_debug_opts[] in eval.c.  That
> value is supposed to be overwritten by init_stack_limit() in debug.c,
> but it will be left unchanged if getrlimit() fails, or if both the
> hard and soft stack limits are set to infinity.
> 
> I suggest increasing the initial value of 40000 to something more
> reasonable, perhaps 128000 words (which is approximately what my
> system needs to successfully compile the ecmascript code), and
> printing an error message if getrlimit fails.
> 
>      Mark
> 

Re: autocompilation support in master

[Neil Jerram]

Andy Wingo <wingo@pobox.com> writes:

> On Tue 09 Jun 2009 20:47, Mark H Weaver <mhw@netris.org> writes:
>
>> To generate the backtrace, I added the following lines near the top of
>> guile-tools.  Is there a better way?
>>
>>   (debug-enable 'debug)
>>   (debug-enable 'backtrace)
>>   (debug-set! depth 100)
>>   (write (debug-options-interface))
>>   (newline)
>
> No, this is the best way we have currently. As I understand it, the
> debugging evaluator is slower than the normal one -- but at this point,
> who cares? With the VM we are already faster and we keep backtraces. We
> should have backtraces on, always.
>
> Does anyone have objections to that? If not, I'll make that change
> within the week.

Sorry, but I'm not clear what change you're proposing here.

     Neil

Re: autocompilation support in master

[Neil Jerram]

Mark H Weaver <mhw@netris.org> writes:

> Apologies for sending so many emails in so short a time, but I've put
> together the final pieces of this puzzle.

Fantastic puzzling!

With Andy's change to increase the default stack limit to 160000,
`make check' passes for me now.  I've also unblocked the overnight
snapshot (which was stuck failing to `git pull' because of
lib/iconv*.h), so hopefully that will succeed tonight also.

Regards,
        Neil