GSOC PEG project

GSOC PEG project

[Michael Lucy]

Hi,

I finally got things into a usable state, so I pushed out what I have
to origin/mlucy.  Many apologies if I did something wrong; I've never
used git with remote repositories before.

Files I've added so far:
guile/modules/ice-9/peg.scm (I assume this is the right place to put this?)
guile/test-suite/tests/peg.test

Status of things right now:
PEG compiler: Works for all the grammars I've tested, no known bugs.
Currently compiles to Scheme code rather than Tree-IL for debugging.
Comments: Pretty good I think, but I could stand to have more.
External documentation: None so far.
Syntactic Sugar: Canonical string-based PEGs can be parsed just fine.
Testing: Small test suite.  Needs to be standardized with the other .test files.
Benchmarks: None so far, but I haven't really done any optimization so
I'd bet it performs poorly.

Problems:

1. PEG compiler compiles to scheme code rather than TREE-IL, and uses
some custom pieces of syntax that I then have to export.  I've been
meaning to ask, how big a deal is this?  I originally had it compile
to Scheme while I was debugging things, but the Scheme gets compiled
down to TREE-IL form, right?  Should I fix it to compile straight to
TREE-IL anyway rather than just finding a way to get around having to
export all that syntax?

2. Tests aren't standardized (peg.test looks nothing like the other
.test files).

3. This:

scheme@(guile-user)> (use-modules (ice-9 peg))
;;; note: source file /home/zededarian/guile/module/ice-9/peg.scm
;;;       newer than compiled
/home/zededarian/guile/cache/guile/ccache/2.0-0.R-LE-4/home/zededarian/guile/module/ice-9/peg.scm.go
;;; note: autocompilation is enabled, set GUILE_AUTO_COMPILE=0
;;;       or pass the --no-autocompile argument to disable.
;;; compiling /home/zededarian/guile/module/ice-9/peg.scm
;;; WARNING: compilation of /home/zededarian/guile/module/ice-9/peg.scm failed:
;;; key wrong-type-arg, throw args ("vm-debug-engine" "Wrong type to
apply: ~S" (#f) (#f))

Somehow it works fine despite the warnings.  Can anybody shed light on
what this means?

4. External documentation hasn't even been started yet.

5. The pretty-print module doesn't get loaded when peg does, and some
of my functions use it.  I have:

:use-module (ice-9 pretty-print)

in my define-module form at the top of peg.scm; shouldn't that take care of it?

Any thoughts or comments?  I'm sorry I didn't get this into a branch earlier.

Re: GSOC PEG project

[Ludovic Courtès]

Hi!

Thanks for the status report!

Michael Lucy <MichaelGLucy@Gmail.com> writes:

> Files I've added so far:
> guile/modules/ice-9/peg.scm (I assume this is the right place to put this?)

No, it should rather go under ‘module/language/peg.scm’, for the sake of
consistency with other compiler front-ends.  But see below.

> PEG compiler: Works for all the grammars I've tested, no known bugs.
> Currently compiles to Scheme code rather than Tree-IL for debugging.

It should rather compiler to tree-IL, like other front-ends, which is
very close to Scheme anyway.

> 2. Tests aren't standardized (peg.test looks nothing like the other
> .test files).

Well, you know what to do then.  ;-)

> 3. This:
>
> scheme@(guile-user)> (use-modules (ice-9 peg))
> ;;; note: source file /home/zededarian/guile/module/ice-9/peg.scm
> ;;;       newer than compiled
> /home/zededarian/guile/cache/guile/ccache/2.0-0.R-LE-4/home/zededarian/guile/module/ice-9/peg.scm.go
> ;;; note: autocompilation is enabled, set GUILE_AUTO_COMPILE=0
> ;;;       or pass the --no-autocompile argument to disable.
> ;;; compiling /home/zededarian/guile/module/ice-9/peg.scm
> ;;; WARNING: compilation of /home/zededarian/guile/module/ice-9/peg.scm failed:
> ;;; key wrong-type-arg, throw args ("vm-debug-engine" "Wrong type to
> apply: ~S" (#f) (#f))
>
> Somehow it works fine despite the warnings.  Can anybody shed light on
> what this means?

Actually it doesn’t work.  Try:

  $ ./meta/guile --no-autocompile
  > (use-modules (ice-9 peg))

It should trigger the debugger from where you can get a backtrace.

Thanks,
Ludo’.

Re: GSOC PEG project

[Michael Lucy]

On Mon, Jul 5, 2010 at 5:40 PM, Ludovic Courtès <ludo@gnu.org> wrote:
> Hi!
>
> Thanks for the status report!
>
> Michael Lucy <MichaelGLucy@Gmail.com> writes:
>
>> Files I've added so far:
>> guile/modules/ice-9/peg.scm (I assume this is the right place to put this?)
>
> No, it should rather go under ‘module/language/peg.scm’, for the sake of
> consistency with other compiler front-ends.  But see below.

I'm not quite sure how this qualifies as a compiler front-end.  The
functionality is mostly done (if unoptimized), so it shouldn't be too
hard to turn it into one if that's what you want, but the original
thought was just to build a compiling parser.  E.g.:

(use-modules (ice-9 peg))
(peg-find "'b'+" "aabbcc")
--> (2 4 "bb")

It does this by compiling "'b'+" into a lambda expression that then
gets called on whatever the second argument is (the compilation is
done at read-time).  So after macro-expansion, you have something
like:
((lambda (...) ...) "aabbcc" ...)

I was thinking of this as a sort of alternative to regular expressions
that also does parsing, and I noticed the regular expressions module
was in ice-9, so I put it there.

Are you saying that the functionality that compiles "'b'+" to a lambda
expression should be broken out into a language module that instead
compiles it to TREE-IL code?  Am I fundamentally misunderstanding
something?

>
>> PEG compiler: Works for all the grammars I've tested, no known bugs.
>> Currently compiles to Scheme code rather than Tree-IL for debugging.
>
> It should rather compiler to tree-IL, like other front-ends, which is
> very close to Scheme anyway.

Alright, I'll fix that.

Noob question:
The only way I can figure out to compile tree-il code is the rather ugly:
(compile (parse-tree-il '(code here)) #:from 'tree-il)
Is there a better way?

>
>> 2. Tests aren't standardized (peg.test looks nothing like the other
>> .test files).
>
> Well, you know what to do then.  ;-)
>
>> 3. This:
>>
>> scheme@(guile-user)> (use-modules (ice-9 peg))
>> ;;; note: source file /home/zededarian/guile/module/ice-9/peg.scm
>> ;;;       newer than compiled
>> /home/zededarian/guile/cache/guile/ccache/2.0-0.R-LE-4/home/zededarian/guile/module/ice-9/peg.scm.go
>> ;;; note: autocompilation is enabled, set GUILE_AUTO_COMPILE=0
>> ;;;       or pass the --no-autocompile argument to disable.
>> ;;; compiling /home/zededarian/guile/module/ice-9/peg.scm
>> ;;; WARNING: compilation of /home/zededarian/guile/module/ice-9/peg.scm failed:
>> ;;; key wrong-type-arg, throw args ("vm-debug-engine" "Wrong type to
>> apply: ~S" (#f) (#f))
>>
>> Somehow it works fine despite the warnings.  Can anybody shed light on
>> what this means?
>
> Actually it doesn’t work.  Try:

It can definitely use the functions:

"
~/guile-git/tst/guile/meta $ ./guile
GNU Guile 1.9.11.154-b1066
Copyright (C) 1995-2010 Free Software Foundation, Inc.

Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.

Enter `,help' for help.
scheme@(guile-user)> (use-modules (ice-9 peg))
;;; note: autocompilation is enabled, set GUILE_AUTO_COMPILE=0
;;;       or pass the --no-autocompile argument to disable.
;;; compiling /home/zededarian/guile-git/tst/guile/module/ice-9/peg.scm
;;; WARNING: compilation of
/home/zededarian/guile-git/tst/guile/module/ice-9/peg.scm failed:
;;; key wrong-type-arg, throw args ("vm-debug-engine" "Wrong type to
apply: ~S" (#f) (#f))
scheme@(guile-user)> (peg-find "'b'+" "aabbcc")
(2 4 "bb")
scheme@(guile-user)>
"

And when I use it with --no-autocompile I don't get any errors:

"
~/guile-git/tst/guile/meta $ ./guile --no-autocompile
GNU Guile 1.9.11.154-b1066
Copyright (C) 1995-2010 Free Software Foundation, Inc.

Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.

Enter `,help' for help.
scheme@(guile-user)> (use-modules (ice-9 peg))
scheme@(guile-user)> (peg-find "'b'+" "aabbcc")
(2 4 "bb")
scheme@(guile-user)>
"

What does this mean?

>
>  $ ./meta/guile --no-autocompile
>  > (use-modules (ice-9 peg))
>
> It should trigger the debugger from where you can get a backtrace.
>
> Thanks,
> Ludo’.
>
>
>

Thanks.

Re: GSOC PEG project

[No Itisnt]

On Mon, Jul 5, 2010 at 6:59 PM, Michael Lucy <MichaelGLucy@gmail.com> wrote:
> On Mon, Jul 5, 2010 at 5:40 PM, Ludovic Courtès <ludo@gnu.org> wrote:
>> Hi!
>>
>> Thanks for the status report!
>>
>> Michael Lucy <MichaelGLucy@Gmail.com> writes:
>>
>>> Files I've added so far:
>>> guile/modules/ice-9/peg.scm (I assume this is the right place to put this?)
>>
>> No, it should rather go under ‘module/language/peg.scm’, for the sake of
>> consistency with other compiler front-ends.  But see below.
>
> I'm not quite sure how this qualifies as a compiler front-end.  The
> functionality is mostly done (if unoptimized), so it shouldn't be too
> hard to turn it into one if that's what you want, but the original
> thought was just to build a compiling parser.  E.g.:
>
> (use-modules (ice-9 peg))
> (peg-find "'b'+" "aabbcc")
> --> (2 4 "bb")
>
> It does this by compiling "'b'+" into a lambda expression that then
> gets called on whatever the second argument is (the compilation is
> done at read-time).  So after macro-expansion, you have something
> like:
> ((lambda (...) ...) "aabbcc" ...)
>
> I was thinking of this as a sort of alternative to regular expressions
> that also does parsing, and I noticed the regular expressions module
> was in ice-9, so I put it there.
>
> Are you saying that the functionality that compiles "'b'+" to a lambda
> expression should be broken out into a language module that instead
> compiles it to TREE-IL code?  Am I fundamentally misunderstanding
> something?
>
>>
>>> PEG compiler: Works for all the grammars I've tested, no known bugs.
>>> Currently compiles to Scheme code rather than Tree-IL for debugging.
>>
>> It should rather compiler to tree-IL, like other front-ends, which is
>> very close to Scheme anyway.

I think you are both misunderstanding eachother:

- The PEG functions, useful in their own right, would be exposed as (ice-9 peg).
- There is an additional syntax for expressing grammars, so it can be
better used as a parser generator. That would go in as (language peg).

That's just my take though.

A personal whinge: I would like to be able to specify grammars in
Scheme, using macros as the sugar instead of a different syntax.

>
> Alright, I'll fix that.
>
> Noob question:
> The only way I can figure out to compile tree-il code is the rather ugly:
> (compile (parse-tree-il '(code here)) #:from 'tree-il)
> Is there a better way?

I'm not sure if you mean generating tree-il code or just compiling the
tree-il literals. If you want to generate it, use the record interface
from (language tree-il), if you want to compile the literals, that's
pretty much the way to do it. You can also do ,language tree-il at the
REPL to experiment with it.

Exciting stuff :) I can't wait to use this, maintaining my own
hand-written recursive descent parser is like watching a trainwreck in
slow motion.

Re: GSOC PEG project

[Ludovic Courtès]

Hello,

No Itisnt <theseaisinhere@gmail.com> writes:

> On Mon, Jul 5, 2010 at 6:59 PM, Michael Lucy <MichaelGLucy@gmail.com> wrote:
>> On Mon, Jul 5, 2010 at 5:40 PM, Ludovic Courtès <ludo@gnu.org> wrote:
>>> Michael Lucy <MichaelGLucy@Gmail.com> writes:
>>>
>>>> Files I've added so far:
>>>> guile/modules/ice-9/peg.scm (I assume this is the right place to put this?)
>>>
>>> No, it should rather go under ‘module/language/peg.scm’, for the sake of
>>> consistency with other compiler front-ends.  But see below.
>>
>> I'm not quite sure how this qualifies as a compiler front-end.  The
>> functionality is mostly done (if unoptimized), so it shouldn't be too
>> hard to turn it into one if that's what you want, but the original
>> thought was just to build a compiling parser.  E.g.:
>>
>> (use-modules (ice-9 peg))
>> (peg-find "'b'+" "aabbcc")
>> --> (2 4 "bb")

[...]

>>>> PEG compiler: Works for all the grammars I've tested, no known bugs.
>>>> Currently compiles to Scheme code rather than Tree-IL for debugging.
>>>
>>> It should rather compiler to tree-IL, like other front-ends, which is
>>> very close to Scheme anyway.
>
> I think you are both misunderstanding eachother:
>
> - The PEG functions, useful in their own right, would be exposed as (ice-9 peg).
> - There is an additional syntax for expressing grammars, so it can be
> better used as a parser generator. That would go in as (language peg).

OK, I understand now.  Sorry for the confusion!

Ludo’.

Re: GSOC PEG project

[Andy Wingo]

Hi Michael,

I am travelling currently, so a bit of asynchronicity in the reply :)

On Mon 05 Jul 2010 07:05, Michael Lucy <MichaelGLucy@Gmail.com> writes:

> I finally got things into a usable state, so I pushed out what I have
> to origin/mlucy.

Great news!

> Files I've added so far:
> guile/modules/ice-9/peg.scm (I assume this is the right place to put this?)

Probably, yes, for parts of PEG to be available as macros and
procedures. You would probably want a language implementation as well,
for string-based PEGs.

> PEG compiler: Works for all the grammars I've tested, no known bugs.
> Currently compiles to Scheme code rather than Tree-IL for debugging.

That is the right thing to do. Most of PEG should be available through
macros and procedures.

> Comments: Pretty good I think, but I could stand to have more.

Yes, it readable at a quick glance.

> External documentation: None so far.

A .texi file will be good, alongside the LALR(1) parser.

> Syntactic Sugar: Canonical string-based PEGs can be parsed just fine.

Cool!

> Testing: Small test suite.  Needs to be standardized with the other .test files.

Great.

> Benchmarks: None so far, but I haven't really done any optimization so
> I'd bet it performs poorly.

Well, it's good to have work left to do :)

> Problems:
>
> 1. PEG compiler compiles to scheme code rather than TREE-IL, and uses
> some custom pieces of syntax that I then have to export.  I've been
> meaning to ask, how big a deal is this?  I originally had it compile
> to Scheme while I was debugging things, but the Scheme gets compiled
> down to TREE-IL form, right?  Should I fix it to compile straight to
> TREE-IL anyway rather than just finding a way to get around having to
> export all that syntax?

No it should compile to Scheme. However the biggest problem with your
code is that it is implemented with define-macro. You should change it
to be implemented with syntax-case. Please let me know if you need help.

Syntax-case is a bit trickier in some ways than define-macro, but it's
simpler in others (e.g., no need for your `safe-bind' macro). Check out
"Macros" in the manual (from git) and tell me what's confusing there; it
was only fleshed out with info on syntax-case recently.

> 2. Tests aren't standardized (peg.test looks nothing like the other
> .test files).

This isn't a big deal

> 3. This:
>
> scheme@(guile-user)> (use-modules (ice-9 peg))
> ;;; note: source file /home/zededarian/guile/module/ice-9/peg.scm
> ;;;       newer than compiled
> /home/zededarian/guile/cache/guile/ccache/2.0-0.R-LE-4/home/zededarian/guile/module/ice-9/peg.scm.go
> ;;; note: autocompilation is enabled, set GUILE_AUTO_COMPILE=0
> ;;;       or pass the --no-autocompile argument to disable.
> ;;; compiling /home/zededarian/guile/module/ice-9/peg.scm
> ;;; WARNING: compilation of /home/zededarian/guile/module/ice-9/peg.scm failed:
> ;;; key wrong-type-arg, throw args ("vm-debug-engine" "Wrong type to
> apply: ~S" (#f) (#f))
>
> Somehow it works fine despite the warnings.  Can anybody shed light on
> what this means?

Dunno! We'll look at it after getting the macro situation sorted
out. You can try:

  (use-modules (system base compile))
  (compile-file "module/ice-9/peg.scm")

and see where the debugger leaves you.

> 5. The pretty-print module doesn't get loaded when peg does, and some
> of my functions use it.  I have:
>
> :use-module (ice-9 pretty-print)
>
> in my define-module form at the top of peg.scm; shouldn't that take
> care of it?

Yep, that should work.

> Any thoughts or comments?  I'm sorry I didn't get this into a branch earlier.

Heh, no problem. I'm looking forward to a lean mean PEG parsing machine!

Andy
-- 
http://wingolog.org/

Re: GSOC PEG project

[Andy Wingo]

Hi Michael,

On Tue 06 Jul 2010 00:59, Michael Lucy <MichaelGLucy@Gmail.com> writes:

> (use-modules (ice-9 peg))
> (peg-find "'b'+" "aabbcc")
> --> (2 4 "bb")

Humm, another thing to think about: (ice-9 regex) returns "match
structures", which are really just vectors; have a look at them, and if
it makes sense to mimic that interface, re-exporting those accessors
somehow, please do.

> And when I use it with --no-autocompile I don't get any errors:
>
> What does this mean?

Probably some eval-when tomfoolery. See "Eval When" in the manual.

Cheers,

Andy
-- 
http://wingolog.org/

Re: GSOC PEG project

[Andy Wingo]

Good day sir or lady,

Agreed regarding all of your points, and particularly:

On Tue 06 Jul 2010 02:41, No Itisnt <theseaisinhere@gmail.com> writes:

> A personal whinge: I would like to be able to specify grammars in
> Scheme, using macros as the sugar instead of a different syntax.

Yes!

> Exciting stuff :) I can't wait to use this, maintaining my own
> hand-written recursive descent parser is like watching a trainwreck in
> slow motion.

Also yes!

Happy hacking,

Andy
-- 
http://wingolog.org/

Re: GSOC PEG project

[Michael Lucy]

On Thu, Jul 8, 2010 at 5:29 AM, Andy Wingo <wingo@pobox.com> wrote:
> Hi Michael,
>
> I am travelling currently, so a bit of asynchronicity in the reply :)
>
> On Mon 05 Jul 2010 07:05, Michael Lucy <MichaelGLucy@Gmail.com> writes:
>
>> I finally got things into a usable state, so I pushed out what I have
>> to origin/mlucy.
>
> Great news!
>
>> Files I've added so far:
>> guile/modules/ice-9/peg.scm (I assume this is the right place to put this?)
>
> Probably, yes, for parts of PEG to be available as macros and
> procedures. You would probably want a language implementation as well,
> for string-based PEGs.
>
>> PEG compiler: Works for all the grammars I've tested, no known bugs.
>> Currently compiles to Scheme code rather than Tree-IL for debugging.
>
> That is the right thing to do. Most of PEG should be available through
> macros and procedures.
>
>> Comments: Pretty good I think, but I could stand to have more.
>
> Yes, it readable at a quick glance.
>
>> External documentation: None so far.
>
> A .texi file will be good, alongside the LALR(1) parser.

Working on this now; should be done soon.

>
>> Syntactic Sugar: Canonical string-based PEGs can be parsed just fine.
>
> Cool!
>
>> Testing: Small test suite.  Needs to be standardized with the other .test files.
>
> Great.
>
>> Benchmarks: None so far, but I haven't really done any optimization so
>> I'd bet it performs poorly.
>
> Well, it's good to have work left to do :)
>
>> Problems:
>>
>> 1. PEG compiler compiles to scheme code rather than TREE-IL, and uses
>> some custom pieces of syntax that I then have to export.  I've been
>> meaning to ask, how big a deal is this?  I originally had it compile
>> to Scheme while I was debugging things, but the Scheme gets compiled
>> down to TREE-IL form, right?  Should I fix it to compile straight to
>> TREE-IL anyway rather than just finding a way to get around having to
>> export all that syntax?
>
> No it should compile to Scheme. However the biggest problem with your
> code is that it is implemented with define-macro. You should change it
> to be implemented with syntax-case. Please let me know if you need help.
>
> Syntax-case is a bit trickier in some ways than define-macro, but it's
> simpler in others (e.g., no need for your `safe-bind' macro). Check out
> "Macros" in the manual (from git) and tell me what's confusing there; it
> was only fleshed out with info on syntax-case recently.

The documentation looks pretty straightforward.  This will probably
take me a bit; some of the translations look a little complicated.
This will also solve my current problem of having to export a bunch of
macros for the compiled PEGs to work.

I've been thinking it might be good to work on the benchmarks
first--once I understand where the speed bottlenecks are I'll have to
rewrite some of the macros anyway, and it would probably be easier to
rewrite them as syntax-case macros then instead of doing it twice.  If
you'd rather I translate them first, though, I'm happy to.

Sorry I didn't do it this way in the first place.

>
>> 2. Tests aren't standardized (peg.test looks nothing like the other
>> .test files).
>
> This isn't a big deal
>
>> 3. This:
>>
>> scheme@(guile-user)> (use-modules (ice-9 peg))
>> ;;; note: source file /home/zededarian/guile/module/ice-9/peg.scm
>> ;;;       newer than compiled
>> /home/zededarian/guile/cache/guile/ccache/2.0-0.R-LE-4/home/zededarian/guile/module/ice-9/peg.scm.go
>> ;;; note: autocompilation is enabled, set GUILE_AUTO_COMPILE=0
>> ;;;       or pass the --no-autocompile argument to disable.
>> ;;; compiling /home/zededarian/guile/module/ice-9/peg.scm
>> ;;; WARNING: compilation of /home/zededarian/guile/module/ice-9/peg.scm failed:
>> ;;; key wrong-type-arg, throw args ("vm-debug-engine" "Wrong type to
>> apply: ~S" (#f) (#f))
>>
>> Somehow it works fine despite the warnings.  Can anybody shed light on
>> what this means?
>
> Dunno! We'll look at it after getting the macro situation sorted
> out. You can try:
>
>  (use-modules (system base compile))
>  (compile-file "module/ice-9/peg.scm")
>
> and see where the debugger leaves you.

Turns out I can fix this by wrapping my entire file in an (eval-when
(compile load eval) ...) form.  Is this one of those fixes that's
going to haunt me down the line?

>
>> 5. The pretty-print module doesn't get loaded when peg does, and some
>> of my functions use it.  I have:
>>
>> :use-module (ice-9 pretty-print)
>>
>> in my define-module form at the top of peg.scm; shouldn't that take
>> care of it?
>
> Yep, that should work.
>
>> Any thoughts or comments?  I'm sorry I didn't get this into a branch earlier.
>
> Heh, no problem. I'm looking forward to a lean mean PEG parsing machine!
>
> Andy
> --
> http://wingolog.org/
>

Re: GSOC PEG project

[Michael Lucy]

On Fri, Jul 9, 2010 at 2:58 AM, Michael Lucy <MichaelGLucy@gmail.com> wrote:
> On Thu, Jul 8, 2010 at 5:29 AM, Andy Wingo <wingo@pobox.com> wrote:
>> A .texi file will be good, alongside the LALR(1) parser.
>
> Working on this now; should be done soon.
>

Pushed a first version of api-peg.texi up to origin/mlucy (and added
it to guile.texi right under the LALR(1) parser).  I plan to let it
sit for a bit and then come back to it with fresh eyes; I'll go back
to working on actual functionality for now.

Re: GSOC PEG project

[Michael Lucy]

On Thu, Jul 8, 2010 at 11:21 AM, Andy Wingo <wingo@pobox.com> wrote:
> Hi Michael,
>
> On Tue 06 Jul 2010 00:59, Michael Lucy <MichaelGLucy@Gmail.com> writes:
>
>> (use-modules (ice-9 peg))
>> (peg-find "'b'+" "aabbcc")
>> --> (2 4 "bb")
>
> Humm, another thing to think about: (ice-9 regex) returns "match
> structures", which are really just vectors; have a look at them, and if
> it makes sense to mimic that interface, re-exporting those accessors
> somehow, please do.

So, there are a few places where the interfaces don't quite match up:

1. match:substring
Problem: It's perfectly legal to pass peg-match a parsing nonterminal
and have it give you a parse tree rather than a substring.
Potential solutions:
1.a. Just have match:substring return either the substring or the
parse tree.  The problem with this is that it may violate
expectations.
1.b. Have match:substring collapse the parse tree into a string, and
have another function match:parse-tree that would return the parse
tree.  The problem with this is that the parse tree might have
discarded text, which would once again violate expectations.

2. match:count
Problem: the notion of a "parenthesized sub-expression" doesn't really
map cleanly onto PEGs.  This information isn't tracked while parsing
and wouldn't be very meaningful.
Potential solutions:
2.a. Ignore it (not that bad a solution).
2.b. Track that information.  I'd rather not do this because it would
slow down the parser and wouldn't be very useful.  Which brings us
to...

3. submatch numbers
Problem: there's no notion of a "submatch" right now.  People should
be getting this information by building a parsing nonterminal and then
traversing the resulting parse-tree.  I'd rather not wire in a whole
separate system just to provide an alternative way of getting
information about what parts of an expression matched what (it would
also slow down parsing).
Potential solutions:
3.a. Ignore it (would violate expectations in a big way).
3.b. Wire it in (I'd really rather not do this).

So, there would be some gaps if I shimmied the match structure
interface onto PEGs.

The problem is that, while it would be useful to have a consistent
interface for matching both regexps and PEGs, they're different things
and naming the accessor functions the same things might lead people to
assume things that aren't true.

So, three potential paths from here:
1. Mimic the match structure interface as much as possible.
2. Have a similar but differently-named "peg-match structure"
interface that behaves mostly the same but has a few different
functions (I think naming them something slightly different would lead
to fewer people assuming they worked exactly the same as match
structures).
3. Just having a different interface.

I'm leaning toward (2); what do other people think?  I'd probably:
1. Not have a peg-match:count function at all.
2. Not have the functions take submatch numbers.
3. Have peg-match:substring return the actual substring.
4. Have another function peg-match:parse-tree that returns the parse tree.

>
>> And when I use it with --no-autocompile I don't get any errors:
>>
>> What does this mean?
>
> Probably some eval-when tomfoolery. See "Eval When" in the manual.
>
> Cheers,
>
> Andy
> --
> http://wingolog.org/
>

Re: GSOC PEG project

[Andy Wingo]

Heya,

Sorry for the delayed reply again! I am trying to drain my email swamp,
but there is a sucking sound there...

On Fri 09 Jul 2010 09:58, Michael Lucy <MichaelGLucy@Gmail.com> writes:

>> You should change it to be implemented with syntax-case. Please let
>> me know if you need help.
>
> Sorry I didn't do it this way in the first place.

No need to apologize! Hacking Scheme is a journey and we all come from
different places.

> wrapping my entire file in an (eval-when (compile load eval) ...)
> form.  Is this one of those fixes that's going to haunt me down the
> line?

No, it will be easy to fix, should the problem remain after your
syntax-case foo.

Cheers,

Andy
-- 
http://wingolog.org/

Re: GSOC PEG project

[Andy Wingo]

On Sun 11 Jul 2010 09:48, Michael Lucy <MichaelGLucy@Gmail.com> writes:

> On Thu, Jul 8, 2010 at 11:21 AM, Andy Wingo <wingo@pobox.com> wrote:
>
>> Humm, another thing to think about: (ice-9 regex) returns "match
>> structures", which are really just vectors; have a look at them, and if
>> it makes sense to mimic that interface, re-exporting those accessors
>> somehow, please do.
>
> So, three potential paths from here:
> 1. Mimic the match structure interface as much as possible.
> 2. Have a similar but differently-named "peg-match structure"
> interface that behaves mostly the same but has a few different
> functions (I think naming them something slightly different would lead
> to fewer people assuming they worked exactly the same as match
> structures).
> 3. Just having a different interface.
>
> I'm leaning toward (2); what do other people think?  I'd probably:
> 1. Not have a peg-match:count function at all.
> 2. Not have the functions take submatch numbers.
> 3. Have peg-match:substring return the actual substring.
> 4. Have another function peg-match:parse-tree that returns the parse
> tree.

Yes, if the needs are different, there's no sense in trying to horn the
present into the past's shoe. Take the good conventions from (ice-9
match), but there is no strict need for compatibility.

Andy
-- 
http://wingolog.org/

Re: GSOC PEG project

[Ludovic Courtès]

Hi,

Andy Wingo <wingo@pobox.com> writes:

> On Sun 11 Jul 2010 09:48, Michael Lucy <MichaelGLucy@Gmail.com> writes:
>
>> On Thu, Jul 8, 2010 at 11:21 AM, Andy Wingo <wingo@pobox.com> wrote:
>>
>>> Humm, another thing to think about: (ice-9 regex) returns "match
>>> structures", which are really just vectors; have a look at them, and if
>>> it makes sense to mimic that interface, re-exporting those accessors
>>> somehow, please do.
>>
>> So, three potential paths from here:
>> 1. Mimic the match structure interface as much as possible.
>> 2. Have a similar but differently-named "peg-match structure"
>> interface that behaves mostly the same but has a few different
>> functions (I think naming them something slightly different would lead
>> to fewer people assuming they worked exactly the same as match
>> structures).
>> 3. Just having a different interface.
>>
>> I'm leaning toward (2); what do other people think?  I'd probably:
>> 1. Not have a peg-match:count function at all.
>> 2. Not have the functions take submatch numbers.
>> 3. Have peg-match:substring return the actual substring.
>> 4. Have another function peg-match:parse-tree that returns the parse
>> tree.
>
> Yes, if the needs are different, there's no sense in trying to horn the
> present into the past's shoe. Take the good conventions from (ice-9
> match), but there is no strict need for compatibility.

This should read (ice-9 regex), I think.

Ludo’.

Re: GSOC PEG project

[Andy Wingo]

On Sat 17 Jul 2010 14:56, ludo@gnu.org (Ludovic Courtès) writes:

> Andy Wingo <wingo@pobox.com> writes:
>
>> Take the good conventions from (ice-9 match), but there is no strict
>> need for compatibility.
>
> This should read (ice-9 regex), I think.

Ah indeed, my mistake. I think my head was swimming at this point with
having read too many old mails ;)

Andy
-- 
http://wingolog.org/