Some more elisp aspects: Reader and documentation

Some more elisp aspects: Reader and documentation

[Daniel Kraft]

Hi Andy and all,

as the main elisp compiler gets more and more complete (well, still a 
lot details missing as well as probably even most built-ins, but 
anyways), I think I'll also work on a real elisp reader (currently, 
Guile's Scheme reader is used) as well as some (internals) documentation 
about what I did, am doing and will do in the future.  I hope this 
sounds like a good plan ;)  BTW, if you want to review the compiler, we 
could agree on a "freeze" to compiler changes on the branch so you can 
do it freely, while I work entirely on the reader or documentation?

Regarding the reader, I think we've got two basic options (this was 
already disussed briefly here at the start of my project, IIRC):

1) Use the existing reader code in C and try to adapt it so that it 
(maybe controlled by some kind of "options") cares correctly about the 
differences between elisp and Scheme.  This might save some work as the 
basic reading of S-expressions, literals and the like can be shared, and 
the code will probably run fast; on the other hand, I'm not sure if we 
might risk messing things up and getting some passages complicated in 
order to have it handle alternatively both Scheme and elisp.  To be 
honest, I've so far not looked at the reader C code thoroughly, so I 
can't judge what this will really lead us to.

2) Write a seperate elisp reader, possibly in Scheme (but could be C as 
well if that's important for performance).  This helps us keep "both" 
readers clean and seperate, but all has to be done from ground up and 
the code is probably slower (when written in Scheme).  But my opinion is 
that performance will not matter that much here anyways (because the 
compiler probably takes most of the time, not the parser), and 
implementing parsing of S-expressions and the like is not really hard, 
so we won't lose much by not sharing existing code here.  So if you 
don't have another opinion because you know the existing code better 
than I do or see problems I don't, I'd go with this approach.

For this and in order to get meaningful debug and error outputs, I guess 
I'll have to find and associate the source code positions myself with 
read objects, right?  It seems that I can use port-line and port-column 
to get the position of something just read, and set-source-properties! 
to associate line/column to the expressions?  Anything else I need to 
care about then?

Documentation:  Currently, I've got the module/language/elisp/README 
file with some notes, mainly about stuff implemented/missing as well as 
the extensions I implemented over original elisp.  Additionally, I tried 
to explain most "internal workings" very briefly directly with comments 
in the source code; but I think it would be nice to have some very basic 
information (like, how dynamic binding or the "void" value of variables 
is implemented, or the different modules for function/value slots) also 
in the real docs.  Where do you suggest I put them 
(chapter/section/subsection of what part of the documentation)?

Yours and have a nice weekend!

Daniel

-- 
Done:  Arc-Bar-Cav-Ran-Rog-Sam-Tou-Val-Wiz
To go: Hea-Kni-Mon-Pri

Re: Some more elisp aspects: Reader and documentation

[Ludovic Courtès]

Hi Daniel,

Sorry for the late reply.

Daniel Kraft <d@domob.eu> writes:

> 2) Write a seperate elisp reader, possibly in Scheme (but could be C
> as well if that's important for performance).  This helps us keep
> "both" readers clean and seperate, but all has to be done from ground
> up and the code is probably slower (when written in Scheme).

This sounds like the best option to me.  You could use SILex to build
the lexer (http://www.iro.umontreal.ca/~dube/) and `(text parse-lalr)'
for the parser.

It may be slower than the C counterpart at first but it's bound to get
faster as Andy implements more crazy compiler optimizations.  :-)

Thanks,
Ludo'.

Re: Some more elisp aspects: Reader and documentation

[Daniel Kraft]

Hi Ludo,

Ludovic Courtès wrote:
> Sorry for the late reply.

no problem, thanks for the reply!  I had a lot of other stuff to do 
anyways, and so didn't really get into the situation of waiting for a 
reply here ;)

>> 2) Write a seperate elisp reader, possibly in Scheme (but could be C
>> as well if that's important for performance).  This helps us keep
>> "both" readers clean and seperate, but all has to be done from ground
>> up and the code is probably slower (when written in Scheme).
> 
> This sounds like the best option to me.  You could use SILex to build
> the lexer (http://www.iro.umontreal.ca/~dube/) and `(text parse-lalr)'
> for the parser.

Hm, ok, I'll look into those.  On the other hand I guess that Lisp has 
such a simple syntax that hand-writing some recursive-descent parser may 
be an equally good option?  Well, I'll simply give it a try!

But it seems that the LALR parser generator is part of guile-lib, and 
will introduce that as a new dependency for Guile; do you think that's 
ok?  (Haven't checked how SILex works and if it may introduce dependencies.)

Yours,
Daniel

Re: Some more elisp aspects: Reader and documentation

[Ludovic Courtès]

Daniel Kraft <d@domob.eu> writes:

>>> 2) Write a seperate elisp reader, possibly in Scheme (but could be C
>>> as well if that's important for performance).  This helps us keep
>>> "both" readers clean and seperate, but all has to be done from ground
>>> up and the code is probably slower (when written in Scheme).
>>
>> This sounds like the best option to me.  You could use SILex to build
>> the lexer (http://www.iro.umontreal.ca/~dube/) and `(text parse-lalr)'
>> for the parser.
>
> Hm, ok, I'll look into those.  On the other hand I guess that Lisp has
> such a simple syntax that hand-writing some recursive-descent parser
> may be an equally good option?  Well, I'll simply give it a try!

Well, yes, you may be right.

> But it seems that the LALR parser generator is part of guile-lib, and
> will introduce that as a new dependency for Guile;

Actually it's already in `master', because it's used for ECMAScript.
I'd need to be made part of the official API, though.

> do you think that's ok?  (Haven't checked how SILex works and if it
> may introduce dependencies.)

SILex generates a source file containing the lexer, so there's no
additional run-time dependency.  Guile's tarball could come with the
generated file, such that end-users don't need to have SILex installed.
For convenience, it may be helpful to have SILex in the repository,
though (that's what Guile-RPC does).

Thanks,
Ludo'.

Re: Some more elisp aspects: Reader and documentation

[Daniel Kraft]

Hi Ludo,

>>>> 2) Write a seperate elisp reader, possibly in Scheme (but could be C
>>>> as well if that's important for performance).  This helps us keep
>>>> "both" readers clean and seperate, but all has to be done from ground
>>>> up and the code is probably slower (when written in Scheme).
>>> This sounds like the best option to me.  You could use SILex to build
>>> the lexer (http://www.iro.umontreal.ca/~dube/) and `(text parse-lalr)'
>>> for the parser.
>> Hm, ok, I'll look into those.  On the other hand I guess that Lisp has
>> such a simple syntax that hand-writing some recursive-descent parser
>> may be an equally good option?  Well, I'll simply give it a try!
> 
> Well, yes, you may be right.

I'll play around a little and see what grammar I'd get and how easy it 
could be without the additional tools, and then make my choice!

>> But it seems that the LALR parser generator is part of guile-lib, and
>> will introduce that as a new dependency for Guile;
> 
> Actually it's already in `master', because it's used for ECMAScript.
> I'd need to be made part of the official API, though.
 >
>> do you think that's ok?  (Haven't checked how SILex works and if it
>> may introduce dependencies.)
> 
> SILex generates a source file containing the lexer, so there's no
> additional run-time dependency.  Guile's tarball could come with the
> generated file, such that end-users don't need to have SILex installed.
> For convenience, it may be helpful to have SILex in the repository,
> though (that's what Guile-RPC does).

Ok thanks for the hints!  I'll try it out!

Cheers,
Daniel