REPL reader of multi-language

REPL reader of multi-language

[Nala Ginrut]

hi folks!
When I'm hacking guile-lua on Guile, I realize that some languages may
need two different reader for REPL(interactive environment) and
compiler.

For an instance, the reader in Lua REPL has little different from its
compiler reader:
------------------------------cut-------------------------------------
In interactive mode, Lua usually interprets each line that you type as a
complete chunk. However, if it detects that the line cannot form a
complete chunk, it waits for more input, until it has a complete chunk.
When Lua is waiting for a line continuation, it shows a different
prompt...
------------------------------end--------------------------------------

But in Lua compiler/interpreter, a chunk(the minimum unit of compiling)
will be detected when the reader encounters EOF. Except for -l option
which let you input multiple chunks from command line.

It's not a problem for Ecmascript, because its statement has to be ended
with semicolon. So it's easier to detect the minimum compiling unit in
REPL. 
Lua accepts semicolon as an optional ending token. But it's strange to
force users to input semicolon or Ctrl+d to see the result in REPL. It's
better to mimic original Lua REPL.


So here's my problem, there's only one reader slot defined in each
language type ,say, in (system base language). I have to implement just
one reader to fit them all:
1. If I provide the interactive reader in Guile, the reader becomes
inefficient. Because each time users input `Enter', the source code
typed so far will be detected in lalr-parser, and exception would be
caught when the line doesn't complete a chunk, then the REPL returns
continue prompt, say '...'.

2. If I provide compiler reader, people have to type semicolon or ctrl+d
to see the result.


Here's my solution:
1. Provide two readers in language type, repl-reader and reader. And let
the frontend writers decide how to use;
2. Another way is to set a flag when we're in interactive mode. But I
didn't find the mechanism in our repl modules;
3. I saw there're various hooks in REPL, but there' no any docs for
them. Maybe there's chance to take advantage of them?

Any help?

Re: REPL reader of multi-language

[Mike Gerwitz]

[-- Attachment #1: Type: text/plain, Size: 1479 bytes --]

On Fri, May 30, 2014 at 04:25:40PM +0800, Nala Ginrut wrote:
> It's not a problem for Ecmascript, because its statement has to be ended
> with semicolon. So it's easier to detect the minimum compiling unit in
> REPL. 

ECMAScript uses semicolon insertion---semicolons are optional, but this can
lead to ambiguities. For example:

  function foo() {
    return
      { foo: "bar" }
  }

`foo` will return undefined, not {foo:"bar}, because it inserts a semi-colon
after `return`; the way to get around this is to put the beginning curly
brace of the object literal on the same line as `return`, in which case the
expression would be completed by the closing curly brace of the object
literal, semicolon optional.

If you take a look at Node.js' REPL, you can see how it handles line
continuations.

So this problem would exist for more than just Lua. But is it actually a
problem? Ctrl+D, if needed to end an expression, would represent EOF; I
don't know if this is valid in Lua, but in ES, if an EOF (or semicolon) is
encountered during the parsing of an expression that requires more input,
then it is a syntax error---the compiler and REPL would handle it in the
same way, but the REPL would just display a PS2-style prompt (which readline
seems to do as "..." right in Guile 2.0).

It could be my ignorance of Lua. :)

-- 
Mike Gerwitz
Free Software Hacker | GNU Maintainer
http://mikegerwitz.com
FSF Member #5804 | GPG Key ID: 0x8EE30EAB

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: REPL reader of multi-language

[Nala Ginrut]

On Fri, 2014-05-30 at 08:48 -0400, Mike Gerwitz wrote:
> On Fri, May 30, 2014 at 04:25:40PM +0800, Nala Ginrut wrote:
> > It's not a problem for Ecmascript, because its statement has to be ended
> > with semicolon. So it's easier to detect the minimum compiling unit in
> > REPL. 
> 
> ECMAScript uses semicolon insertion---semicolons are optional, but this can
> lead to ambiguities. For example:
> 
>   function foo() {
>     return
>       { foo: "bar" }
>   }
> 
> `foo` will return undefined, not {foo:"bar}, because it inserts a semi-colon
> after `return`; the way to get around this is to put the beginning curly
> brace of the object literal on the same line as `return`, in which case the
> expression would be completed by the closing curly brace of the object
> literal, semicolon optional.
> 

Oh~I didn't notice that, thanks for pointing out! ;-)

> If you take a look at Node.js' REPL, you can see how it handles line
> continuations.
> 

Thanks! I'm going to read node.js code.

> So this problem would exist for more than just Lua. But is it actually a
> problem?

I want to provide it because the original Lua REPL provide it. Anyway,
it's relative trivial in the whole compiler frontend, so I just throw
the issue here, but with lower priority at present.

>  Ctrl+D, if needed to end an expression, would represent EOF; I
> don't know if this is valid in Lua, but in ES, if an EOF (or semicolon) is
> encountered during the parsing of an expression that requires more input,
> then it is a syntax error---the compiler and REPL would handle it in the
> same way, but the REPL would just display a PS2-style prompt (which readline
> seems to do as "..." right in Guile 2.0).
> 

Yes, the situation you described is right, the EOF shouldn't appear in
the middle of the valid code.
But what I mean is, our REPL reader don't know when to generate EOF if
you don't type it explicitly. The current parser will start to work when
it encounters EOF. So the REPL will wait infinitely till you give it an
EOF. This is the problem I'm considering to fix.
The original Lua REPL will detect if it's a valid production in the
parser which is used to decide if the current token list is a valid
chunk to be parsed. If no, it prints continue prompt, yes for evaluating
it.
I don't know how to provide it in scm-lalr, throw & catch exceptions in
parser seems ugly.

If you use `guild compile' or `guile -c' to run the Lua code, it's OK
because it'll get EOF successfully. 

Thanks!

Re: REPL reader of multi-language

[Ludovic Courtès]

Nala Ginrut <nalaginrut@gmail.com> skribis:

> So here's my problem, there's only one reader slot defined in each
> language type ,say, in (system base language). I have to implement just
> one reader to fit them all:
> 1. If I provide the interactive reader in Guile, the reader becomes
> inefficient. Because each time users input `Enter', the source code
> typed so far will be detected in lalr-parser, and exception would be
> caught when the line doesn't complete a chunk, then the REPL returns
> continue prompt, say '...'.
>
> 2. If I provide compiler reader, people have to type semicolon or ctrl+d
> to see the result.

Commit 65fa60 adjusted the REPL for multi-language support.  In
particular that works fine with both Scheme/elisp and ECMAScript.

What exactly is missing for Lua to be usable at the REPL?

Ludo’.

Re: REPL reader of multi-language

[Nala Ginrut]

Sorry for late reply, I was on my vocation. ;-)

On Fri, 2014-05-30 at 21:08 +0200, Ludovic Courtès wrote:
> Nala Ginrut <nalaginrut@gmail.com> skribis:
> 
> > So here's my problem, there's only one reader slot defined in each
> > language type ,say, in (system base language). I have to implement just
> > one reader to fit them all:
> > 1. If I provide the interactive reader in Guile, the reader becomes
> > inefficient. Because each time users input `Enter', the source code
> > typed so far will be detected in lalr-parser, and exception would be
> > caught when the line doesn't complete a chunk, then the REPL returns
> > continue prompt, say '...'.
> >
> > 2. If I provide compiler reader, people have to type semicolon or ctrl+d
> > to see the result.
> 
> Commit 65fa60 adjusted the REPL for multi-language support.  In
> particular that works fine with both Scheme/elisp and ECMAScript.
> 

Yes, I saw it. It provided dependent `reader' for each language.
Actually, my question was based on it. I requested adding `repl-reader',
but now I don't think it's the best way.  

> What exactly is missing for Lua to be usable at the REPL?
> 

After rethinking this issue, I think I need a method to detect if I'm
under interactive environment(REPL) or not, then I can handle the reader
properly.
I found *repl-stack* maybe a way for that, I guess it'll be '() when I'm
not in REPL. Can I rely on it?


Thanks!

> Ludo’.
> 
>