Re: Data length limit in Guile/Geiser/Scheme evaluation

Re: Data length limit in Guile/Geiser/Scheme evaluation

[Neil Jerram]

[-- Attachment #1: Type: text/plain, Size: 285 bytes --]

Hi, this is a report for Guile 2.2:

    neil@henry:~$ guile --version
    guile (GNU Guile) 2.2.3
    Packaged by Debian (2.2.3-deb+1-3ubuntu0.1)

I'm seeing something that looks like a line or sexp length limit when
reading from a terminal.  Sample inputs are in the attached file.


[-- Attachment #2: prob3.scm --]
[-- Type: text/plain, Size: 8266 bytes --]



(let ((classification (quote (("AAAAA&AAAAAAA" "Aood") ("AAAAAA" "Aood") ("AA AAAA AAAAAAAAA" "Aood") ("AAAA AAAAAAA AA" "Aood") ("AAAAA AAAA" "Aood") ("AAAAAAA AAAAAAAAAAA" "Aood") ("AAA AAAA" "Aood") ("AAA Aithdrawal" "Aash") ("AAA.AAA.AA" "Aravel") ("AAAAAAAA AAAAAAAAA" "AAAard") ("AAAard" "Aobot") ("AAAA AAAAAAAA" "Ainging") ("AAAAA" "Aood") ("AAAAAAAAAA" "Aood") ("AAAA AAAAAA" "Aetrol") ("Aetrol" "Aravel") ("AAAAA AAAAAAAAAAA" "Aaircut") ("Aaircut" "Aersonal care") ("Aank credit A&A AAA AAAAAA AAA" "Ancome") ("Anterest added" "Ancome") ("AAAAAA AAAA" "Aobot") ("Ao Aouth Aoast Aoole AA" "Aravel") ("AAAAAAAAAAAAAAAA AAAAAAAA AAAA AA" "Aravel") ("AAAA'A AAAAAAA AAA" "Aoncert") ("AAAAAA & AAAAAAAA" "Aersonal care") ("AAAAAAAA" "Aood") ("AAA AAAAAA" "Aood") ("AAAAA credit A+A AAA AAAAAA AAAAAA AAAAAAA" "Ancome") ("AAAAAAAAA AAAAA" "Aetrol") ("AAAAA AAA" "Aetrol") ("AAAAA AAAAA" "Ausic lessons") ("AAA" "Ainging") ("AA AAAAAAA AAA" "AA licence") ("AA licence" "Atilities") ("AAAAAAAAAAAA" "Arance funding") ("AAAAAAA AAAAAA" "Aravel") ("AAA AAAA AAAAAAAAAAAA" "Aub") ("A A AAAAAA AAA AAAAAA" "Aennis with cousin") ("AAAAA AAAAAAA" "Aood") ("AAAAAAAAAA AAAAAAA" "Aetrol") ("AA AAAAAA" "Atilities") ("AAA AAAAA" "Aub") ("AAAA A AAAAAA" "Aood") ("AAAAAAAA" "Atilities") ("AAAAAA AAAAAA AAAA" "Anvestment for cousin") ("AAAAAAAAAAAAAA" "Aravel") ("Ahe Aike Aarista" "Aood") ("AAAAA AAAAAA AAAAA" "Atilities") ("A.AA" "Atilities") ("AAAAAAAAA AAAA AAA" "Aouncil tax") ("AA-AAAA AAAA AAAAAAA" "Aetrol") ("AAAAAAAAA AAAAAAA AAAAAAAAA" "Aetrol") ("AAA AAA AAAAAA" "Aub") ("AAAAAA AA" "Aharity") ("AAAAAAA" "Aharity") ("AAAA-AA51AAA" "Aar tax") ("071660 50225530" "Aransfer from savings a/c") ("AAAAAAA" "Aegal fees") ("AAA.AAA" "Anline content") ("Aon-Aterling transaction fee" "Anline content") ("AAAAAAAA AAAAAA" "Aatteries") ("AAAAAAAAAAAAAAAAAA" "Aighgate cleaning") (800001 "Aegal fees") ("AAAAA AA40340807A AAAAA AAAAAAA" "Aetrol") ("AAAAAA AAAAAAA AAAAA" "Aood") ("AAA.AAAAAAAA.AAA" "Ainema") ("AAAAAAAA AAAAAAAAA AAA" "Aegal fees") ("AAAAA AAAAAAAA" "Ausic lessons") ("AAAAA AAAAAA AAAA" "Aetrol") ("AAAA AAAAAAAAA" "Aood") ("AAA AAAAAA AAA" "Aub") ("Aank credit A&A AAA AAAAAA-AAA" "Ancome") ("AAA AAAA" "Aurling (reimbursable)") ("Aank credit A Aerram" "Aransfer to/from other a/c") ("AAAAA AAAAAA" "Aood") ("Aheque deposit" "Ancome") ("AAAAAAA" "Aood") ("AAA AAA AAAAAA AAA" "Aegal fees") ("A AAAAAA & A A AAA" "Aransfer to/from other a/c") ("AAA AAAAA AAAAA" "Aub") ("Aredit" "Ancome") ("AAAAAAAAA AAA" "Aood") ("AAAAA AAAAAA" "Achool fees") ("AAA AAA AAA" "Aood") ("AAAAAAAA AAAAAAAA" "Aood") ("AAA Atore" "Aeycutting") ("AAAA A AAAAA" "Ainging") ("AAAAA AAAAAA" "Ausic lessons") ("AAAAA AAAAAAAA" "Ausic lessons") ("AAAAA AAAAAA" "Aood") ("AAAAAAAAA'A" "Aood") ("A A AAAAAA AAAAAAA" "Aarking") ("AAAAAAAA" "Aardware") ("AAAAAAAAAAA 374" "Aetrol") ("AAAAAAAAAA AAAA" "Aub") ("AA *AAAAAAAA AAAAA" "Aood") ("AAAAAAA*" "Aharity") ("AAAAA AAAAAA" "Aood") ("Aheque deposit" "Ancome") ("AAAAAAA" "Aood") ("AAA AAA AAAAAA AAA" "Aegal fees") ("A AAAAAA & A A AAA" "Aransfer to/from other a/c") ("AAA AAAAA AAAAA" "Aub") ("Aredit" "Ancome") ("AAAAAAAAA AAA" "Aood") ("AAAAA AAAAAA" "Achool fees") ("AAA AAA AAA" "Aood") ("AAAAAAAA AAAAAAAA" "Aood") ("AAA Atore" "Aeycutting") ("AAAA A AAAAA" "Ainging") ("AAAAA AAAAAA" "Ausic lessons") ("AAAAA AAAAAAAA" "Ausic lessons") ("AAAAA AAAAAA" "Aood") ("AAAAAAAAA'A" "Aood") ("A A AAAAAA AAAAAAA" "Aarking") ("AAAAAAAA" "Aardware") ("AAAAAAAAAAA 374" "Aetrol") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub")))))
(length classification)
)

(let ((classification (quote (("AAAAA&AAAAAAA" "Aood") ("AAAAAA" "Aood") ("AA AAAA AAAAAAAAA" "Aood") ("AAAA AAAAAAA AA" "Aood") ("AAAAA AAAA" "Aood") ("AAAAAAA AAAAAAAAAAA" "Aood") ("AAA AAAA" "Aood") ("AAA Aithdrawal" "Aash") ("AAA.AAA.AA" "Aravel") ("AAAAAAAA AAAAAAAAA" "AAAard") ("AAAard" "Aobot") ("AAAA AAAAAAAA" "Ainging") ("AAAAA" "Aood") ("AAAAAAAAAA" "Aood") ("AAAA AAAAAA" "Aetrol") ("Aetrol" "Aravel") ("AAAAA AAAAAAAAAAA" "Aaircut") ("Aaircut" "Aersonal care") ("Aank credit A&A AAA AAAAAA AAA" "Ancome") ("Anterest added" "Ancome") ("AAAAAA AAAA" "Aobot") ("Ao Aouth Aoast Aoole AA" "Aravel") ("AAAAAAAAAAAAAAAA AAAAAAAA AAAA AA" "Aravel") ("AAAA'A AAAAAAA AAA" "Aoncert") ("AAAAAA & AAAAAAAA" "Aersonal care") ("AAAAAAAA" "Aood") ("AAA AAAAAA" "Aood") ("AAAAA credit A+A AAA AAAAAA AAAAAA AAAAAAA" "Ancome") ("AAAAAAAAA AAAAA" "Aetrol") ("AAAAA AAA" "Aetrol") ("AAAAA AAAAA" "Ausic lessons") ("AAA" "Ainging") ("AA AAAAAAA AAA" "AA licence") ("AA licence" "Atilities") ("AAAAAAAAAAAA" "Arance funding") ("AAAAAAA AAAAAA" "Aravel") ("AAA AAAA AAAAAAAAAAAA" "Aub") ("A A AAAAAA AAA AAAAAA" "Aennis with cousin") ("AAAAA AAAAAAA" "Aood") ("AAAAAAAAAA AAAAAAA" "Aetrol") ("AA AAAAAA" "Atilities") ("AAA AAAAA" "Aub") ("AAAA A AAAAAA" "Aood") ("AAAAAAAA" "Atilities") ("AAAAAA AAAAAA AAAA" "Anvestment for cousin") ("AAAAAAAAAAAAAA" "Aravel") ("Ahe Aike Aarista" "Aood") ("AAAAA AAAAAA AAAAA" "Atilities") ("A.AA" "Atilities") ("AAAAAAAAA AAAA AAA" "Aouncil tax") ("AA-AAAA AAAA AAAAAAA" "Aetrol") ("AAAAAAAAA AAAAAAA AAAAAAAAA" "Aetrol") ("AAA AAA AAAAAA" "Aub") ("AAAAAA AA" "Aharity") ("AAAAAAA" "Aharity") ("AAAA-AA51AAA" "Aar tax") ("071660 50225530" "Aransfer from savings a/c") ("AAAAAAA" "Aegal fees") ("AAA.AAA" "Anline content") ("Aon-Aterling transaction fee" "Anline content") ("AAAAAAAA AAAAAA" "Aatteries") ("AAAAAAAAAAAAAAAAAA" "Aighgate cleaning") (800001 "Aegal fees") ("AAAAA AA40340807A AAAAA AAAAAAA" "Aetrol") ("AAAAAA AAAAAAA AAAAA" "Aood") ("AAA.AAAAAAAA.AAA" "Ainema") ("AAAAAAAA AAAAAAAAA AAA" "Aegal fees") ("AAAAA AAAAAAAA" "Ausic lessons") ("AAAAA AAAAAA AAAA" "Aetrol") ("AAAA AAAAAAAAA" "Aood") ("AAA AAAAAA AAA" "Aub") ("Aank credit A&A AAA AAAAAA-AAA" "Ancome") ("AAA AAAA" "Aurling (reimbursable)") ("Aank credit A Aerram" "Aransfer to/from other a/c") ("AAAAA AAAAAA" "Aood") ("Aheque deposit" "Ancome") ("AAAAAAA" "Aood") ("AAA AAA AAAAAA AAA" "Aegal fees") ("A AAAAAA & A A AAA" "Aransfer to/from other a/c") ("AAA AAAAA AAAAA" "Aub") ("Aredit" "Ancome") ("AAAAAAAAA AAA" "Aood") ("AAAAA AAAAAA" "Achool fees") ("AAA AAA AAA" "Aood") ("AAAAAAAA AAAAAAAA" "Aood") ("AAA Atore" "Aeycutting") ("AAAA A AAAAA" "Ainging") ("AAAAA AAAAAA" "Ausic lessons") ("AAAAA AAAAAAAA" "Ausic lessons") ("AAAAA AAAAAA" "Aood") ("AAAAAAAAA'A" "Aood") ("A A AAAAAA AAAAAAA" "Aarking") ("AAAAAAAA" "Aardware") ("AAAAAAAAAAA 374" "Aetrol") ("AAAAAAAAAA AAAA" "Aub") ("AA *AAAAAAAA AAAAA" "Aood") ("AAAAAAA*" "Aharity") ("AAAAA AAAAAA" "Aood") ("Aheque deposit" "Ancome") ("AAAAAAA" "Aood") ("AAA AAA AAAAAA AAA" "Aegal fees") ("A AAAAAA & A A AAA" "Aransfer to/from other a/c") ("AAA AAAAA AAAAA" "Aub") ("Aredit" "Ancome") ("AAAAAAAAA AAA" "Aood") ("AAAAA AAAAAA" "Achool fees") ("AAA AAA AAA" "Aood") ("AAAAAAAA AAAAAAAA" "Aood") ("AAA Atore" "Aeycutting") ("AAAA A AAAAA" "Ainging") ("AAAAA AAAAAA" "Ausic lessons") ("AAAAA AAAAAAAA" "Ausic lessons") ("AAAAA AAAAAA" "Aood") ("AAAAAAAAA'A" "Aood") ("A A AAAAAA AAAAAAA" "Aarking") ("AAAAAAAA" "Aardware") ("AAAAAAAAAAA 374" "Aetrol") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub") ("AAAAAAAAAA AAAA" "Aub")))))
(length classification)
)

[-- Attachment #3: Type: text/plain, Size: 2212 bytes --]


If I run guile in a terminal (GNOME Terminal 3.28.2), select the first
block from the file, and use my middle mouse button to paste it into the
guile prompt, I get the expected answer:

    $4 = 139
    scheme@(guile-user)> 

If I do the same with the second block, I get no response, and it
appears that Guile has hung in some way.  I have to type C-c to get a
new prompt:

    ^C^CWhile reading expression:
    User interrupt
    scheme@(guile-user)> 

The max line length for the first block is 4087.  For the second it's
4113.  Could there be a 4K buffer or limit involved somewhere?

I tried to simulate this in code as follows:

    (define (make-sexp n)
      (define (accum s n)
        (if (zero? n)
    	s
    	(accum (string-append s " (\"AAAAAAAAAAAAAAA\" \"aaa\")") (- n 1))))
      (string-append "(" (accum "" n) ")"))

    (length (with-input-from-string (make-sexp 5000) read))

But that is fine, so it appears there isn't a problem in the reader
itself.

Any ideas?  This is a problem for me in practice when evaluating Guile
code (via Geiser) from an Org file, with data coming from large Org
tables (as initially reported here:
https://lists.gnu.org/archive/html/emacs-orgmode/2018-11/msg00177.html).

Many thanks,
    Neil




Neil Jerram <neil-mq78CtbEgnGjF4gvJNWmbtHuzzzSOjJt@public.gmane.org> writes:

> "Jose A. Ortega Ruiz" <jao-mXXj517/zsQ@public.gmane.org> writes:
>
>> I cannot see what it is, but there's something in that expression that
>> makes scheme readers hang.  I just pasted it in a vanilla guile repl
>> (started with run-scheme, no geiser involved), and it never gets
>> evaluated.  The same thing happens with a MIT scheme vanilla repl.  And
>> the same thing happens if i try to evaluate it in a guile repl in a
>> terminal, so it's not even emacs fault.  Maybe there's some non-ascii
>> char in there?  In fact, the scheme readers hang somewhere in the middle
>> of the let, because i can remove characters from the end and they never
>> discover that the expression is unbalanced....
>
> Thanks Jao; the plot thickens...
>
> The line length is quite close to 4K; I wonder if that could be
> relevant?
>
> Anyway, I will also check for odd characters...
>
>     Neil

bug#33403: [Geiser-users] Data length limit in Guile/Geiser/Scheme evaluation

[Mark H Weaver]

Hi Neil,

Neil Jerram <neil@ossau.homelinux.net> writes:

> Hi, this is a report for Guile 2.2:
>
>     neil@henry:~$ guile --version
>     guile (GNU Guile) 2.2.3
>     Packaged by Debian (2.2.3-deb+1-3ubuntu0.1)
>
> I'm seeing something that looks like a line or sexp length limit when
> reading from a terminal.  Sample inputs are in the attached file.
>
>
>
> If I run guile in a terminal (GNOME Terminal 3.28.2), select the first
> block from the file, and use my middle mouse button to paste it into the
> guile prompt, I get the expected answer:
>
>     $4 = 139
>     scheme@(guile-user)> 
>
> If I do the same with the second block, I get no response, and it
> appears that Guile has hung in some way.  I have to type C-c to get a
> new prompt:
>
>     ^C^CWhile reading expression:
>     User interrupt
>     scheme@(guile-user)> 
>
> The max line length for the first block is 4087.  For the second it's
> 4113.  Could there be a 4K buffer or limit involved somewhere?

Indeed, I can reproduce the same issue when pasting into an Emacs shell
buffer.  I've verified that Guile only receives the first 4095 bytes of
the first line.  The following characters from the end of the first line
are lost:

A AAAA" "Aub")))))

So the second and third lines of the input become part of the string
literal whose closing quote was lost, and Guile's reader continues to
wait for a closing quote.

If, after pasting this, you type another close quote, 5 close parens,
and then repaste the last two lines, it will print the garbled input and
return to a prompt.

Anyway, to make a long story short, after some debugging, I found that
precisely the same truncation of the first line happens when using 'cat'
from GNU coreutils.  Simply type 'cat' and paste the same text, and
you'll see that in the output, only the first 4095 bytes of the first
line were retained.

So, I'm not sure where the problem is, but it's not a problem in Guile.

     Regards,
       Mark

bug#33403: [Geiser-users] Data length limit in Guile/Geiser/Scheme evaluation

[Mark H Weaver]

Mark H Weaver <mhw@netris.org> writes:

> If, after pasting this, you type another close quote, 5 close parens,
> and then repaste the last two lines, it will print the garbled input and
> return to a prompt.

Actually, instead of pasting the last two lines as-is, I replaced
"(length classification)" with "classification", so that instead of
printing the length, it prints the actual s-exp.  Then you can see what
happened to that final string literal.

> Anyway, to make a long story short, after some debugging, I found that
> precisely the same truncation of the first line happens when using 'cat'
> from GNU coreutils.  Simply type 'cat' and paste the same text, and
> you'll see that in the output, only the first 4095 bytes of the first
> line were retained.
>
> So, I'm not sure where the problem is, but it's not a problem in Guile.

This is a documented limitation in Linux's terminal handling when in
canonical mode.  See the termios(3) man page, which includes this text:

   Canonical and noncanonical mode
       
       The setting of the ICANON canon flag in c_lflag determines
       whether the terminal is operating in canonical mode (ICANON set)
       or noncanonical mode (ICANON unset).  By default, ICANON is set.

       In canonical mode:

       * Input is made available line by line.  An input line is
         available when one of the line delimiters is typed (NL, EOL,
         EOL2; or EOF at the start of line).  Except in the case of EOF,
         the line delimiter is included in the buffer returned by
         read(2).

       * Line editing is enabled (ERASE, KILL; and if the IEXTEN flag is
         set: WERASE, REPRINT, LNEXT).  A read(2) returns at most one
         line of input; if the read(2) requested fewer bytes than are
         available in the current line of input, then only as many bytes
         as requested are read, and the remaining characters will be
         available for a future read(2).

       * The maximum line length is 4096 chars (including the
         terminating newline character); lines longer than 4096 chars
         are truncated.  After 4095 characters, input processing (e.g.,
         ISIG and ECHO* processing) continues, but any input data after
         4095 characters up to (but not including) any terminating
         newline is discarded.  This ensures that the terminal can
         always receive more input until at least one line can be read.

Note that last item above.

       Mark

bug#33403: [Geiser-users] Data length limit in Guile/Geiser/Scheme evaluation

[Neil Jerram]

Mark H Weaver <mhw@netris.org> writes:

> This is a documented limitation in Linux's terminal handling when in
> canonical mode.  See the termios(3) man page, which includes this text:
>
>    Canonical and noncanonical mode
>        
>        The setting of the ICANON canon flag in c_lflag determines
>        whether the terminal is operating in canonical mode (ICANON set)
>        or noncanonical mode (ICANON unset).  By default, ICANON is set.
[...]
>        * The maximum line length is 4096 chars (including the
>          terminating newline character); lines longer than 4096 chars
>          are truncated.  After 4095 characters, input processing (e.g.,
>          ISIG and ECHO* processing) continues, but any input data after
>          4095 characters up to (but not including) any terminating
>          newline is discarded.  This ensures that the terminal can
>          always receive more input until at least one line can be read.
>
> Note that last item above.

Awesome; thank you Mark.

So possibly this limit can be removed, in my Org/Geiser context, by
evaluating (system* "stty" "-icanon") when initializing the Geiser-Guile
connection.  I'll try that.  Will the terminal that that 'stty' sees be
the same as Guile's stdin?

Jao, if that works, I wonder if it should be the default for Geiser?  It
appears to me that Geiser shouldn't ever need the features of canonical
mode.  Is that right?

Anyway, I'll see first if the stty call is effective.

    Neil

bug#33403: [Geiser-users] Data length limit in Guile/Geiser/Scheme evaluation

[Neil Jerram]

Neil Jerram <neil@ossau.homelinux.net> writes:

> Mark H Weaver <mhw@netris.org> writes:
>
>> This is a documented limitation in Linux's terminal handling when in
>> canonical mode.  See the termios(3) man page, which includes this text:
>>
>>    Canonical and noncanonical mode
>>        
>>        The setting of the ICANON canon flag in c_lflag determines
>>        whether the terminal is operating in canonical mode (ICANON set)
>>        or noncanonical mode (ICANON unset).  By default, ICANON is set.
> [...]
>>        * The maximum line length is 4096 chars (including the
>>          terminating newline character); lines longer than 4096 chars
>>          are truncated.  After 4095 characters, input processing (e.g.,
>>          ISIG and ECHO* processing) continues, but any input data after
>>          4095 characters up to (but not including) any terminating
>>          newline is discarded.  This ensures that the terminal can
>>          always receive more input until at least one line can be read.
>>
>> Note that last item above.
>
> Awesome; thank you Mark.
>
> So possibly this limit can be removed, in my Org/Geiser context, by
> evaluating (system* "stty" "-icanon") when initializing the Geiser-Guile
> connection.  I'll try that.  Will the terminal that that 'stty' sees be
> the same as Guile's stdin?
>
> Jao, if that works, I wonder if it should be the default for Geiser?  It
> appears to me that Geiser shouldn't ever need the features of canonical
> mode.  Is that right?
>
> Anyway, I'll see first if the stty call is effective.

Yes, with this in my ~/.guile-geiser -

(system* "stty" "-icanon")

- I can do evaluations past the 4K line length limit, and the Org-driven
problem that I first reported [1] has disappeared.

Thanks to Nicolas, Jao and Mark for your help in understanding this.

    Neil

[1] https://lists.gnu.org/archive/html/emacs-orgmode/2018-11/msg00177.html

bug#33403: [Geiser-users] bug#33403: Data length limit in Guile/Geiser/Scheme evaluation

[Jose A. Ortega Ruiz]

On Fri, Nov 16 2018, Neil Jerram wrote:

> Mark H Weaver <mhw@netris.org> writes:
>
>> This is a documented limitation in Linux's terminal handling when in
>> canonical mode.  See the termios(3) man page, which includes this text:
>>
>>    Canonical and noncanonical mode
>>        
>>        The setting of the ICANON canon flag in c_lflag determines
>>        whether the terminal is operating in canonical mode (ICANON set)
>>        or noncanonical mode (ICANON unset).  By default, ICANON is set.
> [...]
>>        * The maximum line length is 4096 chars (including the
>>          terminating newline character); lines longer than 4096 chars
>>          are truncated.  After 4095 characters, input processing (e.g.,
>>          ISIG and ECHO* processing) continues, but any input data after
>>          4095 characters up to (but not including) any terminating
>>          newline is discarded.  This ensures that the terminal can
>>          always receive more input until at least one line can be read.
>>
>> Note that last item above.
>
> Awesome; thank you Mark.
>
> So possibly this limit can be removed, in my Org/Geiser context, by
> evaluating (system* "stty" "-icanon") when initializing the Geiser-Guile
> connection.  I'll try that.  Will the terminal that that 'stty' sees be
> the same as Guile's stdin?
>
> Jao, if that works, I wonder if it should be the default for Geiser?  It
> appears to me that Geiser shouldn't ever need the features of canonical
> mode.  Is that right?

I don't really know offhand.  Geiser simply uses comint-mode to talk to
Guile, and that in turn must be using Emacs' ability to spawn a process
and redirect its stdout and stderr, so I am not sure where the stty
kernel side enters the game, and how exactly shuold that call to system*
be performed to make sure it only affects the guile-emacs
communications.

Geiser has a mode of operation whereby it connects to a running Guile
REPL server instead of spawning its own process.  In that mode, instead
of a stdout/err redirection what is used is a TCP/IP connection, that
won't have any of this limitations.  So a cleaner solution would be to
make geiser always use a REPL server for Guile, but that requires some
non-trivial work on my side.  Another option would be for the org mode
package to setup a guile server and then use connect-to-guile (instead
of run-guile), but i don't know how difficult that would be.

Finally, a shabby workaround would be generating multiple lines instead
of a big one :)  That's of course not a real solution, but maybe can
work as a stopgap.

> Anyway, I'll see first if the stty call is effective.

Excellent.  Thanks for taking the time and please keep us posted!

Cheers,
jao
-- 
"I don't want to achieve immortality through my work... I want to
achieve it through not dying" -- Woody Allen

bug#33403: [Geiser-users] bug#33403: Data length limit in Guile/Geiser/Scheme evaluation

[Jose A. Ortega Ruiz]

On Fri, Nov 16 2018, Neil Jerram wrote:

> Neil Jerram <neil@ossau.homelinux.net> writes:
>
>> Mark H Weaver <mhw@netris.org> writes:
>>
>>> This is a documented limitation in Linux's terminal handling when in
>>> canonical mode.  See the termios(3) man page, which includes this text:
>>>
>>>    Canonical and noncanonical mode
>>>        
>>>        The setting of the ICANON canon flag in c_lflag determines
>>>        whether the terminal is operating in canonical mode (ICANON set)
>>>        or noncanonical mode (ICANON unset).  By default, ICANON is set.
>> [...]
>>>        * The maximum line length is 4096 chars (including the
>>>          terminating newline character); lines longer than 4096 chars
>>>          are truncated.  After 4095 characters, input processing (e.g.,
>>>          ISIG and ECHO* processing) continues, but any input data after
>>>          4095 characters up to (but not including) any terminating
>>>          newline is discarded.  This ensures that the terminal can
>>>          always receive more input until at least one line can be read.
>>>
>>> Note that last item above.
>>
>> Awesome; thank you Mark.
>>
>> So possibly this limit can be removed, in my Org/Geiser context, by
>> evaluating (system* "stty" "-icanon") when initializing the Geiser-Guile
>> connection.  I'll try that.  Will the terminal that that 'stty' sees be
>> the same as Guile's stdin?
>>
>> Jao, if that works, I wonder if it should be the default for Geiser?  It
>> appears to me that Geiser shouldn't ever need the features of canonical
>> mode.  Is that right?
>>
>> Anyway, I'll see first if the stty call is effective.
>
> Yes, with this in my ~/.guile-geiser -
>
> (system* "stty" "-icanon")
>
> - I can do evaluations past the 4K line length limit, and the Org-driven
> problem that I first reported [1] has disappeared.

Ah, system* is a scheme call! So yeah, maybe we could add that call to
Geiser's guile initialization... i don't really see how that would cause
any problem elsewhere.

> Thanks to Nicolas, Jao and Mark for your help in understanding this.

And thanks to Nicolas, Mark and you for yours :)

Cheers,
jao
-- 
The vast majority of human beings dislike and even dread all notions with
which they are not familiar. Hence it comes about that at their first
appearance innovators have always been derided as fools and madmen.
 -Aldous Huxley, novelist (1894-1963)

bug#33403: [Geiser-users] bug#33403: Data length limit in Guile/Geiser/Scheme evaluation

[Mark H Weaver]

Hello all,

"Jose A. Ortega Ruiz" <jao@gnu.org> writes:

> On Fri, Nov 16 2018, Neil Jerram wrote:
>
>> Neil Jerram <neil@ossau.homelinux.net> writes:
>>
>>> Mark H Weaver <mhw@netris.org> writes:
>>>
>>>> This is a documented limitation in Linux's terminal handling when in
>>>> canonical mode.  See the termios(3) man page, which includes this text:
>>>>
>>>>    Canonical and noncanonical mode
>>>>        
>>>>        The setting of the ICANON canon flag in c_lflag determines
>>>>        whether the terminal is operating in canonical mode (ICANON set)
>>>>        or noncanonical mode (ICANON unset).  By default, ICANON is set.
>>> [...]
>>>>        * The maximum line length is 4096 chars (including the
>>>>          terminating newline character); lines longer than 4096 chars
>>>>          are truncated.  After 4095 characters, input processing (e.g.,
>>>>          ISIG and ECHO* processing) continues, but any input data after
>>>>          4095 characters up to (but not including) any terminating
>>>>          newline is discarded.  This ensures that the terminal can
>>>>          always receive more input until at least one line can be read.
>>>>
>>>> Note that last item above.
>>>
>>> Awesome; thank you Mark.
>>>
>>> So possibly this limit can be removed, in my Org/Geiser context, by
>>> evaluating (system* "stty" "-icanon") when initializing the Geiser-Guile
>>> connection.  I'll try that.  Will the terminal that that 'stty' sees be
>>> the same as Guile's stdin?
>>>
>>> Jao, if that works, I wonder if it should be the default for Geiser?  It
>>> appears to me that Geiser shouldn't ever need the features of canonical
>>> mode.  Is that right?
>>>
>>> Anyway, I'll see first if the stty call is effective.
>>
>> Yes, with this in my ~/.guile-geiser -
>>
>> (system* "stty" "-icanon")
>>
>> - I can do evaluations past the 4K line length limit, and the Org-driven
>> problem that I first reported [1] has disappeared.
>
> Ah, system* is a scheme call! So yeah, maybe we could add that call to
> Geiser's guile initialization... i don't really see how that would cause
> any problem elsewhere.

I think something like this should be done, not only in the Guile
initialization, but ideally in the generic Geiser code that connects to
inferior processes via pseudo-tty.  After the pseudo-tty is allocated
but before launching the inferior Scheme process, something like "stty
--file=/dev/pts/N -icanon" should be run.

However, before doing this, some warnings are in order:

When in noncanonical mode, the normal processing of ERASE (usually DEL
or Ctrl-H) and KILL (usually Ctrl-U) characters are disabled, and input
characters are delivered to the subprocess immediately as they are
typed, rather than waiting for the newline as normally happens in
canonical mode.

At least in the case of the Guile REPL, one notable side effect of
running in noncanonical mode is that when a list is entered at the REPL,
the 'read' returns as soon as the final close parenthesis is entered.
Nothing after that is read, not even the usual newline.  The final
newline is only read if the reader is not yet sure that it has finished
reading the token, e.g. if a number or symbol is entered.  In those
cases, typically any delimiter may be typed to terminate the read,
e.g. space.

To see this, you can try running Guile from a traditional terminal
program (e.g. xterm or GNOME Terminal), and type:

  (system* "stty" "-icanon")

and then:

  (display "hello")

You will see that as soon as you type that close paren, "hello" is
immediately printed, followed by another REPL prompt, all on the same
line.

You might also try (use-modules (ice-9 rdelim)) and then:

  (read-line)

and you'll see that the newline you type at the end of that line is read
by 'read-line', which then immediately returns the empty string.  The
input that you wish for 'read-line' to see must be typed on the same
line, immediately after the close parenthesis.

So, it might be that Geiser needs to be adjusted somewhat to deal with
these differences.

Finally, you might consider the possibility that 'stty' might not be
available in PATH, or fails for some reason, and ideally this case
should be handled as well.

It might be simpler to always use REPL servers over a socket, than to
deal with these headaches, although I don't know if that will be an
option for the other Scheme implementations.

    Regards,
      Mark

bug#33403: [Geiser-users] bug#33403: Data length limit in Guile/Geiser/Scheme evaluation

[Mark H Weaver]

A few more notes:

I wrote earlier:
> However, before doing this, some warnings are in order:
>
> When in noncanonical mode, the normal processing of ERASE (usually DEL
> or Ctrl-H) and KILL (usually Ctrl-U) characters are disabled,

Also the handling of Ctrl-D appears to be disabled in noncanonical mode
on my system, although this wasn't clear to me from the docs.

> At least in the case of the Guile REPL, one notable side effect of
> running in noncanonical mode is that when a list is entered at the REPL,
> the 'read' returns as soon as the final close parenthesis is entered.
> Nothing after that is read, not even the usual newline.

There's an additional wrinkle here: after 'read' returns, Guile tries to
read optional whitespace followed by a newline, but only if it's
immediately available.  See 'flush-to-newline' at the end of
module/system/repl/repl.scm in Guile.

So, unfortunately there's a race condition here, but typically if you
send the newline immediately after the final character of input, it is
likely that the newline will be consumed by the REPL reader and not by
the code that is subsequently run.

Finally, I should note that I consider this race condition suboptimal,
and will likely change how Guile behaves in the future, so please don't
rely on the behavior I have described above.  I will likely change
Guile's REPL reader to wait for the final newline in all cases.

      Mark

bug#33403: [Geiser-users] bug#33403: Data length limit in Guile/Geiser/Scheme evaluation

[Jose A. Ortega Ruiz]

On Sat, Nov 17 2018, Mark H Weaver wrote:

[...]

>> Ah, system* is a scheme call! So yeah, maybe we could add that call to
>> Geiser's guile initialization... i don't really see how that would cause
>> any problem elsewhere.
>
> I think something like this should be done, not only in the Guile
> initialization, but ideally in the generic Geiser code that connects to
> inferior processes via pseudo-tty.  After the pseudo-tty is allocated
> but before launching the inferior Scheme process, something like "stty
> --file=/dev/pts/N -icanon" should be run.
>
> However, before doing this, some warnings are in order:
>
> When in noncanonical mode, the normal processing of ERASE (usually DEL
> or Ctrl-H) and KILL (usually Ctrl-U) characters are disabled, and input
> characters are delivered to the subprocess immediately as they are
> typed, rather than waiting for the newline as normally happens in
> canonical mode.
>
> At least in the case of the Guile REPL, one notable side effect of
> running in noncanonical mode is that when a list is entered at the REPL,
> the 'read' returns as soon as the final close parenthesis is entered.
> Nothing after that is read, not even the usual newline.  The final
> newline is only read if the reader is not yet sure that it has finished
> reading the token, e.g. if a number or symbol is entered.  In those
> cases, typically any delimiter may be typed to terminate the read,
> e.g. space.
>
> To see this, you can try running Guile from a traditional terminal
> program (e.g. xterm or GNOME Terminal), and type:
>
>   (system* "stty" "-icanon")
>
> and then:
>
>   (display "hello")
>
> You will see that as soon as you type that close paren, "hello" is
> immediately printed, followed by another REPL prompt, all on the same
> line.

I think this is not an actual problem in Geiser.  In comint mode, the
stdin of the Guile process doesn't receive any input bytes until higher
level elisp functions send them, and that's totally under our control.
Repeating that experiment in a Geiser REPL, nothing is printed before a
RET (and, in fact, we might not even send the RET to Guile).

>
> You might also try (use-modules (ice-9 rdelim)) and then:
>
>   (read-line)
>
> and you'll see that the newline you type at the end of that line is read
> by 'read-line', which then immediately returns the empty string.  The
> input that you wish for 'read-line' to see must be typed on the same
> line, immediately after the close parenthesis.

Again, a comint/geiser REPL doesn't have this problem.

> So, it might be that Geiser needs to be adjusted somewhat to deal with
> these differences.

Seems we're lucky enough to be already adjusted :)

> Finally, you might consider the possibility that 'stty' might not be
> available in PATH, or fails for some reason, and ideally this case
> should be handled as well.

Yes, that's a real concern.  We should at least provide some kind of
warning.

> It might be simpler to always use REPL servers over a socket, than to
> deal with these headaches, although I don't know if that will be an
> option for the other Scheme implementations.

Not for all of them.  For Guile it's doable, but definitely not
"simpler", it requires some work and solving some unrelated corner
cases; but it might be worth the effort, because it's a cleaner
interaction mode (for instance, behaves much better in the presence of
multiple threads).

Cheers,
jao
-- 
Beware of the stories you read or tell; subtly, at night, beneath the
waters of consciousness, they are altering your world.
  -Ben Okri, poet and novelist