bug#2148: In shell mode, comint-send-input seems to cut off at 254 characters

bug#2148: In shell mode, comint-send-input seems to cut off at 254 characters

[Richard Addison-Wood]

I attempted to do this command in shell mode:

echo _234567_10_234567_20_234567_30_234567_40_234567_50_234567_60_234567_70_234567_80_234567_90_23456_100_23456_110_23456_120_23456_130_23456_140_23456_150_23456_160_23456_170_23456_180_23456_190_23456_200_23456_210_23456_220_23456_230_23456_240_23456_250_23456_260

The immediate output was:

0_23456_260: Command not found.

but when I hit enter again without typing anything else, I got:

_234567_10_234567_20_234567_30_234567_40_234567_50_234567_60_234567_70_234567_80_234567_90_23456_100_23456_110_23456_120_23456_130_23456_140_23456_150_23456_160_23456_170_23456_180_23456_190_23456_200_23456_210_23456_220_23456_230_23456_240_23456_25

So the first 254 characters I typed got held back, and the additional 11 characters were sent to the inferior shell.  Further, the held back characters were then sent when I pressed enter again.

So this is what the buffer looked like:

====================
>echo _234567_10_234567_20_234567_30_234567_40_234567_50_234567_60_234567_70_234567_80_234567_90_23456_100_23456_110_23456_120_23456_130_23456_140_23456_150_23456_160_23456_170_23456_180_23456_190_23456_200_23456_210_23456_220_23456_230_23456_240_23456_250_23456_260
echo _234567_10_234567_20_234567_30_234567_40_234567_50_234567_60_234567_70_234567_80_234567_90_23456_100_23456_110_23456_120_23456_130_23456_140_23456_150_23456_160_23456_170_23456_180_23456_190_23456_200_23456_210_23456_220_23456_230_23456_240_23456_250_23456_260

>0_23456_260: Command not found.
>

_234567_10_234567_20_234567_30_234567_40_234567_50_234567_60_234567_70_234567_80_234567_90_23456_100_23456_110_23456_120_23456_130_23456_140_23456_150_23456_160_23456_170_23456_180_23456_190_23456_200_23456_210_23456_220_23456_230_23456_240_23456_25
>
====================

I would count this as very unexpected and dangerous behaviour.


In GNU Emacs 22.3.1 (x86_64-unknown-linux-gnu)
 of 2008-09-15 on lambretta
Windowing system distributor `The X.Org Foundation', version 11.0.70200000
configured using `configure  '--prefix=/vol/apps_master/apps.Linux64/emacs-22.3_64''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: nil
  locale-coding-system: nil
  default-enable-multibyte-characters: t

Major mode: Shell

Minor modes in effect:
  shell-dirtrack-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  unify-8859-on-encoding-mode: t
  utf-translate-cjk-mode: t
  auto-compression-mode: t
  line-number-mode: t

Recent input:
<help-echo> <help-echo> <help-echo> <help-echo> M-x 
s h e l l <return> <down-mouse-2> <mouse-2> <return> 
<return> M-x r e p o r t - e m a c s - b u g <retu
rn>

Recent messages:
("/vol/apps/emacs-22.3_64/bin/emacs" "--no-init")
For information about GNU Emacs and the GNU system, type C-h C-a.
Loading shell...done
Mark set
Loading emacsbug...
Loading regexp-opt...done
Loading emacsbug...done

bug#2148: In shell mode, comint-send-input seems to cut off at 254 characters

[Chong Yidong]

> I attempted to do this command in shell mode:
> echo
> _234567_10....
> The immediate output was:
> 0_23456_260: Command not found.
> but when I hit enter again without typing anything else, I got:
> _234567_10_234567....
> So the first 254 characters I typed got held back, and the additional
> 11 characters were sent to the inferior shell.  Further, the held back
> characters were then sent when I pressed enter again.
>
>In GNU Emacs 22.3.1 (x86_64-unknown-linux-gnu)

I can't reproduce this, on either 22.3.1 or Emacs 23.  Could you see if
the bug is present in the Emacs 23 pretest?  You can find the pretest
tarball at

 http://alpha.gnu.org/gnu/emacs/pretest/

Thanks.

bug#2148: In shell mode, comint-send-input seems to cut off at 254 characters

[Richard Addison-Wood]

I have more information about what is going on.

Can you see if you can reproduce it?

After opening a new shell buffer, I type:

/bin/tcsh -f

to get a tcsh shell without running custom start-up stuff.

Then, I type this:

set filec

to turn on tcsh's file completion mechanisms (since this is one of the 
settings that my start-up stuff does).

And then, when I type this:

echo 
_234567_10_234567_20_234567_30_234567_40_234567_50_234567_60_234567_70_234567_80_234567_90_23456_100_23456_110_23456_120_23456_130_23456_140_23456_150_23456_160_23456_170_23456_180_23456_190_23456_200_23456_210_23456_220_23456_230_23456_240_23456_250_23456_260

I get the strange behaviour I described before.

Alternatively, If I type this:

unset filec

I can enter that long line without a problem.

So, in /bin/tcsh, one of the things that 'set filec' turns on is to use 
control-D to show a list of what matches the prefix of the immediately 
preceding word.

It appears that 'send_process(proc, buf, len, object)' in process.c will 
determine that it should send a maximum of 254 characters and will send 
'\004' at each 254 character interval.

It still seems strange to me that emacs would have this behaviour.  Is 
that really how it should be done?  I wouldn't think that I would be the 
only user who would be using /bin/tcsh with 'set filec'.

bug#2148: In shell mode, comint-send-input seems to cut off at 254 characters

[Richard M Stallman]

This may have to do with truncation in the input buffer
used for line-at-a-time input.

bug#2148: Re: In shell mode, comint-send-input seems to cut off at 254 characters

[Chong Yidong]

> [in shell mode]
> /bin/tcsh -f
> set filec
> echo [very long line]
>
> I get the strange behaviour I described before.
>
> So, in /bin/tcsh, one of the things that 'set filec' turns on is to use 
> control-D to show a list of what matches the prefix of the immediately 
> preceding word.
>
> It appears that 'send_process(proc, buf, len, object)' in process.c will 
> determine that it should send a maximum of 254 characters and will send 
> '\004' at each 254 character interval.
>
> It still seems strange to me that emacs would have this behaviour.  Is 
> that really how it should be done?  I wouldn't think that I would be the 
> only user who would be using /bin/tcsh with 'set filec'.

The ^D is sent in process.c:5781.  If we are splitting a string into
chunks for sending to the process, Emacs puts in an EOF (C-d) to "force
it through".

I can't think of any fix, off the top of my head.  Stefan, can you?  If
not, we could simply document this limitation in PROBLEMS.

bug#2148: In shell mode, comint-send-input seems to cut off at 254 characters

[Stefan Monnier]

> The ^D is sent in process.c:5781.  If we are splitting a string into
> chunks for sending to the process, Emacs puts in an EOF (C-d) to "force
> it through".

> I can't think of any fix, off the top of my head.  Stefan, can you?  If
> not, we could simply document this limitation in PROBLEMS.

The obvious fix is to not add this ^D.  At least I could never
understand what it was supposed to do ("force it though"?  what does
that mean?).


        Stefan

bug#2148: In shell mode, comint-send-input seems to cut off at 254 characters

[Chong Yidong]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> The obvious fix is to not add this ^D.  At least I could never
> understand what it was supposed to do ("force it though"?  what does
> that mean?).

Ah, ok---for some reason, I thought you wrote that code, but on closer
inspection it was written by Gerd in 2000.

The current symptoms don't seem serious enough to risk changing this
now, so I think we should do it after the release.  WDYT?

bug#2148: In shell mode, comint-send-input seems to cut off at 254 characters

[Stefan Monnier]

>> The obvious fix is to not add this ^D.  At least I could never
>> understand what it was supposed to do ("force it though"?  what does
>> that mean?).
> Ah, ok---for some reason, I thought you wrote that code, but on closer
> inspection it was written by Gerd in 2000.

Not close enough: his 2000 patch just changed indentation.  I think the
origin of this problem is:

   revno: 6577
   committer: rms
   branch nick: HEAD
   timestamp: Thu 1994-03-03 05:50:31 +0000
   message:
     Include unistd.h.
     (pty_max_bytes): New variable.
     (send_process): Send an eof after each pty_max_bytes bytes.

And clearly there was a good reason for that.  I don't know enough about
PTY programming to know what we should do.

> The current symptoms don't seem serious enough to risk changing this
> now, so I think we should do it after the release.  WDYT?

Agreed (if at all).


        Stefan

bug#2148: In shell mode, comint-send-input seems to cut off at 254 characters

[Richard M Stallman]

    The obvious fix is to not add this ^D.  At least I could never
    understand what it was supposed to do ("force it though"?  what does
    that mean?).

The problem is that the subprogram is reading from its tty with line
editing, so it won't receive any input until Emacs "types" one of the
few characters that says "give the input to the program".  Until that
occurs, theoretically Emacs could get rid of the pending input by
typing DEL or Backspace or C-u or C-w.

If the system's line-editing buffer gets full, everything hangs.  The
subprogram waits for a complete input line, but Emacs can't finish the
line because it's waiting for space to appear in that buffer (and
anyway the buffer has no room for more).

At least this is what was happening at the time I implemented that code.

If emacs_set_tty turns off the line editing, or turns off the
characters that could cancel input, it would be proper for the kernel
to give the characters to the subprogram right away.  Then the buffer
would never get full.  We could suggest this change in Linux.

bug#2148: In shell mode, comint-send-input seems to cut off at 254 characters

[Dorish Apira]

Sorry to intrude, but it seems like this issue had lost all interest !?!
Is it a problem to change 254 to 8191 in some way ?
Dorish

bug#2148: In shell mode, comint-send-input seems to cut off at 254 characters

[Alexis]

Dorish Apira writes:

> Sorry to intrude, but it seems like this issue had lost all interest !?!
> Is it a problem to change 254 to 8191 in some way ?

Well, are you still observing this issue in Emacs 24.4?

bug#2148: Re: In shell mode, comint-send-input seems to cut off at 254 characters

[Alan J Third]

Chong Yidong <cyd@stupidchicken.com> writes:

>> [in shell mode]
>> /bin/tcsh -f
>> set filec
>> echo [very long line]
>>
>> I get the strange behaviour I described before.
>>
>> So, in /bin/tcsh, one of the things that 'set filec' turns on is to use 
>> control-D to show a list of what matches the prefix of the immediately 
>> preceding word.
>>
>> It appears that 'send_process(proc, buf, len, object)' in process.c will 
>> determine that it should send a maximum of 254 characters and will send 
>> '\004' at each 254 character interval.
>>
>> It still seems strange to me that emacs would have this behaviour.  Is 
>> that really how it should be done?  I wouldn't think that I would be the 
>> only user who would be using /bin/tcsh with 'set filec'.
>
> The ^D is sent in process.c:5781.  If we are splitting a string into
> chunks for sending to the process, Emacs puts in an EOF (C-d) to "force
> it through".
>
> I can't think of any fix, off the top of my head.  Stefan, can you?  If
> not, we could simply document this limitation in PROBLEMS.

I can't replicate this in Emacs 25 (or Emacs 24.5) on OS X, and I can't find the code
described above in the source, or at least in send_process. Can someone
who knows their way around better than me please confirm whether the
offending code has been removed, please?
-- 
Alan Third

bug#2148: Re: In shell mode, comint-send-input seems to cut off at 254 characters

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Decades ago, I encountered on some systems that sending input
to a pty would get hung because the tty's input buffer was full.
I implemented the ^D to get the input delivered.

The right fix is to change the implementation of ptys.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

bug#2148: Re: In shell mode, comint-send-input seems to cut off at 254 characters

[Alan J Third]

Alan J Third <alan@idiocy.org> writes:

> Chong Yidong <cyd@stupidchicken.com> writes:
>
>> The ^D is sent in process.c:5781.  If we are splitting a string into
>> chunks for sending to the process, Emacs puts in an EOF (C-d) to "force
>> it through".
>>
>> I can't think of any fix, off the top of my head.  Stefan, can you?  If
>> not, we could simply document this limitation in PROBLEMS.
>
> I can't replicate this in Emacs 25 (or Emacs 24.5) on OS X, and I can't find the code
> described above in the source, or at least in send_process. Can someone
> who knows their way around better than me please confirm whether the
> offending code has been removed, please?

OK, this code is no longer in Emacs. It was removed sometime in the life
of Emacs 24:

commit 2b0a91e78f83fb446cc38efb99399e83ad2cded3
Author: Stefan Monnier <monnier@iro.umontreal.ca>
Date:   Mon Apr 12 22:07:48 2010 -0400

    Try to solve the problem of spurious EOF chars in long lines of text
    sent to interactive subprocesses.
    * sysdep.c (child_setup_tty): Do not enable ICANON any more.
    (system_process_attributes): Remove unused var `ttotal'.
    * process.c (send_process): Don't bother breaking long line with EOF
    chars when talking to ttys any more.
    (wait_reading_process_output): Output a warning when called in such
    a way that it could block without being interruptible.

So I believe we can close this bug.
-- 
Alan Third

bug#2148: Re: In shell mode, comint-send-input seems to cut off at 254 characters

[Alan J Third]

Alan J Third <alan@idiocy.org> writes:

> Alan J Third <alan@idiocy.org> writes:
>
>> Chong Yidong <cyd@stupidchicken.com> writes:
>>
>>> The ^D is sent in process.c:5781.  If we are splitting a string into
>>> chunks for sending to the process, Emacs puts in an EOF (C-d) to "force
>>> it through".
>>>
>>> I can't think of any fix, off the top of my head.  Stefan, can you?  If
>>> not, we could simply document this limitation in PROBLEMS.
>>
>> I can't replicate this in Emacs 25 (or Emacs 24.5) on OS X, and I can't find the code
>> described above in the source, or at least in send_process. Can someone
>> who knows their way around better than me please confirm whether the
>> offending code has been removed, please?
>
> OK, this code is no longer in Emacs. It was removed sometime in the life
> of Emacs 24:
>
> commit 2b0a91e78f83fb446cc38efb99399e83ad2cded3
> Author: Stefan Monnier <monnier@iro.umontreal.ca>
> Date:   Mon Apr 12 22:07:48 2010 -0400
>
>     Try to solve the problem of spurious EOF chars in long lines of text
>     sent to interactive subprocesses.
>     * sysdep.c (child_setup_tty): Do not enable ICANON any more.
>     (system_process_attributes): Remove unused var `ttotal'.
>     * process.c (send_process): Don't bother breaking long line with EOF
>     chars when talking to ttys any more.
>     (wait_reading_process_output): Output a warning when called in such
>     a way that it could block without being interruptible.
>
> So I believe we can close this bug.

Apologies, I think I closed this bug report without cc'ing you in. As
described above the behaviour you reported was removed some time ago. If
you're still experiencing it in a recent version of Emacs, please let me
know.

-- 
Alan Third