bug#15414: 24.3; piping problems in eshell

bug#15414: 24.3; piping problems in eshell

[Joseph Mingrone]

1. emacs -Q
2. M-x eshell
3. cat a_file | wc -l

The output of 3. will be wrong for most files longer than something like
40 lines and it will often give different results each time 3. is run.
No error messages are given.


In GNU Emacs 24.3.1 (i386-portbld-freebsd9.1, X toolkit)
 of 2013-08-08 on gly.ath.cx
Windowing system distributor `The X.Org Foundation', version 11.0.11204000
Configured using:
 `configure '--localstatedir=/var' '--without-rsvg'
 '--with-x-toolkit=athena' '--without-xaw3d'
 '--without-toolkit-scroll-bars' '--without-gif' '--with-xft'
 '--without-m17n-flt' '--with-otf' '--without-imagemagick'
 '--without-gsettings' '--without-gconf' '--with-xim' '--with-sound'
 '--without-dbus' '--with-xml2' '--with-gnutls'
 '--x-libraries=/usr/local/lib' '--x-includes=/usr/local/include'
 '--prefix=/usr/local' '--mandir=/usr/local/man'
 '--infodir=/usr/local/info/' '--build=i386-portbld-freebsd9.1'
 'build_alias=i386-portbld-freebsd9.1' 'CC=cc' 'CFLAGS=-O2 -pipe
 -fno-strict-aliasing' 'LDFLAGS= -L/usr/local/lib
 -Wl,-rpath=/usr/local/lib' 'CPPFLAGS=-I/usr/local/include' 'CPP=cpp''

Important settings:
  value of $LANG: en_CA.UTF-8
  locale-coding-system: utf-8-unix
  default enable-multibyte-characters: t

Major mode: EShell

Minor modes in effect:
  erc-track-mode: t
  erc-track-minor-mode: t
  erc-spelling-mode: t
  erc-ring-mode: t
  erc-netsplit-mode: t
  erc-menu-mode: t
  erc-match-mode: t
  erc-log-mode: t
  erc-list-mode: t
  erc-pcomplete-mode: t
  erc-button-mode: t
  erc-fill-mode: t
  erc-stamp-mode: t
  erc-autojoin-mode: t
  show-paren-mode: t
  global-auto-revert-mode: t
  multi-web-global-mode: t
  shell-dirtrack-mode: t
  erc-services-mode: t
  erc-networks-mode: t
  erc-irccontrols-mode: t
  erc-noncommands-mode: t
  erc-move-to-prompt-mode: t
  erc-readonly-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
M-x 
r e p o r t <tab> <return>

Recent messages:
Checking 24 files in /usr/local/share/emacs/24.3/lisp/cedet...
Checking 57 files in /usr/local/share/emacs/24.3/lisp/calendar...
Checking 87 files in /usr/local/share/emacs/24.3/lisp/calc...
Checking 77 files in /usr/local/share/emacs/24.3/lisp/obsolete...
Checking 1 files in /usr/local/share/emacs/24.3/leim...
Checking for load-path shadows...done
Mark set
Wrote /tmp/docview1001/modified-lr-supplementary.pdf
Type C-c C-c to toggle between editing or viewing the document.
Mark all unread articles as read? (y or n)  y

Load-path shadows:
/home/jrm/.emacs.d/elpa/slime-20130626.1151/slime hides /home/jrm/quicklisp/dists/quicklisp/software/slime-20130720-cvs/slime
/home/jrm/.emacs.d/elpa/slime-20130626.1151/hyperspec hides /home/jrm/quicklisp/dists/quicklisp/software/slime-20130720-cvs/hyperspec
/home/jrm/.emacs.d/elpa/slime-20130626.1151/slime-autoloads hides /home/jrm/quicklisp/dists/quicklisp/software/slime-20130720-cvs/slime-autoloads

Features:
(shadow emacsbug apropos tutorial two-column iso-transl goto-addr view
multi-term imenu sgml-mode shr-color color mule-util gnus-dup gnus-fun
doctor ido-gnus cperl-mode nxml-uchnm rng-xsd xsd-regexp rng-cmpct
rng-nxml rng-valid rng-loc rng-uri rng-parse nxml-parse rng-match rng-dt
rng-util rng-pttrn nxml-ns nxml-mode nxml-outln nxml-rap nxml-util
nxml-glyph nxml-enc xmltok tabify man org-table org-wl org-w3m org-vm
org-rmail org-mhe org-mew org-irc org-jsinfo org-infojs org-html org-exp
ob-exp org-exp-blocks org-agenda org-info org-gnus org-docview
org-bibtex bibtex org-bbdb org ob-tangle ob-ref ob-lob ob-table
org-footnote org-src ob-comint ob-keys org-pcomplete org-list org-faces
org-entities noutline outline org-version ob-emacs-lisp ob org-compat
org-macs ob-eval org-loaddefs find-func cal-menu calendar cal-loaddefs
cc-langs cc-mode cc-fonts cc-guess cc-menus cc-cmds cc-styles cc-align
cc-engine grep misearch multi-isearch etags mailalias smtpmail sendmail
sh-script smie url-http url-gw url-auth url-queue url-cache url
url-proxy url-privacy url-expand url-methods url-history url-cookie
url-domsuf url-util url-parse url-vars shr qp sort smiley gnus-cite
flow-fill mm-archive gnus-async gnus-bcklg gnus-ml tramp-cache tramp-sh
tramp tramp-compat tramp-loaddefs help-mode pcmpl-unix em-unix em-term
term disp-table ehelp electric em-script em-prompt em-ls em-hist em-pred
em-glob em-dirs em-cmpl em-basic em-banner em-alias esh-var esh-io
esh-cmd esh-opt esh-ext esh-proc esh-arg esh-groups eshell esh-module
esh-mode esh-util multi-eshell gnus-topic nndraft nnmh gnus-agent
gnus-srvr gnus-score score-mode nnvirtual gnus-msg parse-time netrc
gnus-cache bbdb-gnus gnus-art mm-uu mml2015 mm-view mml-smime smime dig
mailcap gnus-sum bbdb-snarf mail-extr bbdb-com nnml nntp gnus-group
gnus-undo nnmail mail-source nnoo gnus-start gnus-spec gnus-int
gnus-range message idna rfc822 mml mml-sec mm-decode mm-bodies mm-encode
mail-parse rfc2231 rfc2047 rfc2045 ietf-drums mailabbrev gmm-utils
mailheader gnus-win gnus gnus-ems nnheader mail-utils gnutls
network-stream starttls server auctex-autoloads tex-site
browse-kill-ring-autoloads dired+-autoloads
exec-path-from-shell-autoloads fill-column-indicator-autoloads
google-translate-autoloads ido-gnus-autoloads finder-inf
mode-compile-autoloads multi-eshell-autoloads multi-term-autoloads
rainbow-delimiters-autoloads smart-tab-autoloads w3m-autoloads info
yasnippet-autoloads erc-track erc-spelling flyspell ispell erc-ring
erc-netsplit erc-menu erc-match erc-log erc-pcomplete erc-button
erc-fill erc-stamp wid-edit erc-join paren hl-line autorevert cus-start
cus-load slime-autoloads multi-web-mode bbdb-autoloads uniquify tls
stumpwm-mode package ido ess-toolbar ess-mouse mouseme browse-url
ess-menu ess-swv ess-noweb ess-noweb-font-lock-mode ess-bugs-l essd-els
ess-sas-d ess-sas-l ess-sas-a shell pcomplete ess-arc-d ess-vst-d
ess-xls-d ess-lsp-l ess-sta-d ess-sta-l cc-vars cc-defs make-regexp
ess-sp6-d ess-sp5-d ess-sp3-d ess-julia ess-r-d compile ess-tracebug
warnings ess-roxy advice advice-preload hideshow ess-help ess-developer
ess-r-args eldoc help-fns ess-s-l ess ess-inf comint ansi-color ring
ess-mode ess-noweb-mode ess-utils ess-custom executable ess-compat
ess-site erc-tex easy-mmode edmacro kmacro cl-macs gv cl cl-lib
erc-services erc-networks erc-goodies erc erc-backend erc-compat
format-spec auth-source eieio byte-opt bytecomp byte-compile cconv
gnus-util time-date mm-util mail-prsvr password-cache thingatpt pp
epa-file epa derived epg epg-config doc-view easymenu jka-compr
image-mode dired bbdb timezone tooltip ediff-hook vc-hooks
lisp-float-type mwheel x-win x-dnd tool-bar dnd fontset image regexp-opt
fringe tabulated-list newcomment lisp-mode register page menu-bar
rfn-eshadow timer select scroll-bar mouse jit-lock font-lock syntax
facemenu font-core frame cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese hebrew greek romanian slovak
czech european ethiopic indian cyrillic chinese case-table epa-hook
jka-cmpr-hook help simple abbrev minibuffer loaddefs button faces
cus-face macroexp files text-properties overlay sha1 md5 base64 format
env code-pages mule custom widget hashtable-print-readable backquote
make-network-process dynamic-setting font-render-setting x-toolkit x
multi-tty emacs)

bug#15414: 24.3; piping problems in eshell

[Glenn Morris]

Joseph Mingrone wrote:

> 1. emacs -Q
> 2. M-x eshell
> 3. cat a_file | wc -l
>
> The output of 3. will be wrong for most files longer than something like
> 40 lines and it will often give different results each time 3. is run.

FWIW, I did not see this in 100 repetitions on a 100 line file, on
either of two systems. I got the correct answer every time.

I did notice that

  cat does-not-exist | wc -l 

returns 1 though.

bug#15414: 24.3; piping problems in eshell

[Joseph Mingrone]

Glenn Morris <rgm@gnu.org> writes:

> Joseph Mingrone wrote:
>
>> 1. emacs -Q
>> 2. M-x eshell
>> 3. cat a_file | wc -l
>>
>> The output of 3. will be wrong for most files longer than something like
>> 40 lines and it will often give different results each time 3. is run.
>
> FWIW, I did not see this in 100 repetitions on a 100 line file, on
> either of two systems. I got the correct answer every time.
>
> I did notice that
>
>   cat does-not-exist | wc -l 
>
> returns 1 though.

Were either of those systems running FreeBSD?

~ $ uname -a
Linux node8 3.5.0-27-generic #46~precise1-Ubuntu SMP Tue Mar 26 19:33:21
UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
$ wc -l q08-02a.qzp
82 q08-02a.qzp
~ $ cat q08-02a.qzp | wc -l
82


$ uname -a
FreeBSD phe 9.1-STABLE FreeBSD 9.1-STABLE #0 r247012: Tue Feb 19 21:45:31 AST 2013     root@phe.ath.cx:/usr/obj/usr/src/sys/PHE  amd64
$ wc -l q08-02a.qzp 
      82 q08-02a.qzp
$ cat q08-02a.qzp | wc -l
      51

If there is anything else I can test, please let me know.

Joseph

bug#15414: 24.3; piping problems in eshell

[Glenn Morris]

Joseph Mingrone wrote:

> Were either of those systems running FreeBSD?

As it happens, no.

> If there is anything else I can test, please let me know.

Maybe pipe cat to cat -v -n, and see what goes missing / gets added?

bug#15414: 24.3; piping problems in eshell

[Joseph Mingrone]

Glenn Morris <rgm@gnu.org> writes:
> Maybe pipe cat to cat -v -n, and see what goes missing / gets added?

% cat test.in | cat -v > test.out
% diff test.in test.out

The diff shows that test.out gets truncated after 1984 characters
(that's what wc -m test.out reports).

The contents of test.in, test.out and the diff follow.

Joseph

---------------------- test.in ----------------------
// Stat2080 Assignment 8  Winter 2012/2013

// Question 2a

/LET itX = mathBeg_m+"X"+mathEnd_m
/LET itY = mathBeg_m+"Y"+mathEnd_m
/LET itSSxx = mathBeg_m+"SS"+mathEnd_m+subBeg_m+"xx"+subEnd_m
/LET itSSyy = mathBeg_m+"SS"+mathEnd_m+subBeg_m+"yy"+subEnd_m
/LET itSSxy = mathBeg_m+"SS"+mathEnd_m+subBeg_m+"xy"+subEnd_m
/LET itH0 = mathBeg_m+"H"+mathEnd_m+subBeg_m+"0"+subEnd_m
/LET itHa = mathBeg_m+"H"+mathEnd_m+subBeg_m+"a"+subEnd_m
/LET itF = mathBeg_m+"F"+mathEnd_m

/LET n = 10
/LET xm = 5.
/LET ym = 12.
/LET SSxy = 121.+random(1,8)
/LET SSxx = 136.+random(1,8)
/LET SSyy = 155.
/LET b1 = SSxy/SSxx	// beta1_hat
/LET b0 = ym-b1*xm	// bet0_hat
/LET b1 = roundto(b1,2)
/LET b0 = roundto(b0,2)

/LET dft = n-1
/LET dfr = 1
/LET dfe = dft-dfr
/LET SSR = SSxy*SSxy/SSxx
/LET SST = SSyy
/LET SSE = SST-SSR
/LET MSR = SSR/dfr
/LET MSE = SSE/dfe
/LET R2  = SSR/SST
/LET SEb1= sqrt(MSE/SSxx)

A straight line is fitted to some data using least squares and the following summary statistics:/DIS(newLine_m)
/DIS(n_gw+eq_m)/DIS(n),/DIS(hSpace1_m) /DIS(xbar_gw+eq_m)/DIS(xm:0f),/DIS(hSpace1_m) /DIS(ybar_gw+eq_m)/DIS(ym:0f),/DIS(hSpace1_m) 
/DIS(itSSxy+eq_m)/DIS(SSxy:0f), /DIS(newLine_m) /DIS(itSSxx+eq_m)/DIS(SSxx:0f),/DIS(hSpace1_m) /DIS(itSSyy+eq_m)/DIS(SSyy:0f)./DIS(newLine_m)
The least squares intercept and slope are /DIS(b0:2f) and /DIS(b1:2f), respectively, and the ANOVA table is 
/DIS(vSpace10_m)
/DIS(web("","\begin{tabular}{|l|c|c|c|} \hline","<table border='1' cellpadding='4'><tr><th>")) \
Source /DIS(seph_m) DF /DIS(seph_m) SS /DIS(seph_m) MS 
/DIS(nllh_m)
/DIS(web("","","<tr align='center'><td>")) \
Regression /DIS(sep_m) /DIS(dfr) /DIS(sep_m) /DIS(SSR:2f) /DIS(sep_m) /DIS(MSR:2f) /DIS(nll_m)
/DIS(web("","","<tr align='center'><td>")) \
Residual /DIS(sep_m) /DIS(dfe) /DIS(sep_m) /DIS(SSE:2f) /DIS(sep_m) /DIS(MSE:2f)  /DIS(nll_m)
/DIS(web("","","<tr align='center'><td>")) \
Total /DIS(sep_m) /DIS(dft) /DIS(sep_m) /DIS(SST:2f)/DIS(sep_m)/DIS(nbsp_m) /DIS(nll_m)
/DIS(web("","\end{tabular}","</table>"))
/DIS(vSpace10_m)

/LET xstar = random(7,10)
/LET yhat = b0+b1*xstar

/IMP "/dalphysicslib/Tools/Problem"
What is the estimated mean value for /DIS(y_gw) when /DIS(x_gw+eq_m)/DIS(xstar)?

/ANS(yhat:2f, tol=0.03, wgt=wgt_v, tries=tries_v)

/LET yhat = random(18,22)
/LET xstar = (yhat-b0)/b1

/IMP "/dalphysicslib/Tools/Problem"
If the fitted value for /DIS(y_gw) is /DIS(yhat), what is the corresponding value for /DIS(x_gw)?

/ANS(xstar:2f, tol=0.03, wgt=wgt_v, tries=tries_v)

/LET percent = R2*100

/IMP "/dalphysicslib/Tools/Problem"
What percent of the variation in /DIS(y_gw) is explained by the regression in /DIS(x_gw)? Enter an integer, without the /DIS(pct_u) sign.

/ANS(percent:0f, tol=0.6, wgt=wgt_v, tries=tries_v)

/IMP "/dalphysicslib/Tools/Problem"
What is the standard error of the slope estimate? Enter 3 decimals.

/ANS(SEb1:3f, tol=0.01, wgt=wgt_v, tries=tries_v)

---------------------- test.out ----------------------
// Stat2080 Assignment 8  Winter 2012/2013

// Question 2a

/LET itX = mathBeg_m+"X"+mathEnd_m
/LET itY = mathBeg_m+"Y"+mathEnd_m
/LET itSSxx = mathBeg_m+"SS"+mathEnd_m+subBeg_m+"xx"+subEnd_m
/LET itSSyy = mathBeg_m+"SS"+mathEnd_m+subBeg_m+"yy"+subEnd_m
/LET itSSxy = mathBeg_m+"SS"+mathEnd_m+subBeg_m+"xy"+subEnd_m
/LET itH0 = mathBeg_m+"H"+mathEnd_m+subBeg_m+"0"+subEnd_m
/LET itHa = mathBeg_m+"H"+mathEnd_m+subBeg_m+"a"+subEnd_m
/LET itF = mathBeg_m+"F"+mathEnd_m

/LET n = 10
/LET xm = 5.
/LET ym = 12.
/LET SSxy = 121.+random(1,8)
/LET SSxx = 136.+random(1,8)
/LET SSyy = 155.
/LET b1 = SSxy/SSxx	// beta1_hat
/LET b0 = ym-b1*xm	// bet0_hat
/LET b1 = roundto(b1,2)
/LET b0 = roundto(b0,2)

/LET dft = n-1
/LET dfr = 1
/LET dfe = dft-dfr
/LET SSR = SSxy*SSxy/SSxx
/LET SST = SSyy
/LET SSE = SST-SSR
/LET MSR = SSR/dfr
/LET MSE = SSE/dfe
/LET R2  = SSR/SST
/LET SEb1= sqrt(MSE/SSxx)

A straight line is fitted to some data using least squares and the following summary statistics:/DIS(newLine_m)
/DIS(n_gw+eq_m)/DIS(n),/DIS(hSpace1_m) /DIS(xbar_gw+eq_m)/DIS(xm:0f),/DIS(hSpace1_m) /DIS(ybar_gw+eq_m)/DIS(ym:0f),/DIS(hSpace1_m) 
/DIS(itSSxy+eq_m)/DIS(SSxy:0f), /DIS(newLine_m) /DIS(itSSxx+eq_m)/DIS(SSxx:0f),/DIS(hSpace1_m) /DIS(itSSyy+eq_m)/DIS(SSyy:0f)./DIS(newLine_m)
The least squares intercept and slope are /DIS(b0:2f) and /DIS(b1:2f), respectively, and the ANOVA table is 
/DIS(vSpace10_m)
/DIS(web("","\begin{tabular}{|l|c|c|c|} \hline","<table border='1' cellpadding='4'><tr><th>")) \
Source /DIS(seph_m) DF /DIS(seph_m) SS /DIS(seph_m) MS 
/DIS(nllh_m)
/DIS(web("","","<tr align='center'><td>")) \
Regression /DIS(sep_m) /DIS(dfr) /DIS(sep_m) /DIS(SSR:2f) /DIS(sep_m) /DIS(MSR:2f) /DIS(nll_m)
/DIS(web("","","<tr align='center'><td>")) \
Residual /DIS(sep_m) /DIS(dfe) /DIS(sep_m) /DIS(SSE:2f) /DIS(sep_m) /DIS(MSE:2f)  /DIS(nll_m)
/DIS(web("","","<tr align='center'><td>")) \
Total /DIS(sep_m) /DIS(dft) /DIS(sep_m) /DIS(SST:2f)/DIS(sep_m)/DIS(nbsp_m) /DIS(nll_m)
/DIS(we

---------------------- diff output ----------------------
diff -c /home/jrm/test.in /home/jrm/test.out
*** /home/jrm/test.in	2013-09-21 20:31:23.000000000 -0300
--- /home/jrm/test.out	2013-09-21 20:32:06.000000000 -0300
***************
*** 47,79 ****
  Residual /DIS(sep_m) /DIS(dfe) /DIS(sep_m) /DIS(SSE:2f) /DIS(sep_m) /DIS(MSE:2f)  /DIS(nll_m)
  /DIS(web("","","<tr align='center'><td>")) \
  Total /DIS(sep_m) /DIS(dft) /DIS(sep_m) /DIS(SST:2f)/DIS(sep_m)/DIS(nbsp_m) /DIS(nll_m)
! /DIS(web("","\end{tabular}","</table>"))
! /DIS(vSpace10_m)
! 
! /LET xstar = random(7,10)
! /LET yhat = b0+b1*xstar
! 
! /IMP "/dalphysicslib/Tools/Problem"
! What is the estimated mean value for /DIS(y_gw) when /DIS(x_gw+eq_m)/DIS(xstar)?
! 
! /ANS(yhat:2f, tol=0.03, wgt=wgt_v, tries=tries_v)
! 
! /LET yhat = random(18,22)
! /LET xstar = (yhat-b0)/b1
! 
! /IMP "/dalphysicslib/Tools/Problem"
! If the fitted value for /DIS(y_gw) is /DIS(yhat), what is the corresponding value for /DIS(x_gw)?
! 
! /ANS(xstar:2f, tol=0.03, wgt=wgt_v, tries=tries_v)
! 
! /LET percent = R2*100
! 
! /IMP "/dalphysicslib/Tools/Problem"
! What percent of the variation in /DIS(y_gw) is explained by the regression in /DIS(x_gw)? Enter an integer, without the /DIS(pct_u) sign.
! 
! /ANS(percent:0f, tol=0.6, wgt=wgt_v, tries=tries_v)
! 
! /IMP "/dalphysicslib/Tools/Problem"
! What is the standard error of the slope estimate? Enter 3 decimals.
! 
! /ANS(SEb1:3f, tol=0.01, wgt=wgt_v, tries=tries_v)
--- 47,50 ----
  Residual /DIS(sep_m) /DIS(dfe) /DIS(sep_m) /DIS(SSE:2f) /DIS(sep_m) /DIS(MSE:2f)  /DIS(nll_m)
  /DIS(web("","","<tr align='center'><td>")) \
  Total /DIS(sep_m) /DIS(dft) /DIS(sep_m) /DIS(SST:2f)/DIS(sep_m)/DIS(nbsp_m) /DIS(nll_m)
! /DIS(we
\ No newline at end of file

Diff finished.  Sat Sep 21 20:38:34 2013