bug#59666: 29.0.50; Eshell: comparisons such as {> 3 2} do not work in Eshell context

bug#59666: 29.0.50; Eshell: comparisons such as {> 3 2} do not work in Eshell context

[Milan Zimmermann]

[-- Attachment #1: Type: text/plain, Size: 6340 bytes --]

Note: I am running this with the following hook removed:
(add-hook 'eshell-mode-hook
          (lambda ()
            (remove-hook
             'eshell-alternate-command-hook
             'eshell-fix-bad-commands t)))
However, not removing it makes no difference to the behavior, the error
visibility would be postponed until the 4th try, when eshell asks to
provide an alias (making everything even more confusing).

===========
Bug report:
===========

All behavior of comparisons below is incorrect

# Comparisons using ">"
### 1.
~/tmp $ ${> 3 2}
Actual: nil: command not found: command not found
Expected: t: command not found (same as below in pure elisp)

~/tmp $ $(> 3 2) # pure elisp
Actual equals Expected: t: command not found


### 2.
~/tmp $ {> 3 2}
Actual: nil: command not found
Expected: t

### 3.
~/tmp $ > 3 2
nil: command not found
Expected: t

# the above shows incorrect result, which can be tested also as follows

### 4.
~/tmp $ if {> 3 2} {echo YES} {echo NO}
Actual: NO
Expected: YES

# Reverse comparisons using "<"

### 5.
~/tmp $ ${< 3 2}
Actual: Eshell does not support input redirection
Expected: nil: command not found

### 6.
~/tmp $ {< 3 2}
Actual: Eshell does not support input redirection
Expected: no result, as in (< 3 2)

### 7.
~/tmp $ < 3 2
Actual: Eshell does not support input redirection
Expected: nil: command not found
Expected: no result, as in (< 3 2)

### 8. The above shows incorrect results, which can be tested in
#      variants of the following:
~/tmp $ if {< 3 2} {echo YES} {echo NO}
Actual: Eshell does not support input redirection
Expected: NO



In GNU Emacs 29.0.50 (build 1, x86_64-suse-linux-gnu, GTK+ Version
3.24.34, cairo version 1.17.6)
System Description: openSUSE Tumbleweed

Configured using:
 'configure --host=x86_64-suse-linux-gnu --build=x86_64-suse-linux-gnu
 --program-prefix= --disable-dependency-tracking --prefix=/usr
 --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin
 --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include
 --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var
 --sharedstatedir=/var/lib --mandir=/usr/share/man
 --infodir=/usr/share/info --disable-build-details --without-pop
 --with-mailutils --without-hesiod --with-gameuser=:games
 --with-kerberos --with-kerberos5 --with-file-notification=inotify
 --with-modules --enable-autodepend --prefix=/usr
 --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share
 --localstatedir=/var --sharedstatedir=/var/lib
 --libexecdir=/usr/libexec --with-file-notification=yes
 --enable-locallisppath=/usr/share/emacs/29.0.50/site-lisp:/usr/share/emacs/site-lisp
 --without-x --with-json --without-xim --with-sound --with-xpm
 --with-jpeg --with-tiff --with-gif --with-png --with-rsvg --with-dbus
 --without-xft --without-gpm --with-pgtk --without-native-compilation
 --with-toolkit-scroll-bars --with-libotf --with-m17n-flt --with-cairo
 --without-xwidgets --with-dumping=pdumper 'CFLAGS=-O2 -Wall
 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 -fstack-protector-strong
 -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection
 -Werror=return-type -flto=auto -D_GNU_SOURCE
 -DGDK_DISABLE_DEPRECATION_WARNINGS -DGLIB_DISABLE_DEPRECATION_WARNINGS'
 LDFLAGS=-flto=auto'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG JSON
LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 MODULES NOTIFY INOTIFY
PDUMPER PGTK PNG RSVG SECCOMP SOUND SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS WEBP XIM GTK3 ZLIB

Important settings:
  value of $LANG: en_CA.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Eshell

Minor modes in effect:
  shell-dirtrack-mode: t
  eshell-prompt-mode: t
  eshell-hist-mode: t
  eshell-pred-mode: t
  eshell-cmpl-mode: t
  eshell-proc-mode: t
  eshell-arg-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068
epg-config gnus-util text-property-search time-date mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail
rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils cl-seq rx
em-unix em-term term disp-table shell subr-x ehelp em-script em-prompt
em-ls em-hist em-pred em-glob em-extpipe em-cmpl em-dirs esh-var
pcomplete comint ansi-osc ansi-color ring em-basic em-banner em-alias
esh-mode eshell esh-cmd generator esh-ext esh-opt esh-proc esh-io
esh-arg esh-module esh-groups esh-util cus-edit pp cus-start cus-load
icons wid-edit cl-loaddefs cl-lib files-x rmc iso-transl tooltip cconv
eldoc paren electric uniquify ediff-hook vc-hooks lisp-float-type
elisp-mode mwheel term/pgtk-win pgtk-win term/common-win pgtk-dnd
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode lisp-mode prog-mode register page tab-bar menu-bar
rfn-eshadow isearch easymenu timer select scroll-bar mouse jit-lock
font-lock syntax font-core term/tty-colors frame minibuffer nadvice seq
simple cl-generic indonesian philippine cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite emoji-zwj charscript charprop case-table
epa-hook jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button
loaddefs theme-loaddefs faces cus-face macroexp files window
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget keymap hashtable-print-readable backquote threads dbusbind
inotify dynamic-setting system-font-setting font-render-setting cairo
gtk pgtk lcms2 multi-tty make-network-process emacs)

Memory information:
((conses 16 78484 13781)
 (symbols 48 8942 0)
 (strings 32 24745 2566)
 (string-bytes 1 734947)
 (vectors 16 15111)
 (vector-slots 8 205050 17892)
 (floats 8 35 34)
 (intervals 56 421 0)
 (buffers 984 11))

[-- Attachment #2: Type: text/html, Size: 7239 bytes --]

bug#59666: 29.0.50; Eshell: comparisons such as {> 3 2} do not work in Eshell context

[Jim Porter]

On 11/28/2022 4:15 PM, Milan Zimmermann wrote:
> All behavior of comparisons below is incorrect
> 
> # Comparisons using ">"

This is actually working as intended, though it's another one of those 
surprising behaviors that you tend to get in Eshell due to its fusion of 
Lisp and shell syntax.

The problem is that ">" and "<" are I/O redirection operators (though 
"<" isn't actually implemented yet). If you want to do a less- or 
greater-than comparison using command-style syntax, you'd need to escape it:

   ~ $ \> 3 2
   t

> ### 4.
> ~/tmp $ if {> 3 2} {echo YES} {echo NO}
> Actual: NO
> Expected: YES

One additional note. Even with the proper escapes, that won't work right:

   ~ $ if {\> 3 2} {echo YES} {echo NO}
   YES
   ~ $ if {\> 3 4} {echo YES} {echo NO}
   YES  ;; Wrong!

That's because the above form is actually just checking, "Did the Lisp 
function '>' signal an error?"[1] You'd need to use ${} expansion, which 
expands to the *output* of the subcommand (and then checks that it's 
non-nil), or the Lisp form $()/():

   ~ $ if ${\> 3 2} {echo YES} {echo NO}
   YES
   ~ $ if ${\> 3 4} {echo YES} {echo NO}
   NO
   ~ $ if (> 3 2) {echo YES} {echo NO}
   YES
   ~ $ if (> 3 4) {echo YES} {echo NO}
   NO
   ~ $ if $(> 3 2) {echo YES} {echo NO}
   YES
   ~ $ if $(> 3 4) {echo YES} {echo NO}
   NO

Maybe {...} forms should work more like (...) forms when it finds a Lisp 
function that's not prefixed with 'eshell/'. I'll have to think about 
this some more though...

[1] This is all assuming there's no external program named ">".

bug#59666: 29.0.50; Eshell: comparisons such as {> 3 2} do not work in Eshell context

[Milan Zimmermann]

[-- Attachment #1: Type: text/plain, Size: 2226 bytes --]

Thanks for your detailed follow up.

I understand what is happening, and am closing this issue (please reopen if
you see fit).

In the closing notes, I have a suggestion (slash question), and a summary
of how to use CONDITIONAL in the 'if' statement - in case it can be of use
for someone reading later.

****** Suggestion / question

Would it be reasonable to suggest removing existing functionality as
follows:

In the statement

if     CONDITIONAL { TRUE-COMMANDS } { FALSE-COMMANDS }

The CONDITIONAL can only have the following forms:

1. ${command-call}
2. ${function-call}

The currently allowed

3. {command-call}
4. (function-call)

would represent syntax error.

(3. is outright wrong as it brings silent invalid results)

By banning 3. and 4., Would Eshell lose the ability to express any
semantics that can be achieved using 1. and 2?


****** Math operations using < or < in ~if CONDITIONAL~

~Quote < and >  as \< or \> with backslash OTHERWISE YOU GET AN ERROR, OR
SILENTLY INCORRECT BEHAVIOR~

inside () or $() quotes are not needed but do not hurt

Examples:

if ${\< 3 5} {echo YES} {echo NO}   # YES -- correct
if ${< 3 5} {echo YES} {echo NO}    # Eshell thinks it is redirection, so
WRONG
if ${> 3 5} {echo YES} {echo NO}    # YES - WRONG!!! (silently)

if $(< 3 5) {echo YES} {echo NO} # YES -- correct
if $(> 3 5) {echo YES} {echo NO} # NO  -- correct
if (< 3 5) {echo YES} {echo NO} # YES -- correct
if (> 3 5) {echo YES} {echo NO} # NO -- correct

****** Flow control: TL;DR of ~if CONDITIONAL~

- Bad:  Do NOT use ~if {function-call}~ (abbreviated, ~if {}~)
- Good: Do use     ~if ${function-call}~  or  ~if $(function-call)~ or ~if
(function-call)~


Examples"
- Bad  use:  ~if {= 3 0}  { echo YES } { echo NO }~  # YES -- WRONG!!!
(silently!!)
- Good use:  ~if ${= 3 0} { echo YES } { echo NO }~  # NO -- correct

****** Flow control: Recipe of ~if CONDITIONAL~

In flow control ~if CONDITIONAL {TRUE-COMMANDS} {FALSE-COMMANDS}~: do NOT
USE the ~{}~ BLOCK as CONDITIONAL; instead use ~the ${}~, ~$()~ or ~()~
blocks. The ~$~ versions are preferred.

The above recipy also applies to ~unless~, ~while~ ~until~. (this does not
apply to ~for VAR in LIST~ which is described elsewhere)

Thanks,

Milan

[-- Attachment #2: Type: text/html, Size: 2636 bytes --]

bug#59666: 29.0.50; Eshell: comparisons such as {> 3 2} do not work in Eshell context

[Jim Porter]

On 12/1/2022 5:38 PM, Milan Zimmermann wrote:
> Would it be reasonable to suggest removing existing functionality as 
> follows:
[snip]
> The currently allowed
> 
> 3. {command-call}
> 4. (function-call)
> 
> would represent syntax error.
> 
> (3. is outright wrong as it brings silent invalid results)

Both 3 and 4 are valid forms, although I think (lisp) and $(lisp) are 
equivalent. In particular, 3 is important for being able to check the 
exit status of external programs:

   ~ $ if {sh -c 'exit 0'} {echo yes} {echo no}
   yes
   ~ $ if {sh -c 'exit 1'} {echo yes} {echo no}
   no
   ~ $ if ${sh -c 'exit 0'} {echo yes} {echo no}
   no  ;; Wrong!
   ~ $ if ${sh -c 'exit 1'} {echo yes} {echo no}
   no

I think there's an argument that {lisp-function} should work the same as 
${lisp-function}, but only for "regular" Lisp functions (i.e. excluding 
eshell/FOO ones; since those are designed to imitate external commands, 
they have different semantics).

bug#59666: 29.0.50; Eshell: comparisons such as {> 3 2} do not work in Eshell context

[Milan Zimmermann]

[-- Attachment #1: Type: text/plain, Size: 3182 bytes --]

On Thu, Dec 1, 2022 at 8:53 PM Jim Porter <jporterbugs@gmail.com> wrote:

> On 12/1/2022 5:38 PM, Milan Zimmermann wrote:
> > Would it be reasonable to suggest removing existing functionality as
> > follows:
> [snip]
> > The currently allowed
> >
> > 3. {command-call}
> > 4. (function-call)
> >
> > would represent syntax error.
> >
> > (3. is outright wrong as it brings silent invalid results)
>
> Both 3 and 4 are valid forms, although I think (lisp) and $(lisp) are
> equivalent. In particular, 3 is important for being able to check the
> exit status of external programs:
>
>    ~ $ if {sh -c 'exit 0'} {echo yes} {echo no}
>    yes
>    ~ $ if {sh -c 'exit 1'} {echo yes} {echo no}
>    no
>    ~ $ if ${sh -c 'exit 0'} {echo yes} {echo no}
>    no  ;; Wrong!
>    ~ $ if ${sh -c 'exit 1'} {echo yes} {echo no}
>    no
>
>
Ah, thanks for pointing this out. I kept searching for a practical
semantics that justifies use of if {}. This is definitely important,
non-replaceable syntax. My suggestion to ban it was definitely incorrect
then.


> I think there's an argument that {lisp-function} should work the same as
> ${lisp-function}, but only for "regular" Lisp functions (i.e. excluding
> eshell/FOO ones; since those are designed to imitate external commands,
> they have different semantics).
>

I agree. I did not find a situation where {lisp-function} and
${lisp-function} would behave differently, but I have not looked yet.


So I put the following to my notes, based on your suggestions. If anyone
else is reading, please disregard my earlier suggestion to disallow the "if
{..}" form. Justified by checking external programs status.

1. use ~if {function-call}~  (only?) to check EXIT STATUS of the EXTERNAL
function (program) call.
2. use ~if ${function-call}~ for everything else Eshell, including internal
functions passed exported variables (as they force Eshell syntax)
3. use ~if (elisp-function-call)~  for everything pure-elisp, including
elisp functions passed setq-ed variables
4. use ~if $(elisp-function-call)~ seems equivalent and interchangeable
with the above


Examples of 1. - check exit status of EXTERNAL program
- if {sh -c 'exit 0'} {echo "external succeeded"} {echo "external failed"}
# external succeeded
- if {sh -c 'exit 1'} {echo "external succeeded"} {echo "external failed"}
 # external failed
- if ${sh -c 'exit 0'} {echo "external succeeded"} {echo "external failed"}
# WRONG external failed
- if ${sh -c 'exit 1'} {echo "external succeeded"} {echo "external failed"}
# external failed
- Other example, if we want to check for success/failure in sed before
calling it
  - # just to test: echo "aaa" | sed 's/aaa/bbb/'                # bbb
  - if {echo "aaa" | sed 's/aaa/bbb/'} {echo sed-success} {echo
sed-failure} # sed-success
  - # just to test: echo "aaa" | sed 's/aaa/bbb'      # /usr/bin/sed: -e
expression #1, char 9: unterminated `s' command
  - if {echo "aaa" | sed 's/aaa/bbb'} {echo sed-success} {echo sed-failure}
 # sed-failure


Examples
- export a="3"
- Bad use:  if {equal $a "0"}   { echo YES } { echo NO }   # YES  --
WRONG!!! (silently!!)
- Good use: if ${equal $a "0"}  { echo YES } { echo NO }   # NO -- correct

[-- Attachment #2: Type: text/html, Size: 4301 bytes --]

bug#59666: 29.0.50; Eshell: comparisons such as {> 3 2} do not work in Eshell context

[Jim Porter]

On 12/1/2022 11:33 PM, Milan Zimmermann wrote:
> 1. use ~if {function-call}~  (only?) to check EXIT STATUS of the 
> EXTERNAL function (program) call.

One final note is that this should also work for Eshell commands (ones 
defined as "defun eshell/FOO"), since those are specially-written to 
conform to Eshell's command behavior. (That is, if they want to report 
an error, they set the exit status.)