bug#8531: 24.0.50; shell-quote-argument shouldn't escape special characters

bug#8531: 24.0.50; shell-quote-argument shouldn't escape special characters

[Thierry Volpiatto]

Hi all,

(shell-quote-argument "Vidéos")
==> "Vid\\éos"

eshell:
ls (shell-quote-argument "Vidéos")
Vid\éos: No such file or directory



In GNU Emacs 24.0.50.1 (i686-pc-linux-gnu, GTK+ Version 2.22.0)
 of 2011-04-19 on thierry-MM061
Windowing system distributor `The X.Org Foundation', version 11.0.10900000
Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: fr_FR.utf8
  value of $XMODIFIERS: nil
  locale-coding-system: utf-8-unix
  default enable-multibyte-characters: t

Major mode: Emacs-Lisp

Minor modes in effect:
  TeX-PDF-mode: t
  eldoc-mode: t
  minibuffer-depth-indicate-mode: t
  auto-image-file-mode: t
  show-paren-mode: t
  display-time-mode: t
  diff-auto-refine-mode: t
  recentf-mode: t
  savehist-mode: t
  shell-dirtrack-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
<M-up> <M-up> <M-up> <M-up> <M-up> <M-up> <M-up> <M-up> 
<M-up> <M-up> <M-up> <M-up> <M-up> <M-up> <M-up> <M-up> 
<M-up> <M-up> <up> <up> <up> <up> <up> <up> <up> <up> 
<up> <down> <right> <M-down> <M-up> <M-down> <M-down> 
<M-down> <M-down> <M-down> <M-down> <M-down> <M-down> 
<M-down> <M-down> <M-down> <M-down> <M-down> <M-down> 
<M-down> <M-down> <M-down> <M-down> <M-down> <M-down> 
<M-down> <M-down> <M-down> <M-down> <M-down> <M-down> 
<M-down> <M-down> <M-down> <M-down> <M-down> <M-down> 
<M-up> <M-up> <M-up> <M-up> <M-up> <M-up> <M-up> <M-up> 
<M-up> <M-up> <M-up> <M-up> <M-up> <M-up> <M-up> <M-up> 
<M-up> <M-up> <M-up> <M-up> <M-up> <M-up> <M-up> <M-up> 
<M-up> <M-up> <M-up> <M-up> <M-up> <M-up> <M-up> <M-up> 
<M-up> <M-up> <M-up> <left> <up> <up> <up> <up> <right> 
<M-down> <M-down> <M-down> <M-down> <M-down> <M-down> 
<M-down> <M-down> <M-down> <M-down> <M-down> <M-down> 
<M-down> <M-down> <M-down> <M-down> <M-down> <M-down> 
<M-down> <M-down> <M-down> <M-down> <M-down> <M-down> 
<M-down> <M-down> <M-down> <M-down> <M-down> <left> 
<M-up> <M-up> <M-up> <M-up> <M-up> <M-up> <M-up> <M-up> 
<M-up> <M-up> <up> <up> <up> <up> <up> <right> <M-down> 
<M-down> <M-down> <M-down> <M-down> <M-down> <M-down> 
<M-down> <M-down> <M-down> <M-down> <M-down> <M-down> 
<M-down> <M-down> <M-down> <M-down> <M-down> <M-down> 
<M-down> <M-down> <M-down> <M-down> <M-down> <M-down> 
<M-down> <M-down> <M-down> <M-down> <M-down> <M-down> 
<M-down> <M-down> <M-down> <M-down> <M-down> <M-down> 
<M-down> <M-down> <M-down> <M-down> <M-down> <M-down> 
<M-down> <M-down> <M-down> <M-down> <M-down> <M-down> 
<M-down> <M-down> <M-down> <M-down> <M-down> <M-down> 
<M-down> <M-down> <M-down> <M-down> <M-down> <M-down> 
<M-down> <M-down> <M-down> <M-down> <M-down> <M-down> 
<M-down> <M-down> <M-down> <M-down> <M-down> <M-down> 
<M-down> <M-down> <M-down> <M-down> <M-down> <M-down> 
<M-down> <M-down> <M-down> <M-down> <M-down> <M-down> 
<M-down> <M-down> <M-down> <M-down> <M-down> <M-down> 
<M-down> <M-down> <M-down> <M-down> <M-down> <M-down> 
<M-down> <M-down> <M-down> <M-down> <M-down> <M-down> 
<M-down> <M-down> <M-down> <M-down> <M-down> Q y C-z 
c C-c a a C-x C-² C-x C-b l i s p <down> <down> <C-tab> 
<f7> m <down> <return> <return> q <up> <return> <return> 
q q y <f7> m q y M-x r e p o r t <return>

Recent messages:
nnml: Reading incoming mail (no new mail)...done
Reading active file via nnml...done
Reading active file from archive via nnfolder...done
Reading active file via nndraft...done
Checking new news...done
No Gnus is good news
No more unread newsgroups
Are you sure you want to quit reading news? (y or n)  y
(No changes need to be saved)


Load-path shadows:
~/elisp/ngnus/lisp/.dir-locals hides ~/elisp/org-active/.dir-locals
~/elisp/ngnus/lisp/lpath hides ~/elisp/emacs-wget/lpath
~/elisp/ngnus/lisp/password-cache hides /usr/local/share/emacs/24.0.50/lisp/password-cache
~/elisp/ngnus/lisp/format-spec hides /usr/local/share/emacs/24.0.50/lisp/format-spec
~/elisp/ngnus/lisp/color hides /usr/local/share/emacs/24.0.50/lisp/color
~/elisp/ngnus/lisp/md4 hides /usr/local/share/emacs/24.0.50/lisp/md4
~/elisp/ngnus/lisp/hex-util hides /usr/local/share/emacs/24.0.50/lisp/hex-util
~/elisp/ngnus/lisp/sha1 hides /usr/local/share/emacs/24.0.50/lisp/sha1
~/elisp/ngnus/lisp/dns-mode hides /usr/local/share/emacs/24.0.50/lisp/textmodes/dns-mode
~/elisp/ngnus/lisp/imap hides /usr/local/share/emacs/24.0.50/lisp/net/imap
~/elisp/ngnus/lisp/hmac-md5 hides /usr/local/share/emacs/24.0.50/lisp/net/hmac-md5
~/elisp/ngnus/lisp/dig hides /usr/local/share/emacs/24.0.50/lisp/net/dig
~/elisp/ngnus/lisp/sasl hides /usr/local/share/emacs/24.0.50/lisp/net/sasl
~/elisp/ngnus/lisp/sasl-cram hides /usr/local/share/emacs/24.0.50/lisp/net/sasl-cram
~/elisp/ngnus/lisp/sasl-digest hides /usr/local/share/emacs/24.0.50/lisp/net/sasl-digest
~/elisp/ngnus/lisp/tls hides /usr/local/share/emacs/24.0.50/lisp/net/tls
~/elisp/ngnus/lisp/dns hides /usr/local/share/emacs/24.0.50/lisp/net/dns
~/elisp/ngnus/lisp/hmac-def hides /usr/local/share/emacs/24.0.50/lisp/net/hmac-def
~/elisp/ngnus/lisp/netrc hides /usr/local/share/emacs/24.0.50/lisp/net/netrc
~/elisp/ngnus/lisp/ntlm hides /usr/local/share/emacs/24.0.50/lisp/net/ntlm
~/elisp/ngnus/lisp/sasl-ntlm hides /usr/local/share/emacs/24.0.50/lisp/net/sasl-ntlm
~/elisp/ngnus/lisp/binhex hides /usr/local/share/emacs/24.0.50/lisp/mail/binhex
~/elisp/ngnus/lisp/hashcash hides /usr/local/share/emacs/24.0.50/lisp/mail/hashcash
~/elisp/ngnus/lisp/uudecode hides /usr/local/share/emacs/24.0.50/lisp/mail/uudecode
~/elisp/ngnus/lisp/parse-time hides /usr/local/share/emacs/24.0.50/lisp/calendar/parse-time
~/elisp/ngnus/lisp/time-date hides /usr/local/share/emacs/24.0.50/lisp/calendar/time-date
~/elisp/ngnus/lisp/pgg-parse hides /usr/local/share/emacs/24.0.50/lisp/obsolete/pgg-parse
~/elisp/ngnus/lisp/pgg hides /usr/local/share/emacs/24.0.50/lisp/obsolete/pgg
~/elisp/ngnus/lisp/pgg-pgp5 hides /usr/local/share/emacs/24.0.50/lisp/obsolete/pgg-pgp5
~/elisp/ngnus/lisp/pgg-pgp hides /usr/local/share/emacs/24.0.50/lisp/obsolete/pgg-pgp
~/elisp/ngnus/lisp/pgg-def hides /usr/local/share/emacs/24.0.50/lisp/obsolete/pgg-def
~/elisp/ngnus/lisp/pgg-gpg hides /usr/local/share/emacs/24.0.50/lisp/obsolete/pgg-gpg

Features:
(shadow epa-mail emacsbug qp smiley gnus-cite mail-extr gnus-async
gnus-bcklg gnus-ml nndraft nnmh utf-7 nnml nnimap utf7 nnfolder rot13
netrc network-stream starttls tls gnus-agent gnus-srvr gnus-score
score-mode nnvirtual gnus-msg nntp gnus-cache gnus-dired view cal-iso
w3m-form w3m-cookie w3m-bookmark w3m-tabmenu w3m-session vc-git xgit-dvc
xgit xgit-annotate xgit-log cc-mode cc-fonts cc-menus cc-cmds cc-styles
cc-align cc-engine cc-vars cc-defs conf-mode sh-script preview prv-emacs
tex-buf reftex-vcr reftex-dcr reftex-auc reftex imenu reftex-vars
font-latex latex tex-style tex latexenc vc-rcs vc-bzr bzr-dvc bzr
dvc-annotate xhg-dvc xhg xhg-annotate xhg-mq xhg-log bzr-core
xdarcs-core xgit-core xhg-core xmtn-minimal tla smerge-mode newcomment
dvc-state dvc-config dvc-diff dvc-fileinfo diff dvc-cmenu dvc-about
dvc-version dvc-revlist uniquify align-let server popup googlecl
google-maps google-maps-static google-maps-geocode google-maps-base json
simple-call-tree el-expectations el-mock csv2org iedit eieio-base
zop-to-char mule-util elscreen smallurl mm-url xml-weather
rectangle-utils tv-utils eldoc pcvs pcvs-parse pcvs-info pcvs-defs
auto-document autodoc mb-depth ioccur moz cl-info slime-xref-browser
slime-banner slime-tramp slime-asdf slime-fancy slime-fontifying-fu
slime-package-fu slime-references slime-scratch slime-presentations
slime-fuzzy slime-fancy-inspector slime-c-p-c slime-editing-commands
slime-autodoc slime-parse slime-repl slime apropos hideshow hyperspec
slime-autoloads boxquote rect image-file newsticker newst-treeview
newst-plainview newst-reader newst-ticker newst-backend ledger-config
ledger esh-arg esh-util lpr woman man two-column em-term term ehelp
electric esh-toggle em-xtra flymake pdbtrack no-word regex-tool
whitespace htmlfontify paren time dired-tar dired-extension image-dired
yaoddmuse skeleton sgml-mode emms-mplayer-config emms-playlist-limit
emms-volume emms-volume-amixer emms-i18n emms-history emms-score
emms-stream-info emms-metaplaylist-mode emms-bookmarks
emms-lastfm-client parse-time emms-cue emms-mode-line-icon emms-browser
sort emms-playlist-sort emms-last-played emms-player-xine
emms-player-mpd tq emms-playing-time emms-lyrics emms-url hl-line
emms-tag-editor emms-mark emms-mode-line emms-cache emms-info-ogginfo
emms-info-mp3info emms-playlist-mode emms-player-vlc emms-player-mplayer
emms-info emms-streams later-do emms-source-playlist emms-source-file
emms-player-simple emms-setup emms emms-compat winner dvc-init bzr-gnus
tla-gnus xgit-gnus xhg-gnus gnus-art mm-uu mml2015 mm-view mml-smime
smime dig nnir gnus-sum nnoo gnus-group gnus-undo nnmail mail-source
gnus-start gnus-spec gnus-int gnus-range gnus-win gnus gnus-ems nnheader
dvc-gnus tla-core tla-autoconf tla-defs dvc-log vc vc-dispatcher
dvc-unified dvc-tips dired-x ediff-merg ediff-diff ediff-wind ediff-help
ediff-util ediff-mult ediff-init ediff dvc-autoloads dvc-core dvc-lisp
dvc-buffers dvc-ui dvc-register dvc-utils dvc-emacs ewoc dvc-defs
dvc-site psvn log-edit pcvs-util add-log diff-mode htmlize-hack htmlize
muse-colors muse-docbook muse-texinfo texnfo-upd texinfo muse-latex
muse-html muse-xml-common muse-wiki cus-edit cus-start cus-load
muse-publish muse-project muse-protocols muse-regexps muse
muse-nested-tags muse-mode muse-autoloads org-config-thierry org-crypt
cal-china lunar solar cal-dst cal-bahai cal-islam cal-hebrew holidays
hol-loaddefs vc-hg org-wl org-w3m org-vm org-rmail org-mhe org-mew
org-irc org-jsinfo org-infojs org-html org-exp ob-exp org-exp-blocks
org-info org-gnus org-docview org-bibtex org-bbdb org-agenda appt
diary-lib diary-loaddefs org-annotation-helper org-capture org-mks
remember org-remember org-datetree config-w3m mime-w3m w3m doc-view
jka-compr image-mode timezone w3m-hist w3m-fb w3m-ems w3m-ccl ccl
w3m-favicon w3m-image w3m-proc w3m-util mime eword-decode mel path-util
mime-parse std11 luna mime-def alist mcharset mcs-20 mcs-e20 pces
pces-e20 pces-20 broken pcustom poe pym static apel-ver product w3m-load
addressbook-bookmark message sendmail rfc822 mml mml-sec mm-decode
mm-bodies mm-encode mail-parse rfc2231 rfc2047 rfc2045 ietf-drums
mailabbrev mail-utils gmm-utils mailheader firefox-protocol
bookmark-firefox-handler url url-proxy url-privacy url-expand
url-methods url-history url-cookie url-util url-parse url-vars mailcap
bookmark-extensions org ob-emacs-lisp ob-tangle ob-ref ob-lob ob-table
org-footnote org-src ob-comint ob-keys ob ob-eval org-pcomplete
pcomplete org-list org-faces org-compat org-entities org-macs noutline
outline cal-menu calendar cal-loaddefs bookmark pp recentf tree-widget
wid-edit savehist init-anything-thierry descbinds-anything
anything-ipython ipython python-mode info-look ansi-color executable
shell shell-history anything-complete anything-show-completion
anything-obsolete anything-match-plugin anything-delicious xml
anything-mercurial anything-config browse-url rx grep compile comint
ring tramp tramp-compat format-spec tramp-loaddefs dired-aux ffap
thingatpt anything warnings epa-file epa derived epg epg-config
auth-source eieio byte-opt bytecomp byte-compile cconv macroexp assoc
gnus-util time-date mm-util mail-prsvr password-cache dired regexp-opt
gnus-load usage-memo punycode idna naquadah-theme sha1 hex-util
eev-thierry edmacro kmacro iterator eev-all eev-mini-steps
eev-browse-url eev-langs eev-compose eev-glyphs disp-table eev-insert
eev-steps eev-bounded eev easy-mmode advice help-fns advice-preload
w3m-wget preview-latex tex-site auto-loads info easymenu cl tooltip
ediff-hook vc-hooks lisp-float-type mwheel x-win x-dnd tool-bar dnd
fontset image fringe lisp-mode register page menu-bar rfn-eshadow timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
frame cham georgian utf-8-lang misc-lang vietnamese tibetan thai
tai-viet lao korean japanese hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese case-table epa-hook jka-cmpr-hook help
simple abbrev minibuffer loaddefs button faces cus-face files
text-properties overlay md5 base64 format env code-pages mule custom
widget hashtable-print-readable backquote make-network-process dbusbind
dynamic-setting font-render-setting move-toolbar gtk x-toolkit x
multi-tty emacs)

-- 
A+ Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997 

bug#8531: 24.0.50; shell-quote-argument shouldn't escape special characters

[Eli Zaretskii]

> From: Thierry Volpiatto <thierry.volpiatto@gmail.com>
> Date: Thu, 21 Apr 2011 09:23:13 +0200
> 
> (shell-quote-argument "Vidéos")
> ==> "Vid\\éos"
> 
> eshell:
> ls (shell-quote-argument "Vidéos")
> Vid\éos: No such file or directory

Looks like a problem in Eshell: I have no problem with using the
result of shell-quote-argument in "M-!".

bug#8531: 24.0.50; shell-quote-argument shouldn't escape special characters

[Thierry Volpiatto]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Thierry Volpiatto <thierry.volpiatto@gmail.com>
>> Date: Thu, 21 Apr 2011 09:23:13 +0200
>> 
>> (shell-quote-argument "Vidéos")
>> ==> "Vid\\éos"
>> 
>> eshell:
>> ls (shell-quote-argument "Vidéos")
>> Vid\éos: No such file or directory
>
> Looks like a problem in Eshell:
No it is the same in M-x shell

> I have no problem with using the
> result of shell-quote-argument in "M-!".
Which result?
Which command?
I am speaking of escaping accentued characters "é, à etc..."
Why `shell-quote-argument' escape such characters?

-- 
A+ Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997 

bug#8531: 24.0.50; shell-quote-argument shouldn't escape special characters

[Eli Zaretskii]

> From: Thierry Volpiatto <thierry.volpiatto@gmail.com>
> Date: Thu, 21 Apr 2011 14:10:49 +0200
> 
> > I have no problem with using the
> > result of shell-quote-argument in "M-!".
> Which result?
> Which command?

The command was "ls Vid\éos".  It displayed the files in that
directory.

> I am speaking of escaping accentued characters "é, à etc..."

Yes, so am I.

> Why `shell-quote-argument' escape such characters?

Because it wants to play safe.  In a Posix shell, any non-special
character preceded with a backslash stands for itself, I'm sure you
know that.  Backslash-escaping anything that is not in the Posix
character set avoids too much knowledge about the underlying shell's
special characters, and should be harmless if the escaped characters
are not special.

bug#8531: 24.0.50; shell-quote-argument shouldn't escape special characters

[Thierry Volpiatto]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Thierry Volpiatto <thierry.volpiatto@gmail.com>
>> Date: Thu, 21 Apr 2011 14:10:49 +0200
>> 
>> > I have no problem with using the
>> > result of shell-quote-argument in "M-!".
>> Which result?
>> Which command?
>
> The command was "ls Vid\éos".  It displayed the files in that
> directory.
Yes, but ls "Vid\éos" fail.

The point is being able to do something like this:
(shell-command (format "ls %s" (shell-quote-argument (read-string "Fname: "))))

instead of e.g
(shell-command (format "ls '%s'" (read-string "Fname: ")))


>> I am speaking of escaping accentued characters "é, à etc..."
>
> Yes, so am I.
Ok.
>> Why `shell-quote-argument' escape such characters?
>
> Because it wants to play safe.  In a Posix shell, any non-special
> character preceded with a backslash stands for itself, I'm sure you
> know that.  Backslash-escaping anything that is not in the Posix
> character set avoids too much knowledge about the underlying shell's
> special characters, and should be harmless if the escaped characters
> are not special.
Oh yes, i remember now that "é etc.." are not in posix character set,
thanks.

-- 
A+ Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997 

bug#8531: 24.0.50; shell-quote-argument shouldn't escape special characters

[Eli Zaretskii]

> From: Thierry Volpiatto <thierry.volpiatto@gmail.com>
> Date: Thu, 21 Apr 2011 15:31:41 +0200
> 
> > The command was "ls Vid\éos".  It displayed the files in that
> > directory.
> Yes, but ls "Vid\éos" fail.

Where did the quotes come from?  shell-quote-argument doesn't add
extra quotes.

> The point is being able to do something like this:
> (shell-command (format "ls %s" (shell-quote-argument (read-string "Fname: "))))
> 
> instead of e.g
> (shell-command (format "ls '%s'" (read-string "Fname: ")))

The former works for me without any changes.  Are you sure you use the
default version of shell-quote-argument?

bug#8531: 24.0.50; shell-quote-argument shouldn't escape special characters

[Thierry Volpiatto]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Thierry Volpiatto <thierry.volpiatto@gmail.com>
>> Date: Thu, 21 Apr 2011 15:31:41 +0200
>> 
>> > The command was "ls Vid\éos".  It displayed the files in that
>> > directory.
>> Yes, but ls "Vid\éos" fail.
>
> Where did the quotes come from?  shell-quote-argument doesn't add
> extra quotes.
>
>> The point is being able to do something like this:
>> (shell-command (format "ls %s" (shell-quote-argument (read-string "Fname: "))))
>> 
>> instead of e.g
>> (shell-command (format "ls '%s'" (read-string "Fname: ")))
>
> The former works for me without any changes.  Are you sure you use the
> default version of shell-quote-argument?
Yes you're right, sorry, i was inserting ~/Vidéos.
So yes the problem is related only to eshell.

-- 
A+ Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997 

bug#8531: 24.0.50; shell-quote-argument shouldn't escape special characters

[Glenn Morris]

Thierry Volpiatto wrote:

> So yes the problem is related only to eshell.

Why do you want/need to call shell-quote-argument inside eshell?

bug#8531: 24.0.50; shell-quote-argument shouldn't escape special characters

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Date: Thu, 21 Apr 2011 19:14:58 -0400
> Cc: 8531@debbugs.gnu.org
> 
> Thierry Volpiatto wrote:
> 
> > So yes the problem is related only to eshell.
> 
> Why do you want/need to call shell-quote-argument inside eshell?

He doesn't.  But a literal Vid\éos (with the backslash typed by hand)
doesn't work with Eshell, either.  Doesn't Eshell support the standard
shell feature whereby a non-special character escaped by a backslash
stands for itself?

bug#8531: 24.0.50; shell-quote-argument shouldn't escape special characters

[Thierry Volpiatto]

Glenn Morris <rgm@gnu.org> writes:

> Thierry Volpiatto wrote:
>
>> So yes the problem is related only to eshell.
>
> Why do you want/need to call shell-quote-argument inside eshell?
For compatibility with Windows filenames:

--8<---------------cut here---------------start------------->8---
(goto-char (point-max))
(insert
 (format "cd '%s'" anything-ff-default-directory))
(eshell-send-input)))
--8<---------------cut here---------------end--------------->8---

As there is no evaluation inside the quotes,
this actually works fine, but before that, i wrote it with
shell-quote-argument, which was working fine excepted with accentued char.

-- 
A+ Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997 

bug#8531: 24.0.50; shell-quote-argument shouldn't escape special characters

[Eli Zaretskii]

> From: Thierry Volpiatto <thierry.volpiatto@gmail.com>
> Date: Fri, 22 Apr 2011 08:03:30 +0200
> Cc: 8531@debbugs.gnu.org
> 
> Glenn Morris <rgm@gnu.org> writes:
> 
> > Thierry Volpiatto wrote:
> >
> >> So yes the problem is related only to eshell.
> >
> > Why do you want/need to call shell-quote-argument inside eshell?
> For compatibility with Windows filenames:

You mean, file names with embedded white space?  If not, what other
compatibility issues are there?

bug#8531: 24.0.50; shell-quote-argument shouldn't escape special characters

[Glenn Morris]

Eli Zaretskii wrote:

> He doesn't.  But a literal Vid\éos (with the backslash typed by hand)
> doesn't work with Eshell, either.  Doesn't Eshell support the standard
> shell feature whereby a non-special character escaped by a backslash
> stands for itself?

From reading the comments in eshell-parse-backslash, no.

bug#8531: 24.0.50; shell-quote-argument shouldn't escape special characters

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: thierry.volpiatto@gmail.com,  8531@debbugs.gnu.org
> Date: Fri, 22 Apr 2011 03:10:39 -0400
> 
> Eli Zaretskii wrote:
> 
> > Doesn't Eshell support the standard shell feature whereby a
> > non-special character escaped by a backslash stands for itself?
> 
> From reading the comments in eshell-parse-backslash, no.

IMO, the reasons stated in those comments for not supporting escaped
non-special characters are bad a design decision.  Eshell should only
do that on Windows, and even then there should be a user option for
such behavior, the default being that the directory separator is a
forward slash.

IOW, Eshell should behave like a Posix shell, even on Windows.

bug#8531: 24.0.50; shell-quote-argument shouldn't escape special characters

[Thierry Volpiatto]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Thierry Volpiatto <thierry.volpiatto@gmail.com>
>> Date: Fri, 22 Apr 2011 08:03:30 +0200
>> Cc: 8531@debbugs.gnu.org
>> 
>> Glenn Morris <rgm@gnu.org> writes:
>> 
>> > Thierry Volpiatto wrote:
>> >
>> >> So yes the problem is related only to eshell.
>> >
>> > Why do you want/need to call shell-quote-argument inside eshell?
>> For compatibility with Windows filenames:
>
> You mean, file names with embedded white space?
Yes.

> If not, what other compatibility issues are there?
Nothing else for the moment.

-- 
A+ Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997 

bug#8531: 24.0.50;

[samer]

Hi,

I found this bug on debbugs thanks to Lars' getting started guide. This 
change simplifies the code by making the behavior that used occur for 
newlines following a backslash, returning the character literal after 
the backslash, the behavior that occurs for _all_ non-special 
characters.

This is my first contribution to emacs, so don't hesitate to correct 
anything nonstandard in the patch I've included. I'm interested in doing 
more work on eshell, too. What eshell bugs would benefit from a fresh 
pair of eyes?

Best,
Samer

Patch for esh-arg.el:

*** 
/home/samer/build/emacs120714/share/emacs/25.0.50/lisp/eshell/esh-arg.el.gz.old
--- 
/home/samer/build/emacs120714/share/emacs/25.0.50/lisp/eshell/esh-arg.el.gz
***************
*** 89,95 ****
   	  (goto-char (match-end 0))
   	  (eshell-finish-arg)))))

!    ;; backslash before a special character means escape it
      'eshell-parse-backslash

      ;; text beginning with ' is a literally quoted
--- 89,96 ----
   	  (goto-char (match-end 0))
   	  (eshell-finish-arg)))))

!    ;; backslash before a character escapes it if the character is
!    ;; special, and returns the character literal if it is non-special
      'eshell-parse-backslash

      ;; text beginning with ' is a literally quoted
***************
*** 282,294 ****
     "A stub function that generates an error if a floating operator is 
found."
     (error "Unhandled operator in input text"))

- (defsubst eshell-looking-at-backslash-return (pos)
-   "Test whether a backslash-return sequence occurs at POS."
-   (and (eq (char-after pos) ?\\)
-        (or (= (1+ pos) (point-max))
- 	   (and (eq (char-after (1+ pos)) ?\n)
- 		(= (+ pos 2) (point-max))))))
-
   (defun eshell-quote-backslash (string &optional index)
     "Intelligently backslash the character occurring in STRING at INDEX.
   If the character is itself a backslash, it needs no escaping."
--- 283,288 ----
***************
*** 305,313 ****
   	  (string ?\\ char)))))

   (defun eshell-parse-backslash ()
!   "Parse a single backslash (\) character, which might mean escape.
! It only means escape if the character immediately following is a
! special character that is not itself a backslash."
     (when (eq (char-after) ?\\)
       (if (eshell-looking-at-backslash-return (point))
   	(throw 'eshell-incomplete ?\\)
--- 299,309 ----
   	  (string ?\\ char)))))

   (defun eshell-parse-backslash ()
!   "Parse a single backslash (\) character to escape the character 
after.
! If the character immediately following the backslash is a special
! character, it returns the escaped version of that character.
! Else, the character has no meaning and is returned as the literal
! character. This conforms with the behavior of bash."
     (when (eq (char-after) ?\\)
       (if (eshell-looking-at-backslash-return (point))
   	(throw 'eshell-incomplete ?\\)
***************
*** 321,338 ****
   	    (forward-char 2)
   	    (list 'eshell-escape-arg
   		  (char-to-string (char-before))))
- 	;; allow \\<RET> to mean a literal "\" character followed by a
- 	;; normal return, rather than a backslash followed by a line
- 	;; continuation (i.e., "\\ + \n" rather than "\ + \\n").  This
- 	;; is necessary because backslashes in Eshell are not special
- 	;; unless they either precede something special, or precede a
- 	;; backslash that precedes something special.  (Mainly this is
- 	;; done to make using backslash on Windows systems more
- 	;; natural-feeling).
- 	(if (eshell-looking-at-backslash-return (1+ (point)))
- 	    (forward-char))
   	(forward-char)
! 	"\\"))))

   (defun eshell-parse-literal-quote ()
     "Parse a literally quoted string.  Nothing has special meaning!"
--- 317,325 ----
   	    (forward-char 2)
   	    (list 'eshell-escape-arg
   		  (char-to-string (char-before))))
   	(forward-char)
! 	(forward-char)
! 	(char-before)))))

   (defun eshell-parse-literal-quote ()
     "Parse a literally quoted string.  Nothing has special meaning!"

Diff finished.  Mon Dec  8 01:23:11 2014
*** 
/home/samer/build/emacs120714/share/emacs/25.0.50/lisp/eshell/esh-arg.el.gz.old
--- 
/home/samer/build/emacs120714/share/emacs/25.0.50/lisp/eshell/esh-arg.el.gz
***************
*** 89,95 ****
   	  (goto-char (match-end 0))
   	  (eshell-finish-arg)))))

!    ;; backslash before a special character means escape it
      'eshell-parse-backslash

      ;; text beginning with ' is a literally quoted
--- 89,96 ----
   	  (goto-char (match-end 0))
   	  (eshell-finish-arg)))))

!    ;; backslash before a character escapes it if the character is
!    ;; special, and returns the character literal if it is non-special
      'eshell-parse-backslash

      ;; text beginning with ' is a literally quoted
***************
*** 282,294 ****
     "A stub function that generates an error if a floating operator is 
found."
     (error "Unhandled operator in input text"))

- (defsubst eshell-looking-at-backslash-return (pos)
-   "Test whether a backslash-return sequence occurs at POS."
-   (and (eq (char-after pos) ?\\)
-        (or (= (1+ pos) (point-max))
- 	   (and (eq (char-after (1+ pos)) ?\n)
- 		(= (+ pos 2) (point-max))))))
-
   (defun eshell-quote-backslash (string &optional index)
     "Intelligently backslash the character occurring in STRING at INDEX.
   If the character is itself a backslash, it needs no escaping."
--- 283,288 ----
***************
*** 305,313 ****
   	  (string ?\\ char)))))

   (defun eshell-parse-backslash ()
!   "Parse a single backslash (\) character, which might mean escape.
! It only means escape if the character immediately following is a
! special character that is not itself a backslash."
     (when (eq (char-after) ?\\)
       (if (eshell-looking-at-backslash-return (point))
   	(throw 'eshell-incomplete ?\\)
--- 299,309 ----
   	  (string ?\\ char)))))

   (defun eshell-parse-backslash ()
!   "Parse a single backslash (\) character to escape the character 
after.
! If the character immediately following the backslash is a special
! character, it returns the escaped version of that character.
! Else, the character has no meaning and is returned as the literal
! character. This conforms with the behavior of bash."
     (when (eq (char-after) ?\\)
       (if (eshell-looking-at-backslash-return (point))
   	(throw 'eshell-incomplete ?\\)
***************
*** 321,338 ****
   	    (forward-char 2)
   	    (list 'eshell-escape-arg
   		  (char-to-string (char-before))))
- 	;; allow \\<RET> to mean a literal "\" character followed by a
- 	;; normal return, rather than a backslash followed by a line
- 	;; continuation (i.e., "\\ + \n" rather than "\ + \\n").  This
- 	;; is necessary because backslashes in Eshell are not special
- 	;; unless they either precede something special, or precede a
- 	;; backslash that precedes something special.  (Mainly this is
- 	;; done to make using backslash on Windows systems more
- 	;; natural-feeling).
- 	(if (eshell-looking-at-backslash-return (1+ (point)))
- 	    (forward-char))
   	(forward-char)
! 	"\\"))))

   (defun eshell-parse-literal-quote ()
     "Parse a literally quoted string.  Nothing has special meaning!"
--- 317,325 ----
   	    (forward-char 2)
   	    (list 'eshell-escape-arg
   		  (char-to-string (char-before))))
   	(forward-char)
! 	(forward-char)
! 	(char-before)))))

   (defun eshell-parse-literal-quote ()
     "Parse a literally quoted string.  Nothing has special meaning!"

bug#8531: 24.0.50; shell-quote-argument shouldn't escape special characters

[samer]

Hi,

I found this bug on debbugs thanks to Lars' getting started guide. This 
change simplifies the code by making the behavior that used occur for 
newlines following a backslash, returning the character literal after 
the backslash, the behavior that occurs for _all_ non-special 
characters.

This is my first contribution to emacs, so don't hesitate to correct 
anything nonstandard in the patch I've included. I'm interested in doing 
more work on eshell, too. What eshell bugs would benefit from a fresh 
pair of eyes?

Also, I sent this email to 8531@debbugs.gnu.org five hours and I didn't 
see my email on debbugs. Apologies for the duplication if that email 
actually went through.

Best,
Samer

Patch for esh-arg.el:

*** 
/home/samer/build/emacs120714/share/emacs/25.0.50/lisp/eshell/esh-arg.el.gz.old
--- 
/home/samer/build/emacs120714/share/emacs/25.0.50/lisp/eshell/esh-arg.el.gz
***************
*** 89,95 ****
   	  (goto-char (match-end 0))
   	  (eshell-finish-arg)))))

!    ;; backslash before a special character means escape it
      'eshell-parse-backslash

      ;; text beginning with ' is a literally quoted
--- 89,96 ----
   	  (goto-char (match-end 0))
   	  (eshell-finish-arg)))))

!    ;; backslash before a character escapes it if the character is
!    ;; special, and returns the character literal if it is non-special
      'eshell-parse-backslash

      ;; text beginning with ' is a literally quoted
***************
*** 282,294 ****
     "A stub function that generates an error if a floating operator is 
found."
     (error "Unhandled operator in input text"))

- (defsubst eshell-looking-at-backslash-return (pos)
-   "Test whether a backslash-return sequence occurs at POS."
-   (and (eq (char-after pos) ?\\)
-        (or (= (1+ pos) (point-max))
- 	   (and (eq (char-after (1+ pos)) ?\n)
- 		(= (+ pos 2) (point-max))))))
-
   (defun eshell-quote-backslash (string &optional index)
     "Intelligently backslash the character occurring in STRING at INDEX.
   If the character is itself a backslash, it needs no escaping."
--- 283,288 ----
***************
*** 305,313 ****
   	  (string ?\\ char)))))

   (defun eshell-parse-backslash ()
!   "Parse a single backslash (\) character, which might mean escape.
! It only means escape if the character immediately following is a
! special character that is not itself a backslash."
     (when (eq (char-after) ?\\)
       (if (eshell-looking-at-backslash-return (point))
   	(throw 'eshell-incomplete ?\\)
--- 299,309 ----
   	  (string ?\\ char)))))

   (defun eshell-parse-backslash ()
!   "Parse a single backslash (\) character to escape the character 
after.
! If the character immediately following the backslash is a special
! character, it returns the escaped version of that character.
! Else, the character has no meaning and is returned as the literal
! character. This conforms with the behavior of bash."
     (when (eq (char-after) ?\\)
       (if (eshell-looking-at-backslash-return (point))
   	(throw 'eshell-incomplete ?\\)
***************
*** 321,338 ****
   	    (forward-char 2)
   	    (list 'eshell-escape-arg
   		  (char-to-string (char-before))))
- 	;; allow \\<RET> to mean a literal "\" character followed by a
- 	;; normal return, rather than a backslash followed by a line
- 	;; continuation (i.e., "\\ + \n" rather than "\ + \\n").  This
- 	;; is necessary because backslashes in Eshell are not special
- 	;; unless they either precede something special, or precede a
- 	;; backslash that precedes something special.  (Mainly this is
- 	;; done to make using backslash on Windows systems more
- 	;; natural-feeling).
- 	(if (eshell-looking-at-backslash-return (1+ (point)))
- 	    (forward-char))
   	(forward-char)
! 	"\\"))))

   (defun eshell-parse-literal-quote ()
     "Parse a literally quoted string.  Nothing has special meaning!"
--- 317,325 ----
   	    (forward-char 2)
   	    (list 'eshell-escape-arg
   		  (char-to-string (char-before))))
   	(forward-char)
! 	(forward-char)
! 	(char-before)))))

   (defun eshell-parse-literal-quote ()
     "Parse a literally quoted string.  Nothing has special meaning!"

Diff finished.  Mon Dec  8 01:23:11 2014
*** 
/home/samer/build/emacs120714/share/emacs/25.0.50/lisp/eshell/esh-arg.el.gz.old
--- 
/home/samer/build/emacs120714/share/emacs/25.0.50/lisp/eshell/esh-arg.el.gz
***************
*** 89,95 ****
   	  (goto-char (match-end 0))
   	  (eshell-finish-arg)))))

!    ;; backslash before a special character means escape it
      'eshell-parse-backslash

      ;; text beginning with ' is a literally quoted
--- 89,96 ----
   	  (goto-char (match-end 0))
   	  (eshell-finish-arg)))))

!    ;; backslash before a character escapes it if the character is
!    ;; special, and returns the character literal if it is non-special
      'eshell-parse-backslash

      ;; text beginning with ' is a literally quoted
***************
*** 282,294 ****
     "A stub function that generates an error if a floating operator is 
found."
     (error "Unhandled operator in input text"))

- (defsubst eshell-looking-at-backslash-return (pos)
-   "Test whether a backslash-return sequence occurs at POS."
-   (and (eq (char-after pos) ?\\)
-        (or (= (1+ pos) (point-max))
- 	   (and (eq (char-after (1+ pos)) ?\n)
- 		(= (+ pos 2) (point-max))))))
-
   (defun eshell-quote-backslash (string &optional index)
     "Intelligently backslash the character occurring in STRING at INDEX.
   If the character is itself a backslash, it needs no escaping."
--- 283,288 ----
***************
*** 305,313 ****
   	  (string ?\\ char)))))

   (defun eshell-parse-backslash ()
!   "Parse a single backslash (\) character, which might mean escape.
! It only means escape if the character immediately following is a
! special character that is not itself a backslash."
     (when (eq (char-after) ?\\)
       (if (eshell-looking-at-backslash-return (point))
   	(throw 'eshell-incomplete ?\\)
--- 299,309 ----
   	  (string ?\\ char)))))

   (defun eshell-parse-backslash ()
!   "Parse a single backslash (\) character to escape the character 
after.
! If the character immediately following the backslash is a special
! character, it returns the escaped version of that character.
! Else, the character has no meaning and is returned as the literal
! character. This conforms with the behavior of bash."
     (when (eq (char-after) ?\\)
       (if (eshell-looking-at-backslash-return (point))
   	(throw 'eshell-incomplete ?\\)
***************
*** 321,338 ****
   	    (forward-char 2)
   	    (list 'eshell-escape-arg
   		  (char-to-string (char-before))))
- 	;; allow \\<RET> to mean a literal "\" character followed by a
- 	;; normal return, rather than a backslash followed by a line
- 	;; continuation (i.e., "\\ + \n" rather than "\ + \\n").  This
- 	;; is necessary because backslashes in Eshell are not special
- 	;; unless they either precede something special, or precede a
- 	;; backslash that precedes something special.  (Mainly this is
- 	;; done to make using backslash on Windows systems more
- 	;; natural-feeling).
- 	(if (eshell-looking-at-backslash-return (1+ (point)))
- 	    (forward-char))
   	(forward-char)
! 	"\\"))))

   (defun eshell-parse-literal-quote ()
     "Parse a literally quoted string.  Nothing has special meaning!"
--- 317,325 ----
   	    (forward-char 2)
   	    (list 'eshell-escape-arg
   		  (char-to-string (char-before))))
   	(forward-char)
! 	(forward-char)
! 	(char-before)))))

   (defun eshell-parse-literal-quote ()
     "Parse a literally quoted string.  Nothing has special meaning!"

bug#8531: 24.0.50;

[Stefan Monnier]

> I found this bug on debbugs thanks to Lars' getting started guide. This
> change simplifies the code by making the behavior that used occur for
> newlines following a backslash, returning the character literal after the
> backslash, the behavior that occurs for _all_ non-special characters.

This looks OK, thanks.  Please include an entry for etc/NEWS mentioning
the change in behavior and a description for the ChangeLog file.


        Stefan

bug#8531: 24.0.50;

[samer]

Forgot to CC debbugs in my previous email:

This is the full patch, generated with magit because I couldn't commit 
this to a private branch (I get "Empty change log entry" even though 
I've changed both ChangeLog files. I don't know awk, do you have any 
idea why I would get that error?)

Best,
Samer

Changes from master to working tree
4 files changed, 26 insertions(+), 22 deletions(-)
  etc/ChangeLog          |  4 ++++
  etc/NEWS               |  6 ++++++
  lisp/ChangeLog         |  7 +++++++
  lisp/eshell/esh-arg.el | 31 +++++++++----------------------

	Modified   etc/ChangeLog
diff --git a/etc/ChangeLog b/etc/ChangeLog
index 309c01f..3e76256 100644
--- a/etc/ChangeLog
+++ b/etc/ChangeLog
@@ -1,3 +1,7 @@
+2014-12-08  Samer Masterson  <samer@dark-horse>
+
+	* NEWS: Mention change in backslash expand behavior for eshell.
+
  2014-12-08  Lars Magne Ingebrigtsen  <larsi@gnus.org>

  	* NEWS: Mention the new eww `S' command.
	Modified   etc/NEWS
diff --git a/etc/NEWS b/etc/NEWS
index 56036f8..e6d9aab 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -192,6 +192,12 @@ result of the calculation into the current buffer.
  *** New minor mode global-eldoc-mode
  *** eldoc-documentation-function now defaults to nil

+** eshell
+
+*** Backslash (\) expands to the character literal after it if that
+character is non-special (e.g. 'b\in' expands to 'bin', because 'i' is
+not a special character). This behavior conforms with bash.
+
  ** eww

  +++
	Modified   lisp/ChangeLog
diff --git a/lisp/ChangeLog b/lisp/ChangeLog
index 2669e07..0ec9b35 100644
--- a/lisp/ChangeLog
+++ b/lisp/ChangeLog
@@ -1,3 +1,10 @@
+2014-12-08  Samer Masterson  <samer@samertm.com>
+
+	* eshell/esh-arg.el (eshell-parse-backslash): Return the literal
+	character after the backslash if the character is non-special
+	(bug#8531).
+	(eshell-looking-at-backslash-return): Unused, remove.
+
  2014-12-08  Lars Magne Ingebrigtsen  <larsi@gnus.org>

  	* net/nsm.el (nsm-check-protocol): Test for RC4 on `high'.
	Modified   lisp/eshell/esh-arg.el
diff --git a/lisp/eshell/esh-arg.el b/lisp/eshell/esh-arg.el
index 704de57..33ff384 100644
--- a/lisp/eshell/esh-arg.el
+++ b/lisp/eshell/esh-arg.el
@@ -89,7 +89,8 @@ yield the values intended."
  	  (goto-char (match-end 0))
  	  (eshell-finish-arg)))))

-   ;; backslash before a special character means escape it
+   ;; backslash before a character escapes it if the character is
+   ;; special, and returns the character literal if it is non-special
     'eshell-parse-backslash

     ;; text beginning with ' is a literally quoted
@@ -282,13 +283,6 @@ Point is left at the end of the arguments."
    "A stub function that generates an error if a floating operator is 
found."
    (error "Unhandled operator in input text"))

-(defsubst eshell-looking-at-backslash-return (pos)
-  "Test whether a backslash-return sequence occurs at POS."
-  (and (eq (char-after pos) ?\\)
-       (or (= (1+ pos) (point-max))
-	   (and (eq (char-after (1+ pos)) ?\n)
-		(= (+ pos 2) (point-max))))))
-
  (defun eshell-quote-backslash (string &optional index)
    "Intelligently backslash the character occurring in STRING at INDEX.
  If the character is itself a backslash, it needs no escaping."
@@ -305,9 +299,11 @@ If the character is itself a backslash, it needs no 
escaping."
  	  (string ?\\ char)))))

  (defun eshell-parse-backslash ()
-  "Parse a single backslash (\) character, which might mean escape.
-It only means escape if the character immediately following is a
-special character that is not itself a backslash."
+  "Parse a single backslash (\) character to escape the character 
after.
+If the character immediately following the backslash is a special
+character, it returns the escaped version of that character.
+Else, the character has no meaning and is returned as the literal
+character. This conforms with the behavior of bash."
    (when (eq (char-after) ?\\)
      (if (eshell-looking-at-backslash-return (point))
  	(throw 'eshell-incomplete ?\\)
@@ -321,18 +317,9 @@ special character that is not itself a backslash."
  	    (forward-char 2)
  	    (list 'eshell-escape-arg
  		  (char-to-string (char-before))))
-	;; allow \\<RET> to mean a literal "\" character followed by a
-	;; normal return, rather than a backslash followed by a line
-	;; continuation (i.e., "\\ + \n" rather than "\ + \\n").  This
-	;; is necessary because backslashes in Eshell are not special
-	;; unless they either precede something special, or precede a
-	;; backslash that precedes something special.  (Mainly this is
-	;; done to make using backslash on Windows systems more
-	;; natural-feeling).
-	(if (eshell-looking-at-backslash-return (1+ (point)))
-	    (forward-char))
  	(forward-char)
-	"\\"))))
+	(forward-char)
+	(char-before)))))

  (defun eshell-parse-literal-quote ()
    "Parse a literally quoted string.  Nothing has special meaning!"

bug#8531: 24.0.50;

[Lars Magne Ingebrigtsen]

samer <samer@samertm.com> writes:

> Forgot to CC debbugs in my previous email:

Looks good, I think.  Do you have FSF copyright assignment papers on file?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#8531: 24.0.50;

[samer]

On 2014-12-08 16:41, Lars Magne Ingebrigtsen wrote:
> samer <samer@samertm.com> writes:
> 
>> Forgot to CC debbugs in my previous email:
> 
> Looks good, I think.  Do you have FSF copyright assignment papers on 
> file?

I don't, but I would be willing to assign my copyright to the FSF. How 
can we start that process?

Best,
Samer

bug#8531: 24.0.50;

[Lars Magne Ingebrigtsen]

samer <samer@samertm.com> writes:

> I don't, but I would be willing to assign my copyright to the FSF. How
> can we start that process?

I've sent the assignment form off-list.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#8531: 24.0.50;

[samer]

I've sent an email to assign@gnu.org. What is the next step for getting 
this patch merged into Emacs?

Best,
Samer

bug#8531: 24.0.50;

[Lars Magne Ingebrigtsen]

samer <samer@samertm.com> writes:

> I've sent an email to assign@gnu.org. What is the next step for
> getting this patch merged into Emacs?

The copyright clerk will get in touch with you.  Once the paperwork is
finalised, follow up to this bug report, and we can get your patches
installed.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#8531: 24.0.50;

[samer]

Alright, thanks!

Best,
Samer

On 2014-12-09 14:21, Lars Magne Ingebrigtsen wrote:
> samer <samer@samertm.com> writes:
> 
>> I've sent an email to assign@gnu.org. What is the next step for
>> getting this patch merged into Emacs?
> 
> The copyright clerk will get in touch with you.  Once the paperwork is
> finalised, follow up to this bug report, and we can get your patches
> installed.

bug#8531: 24.0.50;

[Samer Masterson]

[-- Attachment #1: Type: text/plain, Size: 4646 bytes --]

Hi,

My copyright papers have been processed. Can we apply this to master?

Best,
Samer

Changes from master to working tree
4 files changed, 26 insertions(+), 22 deletions(-)
 etc/ChangeLog          |  4 ++++
 etc/NEWS               |  6 ++++++
 lisp/ChangeLog         |  7 +++++++
 lisp/eshell/esh-arg.el | 31 +++++++++----------------------

	Modified   etc/ChangeLog
diff --git a/etc/ChangeLog b/etc/ChangeLog
index 309c01f..3e76256 100644
--- a/etc/ChangeLog
+++ b/etc/ChangeLog
@@ -1,3 +1,7 @@
+2014-12-08  Samer Masterson  <samer <at> dark-horse>
+
+	* NEWS: Mention change in backslash expand behavior for eshell.
+
 2014-12-08  Lars Magne Ingebrigtsen  <larsi <at> gnus.org>

 	* NEWS: Mention the new eww `S' command.
	Modified   etc/NEWS
diff --git a/etc/NEWS b/etc/NEWS
index 56036f8..e6d9aab 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -192,6 +192,12 @@ result of the calculation into the current buffer.
 *** New minor mode global-eldoc-mode
 *** eldoc-documentation-function now defaults to nil

+** eshell
+
+*** Backslash (\) expands to the character literal after it if that
+character is non-special (e.g. 'b\in' expands to 'bin', because 'i' is
+not a special character). This behavior conforms with bash.
+
 ** eww

 +++
	Modified   lisp/ChangeLog
diff --git a/lisp/ChangeLog b/lisp/ChangeLog
index 2669e07..0ec9b35 100644
--- a/lisp/ChangeLog
+++ b/lisp/ChangeLog
@@ -1,3 +1,10 @@
+2014-12-08  Samer Masterson  <samer <at> samertm.com>
+
+	* eshell/esh-arg.el (eshell-parse-backslash): Return the literal
+	character after the backslash if the character is non-special
+	(bug#8531).
+	(eshell-looking-at-backslash-return): Unused, remove.
+
 2014-12-08  Lars Magne Ingebrigtsen  <larsi <at> gnus.org>

 	* net/nsm.el (nsm-check-protocol): Test for RC4 on `high'.
	Modified   lisp/eshell/esh-arg.el
diff --git a/lisp/eshell/esh-arg.el b/lisp/eshell/esh-arg.el
index 704de57..33ff384 100644
--- a/lisp/eshell/esh-arg.el
+++ b/lisp/eshell/esh-arg.el
@@ -89,7 +89,8 @@ yield the values intended."
 	  (goto-char (match-end 0))
 	  (eshell-finish-arg)))))

-   ;; backslash before a special character means escape it
+   ;; backslash before a character escapes it if the character is
+   ;; special, and returns the character literal if it is non-special
    'eshell-parse-backslash

    ;; text beginning with ' is a literally quoted
@@ -282,13 +283,6 @@ Point is left at the end of the arguments."
   "A stub function that generates an error if a floating operator is
found."
   (error "Unhandled operator in input text"))

-(defsubst eshell-looking-at-backslash-return (pos)
-  "Test whether a backslash-return sequence occurs at POS."
-  (and (eq (char-after pos) ?\\)
-       (or (= (1+ pos) (point-max))
-	   (and (eq (char-after (1+ pos)) ?\n)
-		(= (+ pos 2) (point-max))))))
-
 (defun eshell-quote-backslash (string &optional index)
   "Intelligently backslash the character occurring in STRING at INDEX.
 If the character is itself a backslash, it needs no escaping."
@@ -305,9 +299,11 @@ If the character is itself a backslash, it needs no
escaping."
 	  (string ?\\ char)))))

 (defun eshell-parse-backslash ()
-  "Parse a single backslash (\) character, which might mean escape.
-It only means escape if the character immediately following is a
-special character that is not itself a backslash."
+  "Parse a single backslash (\) character to escape the character
after.
+If the character immediately following the backslash is a special
+character, it returns the escaped version of that character.
+Else, the character has no meaning and is returned as the literal
+character. This conforms with the behavior of bash."
   (when (eq (char-after) ?\\)
     (if (eshell-looking-at-backslash-return (point))
 	(throw 'eshell-incomplete ?\\)
@@ -321,18 +317,9 @@ special character that is not itself a backslash."
 	    (forward-char 2)
 	    (list 'eshell-escape-arg
 		  (char-to-string (char-before))))
-	;; allow \\<RET> to mean a literal "\" character followed by a
-	;; normal return, rather than a backslash followed by a line
-	;; continuation (i.e., "\\ + \n" rather than "\ + \\n").  This
-	;; is necessary because backslashes in Eshell are not special
-	;; unless they either precede something special, or precede a
-	;; backslash that precedes something special.  (Mainly this is
-	;; done to make using backslash on Windows systems more
-	;; natural-feeling).
-	(if (eshell-looking-at-backslash-return (1+ (point)))
-	    (forward-char))
 	(forward-char)
-	"\\"))))
+	(forward-char)
+	(char-before)))))

 (defun eshell-parse-literal-quote ()
   "Parse a literally quoted string.  Nothing has special meaning!"

[-- Attachment #2: Type: text/html, Size: 8584 bytes --]

bug#8531: 24.0.50;

[Eli Zaretskii]

> Date: Tue, 24 Feb 2015 02:51:29 -0800
> From: Samer Masterson <samer@samertm.com>
> 
> Changes from master to working tree
> 4 files changed, 26 insertions(+), 22 deletions(-)
> etc/ChangeLog | 4 ++++
> etc/NEWS | 6 ++++++
> lisp/ChangeLog | 7 +++++++
> lisp/eshell/esh-arg.el | 31 +++++++++----------------------
> 
> Modified etc/ChangeLog
> diff --git a/etc/ChangeLog b/etc/ChangeLog
> index 309c01f..3e76256 100644
> --- a/etc/ChangeLog
> +++ b/etc/ChangeLog
> @@ -1,3 +1,7 @@
> +2014-12-08 Samer Masterson <samer <at> dark-horse>
> +
> + * NEWS: Mention change in backslash expand behavior for eshell.
> +
> 2014-12-08 Lars Magne Ingebrigtsen <larsi <at> gnus.org>
> 
> * NEWS: Mention the new eww `S' command.
> Modified etc/NEWS
> diff --git a/etc/NEWS b/etc/NEWS
> index 56036f8..e6d9aab 100644
> --- a/etc/NEWS
> +++ b/etc/NEWS
> @@ -192,6 +192,12 @@ result of the calculation into the current buffer.
> *** New minor mode global-eldoc-mode
> *** eldoc-documentation-function now defaults to nil
> 
> +** eshell
> +
> +*** Backslash (\) expands to the character literal after it if that
> +character is non-special (e.g. 'b\in' expands to 'bin', because 'i' is
> +not a special character). This behavior conforms with bash.

Thanks.

First, please leave 2 blanks between sentences (we use the US English
conventions for that).

More importantly, this patch doesn't really fix the original bug, does
it?  Apologies if I missed something.

bug#8531: 24.0.50;

[Samer Masterson]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 7410 bytes --]

Hi,

This is the updated patch. Thanks in advance for the review.

Just to clarify, the bug is that

    $ ls Vid\éos

doesn't do the right thing in eshell (it preserves the backslash if
the character after it is non-special). This diverges from POSIX and
Bash.

New behavior:

Outside of a double quoted string, the backslash escapes the following
charater if that character is special. Otherwise it simply returns the
character afterwards (the backslash is ignored).

I expanded `eshell-parse-backslash's behavior to match Bash for quoted
strings as well. Inside a double quoted string, the backslash escapes
the following character if that character is special, otherwise the
backslash is preserved.

Best,
Samer

Patch below:
[PATCH] Fix bug#8531

* eshell/esh-arg.el (eshell-parse-argument-hook): Update comment.
(eshell-parse-backslash): Return escaped character after backslash
if it is special.  Otherwise, if the backslash is not in a quoted
string, ignore the backslash and return the character after; if
the backslash is in a quoted string, return the backslash and the
character after.  (bug#8531)
* automated/eshell.el (eshell-test/escape-nonspecial)
(eshell-test/escape-nonspecial-unicode)
(eshell-test/escape-nonspecial-quoted)
(eshell-test/escape-special-quoted): Add tests for new
`eshell-parse-backslash' behavior.  (bug#8531)
---
 lisp/ChangeLog           |  9 +++++++++
 lisp/eshell/esh-arg.el   | 49 +++++++++++++++++++++---------------------------
 test/ChangeLog           |  8 ++++++++
 test/automated/eshell.el | 31 ++++++++++++++++++++++++++++++
 4 files changed, 69 insertions(+), 28 deletions(-)

diff --git a/lisp/ChangeLog b/lisp/ChangeLog
index a9cf1b0..209382d 100644
--- a/lisp/ChangeLog
+++ b/lisp/ChangeLog
@@ -1,3 +1,12 @@
+2015-03-15  Samer Masterson  <samer@samertm.com>
+
+	* eshell/esh-arg.el (eshell-parse-argument-hook): Update comment.
+	(eshell-parse-backslash): Return escaped character after backslash
+	if it is special.  Otherwise, if the backslash is not in a quoted
+	string, ignore the backslash and return the character after; if
+	the backslash is in a quoted string, return the backslash and the
+	character after.  (bug#8531)
+
 2015-03-14  Michael R. Mauger  <michael@mauger.com>
 
 	* progmodes/sql.el: Version 3.5
diff --git a/lisp/eshell/esh-arg.el b/lisp/eshell/esh-arg.el
index 5c7d7ca..a5f697f 100644
--- a/lisp/eshell/esh-arg.el
+++ b/lisp/eshell/esh-arg.el
@@ -89,7 +89,7 @@ yield the values intended."
 	  (goto-char (match-end 0))
 	  (eshell-finish-arg)))))
 
-   ;; backslash before a special character means escape it
+   ;; parse backslash and the character after
    'eshell-parse-backslash
 
    ;; text beginning with ' is a literally quoted
@@ -305,34 +305,27 @@ If the character is itself a backslash, it needs no escaping."
 	  (string ?\\ char)))))
 
 (defun eshell-parse-backslash ()
-  "Parse a single backslash (\) character, which might mean escape.
-It only means escape if the character immediately following is a
-special character that is not itself a backslash."
+  "Parse a single backslash (\\) character and the character after.
+If the character after the backslash is special, always ignore
+the backslash and return the escaped character.
+
+Otherwise, if the backslash is not in quoted string, the
+backslash is ignored and the character after is returned.  If the
+backslash is in a quoted string, the backslash and the character
+after are both returned."
   (when (eq (char-after) ?\\)
-    (if (eshell-looking-at-backslash-return (point))
-	(throw 'eshell-incomplete ?\\)
-      (if (and (not (eq (char-after (1+ (point))) ?\\))
-	       (if eshell-current-quoted
-		   (memq (char-after (1+ (point)))
-			 eshell-special-chars-inside-quoting)
-		 (memq (char-after (1+ (point)))
-		       eshell-special-chars-outside-quoting)))
-	  (progn
-	    (forward-char 2)
-	    (list 'eshell-escape-arg
-		  (char-to-string (char-before))))
-	;; allow \\<RET> to mean a literal "\" character followed by a
-	;; normal return, rather than a backslash followed by a line
-	;; continuation (i.e., "\\ + \n" rather than "\ + \\n").  This
-	;; is necessary because backslashes in Eshell are not special
-	;; unless they either precede something special, or precede a
-	;; backslash that precedes something special.  (Mainly this is
-	;; done to make using backslash on Windows systems more
-	;; natural-feeling).
-	(if (eshell-looking-at-backslash-return (1+ (point)))
-	    (forward-char))
-	(forward-char)
-	"\\"))))
+    (when (eshell-looking-at-backslash-return (point))
+	(throw 'eshell-incomplete ?\\))
+    (forward-char 2) ; Move one char past the backslash.
+    ;; If the char is in a quote, backslash only has special meaning
+    ;; if it is escaping a special char.
+    (if eshell-current-quoted
+        (if (memq (char-before) eshell-special-chars-inside-quoting)
+            (list 'eshell-escape-arg (char-to-string (char-before)))
+          (concat "\\" (char-to-string (char-before))))
+      (if (memq (char-before) eshell-special-chars-outside-quoting)
+          (list 'eshell-escape-arg (char-to-string (char-before)))
+        (char-to-string (char-before))))))
 
 (defun eshell-parse-literal-quote ()
   "Parse a literally quoted string.  Nothing has special meaning!"
diff --git a/test/ChangeLog b/test/ChangeLog
index 6a474e1..ff2cd60 100644
--- a/test/ChangeLog
+++ b/test/ChangeLog
@@ -1,3 +1,11 @@
+2015-03-15  Samer Masterson  <samer@samertm.com>
+
+	* automated/eshell.el (eshell-test/escape-nonspecial)
+	(eshell-test/escape-nonspecial-unicode)
+	(eshell-test/escape-nonspecial-quoted)
+	(eshell-test/escape-special-quoted): Add tests for new
+	`eshell-parse-backslash' behavior.  (bug#8531)
+
 2015-03-10  Jackson Ray Hamilton  <jackson@jacksonrayhamilton.com>
 
 	* indent/js-indent-init-dynamic.js: Fix spelling error.
diff --git a/test/automated/eshell.el b/test/automated/eshell.el
index d51355f..81898db 100644
--- a/test/automated/eshell.el
+++ b/test/automated/eshell.el
@@ -166,6 +166,37 @@ e.g. \"{(+ 1 2)} 3\" => 3"
    (eshell-command-result-p "+ 1 2; + $_ 4"
                              "3\n6\n")))
 
+(ert-deftest eshell-test/escape-nonspecial ()
+  "Test that \"\\c\" and \"c\" are equivalent when \"c\" is not a
+special character."
+  (with-temp-eshell
+   (eshell-command-result-p "echo he\\llo"
+                            "hello\n")))
+
+(ert-deftest eshell-test/escape-nonspecial-unicode ()
+  "Test that \"\\c\" and \"c\" are equivalent when \"c\" is a
+unicode character (unicode characters are nonspecial by
+definition)."
+  (with-temp-eshell
+   (eshell-command-result-p "echo Vid\\éos"
+                            "Vidéos\n")))
+
+(ert-deftest eshell-test/escape-nonspecial-quoted ()
+  "Test that the backslash is preserved for escaped nonspecial
+chars"
+  (with-temp-eshell
+   (eshell-command-result-p "echo \"h\\i\""
+                            ;; Backslashes are doubled for regexp.
+                            "h\\\\i\n")))
+
+(ert-deftest eshell-test/escape-special-quoted ()
+  "Test that the backslash is not preserved for escaped special
+chars"
+  (with-temp-eshell
+   (eshell-command-result-p "echo \"h\\\\i\""
+                            ;; Backslashes are doubled for regexp.
+                            "h\\\\i\n")))
+
 (ert-deftest eshell-test/command-running-p ()
   "Modeline should show no command running"
   (with-temp-eshell

bug#8531: 24.0.50;

[Samer Masterson]

Hi,

This is the updated patch. Thanks in advance for the review.

Just to clarify, the bug is that

    $ ls Vid\éos

doesn't do the right thing in eshell (it preserves the backslash if
the character after it is non-special). This diverges from POSIX and
Bash.

New behavior:

Outside of a double quoted string, the backslash escapes the following
charater if that character is special. Otherwise it simply returns the
character afterwards (the backslash is ignored).

I expanded `eshell-parse-backslash's behavior to match Bash for quoted
strings as well. Inside a double quoted string, the backslash escapes
the following character if that character is special, otherwise the
backslash is preserved.

Best,
Samer

(The date on the commit below is correct, I just forgot to send it to
the list).

patch below:

900b7a998e210ec260b136ea093c3eba3ce63f34 HEAD bug-8531
Author: Samer Masterson <samer@samertm.com>
Date:   Sun Mar 15 04:01:58 2015 -0700

    Fix bug#8531
    
    * eshell/esh-arg.el (eshell-parse-argument-hook): Update comment.
    (eshell-parse-backslash): Return escaped character after backslash
    if it is special.  Otherwise, if the backslash is not in a quoted
    string, ignore the backslash and return the character after; if
    the backslash is in a quoted string, return the backslash and the
    character after.  (bug#8531)
    * automated/eshell.el (eshell-test/escape-nonspecial)
    (eshell-test/escape-nonspecial-unicode)
    (eshell-test/escape-nonspecial-quoted)
    (eshell-test/escape-special-quoted): Add tests for new
    `eshell-parse-backslash' behavior.  (bug#8531)

4 files changed, 69 insertions(+), 28 deletions(-)
 lisp/ChangeLog           |  9 +++++++++
 lisp/eshell/esh-arg.el   | 49 +++++++++++++++++++++---------------------------
 test/ChangeLog           |  8 ++++++++
 test/automated/eshell.el | 31 ++++++++++++++++++++++++++++++

	Modified   lisp/ChangeLog
diff --git a/lisp/ChangeLog b/lisp/ChangeLog
index a9cf1b0..209382d 100644
--- a/lisp/ChangeLog
+++ b/lisp/ChangeLog
@@ -1,3 +1,12 @@
+2015-03-15  Samer Masterson  <samer@samertm.com>
+
+	* eshell/esh-arg.el (eshell-parse-argument-hook): Update comment.
+	(eshell-parse-backslash): Return escaped character after backslash
+	if it is special.  Otherwise, if the backslash is not in a quoted
+	string, ignore the backslash and return the character after; if
+	the backslash is in a quoted string, return the backslash and the
+	character after.  (bug#8531)
+
 2015-03-14  Michael R. Mauger  <michael@mauger.com>
 
 	* progmodes/sql.el: Version 3.5
	Modified   lisp/eshell/esh-arg.el
diff --git a/lisp/eshell/esh-arg.el b/lisp/eshell/esh-arg.el
index 5c7d7ca..a5f697f 100644
--- a/lisp/eshell/esh-arg.el
+++ b/lisp/eshell/esh-arg.el
@@ -89,7 +89,7 @@ yield the values intended."
 	  (goto-char (match-end 0))
 	  (eshell-finish-arg)))))
 
-   ;; backslash before a special character means escape it
+   ;; parse backslash and the character after
    'eshell-parse-backslash
 
    ;; text beginning with ' is a literally quoted
@@ -305,34 +305,27 @@ If the character is itself a backslash, it needs no escaping."
 	  (string ?\\ char)))))
 
 (defun eshell-parse-backslash ()
-  "Parse a single backslash (\) character, which might mean escape.
-It only means escape if the character immediately following is a
-special character that is not itself a backslash."
+  "Parse a single backslash (\\) character and the character after.
+If the character after the backslash is special, always ignore
+the backslash and return the escaped character.
+
+Otherwise, if the backslash is not in quoted string, the
+backslash is ignored and the character after is returned.  If the
+backslash is in a quoted string, the backslash and the character
+after are both returned."
   (when (eq (char-after) ?\\)
-    (if (eshell-looking-at-backslash-return (point))
-	(throw 'eshell-incomplete ?\\)
-      (if (and (not (eq (char-after (1+ (point))) ?\\))
-	       (if eshell-current-quoted
-		   (memq (char-after (1+ (point)))
-			 eshell-special-chars-inside-quoting)
-		 (memq (char-after (1+ (point)))
-		       eshell-special-chars-outside-quoting)))
-	  (progn
-	    (forward-char 2)
-	    (list 'eshell-escape-arg
-		  (char-to-string (char-before))))
-	;; allow \\<RET> to mean a literal "\" character followed by a
-	;; normal return, rather than a backslash followed by a line
-	;; continuation (i.e., "\\ + \n" rather than "\ + \\n").  This
-	;; is necessary because backslashes in Eshell are not special
-	;; unless they either precede something special, or precede a
-	;; backslash that precedes something special.  (Mainly this is
-	;; done to make using backslash on Windows systems more
-	;; natural-feeling).
-	(if (eshell-looking-at-backslash-return (1+ (point)))
-	    (forward-char))
-	(forward-char)
-	"\\"))))
+    (when (eshell-looking-at-backslash-return (point))
+	(throw 'eshell-incomplete ?\\))
+    (forward-char 2) ; Move one char past the backslash.
+    ;; If the char is in a quote, backslash only has special meaning
+    ;; if it is escaping a special char.
+    (if eshell-current-quoted
+        (if (memq (char-before) eshell-special-chars-inside-quoting)
+            (list 'eshell-escape-arg (char-to-string (char-before)))
+          (concat "\\" (char-to-string (char-before))))
+      (if (memq (char-before) eshell-special-chars-outside-quoting)
+          (list 'eshell-escape-arg (char-to-string (char-before)))
+        (char-to-string (char-before))))))
 
 (defun eshell-parse-literal-quote ()
   "Parse a literally quoted string.  Nothing has special meaning!"
	Modified   test/ChangeLog
diff --git a/test/ChangeLog b/test/ChangeLog
index 6a474e1..ff2cd60 100644
--- a/test/ChangeLog
+++ b/test/ChangeLog
@@ -1,3 +1,11 @@
+2015-03-15  Samer Masterson  <samer@samertm.com>
+
+	* automated/eshell.el (eshell-test/escape-nonspecial)
+	(eshell-test/escape-nonspecial-unicode)
+	(eshell-test/escape-nonspecial-quoted)
+	(eshell-test/escape-special-quoted): Add tests for new
+	`eshell-parse-backslash' behavior.  (bug#8531)
+
 2015-03-10  Jackson Ray Hamilton  <jackson@jacksonrayhamilton.com>
 
 	* indent/js-indent-init-dynamic.js: Fix spelling error.
	Modified   test/automated/eshell.el
diff --git a/test/automated/eshell.el b/test/automated/eshell.el
index d51355f..81898db 100644
--- a/test/automated/eshell.el
+++ b/test/automated/eshell.el
@@ -166,6 +166,37 @@ e.g. \"{(+ 1 2)} 3\" => 3"
    (eshell-command-result-p "+ 1 2; + $_ 4"
                              "3\n6\n")))
 
+(ert-deftest eshell-test/escape-nonspecial ()
+  "Test that \"\\c\" and \"c\" are equivalent when \"c\" is not a
+special character."
+  (with-temp-eshell
+   (eshell-command-result-p "echo he\\llo"
+                            "hello\n")))
+
+(ert-deftest eshell-test/escape-nonspecial-unicode ()
+  "Test that \"\\c\" and \"c\" are equivalent when \"c\" is a
+unicode character (unicode characters are nonspecial by
+definition)."
+  (with-temp-eshell
+   (eshell-command-result-p "echo Vid\\éos"
+                            "Vidéos\n")))
+
+(ert-deftest eshell-test/escape-nonspecial-quoted ()
+  "Test that the backslash is preserved for escaped nonspecial
+chars"
+  (with-temp-eshell
+   (eshell-command-result-p "echo \"h\\i\""
+                            ;; Backslashes are doubled for regexp.
+                            "h\\\\i\n")))
+
+(ert-deftest eshell-test/escape-special-quoted ()
+  "Test that the backslash is not preserved for escaped special
+chars"
+  (with-temp-eshell
+   (eshell-command-result-p "echo \"h\\\\i\""
+                            ;; Backslashes are doubled for regexp.
+                            "h\\\\i\n")))
+
 (ert-deftest eshell-test/command-running-p ()
   "Modeline should show no command running"
   (with-temp-eshell

bug#8531: 24.0.50;

[Stefan Monnier]

> This is the updated patch. Thanks in advance for the review.

Thanks, installed,


        Stefan