bug#57350: 29.0.50; dired and filenames containing newlines

bug#57350: 29.0.50; dired and filenames containing newlines

[Robert Marshall]

I was sent a file whose name contained a newline:


 drwxrwxr-x   9 robert robert    12288 Aug 23 08:22 .
  -rw-rw-r--   1 robert robert  4887658 Aug 23 08:22 21st August Draft Newsletter
 FINAL.pdf

(try not to eyeroll too much at the file name and the combination of
draft and final!) If you delete the file in dired using
dired-flag-file-deletion and then dired-do-flagged-delete the file is
deleted but the file name still appears in the dired buffer.

D -rw-rw-r-- 1 robert robert 4887658 Aug 23 08:23 21st August Draft Newsletter
 FINAL.pdf

(ignore the different time on the file)

Refreshing the buffer does show that the deletion was successful

Also (after getting the file again with the newlined name) renaming
the file in dired then shows the file as existing in both its original
name and the new one. Again refreshing the buffer sorts out the issue.


I tried to rename the file within Wdired and remove the internal
newline, at one point I got an error when exiting Wdired (and the
rename failed) but am currently unable to replicate this bit of the
problem..

Robert

In GNU Emacs 29.0.50 (build 3, x86_64-pc-linux-gnu, GTK+ Version 3.24.30, cairo version 1.16.0)
 of 2022-07-28 built on poulenc
Repository revision: 5999dc1cd925ddf8af0b893432124af7904a6918
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12013000
System Description: Ubuntu 21.10

Configured using:
 'configure --with-xpm=ifavailable'

Configured features:
CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG LCMS2
LIBSELINUX LIBSYSTEMD LIBXML2 MODULES NOTIFY INOTIFY PDUMPER PNG RSVG
SECCOMP SOUND SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS X11 XDBE XIM
XINPUT2 XPM GTK3 ZLIB

Important settings:
  value of $LANG: en_GB.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Dired by date

Minor modes in effect:
  shell-dirtrack-mode: t
  global-hi-lock-mode: t
  hi-lock-mode: t
  recentf-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  buffer-read-only: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
/home/robert/elisp/dired-async hides /home/robert/.emacs.d/elpa/async-20191030.2138/dired-async
/home/robert/elisp/async hides /home/robert/.emacs.d/elpa/async-20191030.2138/async
/home/robert/.emacs.d/elpa/transient-20191115.1356/transient hides /home/robert/emacs-git/new/emacs/lisp/transient

Features:
(shadow emacsbug wdired bookmark shell pcomplete comint pp doc-view
filenotify jka-compr image-mode exif dired-aux url-queue url-cache
dabbrev thingatpt xref project ring shr-color color gnus-fun qp
mule-util flow-fill mm-archive cl-extra help-mode sort smiley
ansi-color gnus-cite mail-extr textsec uni-scripts idna-mapping
ucs-normalize uni-confusable textsec-check gnus-async gnus-bcklg
gnus-ml disp-table gnus-topic cursor-sensor nndraft nnmh nnfolder
gnus-agent gnus-srvr gnus-score score-mode nnvirtual gnus-msg
gnus-cache bbdb-gnus network-stream nntp bbdb-vm bbdb-mua bbdb-com crm
vm-pine gnus-art mm-uu mml2015 mm-view mml-smime smime gnutls dig
gnus-sum shr pixel-fill kinsoku url-file url-dired svg dom gnus-group
gnus-undo gnus-start gnus-dbus dbus gnus-cloud nnimap nnmail
mail-source utf7 netrc nnoo gnus-spec gnus-int gnus-range message
yank-media dired-x dired dired-loaddefs rfc822 mml mml-sec epa derived
epg rfc6068 epg-config mm-decode mm-bodies mm-encode mailabbrev
gmm-utils mailheader gnus-win gnus nnheader gnus-util
text-property-search range bbdb bbdb-site timezone sendmail mail-utils
vm-rfaddons vm-reply vm-imap vm-save vm-virtual vm-summary-faces
vm-delete vm-pop vm-undo vm-sort vm-thread vm-mime vm-toolbar vm-menu
tapestry vm-window vm-folder vm-crypto vm-summary vm-mouse vm-page
vm-motion vm-minibuf vm-message vm-misc vm-macro vm-autoloads vm-vars
vm-version vm misearch multi-isearch twittering-mode advice
identica-mode url-http url-auth mail-parse rfc2231 rfc2047 rfc2045
mm-util ietf-drums mail-prsvr url-gw nsm puny longlines parse-time
iso8601 time-date xml cl cal-china lunar solar cal-dst cal-bahai
cal-islam cal-hebrew holidays holiday-loaddefs diary-lib
diary-loaddefs cal-menu calendar cal-loaddefs server tbemail
org-install hi-lock desktop frameset recentf tree-widget wid-edit
bbdb-loaddefs finder-inf info package browse-url url url-proxy
url-privacy url-expand url-methods url-history url-cookie
generate-lisp-file url-domsuf url-util mailcap url-handlers url-parse
auth-source cl-seq eieio eieio-core cl-macs password-cache json subr-x
map byte-opt gv bytecomp byte-compile cconv url-vars cl-loaddefs
cl-lib rmc iso-transl tooltip eldoc paren electric uniquify ediff-hook
vc-hooks lisp-float-type elisp-mode mwheel term/x-win x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode lisp-mode prog-mode
register page tab-bar menu-bar rfn-eshadow isearch easymenu timer
select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew
greek romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button
loaddefs faces cus-face macroexp files window text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads dbusbind inotify lcms2
dynamic-setting system-font-setting font-render-setting cairo
move-toolbar gtk x-toolkit xinput2 x multi-tty make-network-process
emacs)

Memory information:
((conses 16 1074877 91613)
 (symbols 48 59368 182)
 (strings 32 472465 25550)
 (string-bytes 1 11985789)
 (vectors 16 137125)
 (vector-slots 8 2267698 163527)
 (floats 8 911 505)
 (intervals 56 121608 1899)
 (buffers 992 38))

bug#57350: 29.0.50; dired and filenames containing newlines

[Lars Ingebrigtsen]

Robert Marshall <robert@capuchin.co.uk> writes:

> I was sent a file whose name contained a newline:
>
>  drwxrwxr-x   9 robert robert    12288 Aug 23 08:22 .
>   -rw-rw-r--   1 robert robert  4887658 Aug 23 08:22 21st August Draft Newsletter
>  FINAL.pdf

Unfortunately, dired just isn't designed to work with these kinds of
files.

But you can make it work by changing `dired-listing-switches':

---
If you have files with names with embedded newline characters, adding
`b' to the switches will allow Dired to handle those files better.
---

But that has other side effects some people don't like, which is why we
haven't changed the defaults.

bug#57350: 29.0.50; dired and filenames containing newlines

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > But you can make it work by changing `dired-listing-switches':

  > ---
  > If you have files with names with embedded newline characters, adding
  > `b' to the switches will allow Dired to handle those files better.
  > ---

  > But that has other side effects some people don't like, which is why we
  > haven't changed the defaults.

What do people dislike about -b?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#57350: 29.0.50; dired and filenames containing newlines

[Michael Albinus]

Richard Stallman <rms@gnu.org> writes:

Hi Richard,

>   > But you can make it work by changing `dired-listing-switches':
>
>   > ---
>   > If you have files with names with embedded newline characters, adding
>   > `b' to the switches will allow Dired to handle those files better.
>   > ---
>
>   > But that has other side effects some people don't like, which is why we
>   > haven't changed the defaults.
>
> What do people dislike about -b?

Tramp, for example, has banned newlines in file names:

--8<---------------cut here---------------start------------->8---
(defconst tramp-localname-regexp "[^\n\r]*\\'"
  "Regexp matching localnames.")
--8<---------------cut here---------------end--------------->8---

Best regards, Michael.

bug#57350: 29.0.50; dired and filenames containing newlines

[Lars Ingebrigtsen]

Richard Stallman <rms@gnu.org> writes:

> What do people dislike about -b?

It does more than just newline escapes:

larsi@joga:~/src/emacs/trunk$ touch /tmp/"foo bar"
larsi@joga:~/src/emacs/trunk$ ls -lb /tmp/"foo bar"
-rw-rw-r-- 1 larsi larsi 0 Aug 24 12:24 /tmp/foo\ bar
larsi@joga:~/src/emacs/trunk$ ls -l /tmp/"foo bar"
-rw-rw-r-- 1 larsi larsi 0 Aug 24 12:24 '/tmp/foo bar'

bug#57350: 29.0.50; dired and filenames containing newlines

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I had forgotten about the --dired option.  Now that someone has
reminded us about it, what stands between us and using that option
as the normal way?

Is it a matter of determining whether ls on the current machine
supports --dired?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#57350: 29.0.50; dired and filenames containing newlines

[Eli Zaretskii]

> Cc: robert@capuchin.co.uk, 57350@debbugs.gnu.org
> From: Richard Stallman <rms@gnu.org>
> Date: Thu, 25 Aug 2022 23:37:00 -0400
> 
> I had forgotten about the --dired option.  Now that someone has
> reminded us about it, what stands between us and using that option
> as the normal way?

A lot of coding.  The --dired option tells Emacs where the file name
begins and ends, but it does nothing to remove the literal newline
character in the Dired buffer that breaks the file name entry into two
(or more) lines.  So all the Dired commands that move by lines and
assume a file's entry takes only one line on display become broken and
have to be rewritten almost from scratch.

Patches are welcome to fix all of that.  There's no argument that
using -b is a band-aid, not a solution.

bug#57350: 29.0.50; dired and filenames containing newlines

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > A lot of coding.  The --dired option tells Emacs where the file name
  > begins and ends, but it does nothing to remove the literal newline
  > character in the Dired buffer that breaks the file name entry into two
  > (or more) lines.  So all the Dired commands that move by lines and
  > assume a file's entry takes only one line on display become broken and
  > have to be rewritten almost from scratch.

It seems to me that the newlines in the file names should not be
represented in the Direc buffer as newlines.  Perhaps as \n?  And
backslashes as \\?

dired-readin could search the filenames for newlines and backslashes
and change the representation of them.
-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#57350: 29.0.50; dired and filenames containing newlines

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: larsi@gnus.org, robert@capuchin.co.uk, 57350@debbugs.gnu.org
> Date: Tue, 30 Aug 2022 22:38:34 -0400
> 
>   > A lot of coding.  The --dired option tells Emacs where the file name
>   > begins and ends, but it does nothing to remove the literal newline
>   > character in the Dired buffer that breaks the file name entry into two
>   > (or more) lines.  So all the Dired commands that move by lines and
>   > assume a file's entry takes only one line on display become broken and
>   > have to be rewritten almost from scratch.
> 
> It seems to me that the newlines in the file names should not be
> represented in the Direc buffer as newlines.  Perhaps as \n?  And
> backslashes as \\?
> 
> dired-readin could search the filenames for newlines and backslashes
> and change the representation of them.

Could be done, but that would also need a lot of coding, just
elsewhere: all the features and commands which work on Dired buffers
would need to perform the inverse conversion before actually using the
file names.  I'm not sure which of the jobs is smaller, but it doesn't
look like any of them is small.

Anyway, patches are welcome.

bug#57350: 29.0.50; dired and filenames containing newlines

[Andreas Schwab]

On Aug 31 2022, Eli Zaretskii wrote:

> Could be done, but that would also need a lot of coding, just
> elsewhere: all the features and commands which work on Dired buffers
> would need to perform the inverse conversion before actually using the
> file names.  I'm not sure which of the jobs is smaller, but it doesn't
> look like any of them is small.

Should be centralized to dired-get-filename, though.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

bug#57350: 29.0.50; dired and filenames containing newlines

[Gregory Heytings]

>
> A lot of coding.  The --dired option tells Emacs where the file name 
> begins and ends, but it does nothing to remove the literal newline 
> character in the Dired buffer that breaks the file name entry into two 
> (or more) lines.
>

Is this not an enhancement that we could ask in ls, I mean, escaping only 
newlines with \n and backslashed with \\ (instead of all non-graphic 
characters as --escape does)?  I guess the --dired option is only used by 
Emacs, so this should be a safe change.

bug#57350: 29.0.50; dired and filenames containing newlines

[Eli Zaretskii]

> Cc: robert@capuchin.co.uk, 57350@debbugs.gnu.org
> From: Richard Stallman <rms@gnu.org>
> Date: Tue, 23 Aug 2022 23:53:58 -0400
> 
>   > If you have files with names with embedded newline characters, adding
>   > `b' to the switches will allow Dired to handle those files better.
>   > ---
> 
>   > But that has other side effects some people don't like, which is why we
>   > haven't changed the defaults.
> 
> What do people dislike about -b?

It causes much more than the newline to show up in somewhat awkward
formatting.  Most of those other characters don't need to be
escaped/quoted for Emacs to DTRT, but 'ls' doesn't have any
finer-tuned feature.

bug#57350: 29.0.50; dired and filenames containing newlines

[Andreas Schwab]

On Aug 24 2022, Eli Zaretskii wrote:

>> Cc: robert@capuchin.co.uk, 57350@debbugs.gnu.org
>> From: Richard Stallman <rms@gnu.org>
>> Date: Tue, 23 Aug 2022 23:53:58 -0400
>> 
>>   > If you have files with names with embedded newline characters, adding
>>   > `b' to the switches will allow Dired to handle those files better.
>>   > ---
>> 
>>   > But that has other side effects some people don't like, which is why we
>>   > haven't changed the defaults.
>> 
>> What do people dislike about -b?
>
> It causes much more than the newline to show up in somewhat awkward
> formatting.  Most of those other characters don't need to be
> escaped/quoted for Emacs to DTRT, but 'ls' doesn't have any
> finer-tuned feature.

With support for --dired, there is already everything needed to handle
newlines in file names.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."