bug#36110: find-dired not sorted on any field nor provides a way

bug#36110: find-dired not sorted on any field nor provides a way

[積丹尼 Dan Jacobson]

find-dired (info "(emacs) Dired and Find") is great, except it is not
sorted on any field, except coincidentally.
find-ls-option is a variable defined in ‘find-dired.el’, alas it only
governs each line and not how those lines are sorted.
Yes that is how find(1) works, but there should be a filter (default "|
sort ...") that sorts on the last field (name). Those people who prefer
jumbled order could unset it.

P.S., the modeline says (Dired by name:exit) which looks weird.
emacs-version "26.1"

bug#36110: find-dired not sorted on any field nor provides a way

[Juri Linkov]

> find-dired (info "(emacs) Dired and Find") is great, except it is not
> sorted on any field, except coincidentally.
> find-ls-option is a variable defined in ‘find-dired.el’, alas it only
> governs each line and not how those lines are sorted.
> Yes that is how find(1) works, but there should be a filter (default "|
> sort ...") that sorts on the last field (name). Those people who prefer
> jumbled order could unset it.

Long ago I customized find-ls-option to this value

  (setq find-ls-option '("-exec ls -ld {} \\;" . "-ld"))

and forgot about such problems because this value produces
the same output as Dired.

bug#36110: find-dired not sorted on any field nor provides a way

[Drew Adams]

> Long ago I customized find-ls-option to this value
> 
>   (setq find-ls-option '("-exec ls -ld {} \\;" . "-ld"))
> 
> and forgot about such problems because this value produces
> the same output as Dired.

Same here, although with this (minor) difference:

("-exec ls -ld {} \";\"" . "-ld")

Dunno where I got that or when I customized the value to it.

bug#36110: find-dired not sorted on any field nor provides a way

[積丹尼 Dan Jacobson]

>>>>> "DA" == Drew Adams <drew.adams@oracle.com> writes:
DA> Dunno where I got that or when I customized the value to it.

That's because nobody documented it in (info "(emacs) Dired and Find") !

So somebody please document it, else who likes going around with their
find-dired in "ls -U" order? What sense would that make?

bug#36110: find-dired not sorted on any field nor provides a way

[Juri Linkov]

>>> Long ago I customized find-ls-option to this value
>>>
>>>   (setq find-ls-option '("-exec ls -ld {} \\;" . "-ld"))
>>>
>>> and forgot about such problems because this value produces
>>> the same output as Dired.
>>
>> Same here, although with this (minor) difference:
>>
>> ("-exec ls -ld {} \";\"" . "-ld")
>>
>> Dunno where I got that or when I customized the value to it.
>
> That's because nobody documented it in (info "(emacs) Dired and Find") !
>
> So somebody please document it, else who likes going around with their
> find-dired in "ls -U" order? What sense would that make?

The following patch documents it:

diff --git a/doc/emacs/dired.texi b/doc/emacs/dired.texi
index 9f454ea2ad..2befcf1bbd 100644
--- a/doc/emacs/dired.texi
+++ b/doc/emacs/dired.texi
@@ -1340,7 +1340,9 @@ Dired and Find
   The format of listing produced by these commands is controlled by
 the variable @code{find-ls-option}.  This is a pair of options; the
 first specifying how to call @command{find} to produce the file listing,
-and the second telling Dired to parse the output.
+and the second telling Dired to parse the output.  To sort file names
+lexicographically, you can customize @code{find-ls-option} to the value
+@code{'("-exec ls -ld @{@} +" . "-ld")}

 @findex locate
 @findex locate-with-filter

bug#36110: find-dired not sorted on any field nor provides a way

[Glenn Morris]

Juri Linkov wrote:

>>>> Long ago I customized find-ls-option to this value
>>>>
>>>>   (setq find-ls-option '("-exec ls -ld {} \\;" . "-ld"))

But that has nothing to do with sorting?
find -exec ls -ld {} ";"
will give unsorted output, as always.

> -and the second telling Dired to parse the output.
> +and the second telling Dired to parse the output.  To sort file names
> +lexicographically, you can customize @code{find-ls-option} to the value
> +@code{'("-exec ls -ld @{@} +" . "-ld")}

Oh, I see you've changed it to the "+" form.
It looks like this does give sorted output, but I find it surprising.
Is it assured?

Anyway, this issue has been very recently fixed in bug#29513
(of which this is a duplicate) via find-dired-refine-function.
So I suggest not documenting the above.

bug#36110: find-dired not sorted on any field nor provides a way

[Glenn Morris]

Glenn Morris wrote:

>> +@code{'("-exec ls -ld @{@} +" . "-ld")}
>
> Oh, I see you've changed it to the "+" form.
> It looks like this does give sorted output, but I find it surprising.
> Is it assured?

I verified on a directory with a large number of files that the output
is not sorted. (I didn't see how it could be, given how find works.)
But it does appear as if individual "chunks" are sorted, which can give
the appearance of the whole output being sorted if you don't have a lot
of files.

bug#36110: find-dired not sorted on any field nor provides a way

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Date: Fri, 14 Jun 2019 00:30:05 +0300
> Cc: 36110@debbugs.gnu.org
> 
> > So somebody please document it, else who likes going around with their
> > find-dired in "ls -U" order? What sense would that make?
> 
> The following patch documents it:
> 
> diff --git a/doc/emacs/dired.texi b/doc/emacs/dired.texi
> index 9f454ea2ad..2befcf1bbd 100644
> --- a/doc/emacs/dired.texi
> +++ b/doc/emacs/dired.texi
> @@ -1340,7 +1340,9 @@ Dired and Find
>    The format of listing produced by these commands is controlled by
>  the variable @code{find-ls-option}.  This is a pair of options; the
>  first specifying how to call @command{find} to produce the file listing,
> -and the second telling Dired to parse the output.
> +and the second telling Dired to parse the output.  To sort file names
> +lexicographically, you can customize @code{find-ls-option} to the value
> +@code{'("-exec ls -ld @{@} +" . "-ld")}

This should qualify the suggestion to using GNU Find, right?

bug#36110: find-dired not sorted on any field nor provides a way

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Date: Thu, 13 Jun 2019 19:40:11 -0400
> Cc: 36110@debbugs.gnu.org
> 
> > +@code{'("-exec ls -ld @{@} +" . "-ld")}
> 
> Oh, I see you've changed it to the "+" form.
> It looks like this does give sorted output, but I find it surprising.
> Is it assured?

It is assured because the GNU Find's documentation says this '+'
causes the command line to be created by appending the file names,
like xargs does.  However, problems should be expected when the number
of files becomes very large, subject to limitations of the target
platform.

> Anyway, this issue has been very recently fixed in bug#29513
> (of which this is a duplicate) via find-dired-refine-function.
> So I suggest not documenting the above.

We could document both ways.

Btw, the commit which fixed bug#29513 didn't call out the change in
NEWS, let alone in the manual, so we should definitely improve the
docs regarding this issue anyway.

Thanks.

bug#36110: find-dired not sorted on any field nor provides a way

[Andreas Schwab]

On Jun 13 2019, Glenn Morris <rgm@gnu.org> wrote:

> Glenn Morris wrote:
>
>>> +@code{'("-exec ls -ld @{@} +" . "-ld")}
>>
>> Oh, I see you've changed it to the "+" form.
>> It looks like this does give sorted output, but I find it surprising.
>> Is it assured?
>
> I verified on a directory with a large number of files that the output
> is not sorted. (I didn't see how it could be, given how find works.)
> But it does appear as if individual "chunks" are sorted, which can give
> the appearance of the whole output being sorted if you don't have a lot
> of files.

They are sorted by ls, for each invocation of it.  But it is unspecified
how find will split the invocations if there are more files than fit on
the command line.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#36110: find-dired not sorted on any field nor provides a way

[Juri Linkov]

[-- Attachment #1: Type: text/plain, Size: 2295 bytes --]

>>> +@code{'("-exec ls -ld @{@} +" . "-ld")}
>>
>> Oh, I see you've changed it to the "+" form.

I've changed it to the same value as in `find-exec-terminator'.
So instead of documenting a command with a terminator that depends
on the value of `find-exec-terminator', better would be to provide
different customizable options like in the patch below.

>> It looks like this does give sorted output, but I find it surprising.
>> Is it assured?
>
> I verified on a directory with a large number of files that the output
> is not sorted. (I didn't see how it could be, given how find works.)
> But it does appear as if individual "chunks" are sorted, which can give
> the appearance of the whole output being sorted if you don't have a lot
> of files.

I tried to run different commands on all files in the Emacs source tree:

1. find . -ls

It produces completely unsorted output.

2. find . -exec ls -ld {} +

It splits the output into sizeable chunks and sorts files
inside every chunk, so the boundary between chunks
is clearly visible, e.g.:

-rw-rw-r--  1 juri juri     6191 May  1 23:49 ./test/src/timefns-tests.el
-rw-r--r--  1 juri juri    13623 Jan  2 22:43 ./test/src/undo-tests.el
-rw-r--r--  1 juri juri     2915 Jan  2 22:43 ./test/src/xml-tests.el
drwxr-xr-x   2 juri juri      4096 Jun 11 00:09 ./autom4te.cache
-rw-rw-r--   1 juri juri    945417 Jun 11 00:09 ./autom4te.cache/output.0
-rw-r--r--   1 juri juri      3431 Jun 11 00:09 ./autom4te.cache/requests

3. find . -print0 | sort -z | xargs -0 -e ls -ld

It splits files into chinks, but maintains the sorting order
among all files.   Its only drawback is misaligned chunks, e.g.:

-rw-rw-r--   1 juri juri     62707 May 21 00:04 ./info/forms.info
-rw-rw-r--   1 juri juri   1476691 Jun 11 00:16 ./info/gnus.info
-rw-rw-r--   1 juri juri     74148 May 21 00:04 ./info/htmlfontify.info
-rw-rw-r--  1 juri juri   234730 May 21 00:04 ./info/idlwave.info
-rw-rw-r--  1 juri juri    51892 May 21 00:04 ./info/ido.info
-rw-rw-r--  1 juri juri    84948 May 21 00:04 ./info/info.info

Since an output with more than 5000 files (an approx amount in each chuck
in this experiment) is unmanageable by human users, any of the last 2 options
is sufficiently good.  So here is the patch that allows the user
to choose among these options:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: find-ls-option-default.patch --]
[-- Type: text/x-diff, Size: 2290 bytes --]

diff --git a/lisp/find-dired.el b/lisp/find-dired.el
index 2c76179da0..c563ae533a 100644
--- a/lisp/find-dired.el
+++ b/lisp/find-dired.el
@@ -51,19 +51,23 @@ find-exec-terminator
   :group 'find-dired
   :type 'string)
 
+(defvar find-ls-option-default-ls
+  (cons "-ls" (if (eq system-type 'berkeley-unix) "-gilsb" "-dilsb")))
+
+(defvar find-ls-option-default-exec
+  (cons (format "-exec ls -ld {} %s" find-exec-terminator) "-ld"))
+
+(defvar find-ls-option-default-xargs
+  (cons "-print0 | sort -z | xargs -0 -e ls -ld" "-ld"))
+
 ;; find's -ls corresponds to these switches.
 ;; Note -b, at least GNU find quotes spaces etc. in filenames
 (defcustom find-ls-option
   (if (eq 0
 	  (ignore-errors
 	    (process-file find-program nil nil nil null-device "-ls")))
-      (cons "-ls"
-	    (if (eq system-type 'berkeley-unix)
-		"-gilsb"
-	      "-dilsb"))
-    (cons
-     (format "-exec ls -ld {} %s" find-exec-terminator)
-     "-ld"))
+      find-ls-option-default-ls
+    find-ls-option-default-exec)
   "A pair of options to produce and parse an `ls -l'-type list from `find'.
 This is a cons of two strings (FIND-OPTION . LS-SWITCHES).
 FIND-OPTION is the option (or options) passed to `find' to produce
@@ -78,9 +82,20 @@ find-ls-option
    (\"-ls\" . \"-dilsb\")
 since GNU find's output has the same format as using GNU ls with
 the options \"-dilsb\"."
-  :version "24.1"	       ; add tests for -ls and -exec + support
-  :type '(cons (string :tag "Find Option")
-	       (string :tag "Ls Switches"))
+  :version "27.1"            ; add choice of predefined set of options
+  :type `(choice
+          (cons :tag "find -ls"
+                (string ,(car find-ls-option-default-ls))
+                (string ,(cdr find-ls-option-default-ls)))
+          (cons :tag "find -exec ls -ld"
+                (string ,(car find-ls-option-default-exec))
+                (string ,(cdr find-ls-option-default-exec)))
+          (cons :tag "find -print | sort | xargs"
+                (string ,(car find-ls-option-default-xargs))
+                (string ,(cdr find-ls-option-default-xargs)))
+          (cons :tag "Other values"
+                (string :tag "Find Option")
+                (string :tag "Ls Switches")))
   :group 'find-dired)
 
 (defcustom find-ls-subdir-switches

bug#36110: find-dired not sorted on any field nor provides a way

[Juri Linkov]

[-- Attachment #1: Type: text/plain, Size: 88 bytes --]

In addition to the previous patch, it also makes sense to add similar
options to grep:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: grep-find-use-xargs.patch --]
[-- Type: text/x-diff, Size: 2202 bytes --]

diff --git a/lisp/progmodes/grep.el b/lisp/progmodes/grep.el
index 79178c4346..67222f7862 100644
--- a/lisp/progmodes/grep.el
+++ b/lisp/progmodes/grep.el
@@ -511,14 +511,24 @@ xargs-program
 This variable's value takes effect when `grep-compute-defaults' is called.")
 
 ;;;###autoload
-(defvar grep-find-use-xargs nil
+(defcustom grep-find-use-xargs nil
   "How to invoke find and grep.
 If `exec', use `find -exec {} ;'.
 If `exec-plus' use `find -exec {} +'.
 If `gnu', use `find -print0' and `xargs -0'.
+If `gnu-sort', use `find -print0', `sort -z' and `xargs -0'.
 Any other value means to use `find -print' and `xargs'.
 
-This variable's value takes effect when `grep-compute-defaults' is called.")
+This variable's value takes effect when `grep-compute-defaults' is called."
+  :type '(choice (const :tag "find -exec {} ;" exec)
+                 (const :tag "find -exec {} +" exec-plus)
+                 (const :tag "find -print0 | xargs -0" gnu)
+                 (const :tag "find -print0 | sort -z | xargs -0'" gnu-sort)
+                 string
+		 (const :tag "Not Set" nil))
+  :set 'grep-apply-setting
+  :version "27.1"
+  :group 'grep)
 
 ;; History of grep commands.
 ;;;###autoload
@@ -728,6 +738,9 @@ grep-compute-defaults
 		       ;; forward slashes as directory separators.
 		       (format "%s . -type f -print0 | \"%s\" -0 %s"
 			       find-program xargs-program grep-command))
+		      ((eq grep-find-use-xargs 'gnu-sort)
+		       (format "%s . -type f -print0 | sort -z | \"%s\" -0 %s"
+			       find-program xargs-program grep-command))
 		      ((memq grep-find-use-xargs '(exec exec-plus))
 		       (let ((cmd0 (format "%s . -type f -exec %s"
 					   find-program grep-command))
@@ -752,6 +765,9 @@ grep-compute-defaults
 		  (cond ((eq grep-find-use-xargs 'gnu)
 			 (format "%s <D> <X> -type f <F> -print0 | \"%s\" -0 %s"
 				 find-program xargs-program gcmd))
+			((eq grep-find-use-xargs 'gnu-sort)
+			 (format "%s <D> <X> -type f <F> -print0 | sort -z | \"%s\" -0 %s"
+				 find-program xargs-program gcmd))
 			((eq grep-find-use-xargs 'exec)
 			 (format "%s <D> <X> -type f <F> -exec %s %s %s%s"
 				 find-program gcmd quot-braces null quot-scolon))

bug#36110: find-dired not sorted on any field nor provides a way

[Juri Linkov]

>> Anyway, this issue has been very recently fixed in bug#29513
>> (of which this is a duplicate) via find-dired-refine-function.
>> So I suggest not documenting the above.
>
> We could document both ways.
>
> Btw, the commit which fixed bug#29513 didn't call out the change in
> NEWS, let alone in the manual, so we should definitely improve the
> docs regarding this issue anyway.

Also I see no way to disable its default value with something like:

diff --git a/lisp/find-dired.el b/lisp/find-dired.el
index 2c76179da0..c563ae533a 100644
--- a/lisp/find-dired.el
+++ b/lisp/find-dired.el
@@ -123,7 +138,10 @@ find-dired-refine-function
 output of `find' (one file per line) when this function is called."
   :version "27.1"
   :group 'find-dired
-  :type 'function)
+  :type '(choice (const :tag "Sort file names lexicographically"
+                        find-dired-sort-by-filename)
+                 (function :tag "Refining function")
+                 (const :tag "No refining" nil)))
 
 (defvar find-args nil
   "Last arguments given to `find' by \\[find-dired].")

bug#36110: find-dired not sorted on any field nor provides a way

[Juri Linkov]

>> diff --git a/doc/emacs/dired.texi b/doc/emacs/dired.texi
>> index 9f454ea2ad..2befcf1bbd 100644
>> --- a/doc/emacs/dired.texi
>> +++ b/doc/emacs/dired.texi
>> @@ -1340,7 +1340,9 @@ Dired and Find
>>    The format of listing produced by these commands is controlled by
>>  the variable @code{find-ls-option}.  This is a pair of options; the
>>  first specifying how to call @command{find} to produce the file listing,
>> -and the second telling Dired to parse the output.
>> +and the second telling Dired to parse the output.  To sort file names
>> +lexicographically, you can customize @code{find-ls-option} to the value
>> +@code{'("-exec ls -ld @{@} +" . "-ld")}
>
> This should qualify the suggestion to using GNU Find, right?

grep-find-use-xargs defines such a command as `exec',
whereas GNU Find is defined as a combination of `find -print0'
and `xargs -0'.  From docstring of grep-find-use-xargs:

 If `exec', use `find -exec {} ;'.
 If `exec-plus' use `find -exec {} +'.
 If `gnu', use `find -print0' and `xargs -0'.

bug#36110: find-dired not sorted on any field nor provides a way

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: jidanni@jidanni.org,  36110@debbugs.gnu.org
> Date: Sun, 16 Jun 2019 01:36:45 +0300
> 
> >> diff --git a/doc/emacs/dired.texi b/doc/emacs/dired.texi
> >> index 9f454ea2ad..2befcf1bbd 100644
> >> --- a/doc/emacs/dired.texi
> >> +++ b/doc/emacs/dired.texi
> >> @@ -1340,7 +1340,9 @@ Dired and Find
> >>    The format of listing produced by these commands is controlled by
> >>  the variable @code{find-ls-option}.  This is a pair of options; the
> >>  first specifying how to call @command{find} to produce the file listing,
> >> -and the second telling Dired to parse the output.
> >> +and the second telling Dired to parse the output.  To sort file names
> >> +lexicographically, you can customize @code{find-ls-option} to the value
> >> +@code{'("-exec ls -ld @{@} +" . "-ld")}
> >
> > This should qualify the suggestion to using GNU Find, right?
> 
> grep-find-use-xargs defines such a command as `exec',
> whereas GNU Find is defined as a combination of `find -print0'
> and `xargs -0'.  From docstring of grep-find-use-xargs:
> 
>  If `exec', use `find -exec {} ;'.
>  If `exec-plus' use `find -exec {} +'.
>  If `gnu', use `find -print0' and `xargs -0'.

I meant the "+" part, I think it's GNU-specific?

bug#36110: find-dired not sorted on any field nor provides a way

[Andreas Schwab]

On Jun 16 2019, Eli Zaretskii <eliz@gnu.org> wrote:

>> grep-find-use-xargs defines such a command as `exec',
>> whereas GNU Find is defined as a combination of `find -print0'
>> and `xargs -0'.  From docstring of grep-find-use-xargs:
>> 
>>  If `exec', use `find -exec {} ;'.
>>  If `exec-plus' use `find -exec {} +'.
>>  If `gnu', use `find -print0' and `xargs -0'.
>
> I meant the "+" part, I think it's GNU-specific?

exec-plus is POSIX (originating from SVR4).

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#36110: find-dired not sorted on any field nor provides a way

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: Juri Linkov <juri@linkov.net>,  36110@debbugs.gnu.org,  jidanni@jidanni.org
> Date: Sun, 16 Jun 2019 10:37:50 +0200
> 
> On Jun 16 2019, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > I meant the "+" part, I think it's GNU-specific?
> 
> exec-plus is POSIX (originating from SVR4).

Thanks.  Then there's no need to qualify anything.