bug#36729: 27.0.50; Unclear total in directory listing

bug#36729: 27.0.50; Unclear total in directory listing

[Mattias Engdegård]

The first line of a directory listing reads

  total used in directory 71752 available 65.2 GiB

where it is a mystery what the first number represents. Often it is 512-byte blocks, which is confusing and not very useful, and cannot easily be compared with the 'available' number (which is crystal clear).

It has always been thus, but previously the 'available' number was itself somewhat obscure so the first number didn't stand out.

bug#36729: 27.0.50; Unclear total in directory listing

[Eli Zaretskii]

> From: Mattias Engdegård <mattiase@acm.org>
> Date: Fri, 19 Jul 2019 12:15:58 +0200
> 
> The first line of a directory listing reads
> 
>   total used in directory 71752 available 65.2 GiB
> 
> where it is a mystery what the first number represents. Often it is 512-byte blocks, which is confusing and not very useful, and cannot easily be compared with the 'available' number (which is crystal clear).

Feel free to suggest a clarification in the doc string of 'dired'.

bug#36729: 27.0.50; Unclear total in directory listing

[Stephen Berman]

On Fri, 19 Jul 2019 15:16:02 +0300 Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Mattias Engdegård <mattiase@acm.org>
>> Date: Fri, 19 Jul 2019 12:15:58 +0200
>> 
>> The first line of a directory listing reads
>> 
>>   total used in directory 71752 available 65.2 GiB
>> 
>> where it is a mystery what the first number represents. Often it is 512-byte
>> blocks, which is confusing and not very useful, and cannot easily be
>> compared with the 'available' number (which is crystal clear).
>
> Feel free to suggest a clarification in the doc string of 'dired'.

An alternative is to do for "total used" what was done for "available".
On systems that use `ls' (at least the GNU version), users can do that
by setting the environment variable BLOCK_SIZE, but since the change to
the display of "available" was unconditional (if I'm not mistaken), we
could do the same for the display of "total used", e.g. as in the below
patch.  (But since this won't work on systems that don't use GNU `ls',
either another solution just for them or a different general one would
be needed.)

Steve Berman

diff --git a/lisp/dired.el b/lisp/dired.el
index c455a5cde4..8a94080538 100644
--- a/lisp/dired.el
+++ b/lisp/dired.el
@@ -1271,7 +1271,9 @@ dired-insert-directory
 If HDR is non-nil, insert a header line with the directory name."
   (let ((opoint (point))
 	(process-environment (copy-sequence process-environment))
+	(block-size (getenv "BLOCK_SIZE"))
 	end)
+    (setenv "BLOCK_SIZE" "si")
     (if (and
 	 ;; Don't try to invoke `ls' if we are on DOS/Windows where
 	 ;; ls-lisp emulation is used, except if they want to use `ls'
@@ -1385,7 +1387,10 @@ dired-insert-directory
 	  (insert "  wildcard " (or (cdr-safe (insert-directory-wildcard-in-dir-p dir))
                                     (file-name-nondirectory dir))
                   "\n")))
-      (dired-insert-set-properties content-point (point)))))
+      (dired-insert-set-properties content-point (point)))
+    (if block-size
+	(setenv "BLOCK_SIZE" block-size)
+      (setenv-internal process-environment "BLOCK_SIZE" nil t))))
 
 (defun dired-insert-set-properties (beg end)
   "Add various text properties to the lines in the region."

bug#36729: 27.0.50; Unclear total in directory listing

[Drew Adams]

> >> The first line of a directory listing reads
> >>
> >>   total used in directory 71752 available 65.2 GiB
> >>
> >> where it is a mystery what the first number represents. Often it is 512-byte
> >> blocks, which is confusing and not very useful, and cannot easily be
> >> compared with the 'available' number (which is crystal clear).
> >
> > Feel free to suggest a clarification in the doc string of 'dired'.
> 
> An alternative is to do for "total used" what was done for "available".
> On systems that use `ls' (at least the GNU version), users can do that
> by setting the environment variable BLOCK_SIZE, but since the change to
> the display of "available" was unconditional (if I'm not mistaken), we
> could do the same for the display of "total used", e.g. as in the below
> patch.  (But since this won't work on systems that don't use GNU `ls',
> either another solution just for them or a different general one would
> be needed.)

FWIW, my `ls-lisp+.el' code gives this:

 files 38/38 space used 2563 available 103423152

and mouse-over (`help-echo') on the `files 38/38' says:

 Files shown / total files in directory

and mouse-over on the `space used ...' part says:

 Kbytes used in directory, Kbytes available on disk

and `RET' or `mouse-2' anywhere on that line pops up
`*Help* with info like this (` help-fns+.el' is also
needed, for function `describe-file'):

 g:/usr/foo/bar
 --------------

 File Type:                  Directory
 Permissions:                drwxrwxrwx
 Time of last access:        Mon Apr 22 04:48:30 2019 (Pacific Daylight Time)
 Time of last modification:  Mon Apr 22 04:48:30 2019 (Pacific Daylight Time)
 Time of last status change: Wed Jul 25 08:00:46 2018 (Pacific Daylight Time)
 Number of links:            1
 User ID (UID):              68752
 Group ID (GID):             312
 Inode:                      281474976872649
 Device number:              315267003

Maybe some such aids could be added to vanilla Dired.

bug#36729: 27.0.50; Unclear total in directory listing

[Mattias Engdegård]

19 juli 2019 kl. 14.16 skrev Eli Zaretskii <eliz@gnu.org>:
> 
> Feel free to suggest a clarification in the doc string of 'dired'.

Frankly, I wouldn't know what to write. That the number should be ignored? We'd better remove it in the first place.
Of course, someone is bound to complain if such a thing were attempted.

19 juli 2019 kl. 15.28 skrev Stephen Berman <stephen.berman@gmx.net>:
> 
> On systems that use `ls' (at least the GNU version), users can do that
> by setting the environment variable BLOCK_SIZE, but since the change to
> the display of "available" was unconditional (if I'm not mistaken), we
> could do the same for the display of "total used", e.g. as in the below
> patch.  (But since this won't work on systems that don't use GNU `ls',
> either another solution just for them or a different general one would
> be needed.)

Thank you -- specifically, it doesn't work for BSD ls.
We could set BLOCKSIZE to 1024 -- this works on both GNU and BSD ls -- and then post-process the output. Or insert the switch `-k', which is in Posix (in contrast to BLOCKSIZE or BLOCK_SIZE), but I don't know if that would break any customisation.

It's all very fragile. Migrating to a fully built-in directory lister seems to be the only sane way out, and it's considerably more work than I anticipated for this little detail.

bug#36729: 27.0.50; Unclear total in directory listing

[Eli Zaretskii]

> From: Mattias Engdegård <mattiase@acm.org>
> Date: Sun, 21 Jul 2019 10:19:13 +0200
> Cc: 36729@debbugs.gnu.org, Stephen Berman <stephen.berman@gmx.net>,
>         Drew Adams <drew.adams@oracle.com>
> 
> 19 juli 2019 kl. 14.16 skrev Eli Zaretskii <eliz@gnu.org>:
> > 
> > Feel free to suggest a clarification in the doc string of 'dired'.
> 
> Frankly, I wouldn't know what to write. That the number should be ignored? We'd better remove it in the first place.

(Your reaction seems to imply that I said something silly.  Did I?)

Your original report was about the unclear units of the value, so a
useful clarification would be to tell something about those units.
That is what I had in mind when I suggested a doc clarification.

> Migrating to a fully built-in directory lister seems to be the only
> sane way out, and it's considerably more work than I anticipated for
> this little detail.

We already have ls-lisp.el.  It isn't used on platforms where 'ls' is
available because AFAIK using 'ls' is faster.

bug#36729: 27.0.50; Unclear total in directory listing

[Mattias Engdegård]

21 juli 2019 kl. 16.29 skrev Eli Zaretskii <eliz@gnu.org>:
> 
> (Your reaction seems to imply that I said something silly.  Did I?)

Absolutely not, your comment was most reasonable. It was just my honest reaction of exasperation for not being able to fix such a stupid detail properly.

> Your original report was about the unclear units of the value, so a
> useful clarification would be to tell something about those units.
> That is what I had in mind when I suggested a doc clarification.

Quite, but while we could write that the value might be this or that, it wouldn't actually help the user unless he or she is so well-informed that no explanation is needed. As we all know, good design should not need explaining.

> We already have ls-lisp.el.  It isn't used on platforms where 'ls' is
> available because AFAIK using 'ls' is faster.

Is that is still noticeable on modern systems, or just for very big and/or recursive listings?

I understand the history behind Dired's design: at one point in time, using the system `ls' was not only a way to re-use existing system software, but also gave performance advantages as well as presenting the information in a familiar way. In addition, the `ls' output was more or less identical everywhere, making it easy to parse (no localisation, no Unicode, no MS-DOS, no fancy GNU or BSD options).

The `ls -l' format, in turn, hasn't changed perceptibly for 40 years, give or take a few -- not because it was perfect from the start but because nobody dared breaking scripts and shell pipelines for minor usability advantages.

Thus we are stuck with silly design elements like:
- major structure indicated with a discrete 'd' in column 1
- less-important stuff like the link count and group name prominently displayed
- the file name itself relegated to the very end as if an afterthought, often going past the margin
- disk usage counted in 512-byte physical disk sectors (but not including subdirectories, that would be too useful)
- timestamp parts separated in columns equal in importance to other attributes
- directory 'size' column almost completely useless
- little support for file system improvements since 1975

We can do better, while retaining the old format for those who have grown too accustomed to it (not meant as pejorative).
It's just not what I had planned for in order to fix this bug.

bug#36729: 27.0.50; Unclear total in directory listing

[Drew Adams]

I'm not weighing in here about most of what I see
being discussed in the thread.  And note that I
use Emacs mostly on MS Windows these days, so I
use ls-lisp (and am very grateful for it).

> I understand the history behind Dired's design: at one point in time,
> using the system `ls' was not only a way to re-use existing system
> software, but also gave performance advantages as well as presenting
> the information in a familiar way.  In addition, the `ls' output was
> more or less identical everywhere, making it easy to parse (no
> localisation, no Unicode, no MS-DOS, no fancy GNU or BSD options).

Those are 3 good reasons for Dired being based
on `ls' and its listings: (1) performance, (2)
familiarity, and (3) regularity and simplicity
of design/handling (e.g. parsing).

Another good reason is that a given user's system
`ls' command might support switches and behavior
that ls-lisp does not support.  That one you left
out, and it's the one that most directly concerns
ls-lisp.  Ls-lisp is not, and likely will never
be, a complete replacement for all of the possible
`ls' commands out there.

IMHO, Dired should continue to be based on `ls',
at least in the sense of (1) accepting, displaying,
and working with its output, and (2) accepting and
handling its most common switches.

> The `ls -l' format, in turn, hasn't changed perceptibly for 40 years,
> give or take a few -- not because it was perfect from the start but
> because nobody dared breaking scripts and shell pipelines for minor
> usability advantages.
> 
> Thus we are stuck with silly design elements like:
> - major structure indicated with a discrete 'd' in column 1
> - less-important stuff like the link count and group name prominently
>   displayed
> - the file name itself relegated to the very end as if an afterthought,
>   often going past the margin
> - disk usage counted in 512-byte physical disk sectors (but not
>   including subdirectories, that would be too useful)
> - timestamp parts separated in columns equal in importance to other
>   attributes
> - directory 'size' column almost completely useless
> - little support for file system improvements since 1975

I disagree that the "prominence" of such info is a
problem in Dired nowadays.  It is trivial to use
Dired with such info hidden, and it's trivial to
toggle, to hide/show it.  In my case, I keep details
hidden most of the time.  I typically just show them
temporarily to check the time or byte size of a file
or two.

> We can do better, while retaining the old format for those who have
> grown too accustomed to it (not meant as pejorative).
> It's just not what I had planned for in order to fix this bug.

It sounds instead like your complaint is with `ls'
itself - being 1975-ish or something.  Push to make
`ls' become the way you want it, and let Dired then
adapt to support that, if it happens.

There is a _lot_ to Dired, including use of `find'
output and the ability to show info for files and
directories from unrelated parts of the file system
in the same buffer (a little known, useful feature).

We should not be proposing a Dired replacement for
`ls' output.  (And I didn't think this bug was
supposed to be about that, anyway.)

bug#36729: 27.0.50; Unclear total in directory listing

[Eli Zaretskii]

> From: Mattias Engdegård <mattiase@acm.org>
> Date: Sun, 21 Jul 2019 20:36:15 +0200
> Cc: 36729@debbugs.gnu.org, stephen.berman@gmx.net, drew.adams@oracle.com
> 
> > We already have ls-lisp.el.  It isn't used on platforms where 'ls' is
> > available because AFAIK using 'ls' is faster.
> 
> Is that is still noticeable on modern systems, or just for very big and/or recursive listings?

I think its generally slower.  But you can try and see for yourself.

> We can do better, while retaining the old format for those who have grown too accustomed to it (not meant as pejorative).

I think switching Dired to a different layout will be so revolutionary
that it warrants a new name, not Dired.  People are too accustomed to
that format, not in the least because they also use "ls -l" from the
shell prompt.

bug#36729: 27.0.50; Unclear total in directory listing

[Eli Zaretskii]

> Date: Sun, 21 Jul 2019 14:31:07 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: 36729@debbugs.gnu.org, stephen.berman@gmx.net
> 
> Another good reason is that a given user's system
> `ls' command might support switches and behavior
> that ls-lisp does not support.  That one you left
> out, and it's the one that most directly concerns
> ls-lisp.  Ls-lisp is not, and likely will never
> be, a complete replacement for all of the possible
> `ls' commands out there.

That is only true for some switches of GNU 'ls', AFAIK.

bug#36729: 27.0.50; Unclear total in directory listing

[Drew Adams]

> > Another good reason is that a given user's system
> > `ls' command might support switches and behavior
> > that ls-lisp does not support.  That one you left
> > out, and it's the one that most directly concerns
> > ls-lisp.  Ls-lisp is not, and likely will never
> > be, a complete replacement for all of the possible
> > `ls' commands out there.
> 
> That is only true for some switches of GNU 'ls', AFAIK.

And perhaps not just GNU.

ls-lisp supports the `ls' switches that are common.
AFAIK, it doesn't claim to support every switch of
every `ls', even one being added tonight to some
system.

But if I'm wrong, and it commits to keeping up with
all systems that have `ls' and that might add a
switch (or change a switch's behavior), then more
power to it.

bug#36729: 27.0.50; Unclear total in directory listing

[Juri Linkov]

> - directory 'size' column almost completely useless

There is a package dired-du available on ELPA that displays
recursive directory sizes in Dired.

bug#36729: 27.0.50; Unclear total in directory listing

[Lars Ingebrigtsen]

[-- Attachment #1: Type: text/plain, Size: 544 bytes --]

Mattias Engdegård <mattiase@acm.org> writes:

> The first line of a directory listing reads
>
>   total used in directory 71752 available 65.2 GiB
>
> where it is a mystery what the first number represents. Often it is
> 512-byte blocks, which is confusing and not very useful, and cannot
> easily be compared with the 'available' number (which is crystal
> clear).
>
> It has always been thus, but previously the 'available' number was
> itself somewhat obscure so the first number didn't stand out.

Hm...  My buffers read:


[-- Attachment #2: Type: image/png, Size: 14051 bytes --]

[-- Attachment #3: Type: text/plain, Size: 288 bytes --]


But that's because I have:

(setq dired-listing-switches "-alh")

It's confusing that it's doing the human-readable bit on the available
but not on the "used" with the default "-al".

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#36729: 27.0.50; Unclear total in directory listing

[Lars Ingebrigtsen]

Stephen Berman <stephen.berman@gmx.net> writes:

> An alternative is to do for "total used" what was done for "available".
> On systems that use `ls' (at least the GNU version), users can do that
> by setting the environment variable BLOCK_SIZE, but since the change to
> the display of "available" was unconditional (if I'm not mistaken), we
> could do the same for the display of "total used", e.g. as in the below
> patch. 

[...]

> +	(block-size (getenv "BLOCK_SIZE"))
>  	end)
> +    (setenv "BLOCK_SIZE" "si")

[...]

> -      (dired-insert-set-properties content-point (point)))))
> +      (dired-insert-set-properties content-point (point)))
> +    (if block-size
> +	(setenv "BLOCK_SIZE" block-size)
> +      (setenv-internal process-environment "BLOCK_SIZE" nil t))))

Makes sense to me, but this should probably be protected by an
`unwind-protect' to ensure that things get properly reset.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#36729: 27.0.50; Unclear total in directory listing

[Stephen Berman]

On Sat, 22 Aug 2020 16:02:28 +0200 Lars Ingebrigtsen <larsi@gnus.org> wrote:

> Stephen Berman <stephen.berman@gmx.net> writes:
>
>> An alternative is to do for "total used" what was done for "available".
>> On systems that use `ls' (at least the GNU version), users can do that
>> by setting the environment variable BLOCK_SIZE, but since the change to
>> the display of "available" was unconditional (if I'm not mistaken), we
>> could do the same for the display of "total used", e.g. as in the below
>> patch.
>
> [...]
>
>> +	(block-size (getenv "BLOCK_SIZE"))
>>  	end)
>> +    (setenv "BLOCK_SIZE" "si")
>
> [...]
>
>> -      (dired-insert-set-properties content-point (point)))))
>> +      (dired-insert-set-properties content-point (point)))
>> +    (if block-size
>> +	(setenv "BLOCK_SIZE" block-size)
>> +      (setenv-internal process-environment "BLOCK_SIZE" nil t))))
>
> Makes sense to me, but this should probably be protected by an
> `unwind-protect' to ensure that things get properly reset.

Yes, but since this idea won't work with all ls implementations, I
assumed it's not worth pursuing.

Steve Berman

bug#36729: 27.0.50; Unclear total in directory listing

[Lars Ingebrigtsen]

Stephen Berman <stephen.berman@gmx.net> writes:

> Yes, but since this idea won't work with all ls implementations, I
> assumed it's not worth pursuing.

It works with ls on GNU systems, right?  I think that's what we care
about most.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#36729: 27.0.50; Unclear total in directory listing

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Mattias Engdegård <mattiase@acm.org>,  Eli Zaretskii
>  <eliz@gnu.org>,
>   36729@debbugs.gnu.org
> Date: Wed, 07 Oct 2020 06:44:18 +0200
> 
> Stephen Berman <stephen.berman@gmx.net> writes:
> 
> > Yes, but since this idea won't work with all ls implementations, I
> > assumed it's not worth pursuing.
> 
> It works with ls on GNU systems, right?  I think that's what we care
> about most.

I no longer understand what we are trying to solve here.  Can someone
please summarize the current HEAD of this discussion?  E.g., what
would replace that summary line under the latest proposals?

bug#36729: 27.0.50; Unclear total in directory listing

[Mattias Engdegård]

7 okt. 2020 kl. 09.50 skrev Eli Zaretskii <eliz@gnu.org>:

> I no longer understand what we are trying to solve here.

A bit lost here myself, but it seems that no real progress has been made and that the original problem (how to display a meaningful 'total' value in dired) is still at hand.

However given the apparent constraints (running an external 'ls' binary with unclear semantics; uncountable platform variations; user-settable 'ls' switches; sensitivity to environment variables; localised output; &c &c) I don't think it can be solved in any reasonable way. I therefore propose closing the bug as 'wontfix'.
Sorry about not thinking it through before reporting it.

bug#36729: 27.0.50; Unclear total in directory listing

[Eli Zaretskii]

> From: Mattias Engdegård <mattiase@acm.org>
> Date: Wed, 7 Oct 2020 12:03:39 +0200
> Cc: Lars Ingebrigtsen <larsi@gnus.org>, stephen.berman@gmx.net,
>         36729@debbugs.gnu.org
> 
> 7 okt. 2020 kl. 09.50 skrev Eli Zaretskii <eliz@gnu.org>:
> 
> > I no longer understand what we are trying to solve here.
> 
> A bit lost here myself, but it seems that no real progress has been made and that the original problem (how to display a meaningful 'total' value in dired) is still at hand.

Yes.

> However given the apparent constraints (running an external 'ls' binary with unclear semantics; uncountable platform variations; user-settable 'ls' switches; sensitivity to environment variables; localised output; &c &c) I don't think it can be solved in any reasonable way. I therefore propose closing the bug as 'wontfix'.

You mean, we cannot even say in what units those numbers are measured?
AFAIU, the missing units were the original motivation for the bug
report.

bug#36729: 27.0.50; Unclear total in directory listing

[Mattias Engdegård]

tags 36729 wontfix
close 36729
stop

7 okt. 2020 kl. 14.14 skrev Eli Zaretskii <eliz@gnu.org>:

> You mean, we cannot even say in what units those numbers are measured?

That's right. It may be blocks, sectors, KiB, or baker's dozens. There may be ways to influence the unit (switches, environment variables) but not portably, and there is no way of telling what effects such efforts take, if any.

The alternative routes all sound equally infeasible. Sorry about wasting your time; closing.

bug#36729: 27.0.50; Unclear total in directory listing

[Eli Zaretskii]

> From: Mattias Engdegård <mattiase@acm.org>
> Date: Wed, 7 Oct 2020 15:50:36 +0200
> Cc: larsi@gnus.org, stephen.berman@gmx.net, 36729@debbugs.gnu.org
> 
> > You mean, we cannot even say in what units those numbers are measured?
> 
> That's right. It may be blocks, sectors, KiB, or baker's dozens. There may be ways to influence the unit (switches, environment variables) but not portably, and there is no way of telling what effects such efforts take, if any.

Doesn't the result come from get-free-disk-space, which calls
file-system-info, which reports in known units?  Or am I confused?

bug#36729: 27.0.50; Unclear total in directory listing

[Mattias Engdegård]

7 okt. 2020 kl. 16.23 skrev Eli Zaretskii <eliz@gnu.org>:

> Doesn't the result come from get-free-disk-space, which calls
> file-system-info, which reports in known units?

It comes from the first line of 'ls -l' and is somehow related to the sizes of the shown files.

bug#36729: 27.0.50; Unclear total in directory listing

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> That's right. It may be blocks, sectors, KiB, or baker's
>> dozens. There may be ways to influence the unit (switches,
>> environment variables) but not portably, and there is no way of
>> telling what effects such efforts take, if any.
>
> Doesn't the result come from get-free-disk-space, which calls
> file-system-info, which reports in known units?  Or am I confused?

file-system-info does not give you information about used file sizes in
a directory. Something which I've missed also in Tramp.

Best regards, Michael.

bug#36729: 27.0.50; Unclear total in directory listing

[Robert Pluim]

>>>>> On Wed, 07 Oct 2020 18:40:23 +0200, Michael Albinus <michael.albinus@gmx.de> said:

    Michael> Eli Zaretskii <eliz@gnu.org> writes:
    >>> That's right. It may be blocks, sectors, KiB, or baker's
    >>> dozens. There may be ways to influence the unit (switches,
    >>> environment variables) but not portably, and there is no way of
    >>> telling what effects such efforts take, if any.
    >> 
    >> Doesn't the result come from get-free-disk-space, which calls
    >> file-system-info, which reports in known units?  Or am I confused?

    Michael> file-system-info does not give you information about used file sizes in
    Michael> a directory. Something which I've missed also in Tramp.

Tramp could use 'ls-lisp' maybe?

Robert
-- 

bug#36729: 27.0.50; Unclear total in directory listing

[Michael Albinus]

Robert Pluim <rpluim@gmail.com> writes:

>     Michael> file-system-info does not give you information about used
>     Michael> file sizes in
>     Michael> a directory. Something which I've missed also in Tramp.
>
> Tramp could use 'ls-lisp' maybe?

Tramp does, sometimes. But due to the inferior performance, it is rather
used as fallback.

> Robert

Best regards, Michael.