A Modest Proposal

A Modest Proposal

[Lars Ingebrigtsen]

I've been on the couch with a cold for the past few days, so I've been
doing documentation and doc string tweaks to avoid going crazy with
boredom.  (It was either that or watch Murder She Wrote.)

I've been reminded this time over that (apart from the many pieces of
Emacs that I know nothing about), there are two areas I don't really
like touching:  Customize and dired.

Customize:

When I try to chase down a rendering bug or ... anything, I'm seldom
able to.  Everything seems to happen somewhere else.  In addition, it
does a lot of stuff with overlays instead of text overlays (which I find
more convenient to interface with, especially when debugging).

And looking at the number of commit messages, perhaps I'm not the only
one.

Perhaps it's time just to throw in the towel and reimplement it from
scratch?  I'm talking about the user interface only, of course.  The
`defcustom' language etc etc etc are fine as they are.

Perhaps we could just start generating HTML write a new interface on top
of shr or something?  Or gtk+?  I dunno.  But I think it's an idea that
somebody who's looking for a nice little project should consider.

Dired:

I think the basic design is misguided.  There is so much talk about
matching up "ls" parameters to internal stuff that it makes my head
swim.  (Even if I'm not feverish.)  C'mon.  We're not barbarians.  We
can read directories ourselves.

In addition, dired (in some places) tries to use the visual
representation in the buffer for something useful, which limits what can
and can't be displayed.  Specifying what files to work on as a regexp is
just kinda nauseating.  (Did I mention my cold?)

It has been mentioned that the reliance on "ls" would enable a future
more async version of dired, but we can use the async.el library for
that just as well, without all the drawbacks.

So that's another little project for somebody: Implement a dired clone
from a UI interface standpoint, but implement it in a more flexible
way.  Then we can make dired obsolete in a few years.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: A Modest Proposal

[raman]

Given the choice, I'd prefer a customize front-end rendered via eww --
will work universally across X and the terminal. 
-- 

Re: A Modest Proposal

[John Wiegley]

>>>>> Lars Ingebrigtsen <larsi@gnus.org> writes:

> I think the basic design is misguided. There is so much talk about matching
> up "ls" parameters to internal stuff that it makes my head swim. (Even if
> I'm not feverish.) C'mon. We're not barbarians. We can read directories
> ourselves.

Maybe the default behavior for dired should be to just use opendir.  That
sounds like a good idea to me.

However, dired does a heck of a lot more than just call "ls": It lets you act
on anything with the same basic format, such as output from "find" or "ack".
So, the regexps are handy in that they're allowing a common, "group of files"
browsing metaphor on lots of stuff that just looks like, well, a group of
files.

Another random idea that just occurred to me: Given any compilation buffer,
create a dired buffer from it containing only the files that had errors.
Sometimes useful when I want to search/replace across every non-compiling
file.

Resuming: I wonder how would your proposal normalize that information? Would
it use a Dired API to populate the buffer, rather than filling it with text
and applying a bunch of regexps after the fact?

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: A Modest Proposal

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Date: Sat, 30 Apr 2016 17:14:14 -0700
> 
> Maybe the default behavior for dired should be to just use opendir.  That
> sounds like a good idea to me.

That's what ls-lisp.el does.

Re: A Modest Proposal

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> That's what ls-lisp.el does.

Hah, don't know why I wasn't thinking of that. In fact, it's what em-ls.el
does, which can also be used as the backend for dired.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

RE: A Modest Proposal

[Drew Adams]

> That's what ls-lisp.el does.

BTW, I'm guessing that some of the features that `ls-lisp.el'
offers could be extended to Dired for the case where it is
not used, i.e., the case where `ls' is used.

For example, `ls-lisp.el' advises `insert-directory' so that
it uses `ls-lisp--insert-directory', and that lets you do
things like insert "/foo/*", meaning insert all of the files
(and subdirs) in directory foo, that is, its contents and not
just a single line for the directory.  Without using ls-lisp,
trying to do that results in:

 insert-directory: Reading directory: no such file or
                   directory, /foo/*

Perhaps `insert-directory' could be enhanced to do a little
more of what `ls-lisp--insert-directory' does?

Re: A Modest Proposal

[Lars Ingebrigtsen]

Drew Adams <drew.adams@oracle.com> writes:

> Perhaps `insert-directory' could be enhanced to do a little
> more of what `ls-lisp--insert-directory' does?

My modest proposal is kinda suggesting that `insert-directory' would go
away -- to be replaced by (something like) `ls-lisp--insert-directory'.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: A Modest Proposal

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: John Wiegley <jwiegley@gmail.com>,  Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Sun, 01 May 2016 17:01:11 +0200
> 
> My modest proposal is kinda suggesting that `insert-directory' would go
> away -- to be replaced by (something like) `ls-lisp--insert-directory'.

Users of Posix platforms will curse you if you do.

Re: A Modest Proposal

[Dmitry Gutov]

On 05/01/2016 06:16 PM, Eli Zaretskii wrote:

>> My modest proposal is kinda suggesting that `insert-directory' would go
>> away -- to be replaced by (something like) `ls-lisp--insert-directory'.
>
> Users of Posix platforms will curse you if you do.

Yes. Listing files using external tools is usually faster.

After digging into grep.el and using it a bit, I've been hoping we could 
construct a high-level pluggable API inside it, so that instead of 
customizing grep-program, etc, it would be possible to use third-party 
alternatives (such as Ag) to list files, and to search through files, 
from the core packages.

Using ls-files instead of find could work like one such alternative 
(used by default as fallback).

Re: A Modest Proposal

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 1 May 2016 18:28:04 +0300
> Cc: jwiegley@gmail.com, drew.adams@oracle.com, emacs-devel@gnu.org
> 
> On 05/01/2016 06:16 PM, Eli Zaretskii wrote:
> 
> >> My modest proposal is kinda suggesting that `insert-directory' would go
> >> away -- to be replaced by (something like) `ls-lisp--insert-directory'.
> >
> > Users of Posix platforms will curse you if you do.
> 
> Yes. Listing files using external tools is usually faster.

Not only faster, more flexible: the sheer number of options supported
by modern variants of 'ls' is overwhelming.  ls-lisp supports only a
subset of them.

Re: A Modest Proposal

[Alan Third]

On Sun, May 01, 2016 at 06:28:04PM +0300, Dmitry Gutov wrote:
> On 05/01/2016 06:16 PM, Eli Zaretskii wrote:
> 
> >>My modest proposal is kinda suggesting that `insert-directory' would go
> >>away -- to be replaced by (something like) `ls-lisp--insert-directory'.
> >
> >Users of Posix platforms will curse you if you do.
> 
> Yes. Listing files using external tools is usually faster.

I semi-regularly list large directories on remote Windows shares and
Emacs's built-in ls is so slow as to be useless. I have to use a
Windows native version of ls that someone's written. If that
capability was removed I'd hope that some work was done on Emacs
performance.

(To be fair to Emacs, GNU ls from Cygwin is even slower, in fact, it
hangs before displaying anything.)
-- 
Alan Third

Re: A Modest Proposal

[Eli Zaretskii]

> Date: Sun, 1 May 2016 19:24:38 +0100
> From: Alan Third <alan@idiocy.org>
> Cc: Eli Zaretskii <eliz@gnu.org>, Lars Ingebrigtsen <larsi@gnus.org>,
> 	jwiegley@gmail.com, drew.adams@oracle.com, emacs-devel@gnu.org
> 
> I semi-regularly list large directories on remote Windows shares and
> Emacs's built-in ls is so slow as to be useless. I have to use a
> Windows native version of ls that someone's written. If that
> capability was removed I'd hope that some work was done on Emacs
> performance.

For remote (i.e. networked) volumes, there's nothing that can be done
to speed that up.  Network volumes are slow, period.  Your native 'ls'
simply shortcuts and thuis fails to retrieve some attributes Emacs
doesn't, that's all.  TANSTAAFL.

Re: A Modest Proposal

[Stefan Monnier]

> Yes. Listing files using external tools is usually faster.

Is the difference sufficiently large to be noticeable in dired?


        Stefan

RE: A Modest Proposal

[Drew Adams]

> > Yes. Listing files using external tools is usually faster.
> 
> Is the difference sufficiently large to be noticeable in dired?

Meanwhile, we seem to be headed in the opposite direction for
`A' etc. - now requiring users to have an external `grep' and
`find' commands.  Not a good thing, IMHO.

http://debbugs.gnu.org/cgi/bugreport.cgi?bug=23426#37

Re: A Modest Proposal

[Eli Zaretskii]

> Date: Tue, 3 May 2016 21:29:06 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> 
> Meanwhile, we seem to be headed in the opposite direction for
> `A' etc. - now requiring users to have an external `grep' and
> `find' commands.  Not a good thing, IMHO.

FUD.  We are not moving in any such direction.

Re: A Modest Proposal

[Dmitry Gutov]

On 05/04/2016 06:56 AM, Stefan Monnier wrote:
>> Yes. Listing files using external tools is usually faster.
>
> Is the difference sufficiently large to be noticeable in dired?

It might, if used in the context of a command like find-grep-dired.

So either Dired at least *can* work on the output of 'find -ls', or we'd 
say goodbye to the above command.

Re: A Modest Proposal

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Tue, 03 May 2016 23:56:40 -0400
> 
> > Yes. Listing files using external tools is usually faster.
> 
> Is the difference sufficiently large to be noticeable in dired?

Last time I tried, it was.  But maybe someone should try again, and
show the numbers.

Our algorithm in ls-lisp and the primitives it employs is definitely
less optimal than what 'ls' does, but I cannot estimate the
quantitative effect of that.

Re: A Modest Proposal

[Michael Albinus]

Lars Ingebrigtsen <larsi@gnus.org> writes:

>> Perhaps `insert-directory' could be enhanced to do a little
>> more of what `ls-lisp--insert-directory' does?
>
> My modest proposal is kinda suggesting that `insert-directory' would go
> away -- to be replaced by (something like) `ls-lisp--insert-directory'.

Don't forget that `insert-directory' supports file name handlers like
Tramp. So it isn't just replacing one function by another one, you
should keep this functionality as well.

Best regards, Michael.

Re: A Modest Proposal

[Stefan Monnier]

> Don't forget that `insert-directory' supports file name handlers like
> Tramp. So it isn't just replacing one function by another one, you
> should keep this functionality as well.

AFAIK ls-lisp is implemented with functions that are also handled by
Tramp, so it should work just as well.


        Stefan

Re: A Modest Proposal

[Andy Moreton]

On Tue 03 May 2016, Stefan Monnier wrote:

>> Don't forget that `insert-directory' supports file name handlers like
>> Tramp. So it isn't just replacing one function by another one, you
>> should keep this functionality as well.
>
> AFAIK ls-lisp is implemented with functions that are also handled by
> Tramp, so it should work just as well.

Except for platforms where ls-lisp does not understand the underlying
file system in the same way as the external tool.

For example, on windows the permissions from ls-lisp are fairly
meaningless, and do not match what is reported from MSYS, MSYS2 or
Cygwin versions of ls (all of which have different filesystem paths and
different permissions semantics).

Please keep things the way they are.

    AndyM

Re: A Modest Proposal

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Wed, 04 May 2016 09:51:52 +0100
> 
> On Tue 03 May 2016, Stefan Monnier wrote:
> 
> > AFAIK ls-lisp is implemented with functions that are also handled by
> > Tramp, so it should work just as well.
> 
> Except for platforms where ls-lisp does not understand the underlying
> file system in the same way as the external tool.

ls-lisp understands the underlying filesystem as well as Emacs does.

> For example, on windows the permissions from ls-lisp are fairly
> meaningless

No, they are not meaningless.  I wonder why you say that.

> and do not match what is reported from MSYS, MSYS2 or Cygwin
> versions of ls

Why should they match?  Cygwin emulates Posix user/group/world
permissions using NTFS security features, and in particular encodes
the Posix attribute bits in Windows ACEs using its own private scheme.
There's no reason to expect native calls to match that scheme.

> Please keep things the way they are.

Not sure why do you say that.  If you mean that you'd like to have use
Cygwin/MSYS 'ls' from a native Emacs, then I don't recommend that, due
to the above-mentioned mismatches.  You could have a file listed as
inaccessible, which actually is, or vice versa.  Why is that a good
idea?

Re: A Modest Proposal

[Andy Moreton]

On Wed 04 May 2016, Eli Zaretskii wrote:

>> From: Andy Moreton <andrewjmoreton@gmail.com>
>> Date: Wed, 04 May 2016 09:51:52 +0100
>> 
>> On Tue 03 May 2016, Stefan Monnier wrote:
>> 
>> > AFAIK ls-lisp is implemented with functions that are also handled by
>> > Tramp, so it should work just as well.
>> 
>> Except for platforms where ls-lisp does not understand the underlying
>> file system in the same way as the external tool.
>
> ls-lisp understands the underlying filesystem as well as Emacs does.
>
>> For example, on windows the permissions from ls-lisp are fairly
>> meaningless
>
> No, they are not meaningless.  I wonder why you say that.
>
>> and do not match what is reported from MSYS, MSYS2 or Cygwin
>> versions of ls
>
> Why should they match?  Cygwin emulates Posix user/group/world
> permissions using NTFS security features, and in particular encodes
> the Posix attribute bits in Windows ACEs using its own private scheme.
> There's no reason to expect native calls to match that scheme.

The scheme is publically documented, and closely follows the mappings
previously used by the Interix (aka Services for Unix) subsystem. The
code is GPL, so I don;t understand why you think this is private.

>> Please keep things the way they are.
>
> Not sure why do you say that.  If you mean that you'd like to have use
> Cygwin/MSYS 'ls' from a native Emacs, then I don't recommend that, due
> to the above-mentioned mismatches.  You could have a file listed as
> inaccessible, which actually is, or vice versa.  Why is that a good
> idea?

While it is simpler to use a homogenous environment, there are
significant uses of mixed environments, where the emacs binary and the
platform differ in their view of the filesystem.

It's ok to change the default implementation to prefer using an emacs
internal implementation rather than an external program, but the
proposal was to remove use of external programs, which would make emacs
less useful.

    AndyM

Re: A Modest Proposal

[Michael Albinus]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Don't forget that `insert-directory' supports file name handlers like
>> Tramp. So it isn't just replacing one function by another one, you
>> should keep this functionality as well.
>
> AFAIK ls-lisp is implemented with functions that are also handled by
> Tramp, so it should work just as well.

With a serious performance degradation, as others have said already.

>         Stefan

Best regards, Michael.

Re: A Modest Proposal

[Eli Zaretskii]

> Date: Sun, 1 May 2016 06:44:03 -0800 (GMT-08:00)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: emacs-devel@gnu.org
> 
> For example, `ls-lisp.el' advises `insert-directory' so that
> it uses `ls-lisp--insert-directory', and that lets you do
> things like insert "/foo/*", meaning insert all of the files
> (and subdirs) in directory foo, that is, its contents and not
> just a single line for the directory.  Without using ls-lisp,
> trying to do that results in:
> 
>  insert-directory: Reading directory: no such file or
>                    directory, /foo/*

Not for me, it doesn't; not if I set the 3rd arg non-nil, as the doc
string says I should.

RE: A Modest Proposal

[Drew Adams]

> > For example, `ls-lisp.el' advises `insert-directory' so that
> > it uses `ls-lisp--insert-directory', and that lets you do
> > things like insert "/foo/*", meaning insert all of the files
> > (and subdirs) in directory foo, that is, its contents and not
> > just a single line for the directory.  Without using ls-lisp,
> > trying to do that results in:
> >
> >  insert-directory: Reading directory: no such file or
> >                    directory, /foo/*
> 
> Not for me, it doesn't; not if I set the 3rd arg non-nil, as the doc
> string says I should.

That's true - if you set the 3rd arg, FULL-DIRECTORY-P non-nil.

In fact, my use case was accepting arbitrary file and dir names,
including glob patterns, at a `read-file-name' prompt, and adding
the indicated files and dirs to an existing Dired buffer.

So I wanted the code to automatically interpret the input
"/foo/*" as the contents of dir foo and just "/foo" (or "/foo/")
as a single listing line for the directory.  No indication
needed by the user for each input as to whether to use
FULL-DIRECTORY-P or not.

Without ls-lisp, `insert-directory' apparently does not
do the job (at least as reported by one user -
http://emacs.stackexchange.com/a/21799/105).

(My function that does this is `diredp-dired-union-1'.
Passing it, as arg EXTRA, a list of files and dirs, where
each could be a glob pattern, adds those to a Dired buffer.)

Re: A Modest Proposal

[Lars Ingebrigtsen]

John Wiegley <jwiegley@gmail.com> writes:

> Resuming: I wonder how would your proposal normalize that information? Would
> it use a Dired API to populate the buffer, rather than filling it with text
> and applying a bunch of regexps after the fact?

Yes, indeed.  It would be a mode that takes lists of files (and file
information) and displays that in the way the user wants, instead of
mangling various textual presentations output by various Unix commands.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: A Modest Proposal

[joakim]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> John Wiegley <jwiegley@gmail.com> writes:
>
>> Resuming: I wonder how would your proposal normalize that information? Would
>> it use a Dired API to populate the buffer, rather than filling it with text
>> and applying a bunch of regexps after the fact?
>
> Yes, indeed.  It would be a mode that takes lists of files (and file
> information) and displays that in the way the user wants, instead of
> mangling various textual presentations output by various Unix commands.

That would be nice.

For instance I had a patch that used the filemagic library to return
extended file information to Emacs.

Nowadays I guess you should use the module facility to interface to
filemagic.

I think my point is that lisp could potentially have even better
metadata about a file in a structured way than what is now
available. Something along that line of thought anyway...
-- 
Joakim Verona

Re: A Modest Proposal

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

In the past, running Dired was much faster than generating a directory
listing in Emacs.  This was very important.

Maybe all computers are now so fast that this no longer matters.
But don't fail to check!

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: A Modest Proposal

[raman]

Computers one has on a desk or a laptop are likely plenty fast, but we
still get slower computers e.g. a Raspberry Pi --- and it would be good
to  keep Emacs   from assuming a multicore processor with loads of RAM:-)
-- 

Re: A Modest Proposal

[Ted Zlatanov]

On Sun, 01 May 2016 01:38:00 +0200 Lars Ingebrigtsen <larsi@gnus.org> wrote: 

LI> Customize:
...
LI> Perhaps it's time just to throw in the towel and reimplement it from
LI> scratch?  I'm talking about the user interface only, of course.  The
LI> `defcustom' language etc etc etc are fine as they are.

LI> Perhaps we could just start generating HTML write a new interface on top
LI> of shr or something?  Or gtk+?  I dunno.  But I think it's an idea that
LI> somebody who's looking for a nice little project should consider.

On Sat, 30 Apr 2016 16:50:22 -0700 raman <raman@google.com> wrote: 

r> Given the choice, I'd prefer a customize front-end rendered via eww --
r> will work universally across X and the terminal. 

I agree with Lars and raman. Although eww+shr does not currently look
"native" inside Emacs, that's not a big hurdle. The advantage of better
layouts, better widgets across various platforms and text/GUI, and a
common language are significant. I don't think it's a little project,
unfortunately.

LI> Dired:

LI> I think the basic design is misguided.  There is so much talk about
LI> matching up "ls" parameters to internal stuff that it makes my head
LI> swim.  (Even if I'm not feverish.)  C'mon.  We're not barbarians.  We
LI> can read directories ourselves.

I agree with this too. Performance issues can be resolved; less
dependence on external tools is always a good thing IMHO.

Ted