Too fine design granularity leads to numerous macro/function/command existed in Emacs.

Too fine design granularity leads to numerous macro/function/command existed in Emacs.

[Hongyi Zhao]

 It's well known that Emacs has already implemented numerous
macro/function/command in its current version. But it seems that this
is caused by, to a certain degree, the too fine granularity, i.e.,
there are many functionally similar macro/function/command are
designed separately, for example, `directory-files' and
`directory-files-recursively'. The traditional Unix tool, `find', can
do all the jobs of the above two functions by adjusting its
`-maxdepth' and `-mindepth' arguments.

So, I want to know why these macro/function/command in Emacs are
developed with such a fine granularity.

Regards, Hongyi

Re: Too fine design granularity leads to numerous macro/function/command existed in Emacs.

[Eli Zaretskii]

> From: Hongyi Zhao <hongyi.zhao@gmail.com>
> Date: Fri, 13 Aug 2021 15:12:39 +0800
> 
>  It's well known that Emacs has already implemented numerous
> macro/function/command in its current version. But it seems that this
> is caused by, to a certain degree, the too fine granularity, i.e.,
> there are many functionally similar macro/function/command are
> designed separately, for example, `directory-files' and
> `directory-files-recursively'. The traditional Unix tool, `find', can
> do all the jobs of the above two functions by adjusting its
> `-maxdepth' and `-mindepth' arguments.
> 
> So, I want to know why these macro/function/command in Emacs are
> developed with such a fine granularity.

They aren't.  You provided just one example, and drew far-reaching
conclusions from that single example.  If you want to make such a
general claim, please show more examples of what you consider to be
"fine granularity".

Specifically about the one example you provided: directory-files is a
primitive, written in C, so it provides the basic functionality of
fetching file names from a single directory. The
directory-files-recursively function is written in Lisp, it builds on
that primitive and provides extended functionality.  This paradigm, of
having primitives provide the basics and implement the rest in Lisp on
top of those primitives, is central to Emacs design.

Re: Too fine design granularity leads to numerous macro/function/command existed in Emacs.

[Emanuel Berg via Users list for the GNU Emacs text editor]

Hongyi Zhao wrote:

> It's well known that Emacs has already implemented numerous
> macro/function/command in its current version. But it seems
> that this is caused by, to a certain degree, the too fine
> granularity, i.e., there are many functionally similar
> macro/function/command are designed separately, for example,
> `directory-files' and `directory-files-recursively'.
> The traditional Unix tool, `find', can do all the jobs of
> the above two functions by adjusting its `-maxdepth' and
> `-mindepth' arguments.

OTOH, you can do _a lot_ with find!

GNU tools have a lot of options, Emacs have a lot of
functions, Linux (and zsh) has a lot of features, Gnus has
a lot of source code...

Try BSD (including toolchain) if you want less :)

Well, no disrespect to the BSD people, many of whom use Emacs.

With the maximalist approach it is unavoidable that
sometimes one function can do 1 2 and 3, and another can do 2,
3, and 4, so it seems they have "too fine granularity" in your
phrasing but outside the intersection is actually ... 1 and 4!

The most important thing is that everything makes sense and
works, if it overlaps, has inconsistent names with respect to
each other, yada yada, I mean try to avoid it but if it
happens it is not a problem, really.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Too fine design granularity leads to numerous macro/function/command existed in Emacs.

[Hongyi Zhao]

On Fri, Aug 13, 2021 at 3:58 PM Emanuel Berg via Users list for the
GNU Emacs text editor <help-gnu-emacs@gnu.org> wrote:
> [...]
> With the maximalist approach it is unavoidable that
> sometimes one function can do 1 2 and 3, and another can do 2,
> 3, and 4, so it seems they have "too fine granularity" in your
> phrasing but outside the intersection is actually ... 1 and 4!

Then why not combine the two into one, which can do 1, 2, 3, and 4?

> The most important thing is that everything makes sense and
> works, if it overlaps, has inconsistent names with respect to
> each other, yada yada, I mean try to avoid it but if it
> happens it is not a problem, really.
>
> --
> underground experts united

Why do you use the above sentence as the signature? Does it have any
profound implications?

> https://dataswamp.org/~incal

Hongyi

Re: Too fine design granularity leads to numerous macro/function/command existed in Emacs.

[Emanuel Berg via Users list for the GNU Emacs text editor]

Hongyi Zhao wrote:

>> With the maximalist approach it is unavoidable that
>> sometimes one function can do 1 2 and 3, and another can do
>> 2, 3, and 4, so it seems they have "too fine granularity"
>> in your phrasing but outside the intersection is actually
>> ... 1 and 4!
>
> Then why not combine the two into one, which can do 1, 2, 3,
> and 4?

Well, assuming the situation actually is like that, then
probably because no one feels the need or desire to do that
because after all the work coming up with a common interface
and all the work pruning the function that gets dropped from
all existing code by every Joe Elisp Hacker to ever use it
across the planet, and after fixing every bug that was
introduced in the merge process and calming everyone that gets
upset on mailing lists and so on ... THEN ... you end up with,
not 1 2 3 4, but: 1 2 3 4 :(

>> underground experts united
>
> Why do you use the above sentence as the signature? Does it
> have any profound implications?

Those who know know:

  https://dataswamp.org/~incal/figures/emacs/emacs-uxu-submarine-spetsnaz.png

-- 
underground experts united
https://dataswamp.org/~incal

Re: Too fine design granularity leads to numerous macro/function/command existed in Emacs.

[Emanuel Berg via Users list for the GNU Emacs text editor]

> https://dataswamp.org/~incal/figures/emacs/emacs-uxu-submarine-spetsnaz.png

Or:

  https://dataswamp.org/~incal/figures/opengl-glsl/tri.png

-- 
underground experts united
https://dataswamp.org/~incal

Re: Too fine design granularity leads to numerous macro/function/command existed in Emacs.

[Arthur Miller]

Hongyi Zhao <hongyi.zhao@gmail.com> writes:

>  It's well known that Emacs has already implemented numerous
> macro/function/command in its current version. But it seems that this
> is caused by, to a certain degree, the too fine granularity, i.e.,
> there are many functionally similar macro/function/command are
> designed separately, for example, `directory-files' and
> `directory-files-recursively'. The traditional Unix tool, `find', can
I wouldn't say that these are "very" similar. Sure they both list files,
but a recursive lister vs. plain lister is a big difference. You can
implement recursive lister with directory-files, I have done it myself
sometime, but `directory-files-recursively' is much nicer and time
saving. I wouldn't say it is "fine granularity" there, rather very rough
granularity :).

>                                The traditional Unix tool, `find', can
> do all the jobs of the above two functions by adjusting its
> `-maxdepth' and `-mindepth' arguments.
Sure, so can others too. But nice thing wiht having it in elisp is that
is more straightforward to use. You have only one language to code in as
opposed to shell or whatever your tool needs etc. Also it is portable to
systems that don't have those tools installed. You also save some time
in process creation. Depending on complexity of your directory tree and
what you are trying to accomplish, it can be costly.

Compare to your shell script to give you longest string from the other
day. It involved several unix tools and thus several processes had to be
created, as opposed to none in the pure elisp case as you got from
Thomas or me.

In a case where you have big files with millions of lines, unix
toolchain would probably win performance wise; I don't know where the
cutoff is. But in the case of not so big file, I think the pure elisp
wins, resource and performance wisely. Not to mention, that writing lisp
is almost like writing pseudo-algorithm. If you reflect over the names
chosen in that function, you can almost see that it was codded on the
go, as I was thinking of it. Compare that to pipes and names like tr and
cut and what not in your original shell solution. So it is a kind of
simplicity, in my eyes, to use just one language, and I guess also a bit
of personal preferance of course.

Re: Too fine design granularity leads to numerous macro/function/command existed in Emacs.

[Emanuel Berg via Users list for the GNU Emacs text editor]

Arthur Miller wrote:

> Not to mention, that writing lisp is almost like writing
> pseudo-algorithm. If you reflect over the names chosen in
> that function, you can almost see that it was codded on the
> go, as I was thinking of it. Compare that to pipes and names
> like tr and cut and what not in your original shell
> solution. So it is a kind of simplicity, in my eyes, to use
> just one language, and I guess also a bit of personal
> preferance of course.

As for development time, if that's the right word, it must be
much, much faster using the shell, which is mostly about
combining well known tools often with options and pattern that
return over and over - it is very, very fast.

Compare - this takes as long to do, as it takes to type it:

#! /bin/zsh
longest-line () {
    local f=$1
    awk '{ print length($0) " " $0; }' $f | sort -n | tail -1
}

with this - now, I had a function that worked along similar
lines (ha) so for that reason, it wasn't that slow to write
compared to the shell, without that, I don't know, depends on
the current state/shape :)

;;; -*- lexical-binding: t -*-

(defun sort-lines-length (beg end)
  (interactive (if (use-region-p)
                   (list (region-beginning) (region-end))
                 (list (point-min) (point-max)) ))
  (save-excursion
    (save-restriction
      (narrow-to-region beg end)
      (goto-char (point-min))
      (sort-subr nil
                 #'forward-line
                 #'end-of-line
                 nil nil
                 (lambda (a b) (> (- (cdr a) (car a))
                                  (- (cdr b) (car b)) ))))))
(defalias 'sll #'sort-lines-length)

https://dataswamp.org/~incal/conf/.zsh/text
https://dataswamp.org/~incal/emacs-init/sort-incal.el

-- 
underground experts united
https://dataswamp.org/~incal

[OFF TOPIC] Algorithms (was: Re: Too fine design granularity leads to numerous macro/function/command existed in Emacs.)

[2QdxY4RzWzUUiLuE]

On 2021-08-13 at 12:11:29 +0200,
Emanuel Berg via Users list for the GNU Emacs text editor <help-gnu-emacs@gnu.org> wrote:

> #! /bin/zsh
> longest-line () {
>     local f=$1
>     awk '{ print length($0) " " $0; }' $f | sort -n | tail -1

Don't sort unless you have to.

Don't send all that data through two pipes.

    awk '{if(length($0) > length(x)) {x = $0} END {print x}'

For small files, you won't notice the difference.  For large files, you
will.  Definitions of "small" and "large" depend on a lot of things.

> }

Re: [OFF TOPIC] Algorithms (was: Re: Too fine design granularity leads to numerous macro/function/command existed in Emacs.)

[Emanuel Berg via Users list for the GNU Emacs text editor]

2QdxY4RzWzUUiLuE wrote:

>> #! /bin/zsh
>> longest-line () {
>>     local f=$1
>>     awk '{ print length($0) " " $0; }' $f | sort -n | tail -1
>
> Don't sort unless you have to.
>
> Don't send all that data through two pipes.
>
>     awk '{if(length($0) > length(x)) {x = $0} END {print x}'
>
> For small files, you won't notice the difference. For large
> files, you will. Definitions of "small" and "large" depend
> on a lot of things.

*thumbs down*

-- 
underground experts united
https://dataswamp.org/~incal

Re: [OFF TOPIC] Algorithms (was: Re: Too fine design granularity leads to numerous macro/function/command existed in Emacs.)

[Yuri Khan]

On Fri, 13 Aug 2021 at 20:08, Emanuel Berg via Users list for the GNU
Emacs text editor <help-gnu-emacs@gnu.org> wrote:

> 2QdxY4RzWzUUiLuE wrote:

> > Don't sort unless you have to.

> *thumbs down*

The advice is sound. Asymptotically, sorting in order to extract one
longest (maximum, best) element is like firing a cannon at sparrows:
sorting requires O(N*log(N)) comparisons but a single maximum value
can be found in just O(N).

Re: [OFF TOPIC] Algorithms (was: Re: Too fine design granularity leads to numerous macro/function/command existed in Emacs.)

[Emanuel Berg via Users list for the GNU Emacs text editor]

Yuri Khan wrote:

> The advice is sound. Asymptotically, sorting in order to
> extract one longest (maximum, best) element is like firing
> a cannon at sparrows: sorting requires O(N*log(N))
> comparisons but a single maximum value can be found in just
> O(N).

-1

-- 
underground experts united
https://dataswamp.org/~incal

Re: Too fine design granularity leads to numerous macro/function/command existed in Emacs.

[Arthur Miller]

Emanuel Berg via Users list for the GNU Emacs text editor
<help-gnu-emacs@gnu.org> writes:

> Arthur Miller wrote:
>
>> Not to mention, that writing lisp is almost like writing
>> pseudo-algorithm. If you reflect over the names chosen in
>> that function, you can almost see that it was codded on the
>> go, as I was thinking of it. Compare that to pipes and names
>> like tr and cut and what not in your original shell
>> solution. So it is a kind of simplicity, in my eyes, to use
>> just one language, and I guess also a bit of personal
>> preferance of course.
>
> As for development time, if that's the right word, it must be
> much, much faster using the shell, which is mostly about

Why does it "must be" much much faster to code in shell than
elisp? Who says that? :)

> combining well known tools often with options and pattern that
> return over and over - it is very, very fast.
>
> Compare - this takes as long to do, as it takes to type it:
>
> #! /bin/zsh
> longest-line () {
>     local f=$1
>     awk '{ print length($0) " " $0; }' $f | sort -n | tail -1
> }
>
> with this - now, I had a function that worked along similar
> lines (ha) so for that reason, it wasn't that slow to write
> compared to the shell, without that, I don't know, depends on
> the current state/shape :)

I really like shell too, and unix philosophy and have been coded
shell for like 20 years, compared to elisp which I really started to
learn relatively recently, say one year ago, but I find lisp much faster
to code than shell. That what you coded there requires knowledge of
zshell, awk, sort and tail. If you call it from Emacs you pay four
process penalty compared to none if you code it elisp. Not to mention
the portability, elisp will work without modification in Windows or Mac
too, your shell solution might or might not work as expected. Also if
you need to call it in other lisp functions you will have to wrap your
shell script in shell-command or start/call-process, so I personally
really don't see any point in doing it in shell instead of elisp,
because I don't see benefit. But you should of course, choose whatever
suits *your* heart and mind :). 

Re: Too fine design granularity leads to numerous macro/function/command existed in Emacs.

[Eli Zaretskii]

> From: Arthur Miller <arthur.miller@live.com>
> Date: Fri, 13 Aug 2021 11:26:39 +0200
> Cc: help-gnu-emacs <help-gnu-emacs@gnu.org>
> 
> Hongyi Zhao <hongyi.zhao@gmail.com> writes:
> 
> >  It's well known that Emacs has already implemented numerous
> > macro/function/command in its current version. But it seems that this
> > is caused by, to a certain degree, the too fine granularity, i.e.,
> > there are many functionally similar macro/function/command are
> > designed separately, for example, `directory-files' and
> > `directory-files-recursively'. The traditional Unix tool, `find', can
> I wouldn't say that these are "very" similar. Sure they both list files,
> but a recursive lister vs. plain lister is a big difference.

The comparison of 'find' with the two Emacs APIs is basically invalid:
'find' is an _application_, whereas the 2 APIs are just that: APIs
that could be used to build such an application.  See find-lisp.el for
what the application would look like.

Re: Too fine design granularity leads to numerous macro/function/command existed in Emacs.

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Arthur Miller <arthur.miller@live.com>
>> Date: Fri, 13 Aug 2021 11:26:39 +0200
>> Cc: help-gnu-emacs <help-gnu-emacs@gnu.org>
>> 
>> Hongyi Zhao <hongyi.zhao@gmail.com> writes:
>> 
>> >  It's well known that Emacs has already implemented numerous
>> > macro/function/command in its current version. But it seems that this
>> > is caused by, to a certain degree, the too fine granularity, i.e.,
>> > there are many functionally similar macro/function/command are
>> > designed separately, for example, `directory-files' and
>> > `directory-files-recursively'. The traditional Unix tool, `find', can
>> I wouldn't say that these are "very" similar. Sure they both list files,
>> but a recursive lister vs. plain lister is a big difference.
>
> The comparison of 'find' with the two Emacs APIs is basically invalid:
> 'find' is an _application_, whereas the 2 APIs are just that: APIs
> that could be used to build such an application.  See find-lisp.el for
> what the application would look like.

In a *shell world* one could logically think of cli utlities
as "apis" and options as function arguments, and I guess pipe would be
functional composition. Just a little mind stretch if you don't mind
the pun :). 

Re: Too fine design granularity leads to numerous macro/function/command existed in Emacs.

[Eli Zaretskii]

> From: Arthur Miller <arthur.miller@live.com>
> Cc: help-gnu-emacs@gnu.org
> Date: Fri, 13 Aug 2021 16:42:20 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Arthur Miller <arthur.miller@live.com>
> >> Date: Fri, 13 Aug 2021 11:26:39 +0200
> >> Cc: help-gnu-emacs <help-gnu-emacs@gnu.org>
> >> 
> >> Hongyi Zhao <hongyi.zhao@gmail.com> writes:
> >> 
> >> >  It's well known that Emacs has already implemented numerous
> >> > macro/function/command in its current version. But it seems that this
> >> > is caused by, to a certain degree, the too fine granularity, i.e.,
> >> > there are many functionally similar macro/function/command are
> >> > designed separately, for example, `directory-files' and
> >> > `directory-files-recursively'. The traditional Unix tool, `find', can
> >> I wouldn't say that these are "very" similar. Sure they both list files,
> >> but a recursive lister vs. plain lister is a big difference.
> >
> > The comparison of 'find' with the two Emacs APIs is basically invalid:
> > 'find' is an _application_, whereas the 2 APIs are just that: APIs
> > that could be used to build such an application.  See find-lisp.el for
> > what the application would look like.
> 
> In a *shell world* one could logically think of cli utlities
> as "apis" and options as function arguments, and I guess pipe would be
> functional composition.

But there were no pipes in the original comparison.  It just compared
these Lisp APIs with 'find' the command.

Re: Too fine design granularity leads to numerous macro/function/command existed in Emacs.

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Arthur Miller <arthur.miller@live.com>
>> Cc: help-gnu-emacs@gnu.org
>> Date: Fri, 13 Aug 2021 16:42:20 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> From: Arthur Miller <arthur.miller@live.com>
>> >> Date: Fri, 13 Aug 2021 11:26:39 +0200
>> >> Cc: help-gnu-emacs <help-gnu-emacs@gnu.org>
>> >> 
>> >> Hongyi Zhao <hongyi.zhao@gmail.com> writes:
>> >> 
>> >> >  It's well known that Emacs has already implemented numerous
>> >> > macro/function/command in its current version. But it seems that this
>> >> > is caused by, to a certain degree, the too fine granularity, i.e.,
>> >> > there are many functionally similar macro/function/command are
>> >> > designed separately, for example, `directory-files' and
>> >> > `directory-files-recursively'. The traditional Unix tool, `find', can
>> >> I wouldn't say that these are "very" similar. Sure they both list files,
>> >> but a recursive lister vs. plain lister is a big difference.
>> >
>> > The comparison of 'find' with the two Emacs APIs is basically invalid:
>> > 'find' is an _application_, whereas the 2 APIs are just that: APIs
>> > that could be used to build such an application.  See find-lisp.el for
>> > what the application would look like.
>> 
>> In a *shell world* one could logically think of cli utlities
>> as "apis" and options as function arguments, and I guess pipe would be
>> functional composition.
>
> But there were no pipes in the original comparison.  It just compared
> these Lisp APIs with 'find' the command.

True, they were introduced later on, I was regressing there.

Re: Too fine design granularity leads to numerous macro/function/command existed in Emacs.

[Emanuel Berg via Users list for the GNU Emacs text editor]

Arthur Miller wrote:

> In a *shell world* one could logically think of cli utlities
> as "apis" and options as function arguments, and I guess
> pipe would be functional composition. Just a little mind
> stretch if you don't mind the pun :).

Indeed, in a programming language this is clear, more
well-defined, reusable, functional, more modular, etc - maybe
nicely "refactored" that way (unlikely) or even written that
way to begin with (well it _is_ easier and more
pleasant anyway!)

  fun-1(...)
  fun-2(...)
  (...)
  fun-n(...)

However,

  $ this | is | wasting | forks

and contains ugly pipes. Yukiduk!

Instead one should do it this way:

  $ this -iwf

-- 
underground experts united
https://dataswamp.org/~incal

Re: Too fine design granularity leads to numerous macro/function/command existed in Emacs.

[Emanuel Berg via Users list for the GNU Emacs text editor]

Arthur Miller wrote:

> learn relatively recently, say one year ago, but I find lisp
> much faster to code than shell. That what you coded there
> requires knowledge of zshell, awk, sort and tail.

They are just basic building blocks the basic usage of which
is known to everyone, it is like building Lego or something,
meanwhile the below code requires both some general
programming fluency AND some inside knowledge how the
particulars work in Lisp and even Emacs/Elisp.

Everyone that writes it can also combine a bunch of shell
tools, but everyone who can combine a bunch of shell tools
can't write that - many of them can't program at
all, actually.

(defun sort-lines-length (beg end)
  (interactive (if (use-region-p)
                   (list (region-beginning) (region-end))
                 (list (point-min) (point-max)) ))
  (save-excursion
    (save-restriction
      (narrow-to-region beg end)
      (goto-char (point-min))
      (sort-subr nil
                 #'forward-line
                 #'end-of-line
                 nil nil
                 (lambda (a b) (> (- (cdr a) (car a))
                                  (- (cdr b) (car b)) ))))))
(defalias 'sll #'sort-lines-length)

-- 
underground experts united
https://dataswamp.org/~incal

Re: Too fine design granularity leads to numerous macro/function/command existed in Emacs.

[Arthur Miller]

Emanuel Berg via Users list for the GNU Emacs text editor
<help-gnu-emacs@gnu.org> writes:

> Arthur Miller wrote:
>
>> learn relatively recently, say one year ago, but I find lisp
>> much faster to code than shell. That what you coded there
>> requires knowledge of zshell, awk, sort and tail.
>
> They are just basic building blocks the basic usage of which
> is known to everyone, it is like building Lego or something,
> meanwhile the below code requires both some general
> programming fluency AND some inside knowledge how the
> particulars work in Lisp and even Emacs/Elisp.

You don't believe that programming shell script requires at least some
programming fluency AND some inside knowledge hot the particular tools
works, their options, streams etc? It just comes out of itself? :)

> Everyone that writes it can also combine a bunch of shell
> tools, but everyone who can combine a bunch of shell tools
> can't write that - many of them can't program at
> all, actually.

I don't think I or anyone else claimed that "everyone who writes it"
(it = shellscripts?) writes Elisp. That isn't the point. By the way, you
do realize that we write this in context of Emacs and Elisp and not some
random unix tooling context?

Yes, unix command line tools and piping are like lego blocks, that
was by design, it does not mean that *everyone* can write shell
scripts. Everyone *can learn* to write shell scripts, but not everyone
can write shellscripts. I hope you realize the difference. And that
hypothetical *everyone* can also learn how to write elisp if they choose
so. By the way you know what is also like lego blocks? Lisp functions :). 

Anyway I would even say that majority of computer users working in
offices at hospitals, schools, municipals and other places, have never
even seen a *nix system nor ever heard about zshell, tr, find, sort,
grep or elisp. Saying that unix shell tools are easy for everyone
would be like living in a dream world :). Shell scripts are easy for a
certain audience just like lisp or elisp are easy for a certain audience
too.

Some people simply find writing lisp easier than writing shell scirpts,
checkout gnu guix, don't take just my opinion. 

Re: Too fine design granularity leads to numerous macro/function/command existed in Emacs.

[Hongyi Zhao]

On Fri, Aug 13, 2021 at 10:42 PM Arthur Miller <arthur.miller@live.com> wrote:
> [...]
> In a *shell world* one could logically think of cli utlities
> as "apis" and options as function arguments, and I guess pipe would be
> functional composition.

It seems Emacs lisp don't have pipe at all.

> Just a little mind stretch if you don't mind the pun :).

Hongyi

Re: Too fine design granularity leads to numerous macro/function/command existed in Emacs.

[Emanuel Berg via Users list for the GNU Emacs text editor]

Arthur Miller wrote:

> You don't believe that programming shell script requires at
> least some programming fluency AND some inside knowledge hot
> the particular tools works, their options, streams etc?
> It just comes out of itself? :)

This isn't a script really but a shell function and this
function in particular is just a wrapper so no, you don't need
to know much more than the chaining of GNU tools there...

>> Everyone that writes it can also combine a bunch of shell
>> tools, but everyone who can combine a bunch of shell tools
>> can't write that - many of them can't program at
>> all, actually.
>
> I don't think I or anyone else claimed that "everyone who
> writes it" (it = shellscripts?)

Putting together a bunch of shell tools with pipes in between
to do some parsing, that's much more easy than to write Elisp.

Shell tools are on the user level, not on the
programmer/engineer level.

> By the way, you do realize that we write this in context of
> Emacs and Elisp and not some random unix tooling context?

Yes but no, what?

> Yes, unix command line tools and piping are like lego blocks, that
> was by design, it does not mean that *everyone* can write shell
> scripts. Everyone *can learn* to write shell scripts, but not everyone
> can write shellscripts.

Forget about the shell script, it is just a wrapper, an
interface for convenience. No, what it amounts to is the use
and combination of 3 tools. (BTW I didn't get the other guy's
suggestion to work, that which only used awk and nothing else
- well, that might be better (I agree) but as for how
difficult it is it is probably *more* difficult than mine
because his requires some understanding of awk beyond the
most basic "pile together a bunch of common tool with their
common/very simple and familiar options".)

So again there is no shell programming, just awk (gawk(1)),
sort(1) and tail(1).

  $ awk '{ print length($0) " " $0; }' $file | sort -n | tail -1

> I hope you realize the difference. And that hypothetical
> *everyone* can also learn how to write elisp if they choose
> so. By the way you know what is also like lego blocks?
> Lisp functions :).

True but that Lego is too difficult for a lot of people while
everyone can pipe together three shell tools.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Too fine design granularity leads to numerous macro/function/command existed in Emacs.

[Emanuel Berg via Users list for the GNU Emacs text editor]

Hongyi Zhao wrote:

> It seems Emacs lisp don't have pipe at all.

You can do IPC with Emacs, see/try for example

;;; -*- lexical-binding: t -*-
;;;
;;; this file:
;;;   http://user.it.uu.se/~embe8573/emacs-init/signal.el
;;;   https://dataswamp.org/~incal/emacs-init/signal.el

(defun signal-usr1-f ()
  (interactive)
  (message "got signal: USR1") )

(define-key special-event-map [sigusr1] #'signal-usr1-f)

;; test from Emacs: (signal-process (emacs-pid) 'sigusr1)
;;
;; from zsh: kill -s usr1 $(ps -e | grep emacs | cut -d " " -f 2)

-- 
underground experts united
https://dataswamp.org/~incal

Re: Too fine design granularity leads to numerous macro/function/command existed in Emacs.

[Emanuel Berg via Users list for the GNU Emacs text editor]

Hongyi Zhao wrote:

> It seems Emacs lisp don't have pipe at all.

You can do IPC with Emacs, see/try for example:

;;; -*- lexical-binding: t -*-
;;;
;;; this file:
;;;   http://user.it.uu.se/~embe8573/emacs-init/signal.el
;;;   https://dataswamp.org/~incal/emacs-init/signal.el

(defun signal-usr1-f ()
  (interactive)
  (message "got signal: USR1") )

(define-key special-event-map [sigusr1] #'signal-usr1-f)

;; test:
;;
;; from Emacs:  (signal-process (emacs-pid) 'sigusr1)
;;
;; from zsh: kill -s usr1 $(ps -e | grep emacs | cut -d " " -f 2)

-- 
underground experts united
https://dataswamp.org/~incal

Re: Too fine design granularity leads to numerous macro/function/command existed in Emacs.

[Hongyi Zhao]

On Sat, Aug 14, 2021 at 2:25 AM Emanuel Berg via Users list for the
GNU Emacs text editor <help-gnu-emacs@gnu.org> wrote:
>
> Arthur Miller wrote:
>
> > learn relatively recently, say one year ago, but I find lisp
> > much faster to code than shell. That what you coded there
> > requires knowledge of zshell, awk, sort and tail.
>
> They are just basic building blocks the basic usage of which
> is known to everyone, it is like building Lego or something,
> meanwhile the below code requires both some general
> programming fluency AND some inside knowledge how the
> particulars work in Lisp and even Emacs/Elisp.
>
> Everyone that writes it can also combine a bunch of shell
> tools, but everyone who can combine a bunch of shell tools
> can't write that - many of them can't program at
> all, actually.

So I've heard that "LISP is the language of GOD!". Using LISP
basically means the continuous evolution of pure thought and logic
until the problem is solved.

> (defun sort-lines-length (beg end)
>   (interactive (if (use-region-p)
>                    (list (region-beginning) (region-end))
>                  (list (point-min) (point-max)) ))
>   (save-excursion
>     (save-restriction
>       (narrow-to-region beg end)
>       (goto-char (point-min))
>       (sort-subr nil
>                  #'forward-line
>                  #'end-of-line
>                  nil nil
>                  (lambda (a b) (> (- (cdr a) (car a))
>                                   (- (cdr b) (car b)) ))))))
> (defalias 'sll #'sort-lines-length)
>
> --
> underground experts united
> https://dataswamp.org/~incal

Re: Too fine design granularity leads to numerous macro/function/command existed in Emacs.

[Emanuel Berg via Users list for the GNU Emacs text editor]

Hongyi Zhao wrote:

> So I've heard that "LISP is the language of GOD!".
> Using LISP basically means the continuous evolution of pure
> thought and logic until the problem is solved.

It is the coolest language anyway.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Too fine design granularity leads to numerous macro/function/command existed in Emacs.

[Emanuel Berg via Users list for the GNU Emacs text editor]

> You can do IPC with Emacs, see/try for example [...]

I modernized that just a little bit... a little bit more!
<https://www.youtube.com/watch?v=PnPUKm9ExHo>

;;; -*- lexical-binding: t -*-
;;;
;;; this file:
;;;   http://user.it.uu.se/~embe8573/emacs-init/signal.el
;;;   https://dataswamp.org/~incal/emacs-init/signal.el
;;;
;;; test from Emacs:
;;;   (signal-process (emacs-pid) 'sigusr1)
;;;
;;; test from zsh:
;;;   $ kill -s usr1 $(pidof emacs)

(defun signal-usr1-f ()
  (interactive)
  (message "USR1 signal") )

(define-key special-event-map [sigusr1] #'signal-usr1-f)

-- 
underground experts united
https://dataswamp.org/~incal

Re: Too fine design granularity leads to numerous macro/function/command existed in Emacs.

[Hongyi Zhao]

On Sat, Aug 14, 2021 at 7:43 AM Emanuel Berg via Users list for the
GNU Emacs text editor <help-gnu-emacs@gnu.org> wrote:
> [...]
> Forget about the shell script, it is just a wrapper, an
> interface for convenience. No, what it amounts to is the use
> and combination of 3 tools. (BTW I didn't get the other guy's
> suggestion to work, that which only used awk and nothing else
> - well, that might be better (I agree) but as for how
> difficult it is it is probably *more* difficult than mine
> because his requires some understanding of awk beyond the
> most basic "pile together a bunch of common tool with their
> common/very simple and familiar options".)

It's not difficult for me or maybe some others, just too lazy to think
for a while. After all, we have so many convenient uinx tools and the
excellent (but with performance penalty) pipe method at hand. Anyway,
I've presented a comparison between pure AWK and UNIX toolchains as
follows:

$ time awk 'length > max_length { max_length = length; longest_line =
$0 } END { print longest_line }' american-english-exhaustive
correspond with someone about someone or something correspond with
someone about someone or something

real    0m0.255s
user    0m0.255s
sys    0m0.003s

$ time awk '{ print length($0) " " $0; }' american-english-exhaustive
| sort -n | tail -1| cut -d ' ' -f2-
correspond with someone about someone or something correspond with
someone about someone or something

real    0m1.794s
user    0m1.853s
sys    0m0.053s

Obviously, the former is about 6 times faster than the latter.

> So again there is no shell programming, just awk (gawk(1)),
> sort(1) and tail(1).
>
>   $ awk '{ print length($0) " " $0; }' $file | sort -n | tail -1
>
> > I hope you realize the difference. And that hypothetical
> > *everyone* can also learn how to write elisp if they choose
> > so. By the way you know what is also like lego blocks?
> > Lisp functions :).
>
> True but that Lego is too difficult for a lot of people while
> everyone can pipe together three shell tools.

Hongyi

Re: Too fine design granularity leads to numerous macro/function/command existed in Emacs.

[Emanuel Berg via Users list for the GNU Emacs text editor]

Hongyi Zhao wrote:

> It's not difficult for me or maybe some others, just too
> lazy to think for a while.

A lot of people like to think their lack of [whatever] is
because of laziness, however as for thinking of new reasons to
be lazy, the sure are hard at work. (I don't think that
applies to you but be careful with that word.)

> After all, we have so many
> convenient uinx tools and the excellent (but with
> performance penalty) pipe method at hand. Anyway, I've
> presented a comparison between pure AWK and UNIX toolchains
> as follows:
>
> $ time awk 'length > max_length { max_length = length; longest_line =
> $0 } END { print longest_line }' american-english-exhaustive
> correspond with someone about someone or something correspond with
> someone about someone or something

OK, great, so you got the one-liner to work, what did
you change?

> real    0m0.255s
> user    0m0.255s
> sys    0m0.003s
>
> $ time awk '{ print length($0) " " $0; }' american-english-exhaustive
> | sort -n | tail -1| cut -d ' ' -f2-
> correspond with someone about someone or something correspond with
> someone about someone or something
>
> real    0m1.794s
> user    0m1.853s
> sys    0m0.053s
>
> Obviously, the former is about 6 times faster than the latter.

Cool, try my command as well:

  $ awk '{ print length($0) " " $0; }' $file | sort -n | tail -1

The reason the piped commands are slower are because of the
context-switches so instead of time(1) try this command for
all three:

  $ perf stat sh -c 'COMMAND'

-- 
underground experts united
https://dataswamp.org/~incal

Re: Too fine design granularity leads to numerous macro/function/command existed in Emacs.

[Hongyi Zhao]

On Fri, Aug 13, 2021 at 3:26 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Hongyi Zhao <hongyi.zhao@gmail.com>
> > Date: Fri, 13 Aug 2021 15:12:39 +0800
> >
> >  It's well known that Emacs has already implemented numerous
> > macro/function/command in its current version. But it seems that this
> > is caused by, to a certain degree, the too fine granularity, i.e.,
> > there are many functionally similar macro/function/command are
> > designed separately, for example, `directory-files' and
> > `directory-files-recursively'. The traditional Unix tool, `find', can
> > do all the jobs of the above two functions by adjusting its
> > `-maxdepth' and `-mindepth' arguments.
> >
> > So, I want to know why these macro/function/command in Emacs are
> > developed with such a fine granularity.
>
> They aren't.  You provided just one example, and drew far-reaching
> conclusions from that single example.  If you want to make such a
> general claim, please show more examples of what you consider to be
> "fine granularity".
>
> Specifically about the one example you provided: directory-files is a
> primitive, written in C, so it provides the basic functionality of
> fetching file names from a single directory. The
> directory-files-recursively function is written in Lisp, it builds on
> that primitive and provides extended functionality.

Thank you for letting me know this difference, which I really didn't
notice before. But there's still one question I don't understand:
C is a compiled language, so how does the compilation and invocation
of such C primitives work in Emacs? Are they compiled into libraries
and called in real time?

> This paradigm, of having primitives provide the basics and implement the rest in Lisp on
> top of those primitives, is central to Emacs design.

Thank you for introducing me to this central paradigm.

Hongyi

Re: Too fine design granularity leads to numerous macro/function/command existed in Emacs.

[Hongyi Zhao]

On Sat, Aug 14, 2021 at 8:43 AM Emanuel Berg via Users list for the
GNU Emacs text editor <help-gnu-emacs@gnu.org> wrote:
>
> Hongyi Zhao wrote:
>
> > It's not difficult for me or maybe some others, just too
> > lazy to think for a while.
>
> A lot of people like to think their lack of [whatever] is
> because of laziness, however as for thinking of new reasons to
> be lazy, the sure are hard at work. (I don't think that
> applies to you but be careful with that word.)
>
> > After all, we have so many
> > convenient uinx tools and the excellent (but with
> > performance penalty) pipe method at hand. Anyway, I've
> > presented a comparison between pure AWK and UNIX toolchains
> > as follows:
> >
> > $ time awk 'length > max_length { max_length = length; longest_line =
> > $0 } END { print longest_line }' american-english-exhaustive
> > correspond with someone about someone or something correspond with
> > someone about someone or something
>
> OK, great, so you got the one-liner to work, what did
> you change?

Another reason I prefer UNIX toolchains is that they have been tried
and tested by many people for many years, and using your own temporary
code may expose you to greater risk of errors. Though I have to bear
some performance losses, but compared with today's fast computers,
this is insignificant in many test scenarios.

> > real    0m0.255s
> > user    0m0.255s
> > sys    0m0.003s
> >
> > $ time awk '{ print length($0) " " $0; }' american-english-exhaustive
> > | sort -n | tail -1| cut -d ' ' -f2-
> > correspond with someone about someone or something correspond with
> > someone about someone or something
> >
> > real    0m1.794s
> > user    0m1.853s
> > sys    0m0.053s
> >
> > Obviously, the former is about 6 times faster than the latter.
>
> Cool, try my command as well:
>
>   $ awk '{ print length($0) " " $0; }' $file | sort -n | tail -1
>
> The reason the piped commands are slower are because of the
> context-switches so instead of time(1) try this command for
> all three:
>
>   $ perf stat sh -c 'COMMAND'

See below, but I'm not familiar with this tool, so don't understand
his output very well:

$ sudo perf stat sh -c "awk '{ print length($0) " " $0; }'
american-english-exhaustive  | sort -n | tail -1| cut -d ' ' -f2-"
bash; }' american-english-exhaustive  | sort -n | tail -1| cut -d ' '
-f2-: 1: Syntax error: Unterminated quoted string

 Performance counter stats for 'sh -c awk '{ print length(bash)
bash; }' american-english-exhaustive  | sort -n | tail -1| cut -d ' '
-f2-':

              0.85 msec task-clock                #    0.661 CPUs
utilized
                 0      context-switches          #    0.000 K/sec
                 0      cpu-migrations            #    0.000 K/sec
                62      page-faults               #    0.073 M/sec
         1,012,717      cycles                    #    1.193 GHz
           903,160      instructions              #    0.89  insn per
cycle
           182,165      branches                  #  214.508 M/sec
             8,022      branch-misses             #    4.40% of all
branches

       0.001285545 seconds time elapsed

       0.001390000 seconds user
       0.000000000 seconds sys

Hongyi

Re: Too fine design granularity leads to numerous macro/function/command existed in Emacs.

[Arthur Miller]

Emanuel Berg via Users list for the GNU Emacs text editor
<help-gnu-emacs@gnu.org> writes:

> Arthur Miller wrote:
>
>> You don't believe that programming shell script requires at
>> least some programming fluency AND some inside knowledge hot
>> the particular tools works, their options, streams etc?
>> It just comes out of itself? :)
>
> This isn't a script really but a shell function and this
> function in particular is just a wrapper so no, you don't need
> to know much more than the chaining of GNU tools there...
>
>>> Everyone that writes it can also combine a bunch of shell
>>> tools, but everyone who can combine a bunch of shell tools
>>> can't write that - many of them can't program at
>>> all, actually.
>>
>> I don't think I or anyone else claimed that "everyone who
>> writes it" (it = shellscripts?)
>
> Putting together a bunch of shell tools with pipes in between
> to do some parsing, that's much more easy than to write Elisp.
>
> Shell tools are on the user level, not on the
> programmer/engineer level.
>
>> By the way, you do realize that we write this in context of
>> Emacs and Elisp and not some random unix tooling context?
>
> Yes but no, what?
>
>> Yes, unix command line tools and piping are like lego blocks, that
>> was by design, it does not mean that *everyone* can write shell
>> scripts. Everyone *can learn* to write shell scripts, but not everyone
>> can write shellscripts.
>
> Forget about the shell script, it is just a wrapper, an
> interface for convenience. No, what it amounts to is the use
> and combination of 3 tools. (BTW I didn't get the other guy's
> suggestion to work, that which only used awk and nothing else
> - well, that might be better (I agree) but as for how
> difficult it is it is probably *more* difficult than mine
> because his requires some understanding of awk beyond the
> most basic "pile together a bunch of common tool with their
> common/very simple and familiar options".)
>
> So again there is no shell programming, just awk (gawk(1)),
> sort(1) and tail(1).
>
>   $ awk '{ print length($0) " " $0; }' $file | sort -n | tail -1
>
>> I hope you realize the difference. And that hypothetical
>> *everyone* can also learn how to write elisp if they choose
>> so. By the way you know what is also like lego blocks?
>> Lisp functions :).
>
> True but that Lego is too difficult for a lot of people while
> everyone can pipe together three shell tools.

Ehhhh .... good night.

Re: Too fine design granularity leads to numerous macro/function/command existed in Emacs.

[Emanuel Berg via Users list for the GNU Emacs text editor]

Hongyi Zhao wrote:

> Another reason I prefer UNIX toolchains is that they have
> been tried and tested by many people for many years, and
> using your own temporary code may expose you to greater risk
> of errors.

Yes maybe but why do it if the toolchain already does it?

OTOH if you do it why do it if it doesn't work?

So one must assume it works, if the programs don't work
somewhere it doesn't matter what you say or what discussion
that's always gonna be a problem.

> Though I have to bear some performance losses, but compared
> with today's fast computers, this is insignificant in many
> test scenarios.

Yes, but here it was quite a big difference and you can also
se on the command that it is actually better, if you have
measured advantage and observed/theoretical than that means
the better one is ... better!

But we can save the day and my function by finding a new use
case for it - see below...

# thanks to gmane.emacs.help
longest-line () {
    local f=$1
    awk 'length > max_length { max_length = length; longest_line = $0 } END { print longest_line }' $f
}

longest-lines () {
    local f=$1
    local lines=${2:-10}
    awk '{ print length($0) " " $0; }' $f | sort -n -r | head -n $lines
}

$ longest-lines /usr/share/dict/american-english-insane 5
60 Llanfairpwllgwyngyllgogerychwyrndrobwllllantysiliogogogoch's
58 Llanfairpwllgwyngyllgogerychwyrndrobwllllantysiliogogogoch
45 pneumonoultramicroscopicsilicovolcanoconiosis
45 pneumonoultramicroscopicsilicovolcanoconioses
34 supercalifragilisticexpialidocious

https://dataswamp.org/~incal/conf/.zsh/text
https://dataswamp.org/~incal/emacs-init/sort-incal.el

-- 
underground experts united
https://dataswamp.org/~incal

Re: Too fine design granularity leads to numerous macro/function/command existed in Emacs.

[Eli Zaretskii]

> From: Hongyi Zhao <hongyi.zhao@gmail.com>
> Date: Sat, 14 Aug 2021 08:46:27 +0800
> Cc: help-gnu-emacs <help-gnu-emacs@gnu.org>
> 
> > Specifically about the one example you provided: directory-files is a
> > primitive, written in C, so it provides the basic functionality of
> > fetching file names from a single directory. The
> > directory-files-recursively function is written in Lisp, it builds on
> > that primitive and provides extended functionality.
> 
> Thank you for letting me know this difference, which I really didn't
> notice before. But there's still one question I don't understand:
> C is a compiled language, so how does the compilation and invocation
> of such C primitives work in Emacs? Are they compiled into libraries
> and called in real time?

The low-level infrastructure of the Lisp machine in Emacs is also
written in C, and that includes the primitives used to call Lisp
functions.  Those primitives have a special way of calling other
primitives written in C.  Basically, a primitive written in C is
wrapped in a special 'subr' object that tells Emacs about this, and
then such primitives are called via the normal C mechanisms of calling
functions.

For more details, I suggest to study the node "Writing Emacs
Primitives" in the ELisp manual.

Re: Too fine design granularity leads to numerous macro/function/command existed in Emacs.

[Hongyi Zhao]

On Sat, Aug 14, 2021 at 2:19 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Hongyi Zhao <hongyi.zhao@gmail.com>
> > Date: Sat, 14 Aug 2021 08:46:27 +0800
> > Cc: help-gnu-emacs <help-gnu-emacs@gnu.org>
> >
> > > Specifically about the one example you provided: directory-files is a
> > > primitive, written in C, so it provides the basic functionality of
> > > fetching file names from a single directory. The
> > > directory-files-recursively function is written in Lisp, it builds on
> > > that primitive and provides extended functionality.
> >
> > Thank you for letting me know this difference, which I really didn't
> > notice before. But there's still one question I don't understand:
> > C is a compiled language, so how does the compilation and invocation
> > of such C primitives work in Emacs? Are they compiled into libraries
> > and called in real time?
>
> The low-level infrastructure of the Lisp machine

IMO, by saying ``Lisp machine'', it always means the operating system
is written in lisp, but this is obviously not the case of Emacs.

> in Emacs is also written in C, and that includes the primitives used to call Lisp
> functions.  Those primitives have a special way of calling other
> primitives written in C.  Basically, a primitive written in C is
> wrapped in a special 'subr' object that tells Emacs about this, and
> then such primitives are called via the normal C mechanisms of calling
> functions.
>
> For more details, I suggest to study the node "Writing Emacs
> Primitives" in the ELisp manual.

Thank you very much for your valuable comments.

Hongyi

Re: Too fine design granularity leads to numerous macro/function/command existed in Emacs.

[Eli Zaretskii]

> From: Hongyi Zhao <hongyi.zhao@gmail.com>
> Date: Sat, 14 Aug 2021 14:29:38 +0800
> Cc: help-gnu-emacs <help-gnu-emacs@gnu.org>
> 
> > The low-level infrastructure of the Lisp machine
> 
> IMO, by saying ``Lisp machine'', it always means the operating system
> is written in lisp, but this is obviously not the case of Emacs.

No, I meant the Lisp machine that is part of Emacs core.  IOW, the
Emacs Lisp interpreter and bytecode-running code.

Re: Too fine design granularity leads to numerous macro/function/command existed in Emacs.

[Hongyi Zhao]

On Sat, Aug 14, 2021 at 3:19 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Hongyi Zhao <hongyi.zhao@gmail.com>
> > Date: Sat, 14 Aug 2021 14:29:38 +0800
> > Cc: help-gnu-emacs <help-gnu-emacs@gnu.org>
> >
> > > The low-level infrastructure of the Lisp machine
> >
> > IMO, by saying ``Lisp machine'', it always means the operating system
> > is written in lisp, but this is obviously not the case of Emacs.
>
> No, I meant the Lisp machine that is part of Emacs core.  IOW, the
> Emacs Lisp interpreter and bytecode-running code.

Can these components be called a ``Lisp machine''? I mean, if without
the OS it runs on, it can also be called a ``machine''?

Hongyi

Re: Too fine design granularity leads to numerous macro/function/command existed in Emacs.

[Eli Zaretskii]

> From: Hongyi Zhao <hongyi.zhao@gmail.com>
> Date: Sat, 14 Aug 2021 15:25:36 +0800
> Cc: help-gnu-emacs <help-gnu-emacs@gnu.org>
> 
> On Sat, Aug 14, 2021 at 3:19 PM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > > From: Hongyi Zhao <hongyi.zhao@gmail.com>
> > > Date: Sat, 14 Aug 2021 14:29:38 +0800
> > > Cc: help-gnu-emacs <help-gnu-emacs@gnu.org>
> > >
> > > > The low-level infrastructure of the Lisp machine
> > >
> > > IMO, by saying ``Lisp machine'', it always means the operating system
> > > is written in lisp, but this is obviously not the case of Emacs.
> >
> > No, I meant the Lisp machine that is part of Emacs core.  IOW, the
> > Emacs Lisp interpreter and bytecode-running code.
> 
> Can these components be called a ``Lisp machine''? I mean, if without
> the OS it runs on, it can also be called a ``machine''?

Well, that's what I call them.