Re: What's missing in ELisp that makes, people want to use cl-lib?

Re: What's missing in ELisp that makes, people want to use cl-lib?

[Pedro Andres Aranda Gutierrez]

[-- Attachment #1: Type: text/plain, Size: 877 bytes --]

My own mea culpa:

I've written a couple of things using cl-lib features in the past, mainly
because I came across
the cl-lib documentation and found ways of doing things that were also
present in "regular Emacs LISP". Mainly a backend for company interacting
with different intermediate LaTEX files before the texlab LSP server was
"good enough" for me. Looking at them now, I blush at the fact that I could
have written them with seq-filter & friends.

So, why did I go for cl-lib?

Honest answer: it looked fancy enough to make me try it and I'm always
trying to learn new things to keep my neurons active

Mea culpa, /PA
-- 
Fragen sind nicht da, um beantwortet zu werden,
Fragen sind da um gestellt zu werden
Georg Kreisler

Headaches with a Juju log:
unit-basic-16: 09:17:36 WARNING juju.worker.uniter.operation we should run
a leader-deposed hook here, but we can't yet

[-- Attachment #2: Type: text/html, Size: 1307 bytes --]

Re: What's missing in ELisp that makes, people want to use cl-lib?

[João Távora]

On Fri, Nov 3, 2023 at 8:22 AM Pedro Andres Aranda Gutierrez
<paaguti@gmail.com> wrote:

> Looking at them now, I blush at the fact that I could have written them with seq-filter & friends.

If anything or anyone could be blushing it's seq-filter for
reinventing a ~25 year old perfectly good wheel called
cl-remove-if.

Note it I said it could, not that it should ;-)

João

Re: What's missing in ELisp that makes, people want to use cl-lib?

[Pedro Andres Aranda Gutierrez]

[-- Attachment #1: Type: text/plain, Size: 772 bytes --]

On Fri, 3 Nov 2023 at 10:27, João Távora <joaotavora@gmail.com> wrote:

> On Fri, Nov 3, 2023 at 8:22 AM Pedro Andres Aranda Gutierrez
> <paaguti@gmail.com> wrote:
>
> > Looking at them now, I blush at the fact that I could have written them
> with seq-filter & friends.
>
> If anything or anyone could be blushing it's seq-filter for
> reinventing a ~25 year old perfectly good wheel called
> cl-remove-if.
>
> Note it I said it could, not that it should ;-)
>
> João
>

<LoL> ;-)
/PA

-- 
Fragen sind nicht da, um beantwortet zu werden,
Fragen sind da um gestellt zu werden
Georg Kreisler

Headaches with a Juju log:
unit-basic-16: 09:17:36 WARNING juju.worker.uniter.operation we should run
a leader-deposed hook here, but we can't yet

[-- Attachment #2: Type: text/html, Size: 1425 bytes --]

Re: What's missing in ELisp that makes, people want to use cl-lib?

[Gerd Möllmann]

Pedro Andres Aranda Gutierrez <paaguti@gmail.com> writes:

> So, why did I go for cl-lib? 
>
> Honest answer: it looked fancy enough to make me try it and I'm always
> trying to learn new things to keep my neurons active
>
> Mea culpa, /PA

The reason I will not use seq.el or map.el in the forseeable future is
quite simple: I haven't ever needed an abstraction over sequence types
using generic functions, and I never have CPU cycles to give away for
free.

Re: What's missing in ELisp that makes, people want to use cl-lib?

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: emacs-devel <emacs-devel@gnu.org>
> Date: Fri, 03 Nov 2023 14:37:14 +0100
> 
> The reason I will not use seq.el or map.el in the forseeable future is
> quite simple: I haven't ever needed an abstraction over sequence types
> using generic functions, and I never have CPU cycles to give away for
> free.

What prevents us from implementing in seq.el methods for specific
types of sequences?  AFAIU, the only reason we didn't is because no
one has yet came up with important situations where the generics in
seq.el are not performant enough.

Re: What's missing in ELisp that makes, people want to use cl-lib?

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>> Cc: emacs-devel <emacs-devel@gnu.org>
>> Date: Fri, 03 Nov 2023 14:37:14 +0100
>> 
>> The reason I will not use seq.el or map.el in the forseeable future is
>> quite simple: I haven't ever needed an abstraction over sequence types
>> using generic functions, and I never have CPU cycles to give away for
>> free.
>
> What prevents us from implementing in seq.el methods for specific
> types of sequences?

I won't :-).

> AFAIU, the only reason we didn't is because no
> one has yet came up with important situations where the generics in
> seq.el are not performant enough.

I don't get that line of reasoning, sorry. The Elisp manual kind of
advertises its abstraction over sequence type, you kind of say it is
isn't needed because it's kind of fast enough, usually.

What I'm missing is a positive reason why to use it. I don't see one.

Re: What's missing in ELisp that makes, people want to use cl-lib?

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: paaguti@gmail.com,  emacs-devel@gnu.org
> Date: Fri, 03 Nov 2023 16:08:09 +0100
> 
> What I'm missing is a positive reason why to use it. I don't see one.

It's preloaded.  I think this is a good reason for using it.  The fact
that it's preloaded should also say something about how the Emacs
maintainers see seq.el and its future.

Re: What's missing in ELisp that makes, people want to use cl-lib?

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>> Cc: paaguti@gmail.com,  emacs-devel@gnu.org
>> Date: Fri, 03 Nov 2023 16:08:09 +0100
>> 
>> What I'm missing is a positive reason why to use it. I don't see one.
>
> It's preloaded.  I think this is a good reason for using it.

Sparing 10^-7 seconds, when cl-lib is already loaded anyway?

> The fact
> that it's preloaded should also say something about how the Emacs
> maintainers see seq.el and its future.

I don't want to hurt anyone's feelings, but that's kind of a weak
argument. And an entirely non-technical one.

Re: What's missing in ELisp that makes, people want to use cl-lib?

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: paaguti@gmail.com,  emacs-devel@gnu.org
> Date: Fri, 03 Nov 2023 16:30:47 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> >> Cc: paaguti@gmail.com,  emacs-devel@gnu.org
> >> Date: Fri, 03 Nov 2023 16:08:09 +0100
> >> 
> >> What I'm missing is a positive reason why to use it. I don't see one.
> >
> > It's preloaded.  I think this is a good reason for using it.
> 
> Sparing 10^-7 seconds, when cl-lib is already loaded anyway?

These are not the only situations in Emacs.  For example, in preloaded
code.

> > The fact
> > that it's preloaded should also say something about how the Emacs
> > maintainers see seq.el and its future.
> 
> I don't want to hurt anyone's feelings, but that's kind of a weak
> argument. And an entirely non-technical one.

When we decided to preload it, the reasons were technical.  So I don't
see why the above is not a technical argument.

Re: What's missing in ELisp that makes, people want to use cl-lib?

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>> Cc: paaguti@gmail.com,  emacs-devel@gnu.org
>> Date: Fri, 03 Nov 2023 16:30:47 +0100
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>> >> Cc: paaguti@gmail.com,  emacs-devel@gnu.org
>> >> Date: Fri, 03 Nov 2023 16:08:09 +0100
>> >> 
>> >> What I'm missing is a positive reason why to use it. I don't see one.
>> >
>> > It's preloaded.  I think this is a good reason for using it.
>> 
>> Sparing 10^-7 seconds, when cl-lib is already loaded anyway?
>
> These are not the only situations in Emacs.  For example, in preloaded
> code.
>
>> > The fact
>> > that it's preloaded should also say something about how the Emacs
>> > maintainers see seq.el and its future.
>> 
>> I don't want to hurt anyone's feelings, but that's kind of a weak
>> argument. And an entirely non-technical one.
>
> When we decided to preload it, the reasons were technical.  So I don't
> see why the above is not a technical argument.

I'm sorry, but I think this leads nowhere. 

seq.el and the complexity of Emacs Lisp.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I looked at seq.el to try to get an idea what those functions do.  It
was difficult to get this from the doc strings, because it wasn't
explicitly stated which ones are for users and which ones are
internal.  Eventually I got somewhat of an idea. 

As for understanding the code itself, I basically couldn't the first
thing I saw used a macro `pcase-defmacro', and I have no idea what
that does.  (I generally find `pcase' cryptic and unclear, even if I
have read its documentation just before.)

Ultimately I did figure out what some of the user-facing features were
and understand what they did.  They seem clean enough -- both simpler
and more natural than the cl- sequence functions.

I expect that they are slower at run-time, because each seq- function
needs to do the generic dispatch for each call.  Is that correct?

Maybe we could speed this up by defining a function that's a no-op but
says that its argument is a list.  Using that function in the sequence
argument to a seq- function could tell all compilers to do the generic
dispatch at compile time.  It could be called `force-list' or `!list'.
There could also be `force-vector' too if it's worth the effort.

However, as regards the overall size or complexity of Emacs Lisp, it's
not a matter of comparing the seq- functions with the cl-lib functions
and the traditoinal Emacs Lisp functions and asking which is best.
That's the question you would face when deciding how to write a piece
of code.  However, as regards the language itself, what we have is
both the seq- fuctions and the traditoinal Emacs Lisp functions, with
some developers pushing the cl-lib functions into the combination.

If that continues, the complexity of Emacs Lisp will be the sum of the
three different ways or doing these things.  That is a big step for
the worse from where we started out.  The cost of these triplicated
language features affects both execution (memory usage) and human
thought (needing to know them all).

Could we replace all the cl-lib sequence function calls with seq-
calls, in core and GNU ELPA code?  Seq is simpler and cleaner, so that
would be an improvement.  We could keep cl-lib permanently for
compatibility for external code, but it would not need to be loaded
(into Emacs or your brain) very often.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: seq.el and the complexity of Emacs Lisp.

[Philip Kaludercic]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> I looked at seq.el to try to get an idea what those functions do.  It
> was difficult to get this from the doc strings, because it wasn't
> explicitly stated which ones are for users and which ones are
> internal.  Eventually I got somewhat of an idea. 
>

[...]

> I expect that they are slower at run-time, because each seq- function
> needs to do the generic dispatch for each call.  Is that correct?

Yes, I believe this has been empirically verified on multiple occasions.

[...]

> Could we replace all the cl-lib sequence function calls with seq-
> calls, in core and GNU ELPA code?  Seq is simpler and cleaner, so that
> would be an improvement.  We could keep cl-lib permanently for
> compatibility for external code, but it would not need to be loaded
> (into Emacs or your brain) very often.

Cl-lib would still be used, because it provides more than just functions
to operate on sequences.  There are a number of macros (cl-flet,
cl-loop, cl-incf, ...) and functions (cl-map accepting multiple lists,
cl-subst) that are frequently used and to my recollection to not yet
have analogous replacements in seq.

Re: seq.el and the complexity of Emacs Lisp.

[Gerd Möllmann]

Richard Stallman <rms@gnu.org> writes:

Can't say much about what you write before this. I wasn't there when
seq/map, pcase, or other stuff, was added, but...

> Could we replace all the cl-lib sequence function calls with seq-
> calls, in core and GNU ELPA code?  Seq is simpler and cleaner, so that
> would be an improvement.  We could keep cl-lib permanently for
> compatibility for external code, but it would not need to be loaded
> (into Emacs or your brain) very often.

...you are loosing me here. I'm still wondering what seq/map are good
for in the first place. Considering something "cleaner" is just a
feeling, isn't it? A feeling I don't share.

Re: seq.el and the complexity of Emacs Lisp.

[João Távora]

On Mon, Nov 6, 2023 at 8:54 AM Gerd Möllmann <gerd.moellmann@gmail.com> wrote:
>
> Richard Stallman <rms@gnu.org> writes:
>
> Can't say much about what you write before this. I wasn't there when
> seq/map, pcase, or other stuff, was added, but...
>
> > Could we replace all the cl-lib sequence function calls with seq-
> > calls, in core and GNU ELPA code?  Seq is simpler and cleaner, so that
> > would be an improvement.  We could keep cl-lib permanently for
> > compatibility for external code, but it would not need to be loaded
> > (into Emacs or your brain) very often.
>
> ...you are losing me here. I'm still wondering what seq/map are good
> for in the first place.

Presumably, it's good for handling custom sequence types, such as lazy
lists.  That's a worthy goal, very worthy even, but this polymorphism comes
at a runtime cost, of course.  And worthy as it may be, there's no use
of that feature in the Emacs tree (as far as I can tell from quick greps),
so it can't really be the "technical reason" for why seq.el is preloaded
and or recommended.

> Considering something "cleaner" is just a
> feeling, isn't it? A feeling I don't share.

Absolutely.  I also think "cleaner" is a very weak argument.

Anyway, I think this discussion could use some data:

(require 'cl-lib)

(defun bench-seq-some (seq)
  (seq-some #'identity seq))

(defun bench-cl-some (seq)
  (cl-some #'identity seq))

(defun bench-cl-loop-list (l)
  ;; checks for some types of improper lists
  (cl-loop for e in l thereis (identity e)))

(defun bench-cl-loop-vec (v)
  (cl-loop for e across v thereis (identity e)))


(when nil
  ;; Small lists
  (let ((l (list nil nil nil nil t)))
    (benchmark-run 1000000 (bench-cl-some l))) ;; (0.179339 0 0.0)

  (let ((l (list nil nil nil nil t)))
    (benchmark-run 1000000 (bench-seq-some l))) ;; (2.726298 19
0.7982190000000031)

  (let ((l (list nil nil nil nil t)))
    (benchmark-run 1000000 (bench-cl-loop-list l))) ;; (0.29131 0 0.0)

  ;; Big lists
  (let ((l (make-list 10000000 nil)))
    (benchmark-run 1 (bench-cl-some l))) ;; (0.142612 0 0.0)

  (let ((l (make-list 10000000 nil)))
    (benchmark-run 1 (bench-seq-some l))) ;; (0.379376 0 0.0)

  (let ((l (make-list 10000000 nil)))
    (benchmark-run 1 (bench-cl-loop-list l))) ;; (0.292876 0 0.0)

  ;; Small vectors
  (let ((v (vector nil nil nil nil t)))
    (benchmark-run 1000000 (bench-cl-some l))) ;; (4.058951 53
1.6585750000000008)

  (let ((v (vector nil nil nil nil t)))
    (benchmark-run 1000000 (bench-seq-some l))) ;; (2.61798 25
0.7980089999999986)

  (let ((v (vector nil nil nil nil t)))
    (benchmark-run 1000000 (bench-cl-loop-vec v))) ;; (0.306206 0 0.0)

  ;; Big vectors
  (let ((v (make-vector 10000000 nil)))
    (benchmark-run 1 (bench-cl-some v))) ;; (1.910587 14
1.2171730000000025) (DREADFUL, EASY TO FIX?)

  (let ((v (make-vector 10000000 nil)))
    (benchmark-run 1 (bench-seq-some v))) ;; (0.33895600000000004 0 0.0)

  (let ((v (make-vector 10000000 nil)))
    (benchmark-run 1 (bench-cl-loop-vec v))) ;; (0.325975 0 0.0)


  )

Re: seq.el and the complexity of Emacs Lisp.

[Jim Porter]

On 11/7/2023 2:24 AM, João Távora wrote:
> Presumably, it's good for handling custom sequence types, such as lazy
> lists.  That's a worthy goal, very worthy even, but this polymorphism comes
> at a runtime cost, of course.  And worthy as it may be, there's no use
> of that feature in the Emacs tree (as far as I can tell from quick greps),
> so it can't really be the "technical reason" for why seq.el is preloaded
> and or recommended.

I'd say this feature of seq is already in use, albeit not heavy use: 
Eshell uses some 'seq-FOO' functions in lisp/eshell/esh-var.el 
(specifically 'eshell-index-value') so that Eshell users can index into 
*any* sequence type, no matter what that type actually is.

(Careful observers will also note that 'eshell-index-value' underscores 
one of seq.el's limitations: it doesn't work with rings, so Eshell needs 
to special-case that.)

Overall, I think seq.el's abstractions are very useful, but perhaps they 
shouldn't be the only way to perform all the various algorithms they 
support. That is, seq.el is great when you want to accept any sequence 
type, but not so great if you know for sure that you have (say) a list. 
Some sort of type-annotation like Richard proposed could resolve this 
issue though: it would essentially allow us to turn what's usually a 
runtime dispatch into a compile-time dispatch. That's probably useful 
for all generic functions; not just seq.el.

Re: seq.el and the complexity of Emacs Lisp.

[João Távora]

On Thu, Nov 9, 2023 at 9:02 PM Jim Porter <jporterbugs@gmail.com> wrote:
>
> On 11/7/2023 2:24 AM, João Távora wrote:
> > Presumably, it's good for handling custom sequence types, such as lazy
> > lists.  That's a worthy goal, very worthy even, but this polymorphism comes
> > at a runtime cost, of course.  And worthy as it may be, there's no use
> > of that feature in the Emacs tree (as far as I can tell from quick greps),
> > so it can't really be the "technical reason" for why seq.el is preloaded
> > and or recommended.
>
> I'd say this feature of seq is already in use, albeit not heavy use:
> Eshell uses some 'seq-FOO' functions in lisp/eshell/esh-var.el
> (specifically 'eshell-index-value') so that Eshell users can index into
> *any* sequence type, no matter what that type actually is.

Sure that's what seq.el does, but I see no definition of new sequence
types in the core.  Whatever those "whatever sequence" types are,
they have to be defined somewhere with implementations for a bunch of
generic functions and I see none of that in the Emacs tree (again, from
my quick greps).  Examples could be an implementation of lazy sequences
for example. So it stands to reason that there's no (obvious) reason
to preload seq.el,

Also I'm not an eshell uses and I don't understand how eshell users
benefit from this, i.e. why using any other sequence-manipulation
library, such as cl-lib.el, in esh-vars.el wouldn't  do the job just
as well for them.  Maybe you could explain?

João

Re: seq.el and the complexity of Emacs Lisp.

[Jim Porter]

On 11/9/2023 1:20 PM, João Távora wrote:
> Examples could be an implementation of lazy sequences
> for example. So it stands to reason that there's no (obvious) reason
> to preload seq.el,

In fact, I was considering adding support to seq.el for generators, 
which we could consider a kind of lazy sequence. I'm not entirely sure I 
*need* it yet, but it could make a few things in Eshell easier.

For example, Eshell has a for loop. It'd be nice if you could iterate 
over an Elisp generator, or even just repeatedly call a regular function 
that returns some new value each time[1]. If seq.el supported 
generators, I *think* Eshell could just call 'seq-map' (really, we'd 
want 'seq-mapc', but that's another story), and the code would do the 
right thing. This would make it easy for Eshell for loops to iterate 
over a range of integers without having to pre-build a list of all those 
integers.

I haven't done this yet though, so it's possible there's something else 
that would throw a wrench into this. (In particular, Eshell's internals 
are a bit finicky at times, and I'm not sure how they'd interact with 
generators.)

> Also I'm not an eshell uses and I don't understand how eshell users
> benefit from this, i.e. why using any other sequence-manipulation
> library, such as cl-lib.el, in esh-vars.el wouldn't  do the job just
> as well for them.  Maybe you could explain?

Currently, Eshell's use of seq.el is pretty basic, but 'seq-elt' in 
particular is nice since it automatically decides whether to use 'nth' 
or 'aref' as appropriate. I don't see any similar cl-lib function that 
would do this. This lets an Eshell user use a subscript like "$foo[1]" 
for anything that would make sense to index. (Though again, we need to 
special-case rings.)

[1] Though this would probably necessitate a "break" statement to exit 
the loop.

Re: seq.el and the complexity of Emacs Lisp.

[Jim Porter]

On 11/9/2023 3:49 PM, Jim Porter wrote:
> I *think* Eshell could just call 'seq-map' (really, we'd want 
> 'seq-mapc', but that's another story)

Er, that's 'seq-do', actually.

Re: seq.el and the complexity of Emacs Lisp.

[João Távora]

On Thu, Nov 9, 2023 at 11:49 PM Jim Porter <jporterbugs@gmail.com> wrote:
>
> On 11/9/2023 1:20 PM, João Távora wrote:
> > Examples could be an implementation of lazy sequences
> > for example. So it stands to reason that there's no (obvious) reason
> > to preload seq.el,
>
> In fact, I was considering adding support to seq.el for generators,
> which we could consider a kind of lazy sequence. I'm not entirely sure I
> *need* it yet, but it could make a few things in Eshell easier.

Have fun!  Interested to see the result.

> Currently, Eshell's use of seq.el is pretty basic, but 'seq-elt' in
> particular is nice since it automatically decides whether to use 'nth'
> or 'aref' as appropriate. I don't see any similar cl-lib function that

Right, probably because in both Common Lisp and Elisp that function
is called ELT and is a very old function.  So there was no need to
add it to cl-lib.el.  keep in mind cl-lib.el comes from cl.el which
didn't have the "cl-" prefix.  So Common Lisp functions that were
exactly the same in Emacs Lisp obviously weren't in cl.el, so that's
why there's no cl-elt.

João

Re: seq.el and the complexity of Emacs Lisp.

[Jim Porter]

On 11/9/2023 6:31 PM, João Távora wrote:
> Right, probably because in both Common Lisp and Elisp that function
> is called ELT and is a very old function.  So there was no need to
> add it to cl-lib.el.  keep in mind cl-lib.el comes from cl.el which
> didn't have the "cl-" prefix.  So Common Lisp functions that were
> exactly the same in Emacs Lisp obviously weren't in cl.el, so that's
> why there's no cl-elt.

I suppose that's what you get when you just try to learn the sum total 
of (Emacs) Lisp via osmosis. :) Now I know about a new function, though.

Re: seq.el and the complexity of Emacs Lisp.

[Alan Mackenzie]

Hello, Jim.

On Thu, Nov 09, 2023 at 15:49:01 -0800, Jim Porter wrote:
> On 11/9/2023 1:20 PM, João Távora wrote:
> > Examples could be an implementation of lazy sequences
> > for example. So it stands to reason that there's no (obvious) reason
> > to preload seq.el,

> In fact, I was considering adding support to seq.el for generators, 
> which we could consider a kind of lazy sequence. I'm not entirely sure I 
> *need* it yet, but it could make a few things in Eshell easier.

Please don't.  This would be yet one more unnecessary complicated
abstraction for which future maintainers would curse you for introducing.
I don't know what a "generator" is, exactly, but it would likely involve
a plethora of backticks, funcalls, generic functions, and the like, all
things which make debugging difficult, if not very difficult.

The Subject: line of this thread mentions complexity.  This is something
we should be trying to reduce in Emacs, not increase.

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: seq.el and the complexity of Emacs Lisp.

[Eli Zaretskii]

> Date: Fri, 10 Nov 2023 10:03:48 +0000
> Cc: João Távora <joaotavora@gmail.com>,
>  Gerd Möllmann <gerd.moellmann@gmail.com>,
>  Richard Stallman <rms@gnu.org>, paaguti@gmail.com, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > In fact, I was considering adding support to seq.el for generators, 
> > which we could consider a kind of lazy sequence. I'm not entirely sure I 
> > *need* it yet, but it could make a few things in Eshell easier.
> 
> Please don't.  This would be yet one more unnecessary complicated
> abstraction for which future maintainers would curse you for introducing.
> I don't know what a "generator" is, exactly, but it would likely involve
> a plethora of backticks, funcalls, generic functions, and the like, all
> things which make debugging difficult, if not very difficult.
> 
> The Subject: line of this thread mentions complexity.  This is something
> we should be trying to reduce in Emacs, not increase.

IMSNHO, we should see the code, or at least its prototype, before
passing judgment on whether we like it, let alone whether we will
"curse" the author.  It sounds strange to me to read that you don't
know what a "generator" is, but you already know that you don't like
it because it will involve plethora of stuff you don't want to see in
Emacs.

(Btw, if you dislike backticks so much, why is CC Mode so full of
them?)

To Jim: if you think about extending seq.el, please also think whether
the extensions must be preloaded, or can be on a separate file, say,
seq-x.el, where the extensions will be autoloaded when needed.

Re: seq.el and the complexity of Emacs Lisp.

[Michael Heerdegen]

João Távora <joaotavora@gmail.com> writes:

> Sure that's what seq.el does, but I see no definition of new sequence
> types in the core.  Whatever those "whatever sequence" types are,
> they have to be defined somewhere with implementations for a bunch of
> generic functions and I see none of that in the Emacs tree (again, from
> my quick greps).  Examples could be an implementation of lazy sequences
> for example.

We have lazy sequences in Gnu Elpa: stream.el.

Michael.

Re: seq.el and the complexity of Emacs Lisp.

[João Távora]

On Fri, Nov 10, 2023 at 10:13 AM Michael Heerdegen
<michael_heerdegen@web.de> wrote:
>
> João Távora <joaotavora@gmail.com> writes:
>
> > Sure that's what seq.el does, but I see no definition of new sequence
> > types in the core.  Whatever those "whatever sequence" types are,
> > they have to be defined somewhere with implementations for a bunch of
> > generic functions and I see none of that in the Emacs tree (again, from
> > my quick greps).  Examples could be an implementation of lazy sequences
> > for example.
>
> We have lazy sequences in Gnu Elpa: stream.el.

Right, and they're using seq.el in the sense I described
if I'm not mistaken.

Meaning if such object created by stream.el is somehow given
to a library in core what uses seq-* functions, that
library will do the right thing polymorphically I suppose.

João

Re: seq.el and the complexity of Emacs Lisp.

[Michael Heerdegen]

João Távora <joaotavora@gmail.com> writes:

> > We have lazy sequences in Gnu Elpa: stream.el.
>
> Right, and they're using seq.el in the sense I described
> if I'm not mistaken.
>
> Meaning if such object created by stream.el is somehow given
> to a library in core what uses seq-* functions, that
> library will do the right thing polymorphically I suppose.

Indeed.  These streams are fully-fledged sequences in the sense of
seq.el, that was one of the basic design ideas.

Michael.

Re: seq.el and the complexity of Emacs Lisp.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > ...you are loosing me here. I'm still wondering what seq/map are good
  > for in the first place. Considering something "cleaner" is just a
  > feeling, isn't it? A feeling I don't share.

I don't concretely know, as I have not used them myself.  But I suppose
that the seq- functions do the same jobs as the cl- sequence functions.

  > > Could we replace all the cl-lib sequence function calls with seq-
  > > calls, in core and GNU ELPA code?  Seq is simpler and cleaner, so that
  > > would be an improvement.  We could keep cl-lib permanently for
  > > compatibility for external code, but it would not need to be loaded
  > > (into Emacs or your brain) very often.

  > But does that really make sense? Using either is somewhat subjective.

It is ok for a decision like this, about which of two alternative
solutions is better, to be somewhat subjective.  But it is clearly
somewhat objective too/

  > Saying lets just drop these in a sec to all package authors seems
  > overreaching.

The concept of "overreach" is not applicable to a technical decision
like this.  This isn't about respecting or not respecting anyone's
personal rights.

Surely we should not accept a moral principle that Emacs should
contain all the alternative solutions that are at least halfway
acceptable.  We must be able to limit what we include.

  > You write like lets just remove cl-lib and forget that is existed.
  > As mentioned elsewhere cl-lib isn't just sequence functions..

That is true.  This would affect the CL sequence functions, of which
it seems many or all are defined in cl-seq.el.

The overall issue concerns the CL functions in general, because of the
way they tend to be alternate versions of important facilities defined
by Emacs Lisp.  Having those sets of alternatives is a kind of bloat.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: seq.el and the complexity of Emacs Lisp.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > ...you are loosing me here. I'm still wondering what seq/map are good
  > for in the first place. Considering something "cleaner" is just a
  > feeling, isn't it? A feeling I don't share.

I don't concretely know, as I have not used them myself.  But I think
that the seq- functions do roughly the same jobs as the cl- sequence
functions.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: seq.el and the complexity of Emacs Lisp.

[Gerd Möllmann]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > ...you are loosing me here. I'm still wondering what seq/map are good
>   > for in the first place. Considering something "cleaner" is just a
>   > feeling, isn't it? A feeling I don't share.
>
> I don't concretely know, as I have not used them myself.  But I think
> that the seq- functions do roughly the same jobs as the cl- sequence
> functions.

This was about seq/map's abstraction over sequence types, using generic
functions, which AFAICT is good for nothing.

Re: seq.el and the complexity of Emacs Lisp.

[Björn Bidar]

Richard Stallman <rms@gnu.org> writes:

> Could we replace all the cl-lib sequence function calls with seq-
> calls, in core and GNU ELPA code?  Seq is simpler and cleaner, so that
> would be an improvement.  We could keep cl-lib permanently for
> compatibility for external code, but it would not need to be loaded
> (into Emacs or your brain) very often.

But does that really make sense? Using either is somewhat subjective.
Saying lets just drop these in a sec to all package authors seems
overreaching.

You write like lets just remove cl-lib and forget that is existed.
As mentioned elsewhere cl-lib isn't just sequence functions..
It does sound again like there's someting wrong with cl-lib because it's
cl-lib.

Re: seq.el and the complexity of Emacs Lisp.

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: paaguti@gmail.com, emacs-devel@gnu.org
> Date: Sun, 05 Nov 2023 21:27:25 -0500
> 
> I looked at seq.el to try to get an idea what those functions do.  It
> was difficult to get this from the doc strings, because it wasn't
> explicitly stated which ones are for users and which ones are
> internal.

Our convention nowadays is that any function/variable that has 2
dashes after the prefix, in this case seq--SOMETHING, is internal, and
all the others are public.

Re: seq.el and the complexity of Emacs Lisp.

[Emanuel Berg]

Eli Zaretskii wrote:

>> I looked at seq.el to try to get an idea what those
>> functions do. It was difficult to get this from the doc
>> strings, because it wasn't explicitly stated which ones are
>> for users and which ones are internal.
>
> Our convention nowadays is that any function/variable that
> has 2 dashes after the prefix, in this case seq--SOMETHING,
> is internal, and all the others are public.

Are you only allowed to use some--private-function in the same
file where it is defined?

Even as it is actually available everywhere, just like any
some-public-function defined?

To what extent has this convention been upheld in the
Emacs source?

Maybe the byte-compiler should warn when such functions are
used from elsewhere? But maybe then people would just rename
them with a single dash instead of two? But that, by all
means, at least would uphold the convention.

-- 
underground experts united
https://dataswamp.org/~incal

Re: seq.el and the complexity of Emacs Lisp.

[tomas]

[-- Attachment #1: Type: text/plain, Size: 602 bytes --]

On Mon, Nov 06, 2023 at 10:43:49PM +0100, Emanuel Berg wrote:
> Eli Zaretskii wrote:

[...]

> > Our convention nowadays is that any function/variable that
> > has 2 dashes after the prefix, in this case seq--SOMETHING,
> > is internal, and all the others are public.
> 
> Are you only allowed to use some--private-function in the same
> file where it is defined?

You are allowed to do anything. But then, if you complain
that the function calling signature has changed from under
you, the author is allowed to tell you "told you so".

Seems like a fair deal to me.

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: seq.el and the complexity of Emacs Lisp.

[Emanuel Berg]

tomas wrote:

>>> Our convention nowadays is that any function/variable that
>>> has 2 dashes after the prefix, in this case
>>> seq--SOMETHING, is internal, and all the others
>>> are public.
>> 
>> Are you only allowed to use some--private-function in the
>> same file where it is defined?
>
> You are allowed to do anything. But then, if you complain
> that the function calling signature has changed from under
> you, the author is allowed to tell you "told you so".
>
> Seems like a fair deal to me.

Yes, but if there is a convention in code, shouldn't one be
able to say about some piece of code, "this code sticks to the
convention"?

Also, saying that something is "internal" or "public" based on
a convention - then all the more reason to explain exactly
what the convention means - as, in terms of technology, they
are both as public, actually?

-- 
underground experts united
https://dataswamp.org/~incal

Re: seq.el and the complexity of Emacs Lisp.

[Emanuel Berg]

tomas wrote:

>>> Our convention nowadays is that any function/variable that
>>> has 2 dashes after the prefix, in this case
>>> seq--SOMETHING, is internal, and all the others
>>> are public.
>> 
>> Are you only allowed to use some--private-function in the
>> same file where it is defined?
>
> You are allowed to do anything. But then, if you complain
> that the function calling signature has changed from under
> you, the author is allowed to tell you "told you so".
>
> Seems like a fair deal to me.

Do we have this kind of document?

  https://www.kernel.org/doc/html/v4.10/process/coding-style.html

Search that text for GNU and Emacs (emacs) BTW :D

-- 
underground experts united
https://dataswamp.org/~incal

Re: seq.el and the complexity of Emacs Lisp.

[Ihor Radchenko]

Emanuel Berg <incal@dataswamp.org> writes:

> Do we have this kind of document?
>
>   https://www.kernel.org/doc/html/v4.10/process/coding-style.html

Section "D.1 Emacs Lisp Coding Conventions" in Elisp manual.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: seq.el and the complexity of Emacs Lisp.

[Emanuel Berg]

Ihor Radchenko wrote:

>> Do we have this kind of document?
>>
>>   https://www.kernel.org/doc/html/v4.10/process/coding-style.html
>
> Section "D.1 Emacs Lisp Coding Conventions" in Elisp manual.

Ah, great!

(info "(elisp) Coding Conventions")

Maybe some answers to the questions are there.

-- 
underground experts united
https://dataswamp.org/~incal

Re: seq.el and the complexity of Emacs Lisp.

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Tue, 07 Nov 2023 07:21:03 +0100
> 
> Do we have this kind of document?
> 
>   https://www.kernel.org/doc/html/v4.10/process/coding-style.html

See Appendix D "Tips and Conventions" in the ELisp manual.

Re: seq.el and the complexity of Emacs Lisp.

[Emanuel Berg]

Eli Zaretskii wrote:

>> Do we have this kind of document?
>> 
>>   https://www.kernel.org/doc/html/v4.10/process/coding-style.html
>
> See Appendix D "Tips and Conventions" in the ELisp manual.

(info "(elisp) Tips")

-- 
underground experts united
https://dataswamp.org/~incal

Re: seq.el and the complexity of Emacs Lisp.

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Mon, 06 Nov 2023 22:43:49 +0100
> 
> Eli Zaretskii wrote:
> 
> >> I looked at seq.el to try to get an idea what those
> >> functions do. It was difficult to get this from the doc
> >> strings, because it wasn't explicitly stated which ones are
> >> for users and which ones are internal.
> >
> > Our convention nowadays is that any function/variable that
> > has 2 dashes after the prefix, in this case seq--SOMETHING,
> > is internal, and all the others are public.
> 
> Are you only allowed to use some--private-function in the same
> file where it is defined?

That's the convention, yes.

> To what extent has this convention been upheld in the
> Emacs source?

It is upheld quite closely.

Re: seq.el and the complexity of Emacs Lisp.

[Emanuel Berg]

Eli Zaretskii wrote:

>>>> I looked at seq.el to try to get an idea what those
>>>> functions do. It was difficult to get this from the doc
>>>> strings, because it wasn't explicitly stated which ones
>>>> are for users and which ones are internal.
>>>
>>> Our convention nowadays is that any function/variable that
>>> has 2 dashes after the prefix, in this case
>>> seq--SOMETHING, is internal, and all the others
>>> are public.
>> 
>> Are you only allowed to use some--private-function in the
>> same file where it is defined?
>
> That's the convention, yes.

Maybe the byte-compiler should warn when using such functions
in/from other files?

>> To what extent has this convention been upheld in the
>> Emacs source?
>
> It is upheld quite closely.

Okay, great!

-- 
underground experts united
https://dataswamp.org/~incal

Re: seq.el and the complexity of Emacs Lisp.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Our convention nowadays is that any function/variable that has 2
  > dashes after the prefix, in this case seq--SOMETHING, is internal, and
  > all the others are public.

Yes, and seq.el follows that.  But some of the things defined near the
top of seq.el are not functions or variables, and they don't follow
that convention.  It wasn't immediately clear to me what kind of
things they were.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: seq.el and the complexity of Emacs Lisp.

[Eli Zaretskii]

> From: Björn Bidar <bjorn.bidar@thaodan.de>
> Cc: Gerd Möllmann <gerd.moellmann@gmail.com>,
>  paaguti@gmail.com, emacs-devel@gnu.org
> Date: Mon, 06 Nov 2023 10:11:08 +0200
> 
> Saying lets just drop these in a sec to all package authors seems
> overreaching.

Richard's question was about use of cl-lib in the Emacs tree, it will
not (and cannot) affect third-party packages.

Re: seq.el and the complexity of Emacs Lisp.

[Björn Bidar]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Björn Bidar <bjorn.bidar@thaodan.de>
>> Cc: Gerd Möllmann <gerd.moellmann@gmail.com>,
>>  paaguti@gmail.com, emacs-devel@gnu.org
>> Date: Mon, 06 Nov 2023 10:11:08 +0200
>> 
>> Saying lets just drop these in a sec to all package authors seems
>> overreaching.
>
> Richard's question was about use of cl-lib in the Emacs tree, it will
> not (and cannot) affect third-party packages.

He also mentioned about ELPA. Even if it's just the Emacs tree it does
seem overreaching unless there's a direct benefit.

Re: seq.el and the complexity of Emacs Lisp.

[Emanuel Berg]

Björn Bidar wrote:

>>> Saying lets just drop these in a sec to all package
>>> authors seems overreaching.
>>
>> Richard's question was about use of cl-lib in the Emacs
>> tree, it will not (and cannot) affect third-party packages.
>
> He also mentioned about ELPA. Even if it's just the Emacs
> tree it does seem overreaching unless there's
> a direct benefit.

On the whole, changing code that works, for that to make sense
one must have a distinct improvement in mind, that is almost
beyond discussion, otherwise it isn't worth the effort to do
it, with everything negative and unexpected that can follow -
in my experience.

-- 
underground experts united
https://dataswamp.org/~incal

Re: seq.el and the complexity of Emacs Lisp.

[Harald Judt]

[-- Attachment #1.1: Type: text/plain, Size: 1271 bytes --]

Am 06.11.23 um 22:30 schrieb Emanuel Berg:
> Björn Bidar wrote:
> 
>>>> Saying lets just drop these in a sec to all package
>>>> authors seems overreaching.
>>>
>>> Richard's question was about use of cl-lib in the Emacs
>>> tree, it will not (and cannot) affect third-party packages.
>>
>> He also mentioned about ELPA. Even if it's just the Emacs
>> tree it does seem overreaching unless there's
>> a direct benefit.
> 
> On the whole, changing code that works, for that to make sense
> one must have a distinct improvement in mind, that is almost
> beyond discussion, otherwise it isn't worth the effort to do
> it, with everything negative and unexpected that can follow -
> in my experience.

That seems to be a common but unfortunate practice nowadays, that you should 
not touch code that is working, especially rooted in corporate environments. 
Every improvement to the code will make it easier to maintain in the long 
term, fighting bit-rot and inconsistencies. If you touch it and it fails, you 
simply fix it, there is nothing preventing you from making changes to the code 
again.

Harald

-- 
`Experience is the best teacher.'

PGP Key ID: 4FFFAB21B8580ABD
Fingerprint: E073 6DD8 FF40 9CF2 0665 11D4 4FFF AB21 B858 0ABD


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: seq.el and the complexity of Emacs Lisp.

[Emanuel Berg]

Harald Judt wrote:

>>>>> Saying lets just drop these in a sec to all package
>>>>> authors seems overreaching.
>>>>
>>>> Richard's question was about use of cl-lib in the Emacs
>>>> tree, it will not (and cannot) affect
>>>> third-party packages.
>>>
>>> He also mentioned about ELPA. Even if it's just the Emacs
>>> tree it does seem overreaching unless there's
>>> a direct benefit.
>>
>> On the whole, changing code that works, for that to make
>> sense one must have a distinct improvement in mind, that is
>> almost beyond discussion, otherwise it isn't worth the
>> effort to do it, with everything negative and unexpected
>> that can follow - in my experience.
>
> That seems to be a common but unfortunate practice nowadays,
> that you should not touch code that is working, especially
> rooted in corporate environments. Every improvement to the
> code will make it easier to maintain in the long term,
> fighting bit-rot and inconsistencies. If you touch it and it
> fails, you simply fix it, there is nothing preventing you
> from making changes to the code again.

Fighting bit-rot and inconsistencies is okay if those are
small, easy and uniform to fix, and one can see a clear
benefit of doing so (be it small in size as well).

But cl-lib is huge, any action to reduce its influence will be
an equally huge effort - an effort that will seldom look the
same twice - and, worse, can anyone tell for sure what the
benefit would be again? Code that is easier to maintain?
Why exactly, and how do we even know that?

What are we going to replace cl-1, cl-2, ..., cl-n with to get
what kind of advantage-1, advantage-2, ..., advantage-n
exactly? Can anyone tell, really?

No, this whole case against cl-lib doesn't add up, sorry.

-- 
underground experts united
https://dataswamp.org/~incal

Re: seq.el and the complexity of Emacs Lisp.

[Harald Judt]

[-- Attachment #1.1: Type: text/plain, Size: 3179 bytes --]

Am 07.11.23 um 11:14 schrieb Emanuel Berg:
> Harald Judt wrote:
> 
>>>>>> Saying lets just drop these in a sec to all package
>>>>>> authors seems overreaching.
>>>>>
>>>>> Richard's question was about use of cl-lib in the Emacs
>>>>> tree, it will not (and cannot) affect
>>>>> third-party packages.
>>>>
>>>> He also mentioned about ELPA. Even if it's just the Emacs
>>>> tree it does seem overreaching unless there's
>>>> a direct benefit.
>>>
>>> On the whole, changing code that works, for that to make
>>> sense one must have a distinct improvement in mind, that is
>>> almost beyond discussion, otherwise it isn't worth the
>>> effort to do it, with everything negative and unexpected
>>> that can follow - in my experience.
>>
>> That seems to be a common but unfortunate practice nowadays,
>> that you should not touch code that is working, especially
>> rooted in corporate environments. Every improvement to the
>> code will make it easier to maintain in the long term,
>> fighting bit-rot and inconsistencies. If you touch it and it
>> fails, you simply fix it, there is nothing preventing you
>> from making changes to the code again.
> 
> Fighting bit-rot and inconsistencies is okay if those are
> small, easy and uniform to fix, and one can see a clear
> benefit of doing so (be it small in size as well).
> 
> But cl-lib is huge, any action to reduce its influence will be
> an equally huge effort - an effort that will seldom look the
> same twice - and, worse, can anyone tell for sure what the
> benefit would be again? Code that is easier to maintain?
> Why exactly, and how do we even know that?
> 
> What are we going to replace cl-1, cl-2, ..., cl-n with to get
> what kind of advantage-1, advantage-2, ..., advantage-n
> exactly? Can anyone tell, really?

My response was targeted more at the general denial to code changes, not about 
this discussion about cl-lib. Your assumption that every change needs to be 
driven by benefit seems a bit short-sighted to me. It is probably good enough 
when you want to build something fast without having to care about it much 
later. Maybe it holds when you lack time because then not wasting time is a 
priority. But if you want to build something for long-term use, it might not 
be an optimal solution. Having to depend on immediate benefits to change 
anything also loses importance when time is not that much of a priority.

If cl-lib is huge, then one should ask, why has it grown that huge? Is it just 
a badly documented one-pot of several ideas thrown together instead of someone 
doing a library using careful planning? Wouldn't it have been better then to 
replace those functions with something better? Could the situation be improved 
by revising the cl-lib documentation, or splitting up the library and 
progressively replacing it with something better?

 > No, this whole case against cl-lib doesn't add up, sorry.

Probably it is a bad idea to treat this as dragging someone to court?

Harald

-- 
`Experience is the best teacher.'

PGP Key ID: 4FFFAB21B8580ABD
Fingerprint: E073 6DD8 FF40 9CF2 0665 11D4 4FFF AB21 B858 0ABD


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: seq.el and the complexity of Emacs Lisp.

[Ihor Radchenko]

Harald Judt <h.judt@gmx.at> writes:

> My response was targeted more at the general denial to code changes, not about 
> this discussion about cl-lib. Your assumption that every change needs to be 
> driven by benefit seems a bit short-sighted to me. It is probably good enough 
> when you want to build something fast without having to care about it much 
> later. Maybe it holds when you lack time because then not wasting time is a 
> priority. But if you want to build something for long-term use, it might not 
> be an optimal solution. Having to depend on immediate benefits to change 
> anything also loses importance when time is not that much of a priority.

AFAIU, Eli's position in the past was exactly as Emanuel described - we
do not change things unless there is a clear benefit of such change.

Eli, please correct me if my understanding is wrong.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: seq.el and the complexity of Emacs Lisp.

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: Emanuel Berg <incal@dataswamp.org>, emacs-devel@gnu.org
> Date: Tue, 07 Nov 2023 12:38:49 +0000
> 
> Harald Judt <h.judt@gmx.at> writes:
> 
> > My response was targeted more at the general denial to code changes, not about 
> > this discussion about cl-lib. Your assumption that every change needs to be 
> > driven by benefit seems a bit short-sighted to me. It is probably good enough 
> > when you want to build something fast without having to care about it much 
> > later. Maybe it holds when you lack time because then not wasting time is a 
> > priority. But if you want to build something for long-term use, it might not 
> > be an optimal solution. Having to depend on immediate benefits to change 
> > anything also loses importance when time is not that much of a priority.
> 
> AFAIU, Eli's position in the past was exactly as Emanuel described - we
> do not change things unless there is a clear benefit of such change.
> 
> Eli, please correct me if my understanding is wrong.

You are not wrong, but I respectfully ask to take this discussion off
this list to emacs-tangents, since this kind of semi-philosophical
argument is not really on-topic here, and certainly doesn't belong to
the current thread.

Re: seq.el and the complexity of Emacs Lisp.

[Emanuel Berg]

Harald Judt wrote:

> My response was targeted more at the general denial to code
> changes, not about this discussion about cl-lib.
> Your assumption that every change needs to be driven by
> benefit seems a bit short-sighted to me. It is probably good
> enough when you want to build something fast without having
> to care about it much later. Maybe it holds when you lack
> time because then not wasting time is a priority. But if you
> want to build something for long-term use, it might not be
> an optimal solution. Having to depend on immediate benefits
> to change anything also loses importance when time is not
> that much of a priority.

Well, now we are getting into the whole "revolution vs
evolution" discussion.

It is true that you can't expect to achieve something truly
original and great by endlessly piling up very small
incremental gains. But the great leaps are made more easily if
you have those incremental gains already.

And it is often indicative that a great leap is truly great,
if the way to it are big and small incremental gains.

> If cl-lib is huge, then one should ask, why has it grown
> that huge? Is it just a badly documented one-pot of several
> ideas thrown together instead of someone doing a library
> using careful planning? Wouldn't it have been better then to
> replace those functions with something better? Could the
> situation be improved by revising the cl-lib documentation,
> or splitting up the library and progressively replacing it
> with something better?

I think a lot of people have had a very positive experience
from it, and this is the reason for its proliferation.

But of course, incremental gains and great leaps are welcome
there as well :)

-- 
underground experts united
https://dataswamp.org/~incal

Re: seq.el and the complexity of Emacs Lisp.

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Tue, 07 Nov 2023 14:50:56 +0100
> 
> Harald Judt wrote:
> 
> > My response was targeted more at the general denial to code
> > changes, not about this discussion about cl-lib.
> > Your assumption that every change needs to be driven by
> > benefit seems a bit short-sighted to me. It is probably good
> > enough when you want to build something fast without having
> > to care about it much later. Maybe it holds when you lack
> > time because then not wasting time is a priority. But if you
> > want to build something for long-term use, it might not be
> > an optimal solution. Having to depend on immediate benefits
> > to change anything also loses importance when time is not
> > that much of a priority.
> 
> Well, now we are getting into the whole "revolution vs
> evolution" discussion.

Once again, please take this discussion to emacs-tangents.  It doesn't
belong here.

Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I've changed the subject line to match what we are talking about.

  > 226 files, or 14%, use cl-lib at run time.

These cases may be more or less problematical, but not necessarily
equally so.

Which of those files are usually loaded when you start Emacs with
no arguments?  Those we should certainly fix.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Lisp files that load cl-lib in problematical ways

[Alan Mackenzie]

Hello, Richard.

On Wed, Oct 18, 2023 at 21:28:41 -0400, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]

> I've changed the subject line to match what we are talking about.

>   > 226 files, or 14%, use cl-lib at run time.

> These cases may be more or less problematical, but not necessarily
> equally so.

> Which of those files are usually loaded when you start Emacs with
> no arguments?  Those we should certainly fix.

With the following command, started from the lisp directory:

$ sed 's/^[^;]*(load "\(.*\)".*$/\1.el/; s/\.el\.el/.el/; t; d' loadup.el | \
sort | xargs grep -c "^[^;]*(cl-" | sed '/:0/d'

, I get the following files, with counts of '(cl-' usages:

abbrev.el:2
dnd.el:4
emacs-lisp/byte-run.el:1
emacs-lisp/cconv.el:15
emacs-lisp/cl-generic.el:178
emacs-lisp/cl-preloaded.el:45
emacs-lisp/eldoc.el:12
emacs-lisp/lisp-mode.el:14
emacs-lisp/nadvice.el:7
emacs-lisp/oclosure.el:31
emacs-lisp/seq.el:51
emacs-lisp/shorthands.el:1
emacs-lisp/syntax.el:6
emacs-lisp/tabulated-list.el:1
emacs-lisp/timer.el:1
font-lock.el:1
frame.el:6
international/mule-cmds.el:4
international/ucs-normalize.el:3
isearch.el:3
ldefs-boot.el:3
loaddefs.el:3
minibuffer.el:12
progmodes/elisp-mode.el:23
progmodes/prog-mode.el:1
register.el:23
replace.el:1
select.el:6
simple.el:9
startup.el:3
subr.el:1
tab-bar.el:4
term/android-win.el:7
term/haiku-win.el:11
term/ns-win.el:9
term/pc-win.el:7
term/pgtk-win.el:9
term/w32-win.el:12
term/x-win.el:8
uniquify.el:2
vc/vc-hooks.el:1

..  That's a total of 541 occurrences, just in the files that get dumped
with the Emacs binary.

I fear that that's the way things will stay, because it would be too
much work to fix.  But more because there are enough contributors who
think that cl-lib is just an ordinary part of Emacs to be used freely
without restraint.  I am not among this group of contributors.

> -- 
> Dr Richard Stallman (https://stallman.org)
> Chief GNUisance of the GNU Project (https://gnu.org)
> Founder, Free Software Foundation (https://fsf.org)
> Internet Hall-of-Famer (https://internethalloffame.org)

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> Date: Thu, 19 Oct 2023 05:27:03 +0000
> Cc: Emanuel Berg <incal@dataswamp.org>, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> With the following command, started from the lisp directory:
> 
> $ sed 's/^[^;]*(load "\(.*\)".*$/\1.el/; s/\.el\.el/.el/; t; d' loadup.el | \
> sort | xargs grep -c "^[^;]*(cl-" | sed '/:0/d'
> 
> , I get the following files, with counts of '(cl-' usages:
> 
> abbrev.el:2
> dnd.el:4
> emacs-lisp/byte-run.el:1
> emacs-lisp/cconv.el:15
> emacs-lisp/cl-generic.el:178
> emacs-lisp/cl-preloaded.el:45
> emacs-lisp/eldoc.el:12
> emacs-lisp/lisp-mode.el:14
> emacs-lisp/nadvice.el:7
> emacs-lisp/oclosure.el:31
> emacs-lisp/seq.el:51
> emacs-lisp/shorthands.el:1
> emacs-lisp/syntax.el:6
> emacs-lisp/tabulated-list.el:1
> emacs-lisp/timer.el:1
> font-lock.el:1
> frame.el:6
> international/mule-cmds.el:4
> international/ucs-normalize.el:3
> isearch.el:3
> ldefs-boot.el:3
> loaddefs.el:3
> minibuffer.el:12
> progmodes/elisp-mode.el:23
> progmodes/prog-mode.el:1
> register.el:23
> replace.el:1
> select.el:6
> simple.el:9
> startup.el:3
> subr.el:1
> tab-bar.el:4
> term/android-win.el:7
> term/haiku-win.el:11
> term/ns-win.el:9
> term/pc-win.el:7
> term/pgtk-win.el:9
> term/w32-win.el:12
> term/x-win.el:8
> uniquify.el:2
> vc/vc-hooks.el:1
> 
> ..  That's a total of 541 occurrences, just in the files that get dumped
> with the Emacs binary.

Those use cl-lib during byte-compilation, they don't load cl-lib at
run time.  So this is not what Richard asked about, AFAIU.

> I fear that that's the way things will stay, because it would be too
> much work to fix.  But more because there are enough contributors who
> think that cl-lib is just an ordinary part of Emacs to be used freely
> without restraint.  I am not among this group of contributors.

We cannot possibly expect people to contribute code if we force them
not to use the macros they are used to.  If cl-lib is not loaded as
result, that is good enough for us, I think.

Re: Lisp files that load cl-lib in problematical ways

[Alan Mackenzie]

Hello, Eli.

On Thu, Oct 19, 2023 at 10:47:37 +0300, Eli Zaretskii wrote:
> > Date: Thu, 19 Oct 2023 05:27:03 +0000
> > Cc: Emanuel Berg <incal@dataswamp.org>, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > With the following command, started from the lisp directory:

> > $ sed 's/^[^;]*(load "\(.*\)".*$/\1.el/; s/\.el\.el/.el/; t; d' loadup.el | \
> > sort | xargs grep -c "^[^;]*(cl-" | sed '/:0/d'

> > , I get the following files, with counts of '(cl-' usages:

> > abbrev.el:2
[ .... ]
> > vc/vc-hooks.el:1

> > ..  That's a total of 541 occurrences, just in the files that get dumped
> > with the Emacs binary.

> Those use cl-lib during byte-compilation, they don't load cl-lib at
> run time.

They do, I think (see below).

> So this is not what Richard asked about, AFAIU.

Yes, you're right.

So, I counted all the occurrences of "cl-" not inside an
eval-when-compile, by piping all the dumped .el files through the
following script:

find-run-time-cl.el:
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defun run-time-cl ()
  (save-excursion
    (goto-char (point-min))
    (let ((count 0))
      (while (and (< (point) (point-max))
		  (re-search-forward "^[^;]*(\\(eval-when-compile\\|cl-\\)"
				     nil 'stop))
	(if (string= (match-string-no-properties 1) "cl-")
	    (progn
	      (setq count (1+ count)))
	      ;; (message "%s:%s %s" (buffer-file-name) (line-number-at-pos)
	      ;; 	       (buffer-substring-no-properties (line-beginning-position)
	      ;; 					       (line-end-position)))
	  (goto-char (1- (match-beginning 1)))
	  (forward-list)))
      (prin1 (format "%s:%s\n" (buffer-file-name) count) t))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

$ for f in `sed 's/^[^;]*(load "\(.*\)".*$/\1.el/; s/\.el\.el/.el/; t; d' \
  loadup.el | sort`; \
  do ../src/emacs -Q -batch -l ~/find-run-time-cl.el \
  --eval "(progn (find-file \"$f\") (run-time-cl))"; \
  done | sed '/:0$/d' | sed 's%^.*/lisp/%%'

and got the following results:

abbrev.el:2
dnd.el:4
emacs-lisp/byte-run.el:1
emacs-lisp/cconv.el:13
emacs-lisp/cl-generic.el:56
emacs-lisp/cl-preloaded.el:16
emacs-lisp/eldoc.el:6
emacs-lisp/lisp-mode.el:6
emacs-lisp/nadvice.el:2
emacs-lisp/oclosure.el:8
emacs-lisp/seq.el:25
emacs-lisp/shorthands.el:1
emacs-lisp/syntax.el:5
emacs-lisp/tabulated-list.el:1
emacs-lisp/timer.el:1
font-lock.el:1
frame.el:5
international/mule-cmds.el:3
international/ucs-normalize.el:1
isearch.el:3
ldefs-boot.el:3
loaddefs.el:3
minibuffer.el:12
progmodes/elisp-mode.el:11
progmodes/prog-mode.el:1
register.el:3
replace.el:1
select.el:3
simple.el:9
startup.el:1
subr.el:1
tab-bar.el:4
term/android-win.el:4
term/haiku-win.el:3
term/ns-win.el:2
term/pc-win.el:6
term/pgtk-win.el:3
term/w32-win.el:4
term/x-win.el:2
uniquify.el:2
vc/vc-hooks.el:1

That may not be 541 occurrences, but it's still 239.  Incidentally, when
I start emacs -Q, cl-lib isn't yet loaded for me, either.

> > I fear that that's the way things will stay, because it would be too
> > much work to fix.  But more because there are enough contributors who
> > think that cl-lib is just an ordinary part of Emacs to be used freely
> > without restraint.  I am not among this group of contributors.

> We cannot possibly expect people to contribute code if we force them
> not to use the macros they are used to.  If cl-lib is not loaded as
> result, that is good enough for us, I think.

We "force" them to use Emacs Lisp, but they still contribute.  The
problem is that use of cl- makes maintenance of other people's code much
harder, at least for me.  I'm sure I'm not alone.

cl- probably multiplies the size of the Lisp language parts of Elisp
(i.e. things like defun, defmacro, if, while, cond, ...) by around three.
I'm not able to learn all these fancy macros which are used rarely enough
not to be learnable, but frequently enough to cause maintenance headaches
- every time I encounter `if-let' and the like, it just stops me in my
tracks.  I either have to guess what it means, which makes me
uncomfortable, or laboriously consult a doc string, if any.

Yes, I know if-let isn't a cl- function, and its doc string is of
reasonable quality.  Many cl- functions don't have doc strings, or have
inadequate ones, however.

There was a time in the medium past when the use of run time cl- was not
allowed.  Emacs was then, from my point of view, much easier to maintain.
At some stage this rule was done away with, but I don't recall there
being any discussion on emacs-devel about this change.  There should have
been such discussion.

But we are where we are, as you said.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > and got the following results:

  > abbrev.el:2

A generally useful minor mode as abbrev.el should not load cl-lib.
Why does it?  I investigated.

abbrev--check-chars uses the macro cl-pushnew.  Generally we
arrange for cl- macros not to load cl-lib.  But cl-pushnew
is _defined_ in cl-lib.  It ought to be defined with and like
other cl- macros that are meant for users to use.

That's a bug, and probably not a deep one.  Would someone please fix
it?

I investigated byte-run.el.  I don't think it uses any cl- facility.
Rather, it declares a let-variable whose name starts with cl-.

I think that if we investigate the other general-purpose packages
in the list you posted (I copied it below), we wlll find many such bugs that
got in because developers were not on guard against them.

Once we fix them all, how about if we add a regression test
to verify that various packages anyone might load at any time
do not use cl-lib.  With that, bugs like this would get caught
right away.

Here's the list, with some files crossed off that I've just handled.

;; abbrev.el:2
dnd.el:4
;; emacs-lisp/byte-run.el:1
emacs-lisp/cconv.el:13
emacs-lisp/cl-generic.el:56
emacs-lisp/cl-preloaded.el:16
emacs-lisp/eldoc.el:6
emacs-lisp/lisp-mode.el:6
emacs-lisp/nadvice.el:2
emacs-lisp/oclosure.el:8
emacs-lisp/seq.el:25
emacs-lisp/shorthands.el:1
emacs-lisp/syntax.el:5
emacs-lisp/tabulated-list.el:1
emacs-lisp/timer.el:1
font-lock.el:1
frame.el:5
international/mule-cmds.el:3
international/ucs-normalize.el:1
isearch.el:3
ldefs-boot.el:3
loaddefs.el:3
minibuffer.el:12
progmodes/elisp-mode.el:11
progmodes/prog-mode.el:1
register.el:3
replace.el:1
select.el:3
simple.el:9
startup.el:1
subr.el:1
tab-bar.el:4
term/android-win.el:4
term/haiku-win.el:3
term/ns-win.el:2
term/pc-win.el:6
term/pgtk-win.el:3
term/w32-win.el:4
term/x-win.el:2
uniquify.el:2
vc/vc-hooks.el:1


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Sun, 22 Oct 2023 22:08:55 -0400
> 
>   > abbrev.el:2
> 
> A generally useful minor mode as abbrev.el should not load cl-lib.
> Why does it?  I investigated.
> 
> abbrev--check-chars uses the macro cl-pushnew.  Generally we
> arrange for cl- macros not to load cl-lib.  But cl-pushnew
> is _defined_ in cl-lib.  It ought to be defined with and like
> other cl- macros that are meant for users to use.

It would not have helped to define cl-pushnew on cl-macs, because
cl-pushnew calls cl-adjoin, which is a function defined on cl-lib.  So
basically cl-pushnew is a (rather thin) wrapper around cl-adjoin, and
thus it is correctly defined on the same file where cl-adjoin is
defined.  I see no problem here.

> I investigated byte-run.el.  I don't think it uses any cl- facility.
> Rather, it declares a let-variable whose name starts with cl-.

byte-run.el indeed doesn't need cl-lib, and loading byte-run.elc
doesn't load cl-lib.  So I see no problem here, either.  IOW, this is
a false positive due to the naïve search pattern used to detect this.

> I think that if we investigate the other general-purpose packages
> in the list you posted (I copied it below), we wlll find many such bugs that
> got in because developers were not on guard against them.

I very much doubt that we will find many such bugs, because we
actually pay close attention to these aspects as part of our patch
review process.  But, of course, there could be places we missed, so
if someone finds them, please do report them and/or propose patches to
fix them.

> Once we fix them all, how about if we add a regression test
> to verify that various packages anyone might load at any time
> do not use cl-lib.  With that, bugs like this would get caught
> right away.

It isn't easy to write such a test, since our test suite
infrastructure, ert.el, itself uses cl-lib extensively.  But if
someone can come up with a test, please do submit it, and TIA.

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > It would not have helped to define cl-pushnew on cl-macs, because
  > cl-pushnew calls cl-adjoin, which is a function defined on cl-lib.  So
  > basically cl-pushnew is a (rather thin) wrapper around cl-adjoin, and
  > thus it is correctly defined on the same file where cl-adjoin is
  > defined.

That is a good point -- for that particular solution I proposed.

But if we let ourselves be a little more flexible, how about this
solution: install this

        (while (string-match "\\W" abbrev pos)
          (or (memq (aref abbrev (match-beginning 0)) badchars)
              (push (aref abbrev (match-beginning 0)) badchars))
          (setq pos (1+ pos)))

instead of the current code:

        (while (string-match "\\W" abbrev pos)
          (cl-pushnew (aref abbrev (match-beginning 0)) badchars)
          (setq pos (1+ pos)))

A CL fan might say the latter is simpler, but for me, who isn't sure
just how to use cl-pushnew, the former seems conceptually simpler.

The two ecxpressions have two different kinds of complexity.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Lisp files that load cl-lib in problematical ways

[Stefan Kangas]

Richard Stallman <rms@gnu.org> writes:

> But if we let ourselves be a little more flexible, how about this
> solution: install this
>
>         (while (string-match "\\W" abbrev pos)
>           (or (memq (aref abbrev (match-beginning 0)) badchars)
>               (push (aref abbrev (match-beginning 0)) badchars))
>           (setq pos (1+ pos)))
>
> instead of the current code:
>
>         (while (string-match "\\W" abbrev pos)
>           (cl-pushnew (aref abbrev (match-beginning 0)) badchars)
>           (setq pos (1+ pos)))
>
> A CL fan might say the latter is simpler, but for me, who isn't sure
> just how to use cl-pushnew, the former seems conceptually simpler.

The `cl-pushnew' docstring explains:

    Add X to the list stored in PLACE unless X is already in the list.

> The two ecxpressions have two different kinds of complexity.

The only difference I see is that new version is harder to read, and
more prone to bugs (as you have to write the same code twice).

I'd suggest avoiding these types of changes.

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > A CL fan might say the latter is simpler, but for me, who isn't sure
  > > just how to use cl-pushnew, the former seems conceptually simpler.

  > The `cl-pushnew' docstring explains:

  >     Add X to the list stored in PLACE unless X is already in the list.

Yes, it does -- but what I said is valid nonetheless.
We're talking about the complexity of different ways of writing code.

A reference to a function that you don't know has a kind of complexity.
The fact that you can look up that function doesn't eliminate that complexity.

  > The only difference I see is that new version is harder to read, and
  > more prone to bugs (as you have to write the same code twice).

Do you frequently use the cl- facilities such as `cl-pushnew'?  If so,
that might explain why you didn't _see_ the extra complexity that
comes from calling `cl-pushnew'.  That extra complexity exists for
people who don't habitually use cl facilities, and not for those who
do habitually use them.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: eliz@gnu.org, acm@muc.de, emacs-devel@gnu.org
> Date: Wed, 25 Oct 2023 22:27:54 -0400
> 
>   > The only difference I see is that new version is harder to read, and
>   > more prone to bugs (as you have to write the same code twice).
> 
> Do you frequently use the cl- facilities such as `cl-pushnew'?  If so,
> that might explain why you didn't _see_ the extra complexity that
> comes from calling `cl-pushnew'.  That extra complexity exists for
> people who don't habitually use cl facilities, and not for those who
> do habitually use them.

I think the need to be familiar with the cl-lib and cl-macs
functionalities is nowadays a requirement for any Emacs maintainer.
It can take time to familiarize oneself with them, but it isn't rocket
science.

Btw, some of the cl-lib facilities should not be used in new code if
they have equivalent implementations ins seq.el (which is nowadays
preloaded) and in map.el.  cl-generic.el is also preloaded, so should
be used in preference to the cl-lib alternatives.  And some frequently
needed cl-lib functions were moved to cl-preloaded.el, and are also
always available.

Re: Lisp files that load cl-lib in problematical ways

[Alan Mackenzie]

Hello, Eli.

On Thu, Oct 26, 2023 at 09:55:45 +0300, Eli Zaretskii wrote:
> > From: Richard Stallman <rms@gnu.org>
> > Cc: eliz@gnu.org, acm@muc.de, emacs-devel@gnu.org
> > Date: Wed, 25 Oct 2023 22:27:54 -0400

> >   > The only difference I see is that new version is harder to read, and
> >   > more prone to bugs (as you have to write the same code twice).

> > Do you frequently use the cl- facilities such as `cl-pushnew'?  If so,
> > that might explain why you didn't _see_ the extra complexity that
> > comes from calling `cl-pushnew'.  That extra complexity exists for
> > people who don't habitually use cl facilities, and not for those who
> > do habitually use them.

> I think the need to be familiar with the cl-lib and cl-macs
> functionalities is nowadays a requirement for any Emacs maintainer.

For me, that is not possible.  I simply don't have the brain power to
memorize such a morass of arbitrary, poorly named, poorly documented
stuff.  I am forever cursed by having to look up doc strings each time I
encounter these objects.  Maybe not things like cl-incf, but
monstrosities such as cl-labels just leave me groaning.  I frequently
spend, perhaps, half of my debugging time trying to understand what some
cl-* does rather than concentrating on the problem to be debugged.

> It can take time to familiarize oneself with them, but it isn't rocket
> science.

How many contributors are actually familiar with the bulk of cl-lib?
I suspect, very few indeed.  Maybe Stefan M.  The people who have written
strongly in favour of cl-lib over the last week or two have not been
those who spend lots of time maintaining others' code.

I further suspect that many contributions using cl-lib are from people
who know a small part of cl-lib, find the functionality intriguing, and
thus use it.  The trouble is, each such contributor knows a different
part of it, so in aggregate the part that an Emacs maintainer needs to
know (or continually to look up) is large.

Note that Common Lisp never took off as a popular language.  There must
be reasons for this.

> Btw, some of the cl-lib facilities should not be used in new code if
> they have equivalent implementations ins seq.el (which is nowadays
> preloaded) and in map.el.  cl-generic.el is also preloaded, so should
> be used in preference to the cl-lib alternatives.  And some frequently
> needed cl-lib functions were moved to cl-preloaded.el, and are also
> always available.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Alan Mackenzie wrote:

> How many contributors are actually familiar with the bulk of
> cl-lib? I suspect, very few indeed. Maybe Stefan M.
> The people who have written strongly in favour of cl-lib
> over the last week or two have not been those who spend lots
> of time maintaining others' code.
>
> I further suspect that many contributions using cl-lib are
> from people who know a small part of cl-lib, find the
> functionality intriguing, and thus use it. The trouble is,
> each such contributor knows a different part of it, so in
> aggregate the part that an Emacs maintainer needs to know
> (or continually to look up) is large.

It is the nature of innovation. People like to be at the
front. They like to try out exotic new gear. And the people at
the rear understand this - even if it means they have to build
new facilities to accomodate for the new stuff once in
a while.

If we tell people at the front to stick with the old ways, the
old gear, they will simply leave our section for something
more interesting to do. And we won't get anywhere.

> Note that Common Lisp never took off as a popular language.
> There must be reasons for this.

Maintaining it is too difficult?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Bob Rogers]

   From: Emanuel Berg <incal@dataswamp.org>
   Date: Thu, 26 Oct 2023 13:32:43 +0200

   Alan Mackenzie wrote:

   . . .

   > Note that Common Lisp never took off as a popular language.  There
   > must be reasons for this.

   Maintaining it is too difficult?

Even before CL, Lisp systems tended to be memory-hungry.  Common Lisp
was designed as a large language, for a future where memory would be
cheap.  That future eventually came, but before it came the PC boom,
when tiny systems with relatively tiny amounts of RAM ruled the roost.
And Lisp was neglected even on more powerful workstations; some of you
will rememeber the Sun Sparcstation Pmeg problem that caused page table
thrashing if you tried to allocate more than 16MB.  (This is an over-
simplification for why Lisp popularity declined, of course.)

   This complexity and the resulting complications, I believe, are what
Richard was trying to avoid when he designed Emacs Lisp to be a simpler
dialect of Lisp, and why he sees cl-lib.el as a trojan horse that is
changing Emacs Lisp.  (And why I hope this bit of history is not quite
off-topic.)

					-- Bob Rogers
					   http://www.rgrjr.com/

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >    This complexity and the resulting complications, I believe, are what
  > Richard was trying to avoid when he designed Emacs Lisp to be a simpler
  > dialect of Lisp, and why he sees cl-lib.el as a trojan horse that is
  > changing Emacs Lisp.

I think you've put your finger on it.

I see nothing wrong with having an optional extension which implements
(more or less) Common Lisp in Emacs Lisp.  That's what I thought we
had done.

However, I object to incorporating (more or less) Common Lisp an essential part
of Emacs Lisp.  And it looks like we've been drifting into that.

There is no sharp line between the one and the other, and no simple
test to determine which of those two our current situation actually
is.  Rather, there is a spectrum that runs from "CL support is
available but you can ignore it" to "the CL constructs are an
essential and unavoidable part of Emacs Lisp."

The specific practical questions I've asked are efforts to evaluate
where we are now along that spectrum.  Of course, the answer to that
isn't precise either.  But I was very surprised to learn how far
Emacs has gone towards the latter end.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

What's missing in ELisp that makes people want to use cl-lib?

[Stefan Kangas]

Richard Stallman <rms@gnu.org> writes:

> The specific practical questions I've asked are efforts to evaluate
> where we are now along that spectrum.  Of course, the answer to that
> isn't precise either.  But I was very surprised to learn how far
> Emacs has gone towards the latter end.

It seems to me that the extent to which Emacs has shifted "towards the
latter end" is still open to interpretation.

More fundamentally, I believe the question we should be asking isn't the
quantity of cl-lib used in Emacs, but rather why these abstractions from
Common Lisp appeal to ELisp developers.  Most ELisp developers, I
presume, do not have a background in Common Lisp, indicating that their
choice to use cl-lib.el isn't driven by a mere affinity for CL.

Is it possible that certain abstractions or functions are filling a gap
in Emacs Lisp itself?  In the case of `cl-pushnew', I happen to think
that the answer is yes.

Discussing `cl-pushnew' and other cases like it might help us identify
areas for improvement in Emacs Lisp proper.  Perhaps we can borrow the
CL name/behavior, or perhaps we can come up with something even better.
That would be more constructive, and more helpful, than spending our
time changing code to not use cl-lib.el just for the sake of it.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Stefan Kangas wrote:

> More fundamentally, I believe the question we should be
> asking isn't the quantity of cl-lib used in Emacs, but
> rather why these abstractions from Common Lisp appeal to
> ELisp developers. Most ELisp developers, I presume, do not
> have a background in Common Lisp, indicating that their
> choice to use cl-lib.el isn't driven by a mere affinity
> for CL.

Good idea, I tried this to find out. Maybe the command can be
improved ...

$ grep -E -h -o 'cl-[-a-z]*' **/*.el | sort | uniq -c | sort -r -b -n | head -n 10

   1416 cl-defmethod
    979 cl-loop
    720 cl-incf
    614 cl-lib
    486 cl-assert
    309 cl-letf
    303 cl-seq
    281 cl-defstruct
    257 cl-defgeneric
    227 cl-case

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Sat, 28 Oct 2023 18:18:53 +0200
> 
> Stefan Kangas wrote:
> 
> > More fundamentally, I believe the question we should be
> > asking isn't the quantity of cl-lib used in Emacs, but
> > rather why these abstractions from Common Lisp appeal to
> > ELisp developers. Most ELisp developers, I presume, do not
> > have a background in Common Lisp, indicating that their
> > choice to use cl-lib.el isn't driven by a mere affinity
> > for CL.
> 
> Good idea, I tried this to find out. Maybe the command can be
> improved ...
> 
> $ grep -E -h -o 'cl-[-a-z]*' **/*.el | sort | uniq -c | sort -r -b -n | head -n 10
> 
>    1416 cl-defmethod
>     979 cl-loop
>     720 cl-incf
>     614 cl-lib
>     486 cl-assert
>     309 cl-letf
>     303 cl-seq
>     281 cl-defstruct
>     257 cl-defgeneric
>     227 cl-case

Some of these are macros in cl-macs, so not relevant to the discussion
about cl-lib.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Eli Zaretskii wrote:

>> $ grep -E -h -o 'cl-[-a-z]*' **/*.el | sort | uniq -c | sort -r -b -n |
>>  head -n 10
>> 
>>    1416 cl-defmethod
>>     979 cl-loop
>>     720 cl-incf
>>     614 cl-lib
>>     486 cl-assert
>>     309 cl-letf
>>     303 cl-seq
>>     281 cl-defstruct
>>     257 cl-defgeneric
>>     227 cl-case
>
> Some of these are macros in cl-macs, so not relevant to the
> discussion about cl-lib.

It is everything, remove the last 'head -n 10' to get it.

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Sat, 28 Oct 2023 21:08:33 +0200
> 
> Eli Zaretskii wrote:
> 
> >> $ grep -E -h -o 'cl-[-a-z]*' **/*.el | sort | uniq -c | sort -r -b -n |
> >>  head -n 10
> >> 
> >>    1416 cl-defmethod
> >>     979 cl-loop
> >>     720 cl-incf
> >>     614 cl-lib
> >>     486 cl-assert
> >>     309 cl-letf
> >>     303 cl-seq
> >>     281 cl-defstruct
> >>     257 cl-defgeneric
> >>     227 cl-case
> >
> > Some of these are macros in cl-macs, so not relevant to the
> > discussion about cl-lib.
> 
> It is everything, remove the last 'head -n 10' to get it.

What I wrote is valid even without 'head -n 10'.  My point is that
this kind of list is not useful for this discussion.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Eli Zaretskii wrote:

>>>> $ grep -E -h -o 'cl-[-a-z]*' **/*.el | sort | uniq -c | sort -r -b -n |
>>>>  head -n 10
>>>> 
>>>>    1416 cl-defmethod
>>>>     979 cl-loop
>>>>     720 cl-incf
>>>>     614 cl-lib
>>>>     486 cl-assert
>>>>     309 cl-letf
>>>>     303 cl-seq
>>>>     281 cl-defstruct
>>>>     257 cl-defgeneric
>>>>     227 cl-case
>>>
>>> Some of these are macros in cl-macs, so not relevant to the
>>> discussion about cl-lib.
>> 
>> It is everything, remove the last 'head -n 10' to get it.
>
> What I wrote is valid even without 'head -n 10'. My point is
> that this kind of list is not useful for this discussion.

Of course what you say is valid, since some part of everything
is always something.

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Jim Porter]

On 10/28/2023 8:39 AM, Stefan Kangas wrote:
> Is it possible that certain abstractions or functions are filling a gap
> in Emacs Lisp itself?  In the case of `cl-pushnew', I happen to think
> that the answer is yes.

I agree with this. I can speak about Eshell in particular, since it's 
historically used a bit of cl-lib, and I've added more usage to it.

By far the number one cl-lib thing Eshell uses is 'cl-assert'. This 
would be reasonably easy to do without cl-lib, but it's a pretty 
convenient syntax for, well... asserting some precondition for a 
function. That's been useful for helping to identify subtle bugs in 
Eshell's logic.

After that, the next most important cl-lib thing (in my opinion) is 
generic functions. I introduced this in Eshell to break apart some 
monolithic I/O code that had large 'cond' forms for each I/O target. 
Now, with generic functions, they're separate, and it's much easier for 
third parties to add new I/O target types. In the future, I'll likely 
use generic functions for a few more bits in Eshell.

Related to the above is 'cl-defstruct'. That's useful for working with 
generic functions, but I think there's also some benefit to using them 
instead of lists (or alists/plists) when you want something with a 
very-specific set of members (similar to a C struct or C++ class). 
However, I could do without CL-style structs for this case, excluding 
the generic function side.

Finally, Eshell uses a few miscellaneous cl-lib functions, some of which 
are probably necessary, and some of which are just conveniences. The 
(probable) necessities are: cl-progv and cl-mapc. The conveniences are: 
cl-loop, cl-flet, cl-list*, cl-remove-if, cl-incf, and cl-decf.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> Date: Sat, 28 Oct 2023 12:10:23 -0700
> Cc: emacs-devel@gnu.org
> From: Jim Porter <jporterbugs@gmail.com>
> 
> On 10/28/2023 8:39 AM, Stefan Kangas wrote:
> > Is it possible that certain abstractions or functions are filling a gap
> > in Emacs Lisp itself?  In the case of `cl-pushnew', I happen to think
> > that the answer is yes.
> 
> I agree with this. I can speak about Eshell in particular, since it's 
> historically used a bit of cl-lib, and I've added more usage to it.
> 
> By far the number one cl-lib thing Eshell uses is 'cl-assert'.

cl-assert is a macro in cl-macs, so it isn't relevant to this
discussion.

> After that, the next most important cl-lib thing (in my opinion) is 
> generic functions.

If you mean cl-defgeneric and cl-defmethod, they again are not in
cl-lib, so not relevant to this discussion.  Moreover, they are
defined in cl-generic.el, which is preloaded.

> Related to the above is 'cl-defstruct'.

cl-defstruct is again a macro in cl-macs, not in cl-lib.

> Finally, Eshell uses a few miscellaneous cl-lib functions, some of which 
> are probably necessary, and some of which are just conveniences. The 
> (probable) necessities are: cl-progv and cl-mapc. The conveniences are: 
> cl-loop, cl-flet, cl-list*, cl-remove-if, cl-incf, and cl-decf.

These are the only ones relevant to this discussion.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Jim Porter]

On 10/28/2023 12:14 PM, Eli Zaretskii wrote:
>> Date: Sat, 28 Oct 2023 12:10:23 -0700
>> Cc: emacs-devel@gnu.org
>> From: Jim Porter <jporterbugs@gmail.com>
>>
>> I agree with this. I can speak about Eshell in particular, since it's
>> historically used a bit of cl-lib, and I've added more usage to it.
>>
>> By far the number one cl-lib thing Eshell uses is 'cl-assert'.
> 
> cl-assert is a macro in cl-macs, so it isn't relevant to this
> discussion.

Thanks. I wasn't aware of this distinction. (I see that it was discussed 
elsewhere in the parent thread, but that thread is very big and I hadn't 
been following it closely.)

Is there any benefit in aliasing macros like this to non 'cl-FOO' names 
anyway? As a mere consumer of the various Common Lisp code in Emacs, 
it's not always clear to me what's ok to use versus what should ideally 
be avoided.

>> Finally, Eshell uses a few miscellaneous cl-lib functions, some of which
>> are probably necessary, and some of which are just conveniences. The
>> (probable) necessities are: cl-progv and cl-mapc. The conveniences are:
>> cl-loop, cl-flet, cl-list*, cl-remove-if, cl-incf, and cl-decf.
> 
> These are the only ones relevant to this discussion.

I see that 'cl-progv' is in cl-macs.el, so I guess that's not relevant. 
'cl-mapc' is in cl-extra.el, which I suppose is relevant here? I find it 
very helpful, but I'd be happy with any function/macro that would make 
it easy to iterate over multiple sequences at once.

Of the conveniences, I think the ones I'd miss the most are 'cl-incf' 
and 'cl-decf', which I use in Eshell for recounting some handles. (I 
could possibly rewrite that code to let the garbage collector do all the 
work, but that code interacts with some of the trickier bits in Eshell, 
so maybe it's best not to change it too much.)

'cl-remove-if' should be easy to replace with 'seq-filter' or somesuch. 
Likewise 'cl-list*' with 'nconc'. I know there's some controversy around 
'cl-loop' (I'm not a big fan myself), and I've thought about adding some 
new hooks to Eshell that would probably obviate the need for 'cl-loop' 
(which is used in em-extpipe.el).

So in conclusion, I guess the relevant functions that I care about are 
'cl-mapc', and to a lesser degree 'cl-incf'/'cl-decf'. Everything else 
could go away if we wanted to avoid them.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> Date: Sat, 28 Oct 2023 14:03:29 -0700
> Cc: stefankangas@gmail.com, rms@gnu.org, rogers@rgrjr.com, emacs-devel@gnu.org
> From: Jim Porter <jporterbugs@gmail.com>
> 
> Is there any benefit in aliasing macros like this to non 'cl-FOO' names 
> anyway?

I don't have an opinion on this, so I'll let others chime in.  In
general, introducing aliases has a minor downside: people will need to
learn two names instead of just one.

> I see that 'cl-progv' is in cl-macs.el, so I guess that's not relevant. 
> 'cl-mapc' is in cl-extra.el, which I suppose is relevant here?

Yes, cl-extra is just another part of cl-lib.

> I find it very helpful, but I'd be happy with any function/macro
> that would make it easy to iterate over multiple sequences at once.

Try looking in seq.el or map.el.

> 'cl-remove-if' should be easy to replace with 'seq-filter' or somesuch. 
> Likewise 'cl-list*' with 'nconc'. I know there's some controversy around 
> 'cl-loop' (I'm not a big fan myself), and I've thought about adding some 
> new hooks to Eshell that would probably obviate the need for 'cl-loop' 
> (which is used in em-extpipe.el).
> 
> So in conclusion, I guess the relevant functions that I care about are 
> 'cl-mapc', and to a lesser degree 'cl-incf'/'cl-decf'. Everything else 
> could go away if we wanted to avoid them.

Since seq.el is preloaded, I think we should prefer it to cl-lib.
Likewise map.el, although that isn't preloaded.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Of the conveniences, I think the ones I'd miss the most are 'cl-incf' 
  > and 'cl-decf', which I use in Eshell for recounting some handles.

If this pattern extends to many files -- if getf, incf and decf are
particularly useful among CL constructs -- making them standard parts of Emacs Lisp would be better than inducing lots of files to load cl-lib.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Jim Porter]

On 10/29/2023 7:09 PM, Richard Stallman wrote:
>    > Of the conveniences, I think the ones I'd miss the most are 'cl-incf'
>    > and 'cl-decf', which I use in Eshell for recounting some handles.
> 
> If this pattern extends to many files -- if getf, incf and decf are
> particularly useful among CL constructs -- making them standard parts of Emacs Lisp would be better than inducing lots of files to load cl-lib.

I don't think I'd fight tooth and nail to get incf/decf, but I'd 
certainly be happy to see them in standard Elisp.

I'm not sure how much others want to see setf-like constructs in 
Elisp[1], but I like them quite a bit. They fit in neatly alongside the 
C/C++ parts of my brain. Specifically, I find this style:

   (cl-incf (cdr output-target))
   - or -
   output_target.tail++;

a lot nicer than this:

   (setcdr output-target (1+ (cdr output-target)))
   - or -
   output_target.tail = output_target.tail + 1;

[1] I recall that some generalized values for setf were deprecated 
recently, but I think that was mainly for things where the setter had 
some non-obvious behavior.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Is it possible that certain abstractions or functions are filling a gap
  > in Emacs Lisp itself?

Yes, it could well be true for some of them.

                           In the case of `cl-pushnew', I happen to think
  > that the answer is yes.

Do people really want `cl-pushnew', with its keyword arguments?
Or would a simple `pushnew' be even more pleasing?

To define `pushnew' in Emacs Lisp, and document it in the Emacs Lisp
Reference Manual, would be much cleaner than spreading the use of
cl-lib to more files.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Adam Porter]

> Do people really want `cl-pushnew', with its keyword arguments?
> Or would a simple `pushnew' be even more pleasing?

It might be more aesthetically or ideologically pleasing, but it would 
be much less useful.  :test is often necessary for using EQUAL (as EQ is 
often the wrong function to use); without it, a hypothetical non-CL 
PUSHNEW function would often be unusable.  And :key makes it easy to use 
with, e.g. structs' accessor functions (which are very helpful for 
developers as they allow compile-time checking, completion, etc).

When CL-PUSHNEW is called without keyword arguments, it's as simple as a 
hypothetical non-CL PUSHNEW that offered no keyword arguments.  The 
keyword arguments provide optional flexibility; without them, one would 
have to write a much larger, more complex form to do the same things.

Finally, if there were an additional, non-CL PUSHNEW function added, I 
would think that would be much more confusing to users, especially new 
ones, for there to be a difference in behavior between PUSHNEW and 
CL-PUSHNEW.

So I would ask that CL-PUSHNEW become PUSHNEW as-is.

> To define `pushnew' in Emacs Lisp, and document it in the Emacs Lisp
> Reference Manual, would be much cleaner than spreading the use of
> cl-lib to more files.

I would be happy to see more cl-lib functions "moved" back into "core" 
Elisp.  Using Elisp without cl-lib feels restrictive to me.

By the way, with regard to CL-PUSHNEW, it seems surprising that no one's 
mentioned the existing counterpart in "core" Elisp, ADD-TO-LIST. 
Notably, its docstring says:

   This is handy to add some elements to configuration variables,
   but please do not abuse it in Elisp code, where you are usually
   better off using ‘push’ or ‘cl-pushnew’.

So even "core", non-cl-lib Elisp recommends the use of some cl-lib 
functions, i.e. CL-PUSHNEW.  That certainly suggests a "gap" in core 
Elisp functionality.

--Adam

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> Date: Mon, 30 Oct 2023 21:31:41 -0500
> Cc: emacs-devel@gnu.org, stefankangas@gmail.com
> From: Adam Porter <adam@alphapapa.net>
> 
> By the way, with regard to CL-PUSHNEW, it seems surprising that no one's 
> mentioned the existing counterpart in "core" Elisp, ADD-TO-LIST. 
> Notably, its docstring says:
> 
>    This is handy to add some elements to configuration variables,
>    but please do not abuse it in Elisp code, where you are usually
>    better off using ‘push’ or ‘cl-pushnew’.
> 
> So even "core", non-cl-lib Elisp recommends the use of some cl-lib 
> functions, i.e. CL-PUSHNEW.  That certainly suggests a "gap" in core 
> Elisp functionality.

This kind of remarks in Emacs doc strings are never about "gaps" of
any kind, they are about cleaner coding practices.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Tue, Oct 31, 2023 at 2:32 AM Adam Porter <adam@alphapapa.net> wrote:
>
> > Do people really want `cl-pushnew', with its keyword arguments?
> > Or would a simple `pushnew' be even more pleasing?
>
> It might be more aesthetically or ideologically pleasing, but it would
> be much less useful.  :test is often necessary for using EQUAL (as EQ is
> often the wrong function to use); without it, a hypothetical non-CL
> PUSHNEW function would often be unusable.  And :key makes it easy to use
> with, e.g. structs' accessor functions (which are very helpful for
> developers as they allow compile-time checking, completion, etc).
>
> When CL-PUSHNEW is called without keyword arguments, it's as simple as a
> hypothetical non-CL PUSHNEW that offered no keyword arguments.  The
> keyword arguments provide optional flexibility; without them, one would
> have to write a much larger, more complex form to do the same things.

Of course.

The other day I noticed we have a C function called 'car-less-than-car'
whose only point is to optimize a super-specific use of 'sort'.  Maybe
in 1993 we didn't have compiler macros but now we do, so this is the
wrong way to go about it.

We should just write

  (cl-sort foo #'< :key #'car)

which is much more modular, readable, reusable, and doesn't require
the programmer to know of this specific vocabulary.

Sure, "but what about the optimization?", I hear you ask.  Well if
you must have it then just use a compiler macro to have your cake and
eat it too.

In this case, something like

  (defun cl-sort-compiler-macro (form &rest args)
    (if (equal (cdr args) '(#'< :key #'car))
        `(cl-sort ,(car args) #'car-less-than-car)
      form))

and add it to the cl-sort declaration with

  (declare (compiler-macro 'cl-sort-compiler-macro))

> Finally, if there were an additional, non-CL PUSHNEW function added, I
> would think that would be much more confusing to users, especially new
> ones, for there to be a difference in behavior between PUSHNEW and
> CL-PUSHNEW.

Exactly.  Please don't do this, we have enough such cases already.
Either migrate the fully functional version or don't.

Also, it's not too late to add keyword arguments to our 'sort'

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

It might be ok to add some keyword arguments to `sort', which are more
unusual and complex to use, but not to simple constructs like
`pushnew'.  This is Emacs Lisp, not Common Lisp.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Jim Porter]

On 11/1/2023 7:28 PM, Richard Stallman wrote:
> It might be ok to add some keyword arguments to `sort', which are more
> unusual and complex to use, but not to simple constructs like
> `pushnew'.  This is Emacs Lisp, not Common Lisp.

I'm not advocating for (or against) this, but if we wanted something 
like 'pushnew' and didn't want to use keyword arguments, then what about 
these two functions?

   (defun pushnew (value place &optional testfun) ...)
   (defun pushnew-key (value place keyfun) ...)

There's no reason I can see that we'd *need* to have both forms in a 
single function. (Unless we specifically wanted to match the Common Lisp 
API, and that's what cl-lib is for.)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Philip Kaludercic]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> It might be ok to add some keyword arguments to `sort', which are more
> unusual and complex to use, but not to simple constructs like
> `pushnew'.  This is Emacs Lisp, not Common Lisp.

What does that last sentence mean?  I hope this is not a too
philosophical question, but what constitutes "Emacs Lisp"?  It would
seem peculiar if it were to be defined by the arbitrary decisions of the
past, constrained by the contingent circumstances of the time.
Certainly part of it is that it is part of a tradition of Lisp
implementations, that gives it a distinctive flavour, but to me the main
aspects and attractions of Emacs Lisp (in contrast to CL and Scheme)
have been:

1. Not standardised; it is possible to extend the language without
   having to worry about how many implementations will follow along

2. Accepting Unix; not trying to abstract over various different
   operating systems and file systems, making it easier to work with GNU
   and other POSIX-based environments.

The first point would be relevant here.  Just like if-let or
thread-first, Emacs Lisp can learn from interesting ideas that other
languages provide, adapt and add them, making them available to
everyone.  If a form expresses something that people frequently want to
do, but either have to elaborate using (unless (memq ...) (push ...)),
then we are making Emacs more useful and expressive by providing the
functionality OOTB.  And isn't that the real point of Emacs Lisp?

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: João Távora <joaotavora@gmail.com>,
>  adam@alphapapa.net, emacs-devel@gnu.org,  stefankangas@gmail.com
> Date: Thu, 02 Nov 2023 08:55:54 +0000
> 
> Richard Stallman <rms@gnu.org> writes:
> 
> > It might be ok to add some keyword arguments to `sort', which are more
> > unusual and complex to use, but not to simple constructs like
> > `pushnew'.  This is Emacs Lisp, not Common Lisp.
> 
> What does that last sentence mean?

AFAIU, it means that Emacs Lisp traditionally doesn't use keyword
arguments, except as relatively rare exceptions.

> what constitutes "Emacs Lisp"?  It would
> seem peculiar if it were to be defined by the arbitrary decisions of the
> past, constrained by the contingent circumstances of the time.

Those "arbitrary decisions" are what got us to where we are now, 40
years later.  So some respect for those "arbitrary decisions" is due,
I think.

> to me the main aspects and attractions of Emacs Lisp (in contrast to
> CL and Scheme) have been:
> 
> 1. Not standardised; it is possible to extend the language without
>    having to worry about how many implementations will follow along

IMNSHO, extending Emacs Lisp as the language is not the main goal of
Emacs development.  Emacs Lisp is not a programming language on its
own, it is a language for implementing and extending features and
extensions in Emacs.  Thus, the main goal of Emacs development is to
develop applications and application-level features, and provide more
opportunities for extending the core where that is considered useful.
What we have in Emacs Lisp is IMO ample for that purpose.  Moreover,
most participants in Emacs development are not experts in programming
languages, their expertise is elsewhere (which is definitely a Good
Thing).

Objectively, adding new syntax and semantics to Emacs Lisp does make
the source code harder to read and maintain, because it makes the
language larger and requires familiarization with those new language
features, which more often than not look and feel completely different
from the "traditional" Emacs Lisp.  So even if we conclude that these
additions are worthwhile, we should not pretend they come without a
price, and IMO we should think carefully whether their use is
justified before we use them in each and every case.

> Emacs Lisp can learn from interesting ideas that other
> languages provide, adapt and add them, making them available to
> everyone.

It certainly can.  The question is: should it?  Since we are not
driven by any standard, it is completely up to us to make those
decisions, and we should IMO make them judiciously and carefully,
taking the downsides into consideration.  In particular, I hope people
agree that making a language too large and complex is not a good
thing in the long run.

> If a form expresses something that people frequently want to
> do, but either have to elaborate using (unless (memq ...) (push ...)),
> then we are making Emacs more useful and expressive by providing the
> functionality OOTB.  And isn't that the real point of Emacs Lisp?

AFAIU, what Richard says is that if some form is frequently used, we
should indeed consider adding a primitive or an API for it, but that
primitive or API doesn't have to look like CL.  Cf the pushnew
argument.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: João Távora <joaotavora@gmail.com>,
>>  adam@alphapapa.net, emacs-devel@gnu.org,  stefankangas@gmail.com
>> Date: Thu, 02 Nov 2023 08:55:54 +0000
>> 
>> Richard Stallman <rms@gnu.org> writes:
>> 
>> > It might be ok to add some keyword arguments to `sort', which are more
>> > unusual and complex to use, but not to simple constructs like
>> > `pushnew'.  This is Emacs Lisp, not Common Lisp.
>> 
>> What does that last sentence mean?
>
> AFAIU, it means that Emacs Lisp traditionally doesn't use keyword
> arguments, except as relatively rare exceptions.

It is used rather prominently in `define-minor-mode' or
`define-derived-mode', or do you specifically mean keyword arguments to
functions?

>> what constitutes "Emacs Lisp"?  It would
>> seem peculiar if it were to be defined by the arbitrary decisions of the
>> past, constrained by the contingent circumstances of the time.
>
> Those "arbitrary decisions" are what got us to where we are now, 40
> years later.  So some respect for those "arbitrary decisions" is due,
> I think.

No disrespect meant, but I am not sure we are thinking of the same
things.  An "arbitrary decision" usually doesn't matter much, like
calling a function rplacd or setcdr.  If a decision got us to where we
are now, I would say it wasn't that arbitrary, but a good one?

>> to me the main aspects and attractions of Emacs Lisp (in contrast to
>> CL and Scheme) have been:
>> 
>> 1. Not standardised; it is possible to extend the language without
>>    having to worry about how many implementations will follow along
>
> IMNSHO, extending Emacs Lisp as the language is not the main goal of
> Emacs development.  Emacs Lisp is not a programming language on its
> own, it is a language for implementing and extending features and
> extensions in Emacs.  Thus, the main goal of Emacs development is to
> develop applications and application-level features, and provide more
> opportunities for extending the core where that is considered useful.
> What we have in Emacs Lisp is IMO ample for that purpose.  Moreover,
> most participants in Emacs development are not experts in programming
> languages, their expertise is elsewhere (which is definitely a Good
> Thing).

Of course not extending it for its own sake, but I would assume that
making it easier for users to write practical and useful code should be
something that is valued.

> Objectively, adding new syntax and semantics to Emacs Lisp does make
> the source code harder to read and maintain, because it makes the
> language larger and requires familiarization with those new language
> features, which more often than not look and feel completely different
> from the "traditional" Emacs Lisp.  So even if we conclude that these
> additions are worthwhile, we should not pretend they come without a
> price, and IMO we should think carefully whether their use is
> justified before we use them in each and every case.

Could you explain what you mean by "traditional" Emacs Lisp?  But yes, I
am not advocating for senselessly adding whatever seems remotely
interesting, of course.

>> Emacs Lisp can learn from interesting ideas that other
>> languages provide, adapt and add them, making them available to
>> everyone.
>
> It certainly can.  The question is: should it?  Since we are not
> driven by any standard, it is completely up to us to make those
> decisions, and we should IMO make them judiciously and carefully,
> taking the downsides into consideration.  In particular, I hope people
> agree that making a language too large and complex is not a good
> thing in the long run.
>
>> If a form expresses something that people frequently want to
>> do, but either have to elaborate using (unless (memq ...) (push ...)),
>> then we are making Emacs more useful and expressive by providing the
>> functionality OOTB.  And isn't that the real point of Emacs Lisp?
>
> AFAIU, what Richard says is that if some form is frequently used, we
> should indeed consider adding a primitive or an API for it, but that
> primitive or API doesn't have to look like CL.  Cf the pushnew
> argument.

Right, I agree with everything here.  The functionality, not the style
is of interest.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Bob Rogers]

   From: Philip Kaludercic <philipk@posteo.net>
   Date: Thu, 02 Nov 2023 21:26:00 +0000

   Eli Zaretskii <eliz@gnu.org> writes:

   >> From: Philip Kaludercic <philipk@posteo.net>
   >> Cc: João Távora <joaotavora@gmail.com>,
   >>  adam@alphapapa.net, emacs-devel@gnu.org,  stefankangas@gmail.com
   >> Date: Thu, 02 Nov 2023 08:55:54 +0000
   >> 
   >> Richard Stallman <rms@gnu.org> writes:
   >> 
   >> > It might be ok to add some keyword arguments to `sort', which
   >> > are more unusual and complex to use, but not to simple
   >> > constructs like `pushnew'.  This is Emacs Lisp, not Common Lisp.
   >> 
   >> What does that last sentence mean?
   >
   > . . .

   . . .

   >> what constitutes "Emacs Lisp"?  It would seem peculiar if it were
   >> to be defined by the arbitrary decisions of the past, constrained
   >> by the contingent circumstances of the time.
   >
   > Those "arbitrary decisions" are what got us to where we are now, 40
   > years later.  So some respect for those "arbitrary decisions" is due,
   > I think.

   No disrespect meant, but I am not sure we are thinking of the same
   things.  An "arbitrary decision" usually doesn't matter much, like
   calling a function rplacd or setcdr.  If a decision got us to where we
   are now, I would say it wasn't that arbitrary, but a good one?

I think we are not really talking about arbitrary decisions here, but
about language style decisions, which may seem arbitrary (especially if
you disagree with the style of the resulting language!) but do matter to
the consistency and coherence of the resulting language.  And, although
my experience only goes back to Emacs 18, I think the original Emacs
Lisp had a definite style that set it apart from other Lisp dialects.
But that was a long time ago, and Emacs Lisp has grown enormously (as
someone else pointed out early in the original thread) to support the
enormous growth in Emacs, so that original style is now much harder to
see.  (Indeed, I notice it mostly when updating my old code.)

   So, Richard, I see that you are fighting to preserve something real
and important -- your vision of Emacs Lisp as a coherent language -- and
I believe that may be your prerogative as its creator.  But I also
believe you may be a decade or two too late.

					-- Bob Rogers
					   http://www.rgrjr.com/

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Bob Rogers wrote:

> I think we are not really talking about arbitrary decisions
> here, but about language style decisions, which may seem
> arbitrary (especially if you disagree with the style of the
> resulting language!) but do matter to the consistency and
> coherence of the resulting language. And, although my
> experience only goes back to Emacs 18, I think the original
> Emacs Lisp had a definite style that set it apart from other
> Lisp dialects. But that was a long time ago, and Emacs Lisp
> has grown enormously (as someone else pointed out early in
> the original thread) to support the enormous growth in
> Emacs, so that original style is now much harder to see.
> (Indeed, I notice it mostly when updating my old code.)

So let's hear it then, what is it that sets Emacs 18 Elisp
apart from Emacs 30 Elisp

> So, Richard, I see that you are fighting to preserve
> something real and important -- your vision of Emacs Lisp as
> a coherent language --

and how is the Emacs 30 Elisp less coherent than the
Emacs 18 Elisp?

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Bob Rogers]

   From: Emanuel Berg <incal@dataswamp.org>
   Date: Fri, 03 Nov 2023 03:21:30 +0100

   Bob Rogers wrote:

   > I think we are not really talking about arbitrary decisions here,
   > but about language style decisions . . .

   So let's hear it then, what is it that sets Emacs 18 Elisp
   apart from Emacs 30 Elisp . . .

Briefly, it's now a much larger language . . .

   and how is the Emacs 30 Elisp less coherent than the Emacs 18 Elisp?

   -- 
   underground experts united
   https://dataswamp.org/~incal

. . . and the additions have come from diverse sources.  It started as a
Maclisp subset with extensions to support editing.  The issue is that
the programming features have grown, and not only from Common Lisp,
moving it away from its Maclisp roots.  (This, I believe, is the gist of
Richard's complaint.)  And the language editing features have (not
surprisingly) grown with Emacs' growth as an editor, as they have had to
do, which has unavoidably added to the learning curve you have to climb
before you can contribute to Emacs.  It has also made the language more
about editing and less about programming, which IMHO makes the style
question less acute.

					-- Bob

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Bob Rogers wrote:

>> So let's hear it then, what is it that sets Emacs 18 Elisp
>> apart from Emacs 30 Elisp . . .
>
> Briefly, it's now a much larger language . . .
>
>> and how is the Emacs 30 Elisp less coherent than the Emacs
>> 18 Elisp?
>
> . . . and the additions have come from diverse sources.
> It started as a Maclisp subset with extensions to support
> editing. The issue is that the programming features have
> grown, and not only from Common Lisp, moving it away from
> its Maclisp roots. (This, I believe, is the gist of
> Richard's complaint.) And the language editing features have
> (not surprisingly) grown with Emacs' growth as an editor, as
> they have had to do, which has unavoidably added to the
> learning curve you have to climb before you can contribute
> to Emacs. It has also made the language more about editing
> and less about programming, which IMHO makes the style
> question less acute.

I understand that it has expanded in size, this is the nature
of technology systems in general and perhaps even more so with
Lisp programming, since it is so easy and fun to do.

But I am still uncertain how this has changed the style and
reduced the coherence of Elisp.

Case in point, I don't see any such difference when I look at
cl-lib.el?

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Bob Rogers]

   From: Emanuel Berg <incal@dataswamp.org>
   Date: Fri, 03 Nov 2023 21:32:15 +0100

   Bob Rogers wrote:

   >> So let's hear it then, what is it that sets Emacs 18 Elisp
   >> apart from Emacs 30 Elisp . . .
   >
   > Briefly, it's now a much larger language . . .
   >
   >> and how is the Emacs 30 Elisp less coherent than the Emacs
   >> 18 Elisp?
   >
   > . . . and the additions have come from diverse sources . . .

   I understand that it has expanded in size, this is the nature
   of technology systems in general and perhaps even more so with
   Lisp programming, since it is so easy and fun to do.

But, just to point out the obvious, Emacs Lisp is unusual in that the
language and the application are inseparable.  In other language systems
(e.g. Python, Perl) the language, the application, and packages used to
build the application (even when distributed as part of the language)
are all distinct; in Emacs the boundaries are much fuzzier, undoubtedly
another consequence of Lisp.

   But I am still uncertain how this has changed the style and
   reduced the coherence of Elisp.

I'm sorry I'm apparently unable to be more concrete about explaining my
impressions.  Part of this is due to the elapsed time.  ISTR that, when
I started reading Elisp in the mid-90's, it had a "retro" feel compared
to the other Lisps I was using.  As time went on, I've had to hit C-h f
more and more to understand what was going on, not only for language
features (pcase has been mentioned), but for editing features like
indirect buffers and overlays as they were added -- not strictly part of
the language, but again, where's do you draw the boundary?  (I've also
had to read more C code to follow what's going on, but that's a separate
issue.)

   Case in point, I don't see any such difference when I look at
   cl-lib.el?

   -- 
   underground experts united
   https://dataswamp.org/~incal

Between cl-lib.el and what?

					-- Bob

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Bob Rogers wrote:

> I'm sorry I'm apparently unable to be more concrete about
> explaining my impressions. Part of this is due to the
> elapsed time. ISTR that, when I started reading Elisp in the
> mid-90's, it had a "retro" feel compared to the other Lisps
> I was using. As time went on, I've had to hit C-h f more and
> more to understand what was going on, not only for language
> features (pcase has been mentioned), but for editing
> features like indirect buffers and overlays as they were
> added -- not strictly part of the language, but again,
> where's do you draw the boundary?

To be honest, I have not seen any.

>> Case in point, I don't see any such difference when I look
>> at cl-lib.el?
>
> Between cl-lib.el and what?

We have to get the Emacs 18 source so we can compare :)

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Stephen Berman]

On Fri, 03 Nov 2023 23:46:22 +0100 Emanuel Berg <incal@dataswamp.org> wrote:

> Bob Rogers wrote:
>
>> I'm sorry I'm apparently unable to be more concrete about
>> explaining my impressions. Part of this is due to the
>> elapsed time. ISTR that, when I started reading Elisp in the
>> mid-90's, it had a "retro" feel compared to the other Lisps
>> I was using. As time went on, I've had to hit C-h f more and
>> more to understand what was going on, not only for language
>> features (pcase has been mentioned), but for editing
>> features like indirect buffers and overlays as they were
>> added -- not strictly part of the language, but again,
>> where's do you draw the boundary?
>
> To be honest, I have not seen any.
>
>>> Case in point, I don't see any such difference when I look
>>> at cl-lib.el?
>>
>> Between cl-lib.el and what?
>
> We have to get the Emacs 18 source so we can compare :)

You can find several versions here:

https://github.com/larsbrinkhoff/emacs-history

Steve Berman

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Stephen Berman wrote:

>> We have to get the Emacs 18 source so we can compare :)
>
> You can find several versions here:
>
> https://github.com/larsbrinkhoff/emacs-history

Several, including Emacs 18!

I followed the link to this page:

  http://bitsavers.org/bits/MIT/gnu/emacs_18.41.tar.gz

Now we can see clearly how the style has changed :)

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >    So, Richard, I see that you are fighting to preserve something real
  > and important -- your vision of Emacs Lisp as a coherent language -- and
  > I believe that may be your prerogative as its creator.  But I also
  > believe you may be a decade or two too late.

It is not too late if we find a few more contributors who appreciate
what is at stake.

The consistent style of Emacs Lisp is one part of it.
But there is a more practical part, which I've explained recently:
the duplication, even triplication of features for the same job.
The result of this is to multiply what one needs to know
to understand the code in Emacs.

If Emacs Lisp had only one interface for doing a particular job, it
would be enough to learn that one.  But when it has several different
interfaces for doing the same job, you end up needing to know all
those interfaces.   Any one file probably uses one of the interfaces;
to understand that file, knowing that one interface is emnough.
But the next file you look at probably uses a different interface,
and the next one yet another interface -- so in practice you need to
know all the redundant interfaces.

This puts an avoidable extra a burden on everyone who wants to read
the Lisp code of Emacs.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Richard Stallman wrote:

>> So, Richard, I see that you are fighting to preserve
>> something real and important -- your vision of Emacs Lisp
>> as a coherent language -- and I believe that may be your
>> prerogative as its creator. But I also believe you may be
>> a decade or two too late.
>
> It is not too late if we find a few more contributors who
> appreciate what is at stake [...]

Maybe, but how?

We have `cl-incf' and can't remove it as that would break the
CL emulation interface.

To the people who use it, and don't care about the CL
emulation part necessarily, they just think it is useful, we
can't say instead use

  (setq var (1+ var))

because we can't make the case that that is in any way better
than

  (cl-incf var)

We can however add a new function, "incf", that does the same
thing, only does not rely upon cl-lib in any way.

But how would this help the maintainers? Instead of
familiarizing themselves with `cl-incf', and all other
functions provided by cl-lib that are in use and people will
continue to use, they will have to learn the new functions
replacing them? So what difference will that make?

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Björn Bidar]

Emanuel Berg <incal@dataswamp.org> writes:

> But how would this help the maintainers? Instead of
> familiarizing themselves with `cl-incf', and all other
> functions provided by cl-lib that are in use and people will
> continue to use, they will have to learn the new functions
> replacing them? So what difference will that make?

It does sound more and more the fight for some kind of language purity
to not use cl-lib. Removed from reality like Don Quixote.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Alan Mackenzie]

Hello, Björn.

On Wed, Nov 08, 2023 at 17:00:06 +0200, Björn Bidar wrote:
> Emanuel Berg <incal@dataswamp.org> writes:

> > But how would this help the maintainers? Instead of
> > familiarizing themselves with `cl-incf', and all other
> > functions provided by cl-lib that are in use and people will
> > continue to use, they will have to learn the new functions
> > replacing them? So what difference will that make?

> It does sound more and more the fight for some kind of language purity
> to not use cl-lib. Removed from reality like Don Quixote.

That is verging on the offensively patronising, and I don't think it is
appropriate language for this mailing list.

Just take one look at this "reality" you're so supportive of: the
widespread use of cl-lib, not just in people's own projects, but
throughout the core of Emacs, has multiplied the size of Lisp language
part of Emacs by a factor of around 3.   This is a gross increase in
complexity for maintainers that is not justified by the slight increase
in facility that cl-lib (along with things like seq.el and oclosures)
gives.

Throughout this long discussion, this indiscriminate use of cl-lib has
been supported only by occasional contributers.  Those who actually
maintain other people's code, apart from (I think) Eli, Richard and
myself, have been conspicuously silent.  None of us three have favoured
such use of cl-lib.

Occasional contributors may be fascinated by cl-lib, and learn enough of
it to use random bits of it in their code.  The trouble is, each such
contributor uses a different piece of cl-lib, with the result that those
who end up maintaining it need to know a far greater part of it just to
cope.

This factor of 3 is, I believe, a significant barrier to new programmers
coming into Emacs; Elisp is just that much more difficult than it was in
the past.  And it isn't just for newcomers that it is more difficult.  I
spend a significant amount of debugging time having to look up doc
strings and manual pages for obscure cl-lib (etc.) functions.  This is
lost time, representing a diminution of my capacity to maintain Emacs.
I suspect it is similar for other maintainers.

That is the current reality.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Alan Mackenzie wrote:

> Occasional contributors may be fascinated by cl-lib, and
> learn enough of it to use random bits of it in their code.
> The trouble is, each such contributor uses a different piece
> of cl-lib, with the result that those who end up maintaining
> it need to know a far greater part of it just to cope.

Yes, this sounds like a good explanation how the situation has
appeared. We see how two worlds collide.

But how do you propose to improve the situation, then?

Maybe maintaining can be distributed so that the "occasional
contributors" are maintaining their own code?

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 08/11/2023 19:18, Alan Mackenzie wrote:
> Throughout this long discussion, this indiscriminate use of cl-lib has
> been supported only by occasional contributers.  Those who actually
> maintain other people's code, apart from (I think) Eli, Richard and
> myself, have been conspicuously silent.

I like cl-lib (it could be optimized better still, but it's good for 
certain uses), and this discussion doesn't look like it will reach any 
constructive conclusion, so it didn't seem worth it to spend time on it.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 992 bytes --]

On Wed, Nov 8, 2023, 20:20 Dmitry Gutov <dmitry@gutov.dev> wrote:

> On 08/11/2023 19:18, Alan Mackenzie wrote:
> > Throughout this long discussion, this indiscriminate use of cl-lib has
> > been supported only by occasional contributers.  Those who actually
> > maintain other people's code, apart from (I think) Eli, Richard and
> > myself, have been conspicuously silent.
>
> I like cl-lib (it could be optimized better still, but it's good for
> certain uses), and this discussion doesn't look like it will reach any
> constructive conclusion, so it didn't seem worth it to spend time on it.
>

If it needed any confirmation, I too like cl-lib and I too help maintain
other people's code in the Emacs core tree as well as maintaining a number
of libraries I have authored. I read many styles of Elisp here and am not
really phased by any particular programming technique being discussed here
(cl-lib, seq, pcase, verbose while loops with catch/throw, etc etc).

João

[-- Attachment #2: Type: text/html, Size: 1645 bytes --]

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

João Távora wrote:

>>> Throughout this long discussion, this indiscriminate use
>>> of cl-lib has been supported only by occasional
>>> contributers. Those who actually maintain other people's
>>> code, apart from (I think) Eli, Richard and myself, have
>>> been conspicuously silent.
>>
>> I like cl-lib (it could be optimized better still, but it's
>> good for certain uses), and this discussion doesn't look
>> like it will reach any constructive conclusion, so it
>> didn't seem worth it to spend time on it.
>
> If it needed any confirmation, I too like cl-lib and I too
> help maintain other people's code in the Emacs core tree as
> well as maintaining a number of libraries I have authored.
> I read many styles of Elisp here and am not really phased by
> any particular programming technique being discussed here
> (cl-lib, seq, pcase, verbose while loops with catch/throw,
> etc etc).

I like cl-lib and I maintain an ELPA package [1] - ELPA, as we
know, is part of GNU Emacs. It doesn't use cl-lib. Okay,
so bad example then.

[1] https://elpa.gnu.org/packages/wrap-search.html

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Alan Mackenzie]

Hello, João

On Wed, Nov 08, 2023 at 21:10:00 +0000, João Távora wrote:
>    On Wed, Nov 8, 2023, 20:20 Dmitry Gutov <[1]dmitry@gutov.dev> wrote:

>      On 08/11/2023 19:18, Alan Mackenzie wrote:
>>      > Throughout this long discussion, this indiscriminate use of
>>      > cl-lib has been supported only by occasional contributers.Â
>>      > Those who actually maintain other people's code, apart from (I
>>      > think) Eli, Richard and myself, have been conspicuously
>>      > silent.

>>      I like cl-lib (it could be optimized better still, but it's good
>>      for certain uses), and this discussion doesn't look like it will
>>      reach any constructive conclusion, so it didn't seem worth it to
>>      spend time on it.

>    If it needed any confirmation, I too like cl-lib and I too help
>    maintain other people's code in the Emacs core tree as well as
>    maintaining a number of libraries I have authored.

There's a difference between liking cl-lib and advocating its
indiscriminate use.  I don't think you've done the latter in this (and
related) threads.

Nobody who likes cl-lib has yet addressed the point made by Richard and
(less eloquently) by me, namely that the incorporation and use of cl-lib
swells the size and complexity of Emacs Lisp to the point of making
maintenance difficult.  What is your view on this important point?

>    I read many styles of Elisp here and am not really phased by any
>    particular programming technique being discussed here (cl-lib, seq,
>    pcase, verbose while loops with catch/throw, etc etc).  João

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Thu, Nov 9, 2023 at 10:05 AM Alan Mackenzie <acm@muc.de> wrote:

> >    If it needed any confirmation, I too like cl-lib and I too help
> >    maintain other people's code in the Emacs core tree as well as
> >    maintaining a number of libraries I have authored.
>
> There's a difference between liking cl-lib and advocating its
> indiscriminate use.  I don't think you've done the latter in this (and
> related) threads.

Yes, you're right.  Indeed I don't' advocate for its indiscriminate use,
just as I don't advocate for indiscriminate use of anything, except
perhaps drinking water and brushing teeth.

> Nobody who likes cl-lib has yet addressed the point made by Richard and
> (less eloquently) by me, namely that the incorporation and use of cl-lib
> swells the size and complexity of Emacs Lisp to the point of making
> maintenance difficult.  What is your view on this important point?

That it doesn't make maintenance any more difficult than any other
Elisp construct, be it very old and curiously named like 'rplacd' or
much, much newer like `seq-do` or `pcase-lambda`.

My specific view on cl-lib.el is that it brings us a small part of
the results of  non-trivial design work put in when important figures
in the Lisp world met  regularly for many years to deliver what has
proved to become excellent,  battle-tested, widely understood and
impeccably documented programming abstractions.

What I'm reading so far in this long discussion is that the argument
of its detractors isn't really that cl-lib isn't good, but that
it is superfluous and that learning it is a burden on maintainers.
Well, it's just as superfluous as all of Elisp apart from two handfuls
of primitives, I guess.  Or any programming language for that matter, if
you know enough machine code.  Or any other programming abstraction I
happen not to be familiar with.

Also I seem to hear that Elisp has some kind of ideal hard-to-define
identity or fingerprint and that it shouldn't try to become anything
else.  But this argument is very strange given Elisp is, like any
decent language, easy to extend.  Not to mention I struggle to see
the technical advantage in uniqueness for uniqueness sake.  A good
part of Elisp is about a special purpose function: editing buffers,
and an the equally important part is about doing things with bits in
memory, there's no reason to advocate for singularity here.

I also hear Elisp shouldn't become Common Lisp, but not only is the
use of cl-lib.el nowhere a step in that direction, but also -- somewhat
ironically -- if Elisp _were_ Common Lisp, then that hard-to-define
identity would be much easier to define and language extension would be
much easier to organize into compartments to facilitate policy-making.

Again, the only thing that has brought Elisp any closer to Common
Lisp significantly, was the introduction of lexical binding some 10
years ago.  Elisp looks a lot different now than it did in the 90's.
Closures everywhere, higher-order functions!  Shudder!

There's even talk of a continuation-passing style library, to
be called future.el or promise.el or something.  Oh dear, what
will be of us that we will have to evolve like any other language
in the world!

So I propose we let programmers use their judgement.  Really
respect people who write code for no money and give the copyright
away to the FSF.  Maybe suggest guidelines such as not introduce
cl-loop where a dolist would do the job just as economically and
elegantly. Don't use higher-order functions just because it looks
cool. But maybe do suggest to use cl-position with a :key and a
:test instead of a complex while loop with an auxiliary variable.
Or cl-set-difference instead of nested loops.  Suggest to express
intent, use abstractions.   Suggest to be consistent, be scrutinous,
be "discriminate", be mindful of the style of the current area
you are working on.

But don't suggest anything too hard, especially if it's not
modifying code you have authored.  Don't use arguments of authority
when you can point to specific things.  Be generally respectful of
people putting in any good work even if you don't like the style,
and try to learn a new thing or two every now and then.

BTW here are some nice generic suggestions from the Lisp world,
written by two fenomenal programmers.  I love reading this from
time to time:

   https://www.cs.umd.edu/~nau/cmsc421/norvig-lisp-style.pdf

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Thu, 9 Nov 2023 11:06:02 +0000
> Cc: Dmitry Gutov <dmitry@gutov.dev>,
>  Björn Bidar <bjorn.bidar@thaodan.de>, 
>  emacs-devel <emacs-devel@gnu.org>
> 
> So I propose we let programmers use their judgement.

That's fair enough, provided that those same programmers step up to
become the Emacs maintainers, and thus share the burden to at least
some extent.  Especially those of them who consider witty sarcasm a
useful style in a discussion like this one.

The following is left as an exercise for the interested readers: take
a non-trivial piece of code that you never in your life have laid your
eyes upon, and decide what to do with a serious bug report about that
code, with or without a patch proposed by someone.  A recent case in
point: bug#66970.  Bonus points for doing this with code in a
programming language you don't command 110%, like C if you are mainly
a Lisp programmer or vice versa.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Thu, Nov 9, 2023 at 11:38 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Thu, 9 Nov 2023 11:06:02 +0000
> > Cc: Dmitry Gutov <dmitry@gutov.dev>,
> >  Björn Bidar <bjorn.bidar@thaodan.de>,
> >  emacs-devel <emacs-devel@gnu.org>
> >
> > So I propose we let programmers use their judgement.
>
> That's fair enough, provided that those same programmers step up to
> become the Emacs maintainers, and thus share the burden to at least
> some extent.  Especially those of them who consider witty sarcasm a
> useful style in a discussion like this one.

Did you read the rest of the paragraph from which you extracted that
one line?  I'm advocating for being discriminate and

Also, I would hope this is not a suggestion that I don't step to share the
burden of maintaining things?  I do, certainly for all the features that I
have I added to core in the past 10 years and frequently also to parts
of the code those features interact with.  Do you often have to fix bugs in
flymake.el, elec-pair.el, icomplete.el, eldoc.el, jsonrpc.el, etc...
that I didn't care to take a look at in a reasonable time frame?  More
importantly, to what extent are you and others significantly or often
hindered by the style I use there?

I can't address what I don't know about.  I can't track the bug tracker
as closely as you do.  If I'm missing or late about bug reports about
such things, just ping me like you usually do.

Or are you suggesting I should dumb down my style for when I'm dead and
my code survives me?  Just so you know, I think that's not entirely
unreasonable.  In fact if you read the pdf I linked to, it's a valid
criteria for making decisions.   But it's definitely not the only one.
Who knows if Emacs 40's maintainer wouldn't have appreciated it if I
had smartened it up even more and use _only_ cl-lib.

Anyway, if there's some code of mine that's bothering you and you
find inescrutable, point me to it and we can discuss it, of course.

> The following is left as an exercise for the interested readers: take
> a non-trivial piece of code that you never in your life have laid your
> eyes upon, and decide what to do with a serious bug report about that
> code, with or without a patch proposed by someone.  A recent case in
> point: bug#66970.  Bonus points for doing this with code in a
> programming language you don't command 110%, like C if you are mainly
> a Lisp programmer or vice versa.

Do you need help reading some Elisp construct or debugging?
Is this meaningfully related to style in Lisp code?  You want help
reading hangul.el?  (which doesn't seem to use cl-lib.el btw)

Would you like help rewriting hangul-insert-character to support
overwrite-mode, perhaps by leveraging post-self-insert-hook?  I can
help with that.  If there is some deep required specific domain
knowledge about input methods or how the Korean language works, you
or someone else may have to help out with that.  If C code is involved,
it's not a problem.  I'll have a better look later.

The problem of maintaining other people's code (which btw is 95%
of all my professional experience) is not about programming techniques
or even languages at all, it's about domain knowledge.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Thu, Nov 9, 2023 at 12:34 PM João Távora <joaotavora@gmail.com> wrote:

> > The following is left as an exercise for the interested readers: take
> > a non-trivial piece of code that you never in your life have laid your
> > eyes upon, and decide what to do with a serious bug report about that
> > code, with or without a patch proposed by someone.  A recent case in
> > point: bug#66970.  Bonus points for doing this with code in a
> > programming language you don't command 110%, like C if you are mainly
> > a Lisp programmer or vice versa.

>  I'll have a better look later.

At first sight, didn't seem terribly hard.  I gave it some basic testing
seems to work like the OP in that bug would like it to.  Could be a start
I guess.  Let me know if you don't like the style.

diff --git a/lisp/leim/quail/hangul.el b/lisp/leim/quail/hangul.el
index 46a2e5a6ba2..c97362b1250 100644
--- a/lisp/leim/quail/hangul.el
+++ b/lisp/leim/quail/hangul.el
@@ -146,21 +146,26 @@ hangul-insert-character
       (progn
         (delete-region (region-beginning) (region-end))
         (deactivate-mark)))
-  (quail-delete-region)
-  (let ((first (car queues)))
-    (insert
-     (hangul-character
-      (+ (aref first 0) (hangul-djamo 'cho (aref first 0) (aref first 1)))
-      (+ (aref first 2) (hangul-djamo 'jung (aref first 2) (aref first 3)))
-      (+ (aref first 4) (hangul-djamo 'jong (aref first 4) (aref first 5))))))
-  (move-overlay quail-overlay (overlay-start quail-overlay) (point))
-  (dolist (queue (cdr queues))
-    (insert
-     (hangul-character
-      (+ (aref queue 0) (hangul-djamo 'cho (aref queue 0) (aref queue 1)))
-      (+ (aref queue 2) (hangul-djamo 'jung (aref queue 2) (aref queue 3)))
-      (+ (aref queue 4) (hangul-djamo 'jong (aref queue 4) (aref queue 5)))))
-    (move-overlay quail-overlay (1+ (overlay-start quail-overlay)) (point))))
+  (let* ((things-to-insert
+          (with-temp-buffer
+            (dolist (queue queues (mapcar #'identity (buffer-string)))
+              (insert
+               (hangul-character
+                (+ (aref queue 0) (hangul-djamo 'cho (aref queue 0)
(aref queue 1)))
+                (+ (aref queue 2) (hangul-djamo 'jung (aref queue 2)
(aref queue 3)))
+                (+ (aref queue 4) (hangul-djamo 'jong (aref queue 4)
(aref queue 5))))))))
+         (may-have-to-overwrite-p
+          (or (= (overlay-start quail-overlay) (overlay-end quail-overlay))
+              (cdr things-to-insert))))
+    (quail-delete-region)
+    (dolist (c things-to-insert)
+      (let ((last-command-event c)
+            (overwrite-mode (and overwrite-mode
+                                 may-have-to-overwrite-p
+                                 overwrite-mode)))
+        (self-insert-command 1)
+        (setq may-have-to-overwrite-p nil)))
+    (move-overlay quail-overlay (1- (point)) (point))))

 (defun hangul-djamo (jamo char1 char2)
   "Return the double Jamo index calculated from the arguments.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Thu, 9 Nov 2023 12:34:09 +0000
> Cc: acm@muc.de, dmitry@gutov.dev, bjorn.bidar@thaodan.de, emacs-devel@gnu.org
> 
> On Thu, Nov 9, 2023 at 11:38 AM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > > From: João Távora <joaotavora@gmail.com>
> > > Date: Thu, 9 Nov 2023 11:06:02 +0000
> > > Cc: Dmitry Gutov <dmitry@gutov.dev>,
> > >  Björn Bidar <bjorn.bidar@thaodan.de>,
> > >  emacs-devel <emacs-devel@gnu.org>
> > >
> > > So I propose we let programmers use their judgement.
> >
> > That's fair enough, provided that those same programmers step up to
> > become the Emacs maintainers, and thus share the burden to at least
> > some extent.  Especially those of them who consider witty sarcasm a
> > useful style in a discussion like this one.
> 
> Did you read the rest of the paragraph from which you extracted that
> one line?

You should by now know me well enough to realize that I read
everything here.  However, I respond only to the parts where I have
something useful and non-trivial to say.

> Also, I would hope this is not a suggestion that I don't step to share the
> burden of maintaining things?  I do, certainly for all the features that I
> have I added to core in the past 10 years and frequently also to parts
> of the code those features interact with.  Do you often have to fix bugs in
> flymake.el, elec-pair.el, icomplete.el, eldoc.el, jsonrpc.el, etc...
> that I didn't care to take a look at in a reasonable time frame?  More
> importantly, to what extent are you and others significantly or often
> hindered by the style I use there?

I said nothing about your style.  Neither do I doubt that you know
your own code very well.  We all know our own code well.  I'm asking
you not to judge the burden of the maintainers in maintaining code
they didn't write nor even see, until you get to our place and do it
for a while.  Only then you will be able to realize the role of
obscure programming style and use of unfamiliar syntax in making the
job harder than it has to be.

> The problem of maintaining other people's code (which btw is 95%
> of all my professional experience) is not about programming techniques
> or even languages at all, it's about domain knowledge.

It is both.  In particular, when you need to understand why some code
fragment misbehaves, you need to understand the implementation and
what it intends to do, not only why.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Eli Zaretskii wrote:

> We all know our own code well. I'm asking you not to judge
> the burden of the maintainers in maintaining code they
> didn't write nor even see [...]

We should distribute the maintenance effort so that to
a higher degree, people who wrote the code are maintaining it,
and, whenever that cannot be achieved, at least people who
take on "I'll maintain this package", it will then be
understood, for example, if the file contains a lot of cl-lib,
the maintainer who takes on that file will have to deal
with it, because it comes with the task.

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Thu, 09 Nov 2023 16:29:45 +0100
> 
> Eli Zaretskii wrote:
> 
> > We all know our own code well. I'm asking you not to judge
> > the burden of the maintainers in maintaining code they
> > didn't write nor even see [...]
> 
> We should distribute the maintenance effort so that to
> a higher degree, people who wrote the code are maintaining it,

This is a volunteer-based project, so we have no way of "distributing
the maintenance effort", unless people volunteer to do it.  In
particular, for absolute majority of the Emacs code, the people who
wrote the code are no longer on board.  That's the reality.

If you mean that the authors who are still here should be asked to
take care of their code, then this is already being done, always, and
cannot be done to any higher degree.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Eli Zaretskii wrote:

> This is a volunteer-based project, so we have no way of
> "distributing the maintenance effort", unless people
> volunteer to do it. In particular, for absolute majority of
> the Emacs code, the people who wrote the code are no longer
> on board. That's the reality.
>
> If you mean that the authors who are still here should be
> asked to take care of their code, then this is already being
> done, always, and cannot be done to any higher degree.

Yes, I understand, but we can still just hand out files to
people, "now you are maintaining this file". That person will
have to agree, of course.

For each such file handed out there is one file less for the
"maintainers of everything else" to handle.

Because what we heard was that there are often just a few uses
of cl-lib in some person's code, and that is manageable to
learn for the maintainer. Only in some other person's code,
there are also just a few such cases, but not necessarily the
same, so the maintainers of a lot of files, for them the
complexity increases for each file even tho the individual
files are not complex in their us of, say, cl-lib, really.

So if we give away a bunch of files to a larger group of
people, those people will not have an issue with cl-lib,
really, and at the same time both scope and complexity will be
reduced for the "everything else" maintainers.

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Sat, 11 Nov 2023 14:10:20 +0100
> 
> Eli Zaretskii wrote:
> 
> > This is a volunteer-based project, so we have no way of
> > "distributing the maintenance effort", unless people
> > volunteer to do it. In particular, for absolute majority of
> > the Emacs code, the people who wrote the code are no longer
> > on board. That's the reality.
> >
> > If you mean that the authors who are still here should be
> > asked to take care of their code, then this is already being
> > done, always, and cannot be done to any higher degree.
> 
> Yes, I understand, but we can still just hand out files to
> people, "now you are maintaining this file". That person will
> have to agree, of course.

In the real world, where I live, this is also unbecoming, and thus
practically impossible.  Good luck leading a volunteer project with
these ideas!

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Eli Zaretskii wrote:

>>> This is a volunteer-based project, so we have no way of
>>> "distributing the maintenance effort", unless people
>>> volunteer to do it. In particular, for absolute majority
>>> of the Emacs code, the people who wrote the code are no
>>> longer on board. That's the reality.
>>>
>>> If you mean that the authors who are still here should be
>>> asked to take care of their code, then this is already
>>> being done, always, and cannot be done to any
>>> higher degree.
>> 
>> Yes, I understand, but we can still just hand out files to
>> people, "now you are maintaining this file". That person
>> will have to agree, of course.
>
> In the real world, where I live, this is also unbecoming,
> and thus practically impossible. Good luck leading
> a volunteer project with these ideas!

What do you mean?

Why can't more people be involved?

But if they can't, then yes: the remaining three guys
maintaining hundreds of files will have to deal with a lot
of complexity, for sure.

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Sat, 11 Nov 2023 15:07:44 +0100
> 
> Eli Zaretskii wrote:
> 
> >> Yes, I understand, but we can still just hand out files to
> >> people, "now you are maintaining this file". That person
> >> will have to agree, of course.
> >
> > In the real world, where I live, this is also unbecoming,
> > and thus practically impossible. Good luck leading
> > a volunteer project with these ideas!
> 
> What do you mean?

I mean that's not how things work in this and other volunteer-based
projects.

> Why can't more people be involved?

They can, just not as simply as you put it, and what can be done is
already being done, by and large.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Eli Zaretskii wrote:

>> What do you mean?
>
> I mean that's not how things work in this and other
> volunteer-based projects.

Okay, I think it would be an improvement if more people did,
but each on a smaller number of files.

>> Why can't more people be involved?
>
> They can, just not as simply as you put it, and what can be
> done is already being done, by and large.

Did you ask, for example, on this list?

I have noticed in the package manager there are 70 packages
that are "built-in".

Maybe you can keep up the modularization process of core Emacs
in this way and create more and also see what people are up to
maintaining one or a couple of such packages?

I say this in connection to this issue as maintaining
a "package" sounds more appealing than maintaining some .el
file, I think more people would be attracted to it if phrased
and organized that way.

  allout                         2.3            built-in              extensive outline mode for use alone and with other modes
  allout-widgets                 1.0            built-in              Visually highlight allout outline structure.
  ansi-color                     3.4.2          built-in              translate ANSI escape sequences into faces
  antlr-mode                     2.2.3          built-in              major mode for ANTLR grammar files
  auth-source-pass               5.0.0          built-in              Integrate auth-source with password-store
  backtrace                      1.0            built-in              generic major mode for Elisp backtraces
  bind-key                       2.4.1          built-in              A simple way to manage personal keybindings
  cc-mode                        5.33.1         built-in              major mode for editing C and similar languages
  cedet                          2.0            built-in              Setup CEDET environment
  cfengine                       1.4            built-in              mode for editing Cfengine files
  cl-generic                     1.0            built-in              CLOS-style generic functions for Elisp
  cl-lib                         1.0            built-in              Common Lisp extensions for Emacs
  cl-print                       1.0            built-in              CL-style generic printing
  ede                            2.0            built-in              Emacs Development Environment gloss
  ediff                          2.81.6         built-in              a comprehensive visual interface to diff & patch
  eglot                          1.15           built-in              The Emacs Client for LSP servers
  eieio                          1.4            built-in              Enhanced Implementation of Emacs Interpreted Objects
  eieio-core                     1.4            built-in              Core implementation for eieio
  eldoc                          1.14.0         built-in              Show function arglist or variable docstring in echo area
  epg                            1.0.0          built-in              the EasyPG Library
  erc                            5.6snapshot    built-in              An Emacs Internet Relay Chat client
  eshell                         2.4.2          built-in              the Emacs command shell
  external-completion            0.1            built-in              Let external tools control completion style
  faceup                         0.0.6          built-in              Markup language for faces and font-lock regression testing
  feedmail                       11             built-in              assist other email packages to massage outgoing messages
  find-cmd                       0.6            built-in              Build a valid find(1) command with sexps
  flymake                        1.3.7          built-in              A universal on-the-fly syntax checker
  flymake-proc                   1.0            built-in              Flymake backend for external tools
  foldout                        1.10           built-in              folding extensions for outline-mode and outline-minor-mode
  gnus                           5.13           built-in              a newsreader for GNU Emacs
  htmlfontify                    0.21           built-in              htmlize a buffer/source tree with optional hyperlinks
  idlwave                        6.1.22         built-in              IDL editing mode for GNU Emacs
  image-dired                    0.5            built-in              use dired to browse and manipulate your images
  info-xref                      3              built-in              check external references in an Info document
  isearchb                       1.5            built-in              a marriage between iswitchb and isearch
  js                             9              built-in              Major mode for editing JavaScript
  json                           1.5            built-in              JavaScript Object Notation parser / generator
  jsonrpc                        1.0.17         built-in              JSON-RPC library
  let-alist                      1.0.6          built-in              Easily let-bind values of an assoc-list by their names
  map                            3.3.1          built-in              Map manipulation functions
  meta-mode                      1.0            built-in              major mode for editing Metafont or MetaPost sources
  mh-e                           8.6snapshot    built-in              GNU Emacs interface to the MH mail system
  mixal-mode                     0.4            built-in              Major mode for the mix asm language.
  nadvice                        1.0            built-in              Light-weight advice primitives for Elisp functions
  ntlm                           2.1.0          built-in              NTLM (NT LanManager) authentication support
  org                            9.6.10         built-in              Outline-based notes management and organizer
  package                        1.1.0          built-in              Simple package system for Emacs
  project                        0.10.0         built-in              Operations on the current project
  ps-mode                        1.1.9          built-in              PostScript mode for GNU Emacs
  pulse                          1.0            built-in              Pulsing Overlays
  python                         0.28           built-in              Python's flying circus support for Emacs
  ruby-mode                      1.2            built-in              Major mode for editing Ruby files
  ruby-ts-mode                   0.2            built-in              Major mode for editing Ruby files using tree-sitter
  semantic                       2.2            built-in              Semantic buffer evaluator.
  seq                            2.24           built-in              Sequence manipulation functions
  so-long                        1.1.2          built-in              Say farewell to performance problems with minified code.
  soap-client                    3.2.3          built-in              Access SOAP web services
  sql                            3.6            built-in              specialized comint.el for SQL interpreters
  srecode                        1.2            built-in              Semantic buffer evaluator.
  svg                            1.1            built-in              SVG image creation functions
  tabulated-list                 1.0            built-in              generic major mode for tabulated lists
  thunk                          1.0            built-in              Lazy form evaluation
  tildify                        4.6.1          built-in              adding hard spaces into texts
  tramp                          2.7.0pre       built-in              Transparent Remote Access, Multiple Protocol
  use-package                    2.4.5          built-in              A configuration macro for simplifying your .emacs
  vera-mode                      2.28           built-in              major mode for editing Vera files
  verilog-mode                   2023.6.6.141322628 built-in              major mode for editing verilog source in Emacs
  viper                          3.14.2         built-in              A full-featured Vi emulator for Emacs
  whitespace                     13.2.2         built-in              minor mode to visualize TAB, (HARD) SPACE, NEWLINE
  xref                           1.6.3          built-in              Cross-referencing commands

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Sat, 11 Nov 2023 15:54:29 +0100
> 
> Eli Zaretskii wrote:
> 
> >> Why can't more people be involved?
> >
> > They can, just not as simply as you put it, and what can be
> > done is already being done, by and large.
> 
> Did you ask, for example, on this list?

Yes, when that is appropriate.

> I have noticed in the package manager there are 70 packages
> that are "built-in".

See admin/MAINTAINERS.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Thu, Nov 9, 2023 at 3:05 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > that I didn't care to take a look at in a reasonable time frame?  More
> > importantly, to what extent are you and others significantly or often
> > hindered by the style I use there?
>
> I said nothing about your style.  Neither do I doubt that you know
> your own code very well.  We all know our own code well.  I'm asking
> you not to judge the burden of the maintainers in maintaining code
> they didn't write nor even see, until you get to our place and do it
> for a while.

Okay, and what I tried to say is that I _do_ have a valid judgement
of exactly this burden, to some quite significant degree, because not only
do I maintain "my" code, I have to wade through a lot of code that
I didn't write.  I didn't write any of icomplete.el and yet managed
to significantly enhance it, for example.  Or elisp-mode.el, or
minibuffer.el.  Or flymake.el, which I rewrote, backward compatibly.
Or eldoc.el, where I did likewise.  Or electric.el or so many other
pieces of code I have to visit and read to make sense of Emacs.
These aren't trivial files they're much larger.  I go
wherever M-. takes me.

So I have to read and understand lots of "alien" code in styles that
wouldn't be my first preference if I had written it from scratch,
often styles that lead to many "blerghs" and "yucks" which I know
we all do.  And sometimes that alien code was written by myself a
longer time ago.

And what I'm trying to say is that all those difficulties are much
much less importnat than shortcomings in domain knowledge.

> Only then you will be able to realize the role of
> obscure programming style and use of unfamiliar syntax in making the
> job harder than it has to be.

I am prepared to admit that that impact is heavier to someone
like you who is the first responder for any and every report to
parts of Emacs you didn't even know existed.  But I don't think
a uniform programming style, if that even exists (let alone
being feasible to enforce) would make your job much easier.

Maybe _somewhat_ easier, but to what loss?  Aggravated contributors,
endless hair-splitting of what exactly good style is.   And style
preferences change anyway: what I "liked" 10 years ago is not what
I like now. It's really a bad idea.

But it's definitely a good idea to review any _new_ contributions to
basic Elisp constructs clearly.  I'm not proposing a free-for-all.

I've said this often without any kind of irony or sarcasm that I think
you're an excellent maintainer.  This is not to "butter you up": I want
you to be happy maintaining Emacs for a long time.  IMHO, if I am allowed,
you should keep doing what you do, which is try to ping the specialists
in each domain, and prompt non-specialists too like you did with
bug#66970, to which I hope to have provided at least the start of
a fix.

João Távora

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Thu, 9 Nov 2023 15:39:48 +0000
> Cc: acm@muc.de, dmitry@gutov.dev, bjorn.bidar@thaodan.de, emacs-devel@gnu.org
> 
> And what I'm trying to say is that all those difficulties are much
> much less importnat than shortcomings in domain knowledge.

Not IME, FWIW.

> Maybe _somewhat_ easier, but to what loss?  Aggravated contributors,
> endless hair-splitting of what exactly good style is.   And style
> preferences change anyway: what I "liked" 10 years ago is not what
> I like now. It's really a bad idea.

You will not find any message from me that says we should ban cl-lib.
Maybe you are confusing me with someone else.  All I'm saying is that
people should not dismiss this aspect so easily, when they consider
the subject.  It is not just a coding style issue.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Thu, Nov 9, 2023 at 3:51 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Thu, 9 Nov 2023 15:39:48 +0000
> > Cc: acm@muc.de, dmitry@gutov.dev, bjorn.bidar@thaodan.de, emacs-devel@gnu.org
> >
> > And what I'm trying to say is that all those difficulties are much
> > much less importnat than shortcomings in domain knowledge.
>
> Not IME, FWIW.

Fair enough.  If you ever find my code whose style you think
needs rewriting, or need help translating another piece of
code to more familiar Lisp, let me know.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> That it doesn't make maintenance any more difficult than any other
> Elisp construct, be it very old and curiously named like 'rplacd' or
> much, much newer like `seq-do` or `pcase-lambda`.
>
> My specific view on cl-lib.el is that it brings us a small part of
> the results of  non-trivial design work put in when important figures
> in the Lisp world met  regularly for many years to deliver what has
> proved to become excellent,  battle-tested, widely understood and
> impeccably documented programming abstractions.
>
> What I'm reading so far in this long discussion is that the argument
> of its detractors isn't really that cl-lib isn't good, but that
> it is superfluous and that learning it is a burden on maintainers.
> Well, it's just as superfluous as all of Elisp apart from two handfuls
> of primitives, I guess.  Or any programming language for that matter, if
> you know enough machine code.  Or any other programming abstraction I
> happen not to be familiar with.
>
> Also I seem to hear that Elisp has some kind of ideal hard-to-define
> identity or fingerprint and that it shouldn't try to become anything
> else.  But this argument is very strange given Elisp is, like any
> decent language, easy to extend.  Not to mention I struggle to see
> the technical advantage in uniqueness for uniqueness sake.  A good
> part of Elisp is about a special purpose function: editing buffers,
> and an the equally important part is about doing things with bits in
> memory, there's no reason to advocate for singularity here.
>
> I also hear Elisp shouldn't become Common Lisp, but not only is the
> use of cl-lib.el nowhere a step in that direction, but also -- somewhat
> ironically -- if Elisp _were_ Common Lisp, then that hard-to-define
> identity would be much easier to define and language extension would be
> much easier to organize into compartments to facilitate policy-making.
>
> Again, the only thing that has brought Elisp any closer to Common
> Lisp significantly, was the introduction of lexical binding some 10
> years ago.  Elisp looks a lot different now than it did in the 90's.
> Closures everywhere, higher-order functions!  Shudder!
>
> There's even talk of a continuation-passing style library, to
> be called future.el or promise.el or something.  Oh dear, what
> will be of us that we will have to evolve like any other language
> in the world!
>
> So I propose we let programmers use their judgement.  Really
> respect people who write code for no money and give the copyright
> away to the FSF.  Maybe suggest guidelines such as not introduce
> cl-loop where a dolist would do the job just as economically and
> elegantly. Don't use higher-order functions just because it looks
> cool. But maybe do suggest to use cl-position with a :key and a
> :test instead of a complex while loop with an auxiliary variable.
> Or cl-set-difference instead of nested loops.  Suggest to express
> intent, use abstractions.   Suggest to be consistent, be scrutinous,
> be "discriminate", be mindful of the style of the current area
> you are working on.

Such casuistry and non-sequiturs are the reasons I'm glad I have thus
far given this conversation a generously wide berth.  But one's
restraint has its limits, and I suppose mine have been overstepped,
since I can't keep myself from delving head-first into this madhouse.
Here goes...

> Well, it's just as superfluous as all of Elisp apart from two handfuls
> of primitives, I guess.  Or any programming language for that matter, if
> you know enough machine code.  Or any other programming abstraction I
> happen not to be familiar with.

cl-lib does certainly provide some value for money, by way of constructs
that lend themselves to many general situations, but it does not provide
this value in the lion's share of the circumstances these constructs are
used in.  Consider the blithe misapplication of CL generics throughout
the window system code:

(cl-defmethod gui-backend-set-selection (type value
                                              &context (window-system android))
  ;; First, try to turn value into a string.
  ;; Don't set anything if that did not work.
  (when-let ((string (android-encode-select-string value)))
    (cond ((eq type 'CLIPBOARD)
           (android-set-clipboard string))
          ((eq type 'PRIMARY)
           (setq android-primary-selection string))
          ((eq type 'SECONDARY)
           (setq android-secondary-selection string)))))

do we really expect window-system to be anything _but_ android, so long
as a window system has been initialized, when android-win.el is loaded?
Why was it necessary to convolute the code when g-b-s-s (and its ilk)
could as easily have been defined as an ordinary function in each of the
window system files?

For a second example, how about cl-list*?  Is there really a purpose
for this function, in light of how it can always be replaced by:

  (nconc (list a b) c)

and demands callers load cl-lib.el?  The call to this function in
desktop.el is particularly dubitable: as far as the eye can see, it is
the singular reason that file requires cl-lib, its remainder being
implemented with the cut and dry Emacs Lisp builtins that are well known
to us all.  This abject misuse of cl-lib extensions, not an unqualified
opposition to extending Emacs Lisp, is the catalyst of most misgivings
towards it.

Like begets like, even imitations of like: as the corpus of files which
load cl-lib for legitimate purposes burgeons, so does that of files
which load it for positively frivolous reasons.  Thus usage of cl-lib
should be discouraged in general, and users firmly exhorted to seek
replacements.  Even if there are situations where it is not superfluous,
their existence doesn't outweigh the proliferation of that where it is,
and ultimately it stands us in better stead to treat it as such in
general.

> That it doesn't make maintenance any more difficult than any other
> Elisp construct, be it very old and curiously named like 'rplacd' or
> much, much newer like `seq-do` or `pcase-lambda`.

pcase-lambda and seq-do (plus the boatload of seq and pcase functions)
are not "other Elisp constructs," not in my book.  seq functions are all
generic functions, and their documentation is nothing short of
inscrutable; take the example of seq-do's:

Apply FUNCTION to each element of SEQUENCE.
Presumably, FUNCTION has useful side effects.
Return SEQUENCE.

What is a useful side effect?  Does this imply FUNCTION can modify
SEQUENCE as it is being iterated through?

and each time I read the documentation for pcase, I wind up bewildered
by the needlessly laconic style it is written in.  Most of its users
don't display a persuasive need for it either, a demographic of which
frameset.el is representative: much of its work is also replicated in
the *fns.c files and frame.c, which seem to fare equally well, despite
being authored in a language that largely enjoys an absence of byzantine
constructs.

If doc strings are meant to be incomprehensible without recourse to the
Lisp reference manual, then I suggest they not be written at all.  A doc
string that alludes to and accentuates concepts such as "useful side
effects" without providing the reader with a definition or a reference
as to where he can obtain such a definition is not edifying; it leaves
an unpleasant vacuum in the reader's mind that rankles until he bows to
fate and reads the manual anyway.

rplacd and rplaca are aliases to setcar and setcdr; Emacs Lisp code
ought to use the latter, and aliases and the wholesale introduction of
functions from other languages are plainly incommensurable.

> My specific view on cl-lib.el is that it brings us a small part of
> the results of  non-trivial design work put in when important figures
> in the Lisp world met  regularly for many years to deliver what has
> proved to become excellent,  battle-tested, widely understood and
> impeccably documented programming abstractions.

It's a fair bet that Common Lisp users today number fewer than Emacs
users; of course this does not immediately reflect on the character of
features present in Common Lisp, but it is a consideration to bear in
mind.

The quality of the documentation, which for cl-lib is far from
irreproachable, cannot allay the challenge posed by learning to read and
write a language that is in plenty of respects at variance with the
language Emacs users and developers have learned.  When the latter is in
fact documented beyond reproach (save for rough spots such as the said
pcase), more than adequate for programming in Emacs, and understood by
all Emacs users, the profundity of the deliberations underlying the
design of Common Lisp and the eminence of its designers are not
justifications for its pervading Emacs code, let alone to the degree
that it has by now.

> Also I seem to hear that Elisp has some kind of ideal hard-to-define
> identity or fingerprint and that it shouldn't try to become anything
> else.  But this argument is very strange given Elisp is, like any
> decent language, easy to extend.  Not to mention I struggle to see
> the technical advantage in uniqueness for uniqueness sake.  A good
> part of Elisp is about a special purpose function: editing buffers,
> and an the equally important part is about doing things with bits in
> memory, there's no reason to advocate for singularity here.

The relative ease of extension implies that we should be more
circumspect in judging which extensions to allow--that is the basis of
much of the argumentation here, not the unsupportable notion that there
is an identity to Emacs Lisp which must be preserved at all costs.

> I also hear Elisp shouldn't become Common Lisp, but not only is the
> use of cl-lib.el nowhere a step in that direction, but also -- somewhat
> ironically -- if Elisp _were_ Common Lisp, then that hard-to-define
> identity would be much easier to define and language extension would be
> much easier to organize into compartments to facilitate policy-making.

This is self-evidently wooden language.  I expect the likes of
"compartmentalizing a program for policy-making" from suits, not from
free software developers.

> Again, the only thing that has brought Elisp any closer to Common
> Lisp significantly, was the introduction of lexical binding some 10
> years ago.  Elisp looks a lot different now than it did in the 90's.

[...]

> Closures everywhere, higher-order functions!  Shudder!

This is not ipso facto a net positive and is also beside the point.

> There's even talk of a continuation-passing style library, to
> be called future.el or promise.el or something.  Oh dear, what
> will be of us that we will have to evolve like any other language
> in the world!

I certainly hope never to see the day that becomes widespread in Emacs
code.

> So I propose we let programmers use their judgement.  Really
> respect people who write code for no money and give the copyright
> away to the FSF.  Maybe suggest guidelines such as not introduce
> cl-loop where a dolist would do the job just as economically and
> elegantly. Don't use higher-order functions just because it looks
> cool. But maybe do suggest to use cl-position with a :key and a
> :test instead of a complex while loop with an auxiliary variable.
> Or cl-set-difference instead of nested loops.  Suggest to express
> intent, use abstractions.   Suggest to be consistent, be scrutinous,
> be "discriminate", be mindful of the style of the current area
> you are working on.

Respect for volunteers is no doubt laudable, but it is very much an
position to be considered alone which everything you've presented above
does not impact and nobody takes exception to.  From it, and even
supposing for the sake of argument that everything else you've said is
incontrovertible fact, it doesn't really follow that we should not seek
to reduce the incidence of cl-lib usage in core code.  Now I can't be
certain that is the object of your militating, if you will, though
promoting cl-set-difference and cl-position as alternatives to hallowed
Emacs Lisp constructs does suggest so, but the principle argument
against these constructs is that they are alien to the small group of
individuals charged with trudging through code where people want to make
use of them.  So no contentions resting on their virtues and
distinctions rather than their prevalence among us can hold water...

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Thu, Nov 9, 2023 at 1:46 PM Po Lu <luangruo@yahoo.com> wrote:
>
> João Távora <joaotavora@gmail.com> writes:

> This is self-evidently wooden language.

And all the time I thought I was being cheerful.

> I expect the likes of
> "compartmentalizing a program for policy-making" from suits, not from
> free software developers.

Are _you_ the suit?  I missed the point of most of your very
elegant text, but isn't policy-making exactly what you (and I) are
trying to do, i.e. discourage some things and encourage others?

What's the gripe with compartments?  We don't have all of Elisp in
a single library?  I was talking about namespaces by the way,
if you didn't catch the reference (admittedly obscure)

Also, is really cl-set-difference to take the difference
of two sets worse than writing a "hallowed" multi-line
construct??  Maybe I'm not seeing what the construct is, so
what is your preferred Elisp hallowed way to take the
difference of two sets, say make a list  that contains
all items that appear in LIST1 but not LIST2?  Or to find
the index of a certain element of a sequence...

These comparisons, taken one by one, are the hard evidence of that
allows us to determine what is "alien" or not, not theological
consecrations of personal tastes of a given group, _any_ group.
For example, I don't think your 'nconc'  form is more readable
than 'cl-list*'.  It's slightly less readable IMO.  We could alias
`list*` to nconc though, even better.  But I can agree it's not
worth requiring cl-lib.el just for cl-list* alone.

Much like you prefer setcar to rplacd. Which I also do, btw.  But
I prefer (setf (car ...))  even more.  But you'll frequently see
me using setcar and setcdr (or even rplacwahtevercannevertypeit)
if that's the style of the file I'm working on.  Because I too
have my personal tastes, but contrary to others I can let go of
them without too much fuss.

Stefan M's pcase in particular is great.  Its docstring is tough,
I agree, and could use some examples (I always end up grepping
for examples).  But that macro is a lifesaver.

The way you talk about seq.el and pcase.el makes me think you
want to ban _all the things_, not just cl-lib.  I'm confused about
what you want Elisp code in core to look like.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Thu, Nov 9, 2023 at 2:28 PM João Távora <joaotavora@gmail.com> wrote:

> than 'cl-list*'.  It's slightly less readable IMO.  We could alias
> `list*` to nconc though, even better.

oops, no we couldn't.  Same job, different calling convention.
I just don't use either that much, so here again my suggestion
is to just use what fits better with the style of surrounding code.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> On Thu, Nov 9, 2023 at 2:28 PM João Távora <joaotavora@gmail.com>
> wrote:
>
>> than 'cl-list*'.  It's slightly less readable IMO.  We could alias
>> `list*` to nconc though, even better.
>
> oops, no we couldn't.  Same job, different calling convention.
> I just don't use either that much, so here again my suggestion
> is to just use what fits better with the style of surrounding code.

There is no reason to require a 21 KB file for a six line function.
None.  Yet some of our files do just that; surely this is a
consideration on equal footing with one's own perception of style?

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 10, 2023 at 12:03 AM Po Lu <luangruo@yahoo.com> wrote:
>

> There is no reason to require a 21 KB file for a six line function.
> None.  Yet some of our files do just that; surely this is a
> consideration on equal footing with one's own perception of style?

I said I agree with you.  So go there and replace cl-list* with nconc.
Or put it in a much smaller preloaded file, it doesn't have
dependencies.  Or just define it in terms of nconc if it's really equivalent.

But cl-lib usage and usefulness is really much more than that tiny aspect
you're focusing on.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> I said I agree with you.  So go there and replace cl-list* with nconc.
> Or put it in a much smaller preloaded file, it doesn't have
> dependencies.  Or just define it in terms of nconc if it's really
> equivalent.
>
> But cl-lib usage and usefulness is really much more than that tiny
> aspect you're focusing on.

cl-list* is just one example of a theme that constantly repeats itself
throughout our code.  Removing this one instance is futile, since
there's plenty more where it came from.  The only remedy is for everyone
to develop a general aversion to cl-lib.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 10, 2023 at 2:43 AM Po Lu <luangruo@yahoo.com> wrote:

> cl-list* is just one example of a theme that constantly repeats itself
> throughout our code.  Removing this one instance is futile,

You're just contradicting yourself.  First, you argue that it's easy
to replace and has these immediate benefits and then when someone agrees
with you, you say you don't want to.

Also btw, it might not be easy to replace at all cl-list* can be used
with higher order #'apply in contexts where nconc can't.  Perhaps
you despise the programmer who wrote

  (apply #'cl-list* input-data)

But perhaps they did it to avoid writing a multi-line raw loop choo choo
train again.

So it's not true what you said originally that everytime you find a
cl-list* it can be cleanly replaced by nconc.  Not if you understand
or even appreciate function composition.

> since there's plenty more where it came from.   The only remedy is
> for everyone to develop a general aversion to cl-lib.

Be consistent: find the places using simple forms of cl-list* that
presumably (as you said) are the only motivations for the whole
cl-lib.el 27kb loading.  I didn't check but this was your opening
argument after all.  So do that and you'll have effectively and
consensually helped Emacs's code base.  You'll be closer to your
dream goal, using Emacs without loading cl-lib.el.   Be consistent
with your words, else you're just hand-waving and advocating
for irrationality.  "develop a general aversion"??? I like to think
of programmers as scientists, not acolytes to some sect.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> You're just contradicting yourself.  First, you argue that it's easy
> to replace and has these immediate benefits and then when someone agrees
> with you, you say you don't want to.

I didn't say I don't want to, I said that removing this one instance
will not solve the underlying readiness to resort to _non-preloaded_
files, of which cl-lib is one, and an egregious offender.

> Also btw, it might not be easy to replace at all cl-list* can be used
> with higher order #'apply in contexts where nconc can't.  Perhaps
> you despise the programmer who wrote
>
>   (apply #'cl-list* input-data)
>
> But perhaps they did it to avoid writing a multi-line raw loop choo choo
> train again.
>
> So it's not true what you said originally that everytime you find a
> cl-list* it can be cleanly replaced by nconc.  Not if you understand
> or even appreciate function composition.

I don't appreciate "function combination," whatever that is.

In a language such as C, the function of a single block is clear.  Each
block commences with a list of one declarator for each variable.

Variables are supplied to and operated on by a list of arithmetic or
data operators and procedures, subject to control constructs all
programmers learn, and a value derived from one or more expressions
holding these variables is returned to complete the block's procedure.

Thus as data changes hands between operators, functions and variables,
its movement can be discerned with only a handful of operators and
control constructs in memory.  This is how a computer operates, or what
_you_ do when working out long division on a piece of paper, or when you
set yourself to any other task entailing the observation of a series of
steps while responding to changing circumstances in the process.

Emacs Lisp is well capable of aligning to such requirements as well;
today many would implement the old float-to-string with cl-loop and
floating point numbers with structs, but in the past they were
implemented with cons cells and while loops, without us being any the
worse for wear.

(defun float-to-string (fnum &optional sci)
  "Convert the floating point number to a decimal string.
Optional second argument non-nil means use scientific notation."
  (let* ((value (fabs fnum)) (sign (< (car fnum) 0))
	 (power 0) (result 0) (str "") 
	 (temp 0) (pow10 _f1))

    (if (f= fnum _f0)
	"0"
      (if (f>= value _f1)			; find largest power of 10 <= value
	  (progn				; value >= 1, power is positive
	    (while (f<= (setq temp (f* pow10 highest-power-of-10)) value)
	      (setq pow10 temp
		    power (+ power decimal-digits)))
	    (while (f<= (setq temp (f* pow10 _f10)) value)
	      (setq pow10 temp
		    power (1+ power))))
	(progn				; value < 1, power is negative
	  (while (f> (setq temp (f/ pow10 highest-power-of-10)) value)
	    (setq pow10 temp
		  power (- power decimal-digits)))
	  (while (f> pow10 value)
	    (setq pow10 (f/ pow10 _f10)
		  power (1- power)))))
					  ; get value in range 100000 to 999999
      (setq value (f* (f/ value pow10) all-decimal-digs-minval)
	    result (ftrunc value))
      (let (int)
	(if (f> (f- value result) _f1/2)	; round up if remainder > 0.5
	    (setq int (1+ (fint result)))
	  (setq int (fint result)))
	(setq str (int-to-string int))
	(if (>= int 1000000)
	    (setq power (1+ power))))

      (if sci				; scientific notation
	  (setq str (concat (substring str 0 1) "." (substring str 1)
			    "E" (int-to-string power)))

					  ; regular decimal string
	(cond ((>= power (1- decimal-digits))
					  ; large power, append zeroes
	       (let ((zeroes (- power decimal-digits)))
		 (while (natnump zeroes)
		   (setq str (concat str "0")
			 zeroes (1- zeroes)))))

					  ; negative power, prepend decimal
	      ((< power 0)		; point and zeroes
	       (let ((zeroes (- (- power) 2)))
		 (while (natnump zeroes)
		   (setq str (concat "0" str)
			 zeroes (1- zeroes)))
		 (setq str (concat "0." str))))

	      (t				; in range, insert decimal point
	       (setq str (concat
			  (substring str 0 (1+ power))
			  "."
			  (substring str (1+ power)))))))

      (if sign				; if negative, prepend minus sign
	  (concat "-" str)
	str))))

If it is to these that you apply the moniker "multi-line raw loop choo
choo trains," then they are precisely what should be written more, not
less.

The "Lisp way" (more felicitously the cl-lib way) is to provide
functions to truckloads of other functions which interface them with one
another.

Each such function is a unique control construct.  To glean what a
function does, one must not only commit each of these control constructs
to memory, but as is all too often the case when the functions provided
to the control constructs are beyond its control, also examine each of
its callers, with labyrinths that put C++ to shame into the bargain,
like pcase-lambda and macros which rewrite the control flow of a form
beyond recognition.

> Be consistent: find the places using simple forms of cl-list* that
> presumably (as you said) are the only motivations for the whole
> cl-lib.el 27kb loading.  I didn't check but this was your opening
> argument after all.  So do that and you'll have effectively and
> consensually helped Emacs's code base.  You'll be closer to your
> dream goal, using Emacs without loading cl-lib.el.   Be consistent
> with your words, else you're just hand-waving and advocating
> for irrationality.  "develop a general aversion"??? I like to think
> of programmers as scientists, not acolytes to some sect.

If such invective and an unwillingness to reason from the specific to
the general is the best response you have, I won't condescend to answer.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 10, 2023 at 12:52 PM Po Lu <luangruo@yahoo.com> wrote:

> > So it's not true what you said originally that everytime you find a
> > cl-list* it can be cleanly replaced by nconc.  Not if you understand
> > or even appreciate function composition.
>
> I don't appreciate "function combination," whatever that is.

Composition, not "combination" You find it in mathematics.
Pretty useful many contexts.  IMHO of course.

Anyway, I don't think this argument has any end.  You don't seem to
have any concept of information hiding, abstraction reuse, etc.
Nor do you seem to have any inclination to start actually acting
on that purported plan of yours, even where there is clear
agreement on specific parts of it.  So we don't have a common basis
for discussion in this matter.   That's fine, what can I say?
Go assembly, go machine code!

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> Composition, not "combination" You find it in mathematics.
> Pretty useful many contexts.  IMHO of course.
>
> Anyway, I don't think this argument has any end.  You don't seem to
> have any concept of information hiding, abstraction reuse, etc.

Yes, why should I chase chimeras, when I could as easily get down to
work?

To mention an example that is perhaps a tadge closer to home,
c-forward-decl-or-cast-1 is a function you would certainly advocate the
balkanization of.  But if it is ultimately divvied up into dozens of
private functions, of which it remains the only caller, what use will
the CC Mode user have gotten out of the process?  CC Mode will only have
become that much slower, and, where CC Mode is concerned, speed is like
gold dust.

> Nor do you seem to have any inclination to start actually acting
> on that purported plan of yours, even where there is clear
> agreement on specific parts of it.

Where's the agreement?

Re: What's missing in ELisp that makes people want to use cl-lib?

[Manuel Giraud via Emacs development discussions.]

Po Lu <luangruo@yahoo.com> writes:

[...]

> If it is to these that you apply the moniker "multi-line raw loop choo
> choo trains," then they are precisely what should be written more, not
> less.

Just my 2 cents, but here you have chosen the code for parsing a
floating representation which is, of course, full of peculiarities and
corner cases.  It would not be that much different should it be written
in « pure » CL.

But at the other end of the spectrum, there is lots of code which is
just: walk through this collection and do "this" if "that".
-- 
Manuel Giraud

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 10/11/2023 14:52, Po Lu wrote:
> (defun float-to-string (fnum &optional sci)
>    "Convert the floating point number to a decimal string.
> Optional second argument non-nil means use scientific notation."
>    (let* ((value (fabs fnum)) (sign (< (car fnum) 0))
> 	 (power 0) (result 0) (str "")
> 	 (temp 0) (pow10 _f1))
> 
>      (if (f= fnum _f0)
> 	"0"
>        (if (f>= value _f1)			; find largest power of 10 <= value
> 	  (progn				; value >= 1, power is positive
> 	    (while (f<= (setq temp (f* pow10 highest-power-of-10)) value)
 >      .............
>        (if sign				; if negative, prepend minus sign
> 	  (concat "-" str)
> 	str))))
> 
> If it is to these that you apply the moniker "multi-line raw loop choo
> choo trains," then they are precisely what should be written more, not
> less.

I don't have the corresponding high-level implementation to compare to, 
but the main thing about the code you posted, is that one has to, 
basically, read every line (of which there are ~70) to understand what 
the function is doing and/or to find any irregularities in its 
implementation (e.g. when looking for a bug).

> The "Lisp way" (more felicitously the cl-lib way) is to provide
> functions to truckloads of other functions which interface them with one
> another.

A high-level approach is to use a set of agreed-upon abstractions 
(functional programming in general, across languages, has a basic set, 
but there can be additions) to write programs shorter.

> Each such function is a unique control construct.  To glean what a
> function does, one must not only commit each of these control constructs
> to memory, but as is all too often the case when the functions provided
> to the control constructs are beyond its control, also examine each of
> its callers, with labyrinths that put C++ to shame into the bargain,
> like pcase-lambda and macros which rewrite the control flow of a form
> beyond recognition.

There is, indeed, a mental cost to understanding the code which uses 
them -- one has to know the abstractions and be able to connect the 
names with the concepts. There aren't too many of them in use, though. 
We can argue whether cl-lib is an optimal set, or could be reduced 
(there are indeed many functions and macros in there that I never saw 
the point of using). But your proposed goal (full eventual removal of 
cl-lib, or anything like it) is a non-starter from my POV.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

Dmitry Gutov <dmitry@gutov.dev> writes:

> I don't have the corresponding high-level implementation to compare
> to, but the main thing about the code you posted, is that one has to,
> basically, read every line (of which there are ~70) to understand what
> the function is doing and/or to find any irregularities in its
> implementation (e.g. when looking for a bug).

By contrast, in the archetypal cl-lib user, one must read every line,
the doc strings of every cl-lib function it calls, and the several lines
adjacent to each place where it is called.  To speak nothing of cl-lib
macros which recast a form's control flow into unrecognizability.

> A high-level approach is to use a set of agreed-upon abstractions
> (functional programming in general, across languages, has a basic set,
> but there can be additions) to write programs shorter.

In this respect functional programming can be likened to poetry--after
writing a witty, caustic epigram inveighing against some perceived
iniquity, a poet is always very pleased with himself.  But should he
apply his manner of writing to communication with real people, as by
speaking in epigrams before a clerk at an unemployment office, it is
likely he will starve.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 11/11/2023 03:11, Po Lu wrote:
>> A high-level approach is to use a set of agreed-upon abstractions
>> (functional programming in general, across languages, has a basic set,
>> but there can be additions) to write programs shorter.
> In this respect functional programming can be likened to poetry--after
> writing a witty, caustic epigram inveighing against some perceived
> iniquity, a poet is always very pleased with himself.  But should he
> apply his manner of writing to communication with real people, as by
> speaking in epigrams before a clerk at an unemployment office, it is
> likely he will starve.

Poets don't use a common intermediate language which can be defined, 
studied and use by anybody who has the inclination and time to 
understand it. And makes sentences shorter.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

Dmitry Gutov <dmitry@gutov.dev> writes:

> Poets don't use a common intermediate language which can be defined,
> studied and use by anybody who has the inclination and time to
> understand it. And makes sentences shorter.

Poetic license can be defined, studied and employed by anyone with a
sufficient impetus to do so, which is why it is analogous to functional
programming, or to any other vogue for a particular programming practice
that enjoys a spot in the limelight once in a while.  But such time and
inclination are always in short supply.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Po Lu wrote:

> In this respect functional programming can be likened to
> poetry [...]

Isn't conventional wisdom on the contrary that functional
programming principles make programs shorter and more modular,
so more re-usable and with less overhead reading and
understanding individual parts at the detail-level?

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

Emanuel Berg <incal@dataswamp.org> writes:

> Isn't conventional wisdom on the contrary that functional
> programming principles make programs shorter and more modular,
> so more re-usable and with less overhead reading and
> understanding individual parts at the detail-level?

Of course, much as conventional wisdom generally extols the virtues of
practices that are "in" at any given moment.  I prefer to set store by
my eyes.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Michael Heerdegen]

Po Lu <luangruo@yahoo.com> writes:

> I don't appreciate "function combination," whatever that is.
>
> In a language such as C, the function of a single block is clear.  Each
> block commences with a list of one declarator for each variable.
> [...]
> If it is to these that you apply the moniker "multi-line raw loop choo
> choo trains," then they are precisely what should be written more, not
> less.

Side note: I'm not a "fan" of cl-lib, and I hate cl-loop.  I hate it
when I need to understand code that uses it.

But this discussion has some aspects that I don't like - forcing other
programmers to share the own likes and ways of programming, "without
discussing each decision" although these decision have a very broad
reach, are backward incompatible and throw away the long-year work of
very committed and not unintelligent people.  Be aware that some things
in cl-lib are very well written and you'll need a very long time to to
develop a replacement of even approximate quality.  This discussion has
too much from "Personally I don't like this, let's just throw it all
away and start from scratch" that I only know from maybe a bit naive
people in emacs-help.

I understand most of your points of criticism.  A lot of parts of cl-lib
are not very useful for core Emacs, are harmful when used, should be
avoided as much as possible.  OTOH, there also are the useful things,
`cl-labels' (it had been mentioned as a main example) is definitely
among them for me.  Simple, well-defined and used to implement other
things...  I can't believe that it is an expectation that somebody's
personal preferences and likes should be the only guide of how to
proceed.

cl-lib started as emulation package for Common Lisp.  It was a huge work
to get it where it is now.  We should definitely _not_ simply throw away
this achievement.

I see that some parts of cl-lib are not well documented, and there are
problems when debugging it's stuff.  But honestly, other parts of Emacs
have the same problems.  And the Edebug support of cl stuff has also
improved in the last time - this is possible (this question only applies
to macros anyway).  FWIW, just as a random opinion, I found the original
sparse documentation of `pcase' very fine (and yes, I understood it),
so, please note, we are different people having a different background.
Please, let's discuss this with a bit tolerance and a bit less anger and
sarcasm.  It is an important decision.

My goal would be like this:

 - keep cl-lib available to it's current extent for end users

 - try to get unnecessary cl-lib uses out of the code base

 - make useful cl ideas available in the core language

And: one main problem we have is that cl-lib is more or less a huge
monolithic block (as a `require'able library).  Why not split it up?  I
don't see why it is necessary that when I want to use a small useful cl
thing that cl-loop must be made available.  Such things could go to its
own library, name it cl-obscure when you want.

But I don't see a necessity to, for example, kick keyword support out of
reach for the Emacs code base because, there are still, few, use cases
where it fits.  We don't want to end in a situation where we have 15
places that all reinvent the same wheel in subtly different ways.  This
would not be an improvement.  There is no need to reinvent at all.  This
stuff is well tested and integrated.  As long as the fundamental
guideline is clear that using such features is limited to spare
situations where it really makes things simpler, I don't see a problem,
apart maybe from the fact that cl-lib is big.  I would like us to try to
factor it when the size is a problem (is it a practical, objective, or
more or less mainly only a psychological problem btw? - that has
actually been one of my main questions when following this thread).


Michael.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Date: Sat, 11 Nov 2023 07:01:39 +0100
> 
> My goal would be like this:
> 
>  - keep cl-lib available to it's current extent for end users
> 
>  - try to get unnecessary cl-lib uses out of the code base
> 
>  - make useful cl ideas available in the core language

These are exactly our goals.  We've been doing this for years: that's
how stuff like 'when', 'unless', 'push', 'pop', and others got into
the core.

IOW, the way to make some cl-lib functionality freely available in
core is to have a similar or identical implementation in the likes of
subr.el or seq.el, using our naming convention instead of cl-FOO.
We've been doing that for many years, and we will continue doing it.
People just have to understand that this is the way.

The problem (at least my problem) is that some of the people in this
discussion don't agree with this cautious and slow approach.  They
want the unconstrained liberty of using all or most of cl-lib and its
extra libraries everywhere, and they consider any request to avoid
using these facilities an unacceptable pressure.  They basically want
cl-lib, cl-extra, etc. preloaded and always available for use,
regardless of the aspects you mention above and without any
restrictions.  So your argument is not with Po Lu, your argument is
with others.

> But I don't see a necessity to, for example, kick keyword support out of
> reach for the Emacs code base because, there are still, few, use cases
> where it fits.

We use keyword arguments when there are many of them, or when a few of
them are obscure and rarely used.  From where I stand, the dispute is
not about banishing then -- that will not happen -- the dispute is
about how _much_ to use them.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Sat, Nov 11, 2023 at 7:56 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Michael Heerdegen <michael_heerdegen@web.de>
> > Date: Sat, 11 Nov 2023 07:01:39 +0100
> >
> > My goal would be like this:
> >
> >  - keep cl-lib available to it's current extent for end users
> >
> >  - try to get unnecessary cl-lib uses out of the code base
> >
> >  - make useful cl ideas available in the core language
>
> These are exactly our goals.  We've been doing this for years: that's
> how stuff like 'when', 'unless', 'push', 'pop', and others got into
> the core.
>
> IOW, the way to make some cl-lib functionality freely available in
> core is to have a similar or identical implementation in the likes of
> subr.el or seq.el, using our naming convention instead of cl-FOO.

I generally agree with the goal of bringing in cl-lib.el ideas into
core, but it must be said here that seq.el, despite being a sequence
processing library, should NOT be viewed as a drop-in replacement
for cl-lib.el counterparts

seq.el strength over cl-lib.el is that it is a generic sequence
processing library.  IMO it _should_ be used if there is the suspicion
that the sequences being in that data being processed by a given
function can be custom seq.el sequences.  Or when there is good
reason to suspect that one they might be such custom sequences.

I think that seq.el _should_ be used in those situations because
it expresses exactly this fact, so it becomes self-documenting.

I think the argument that Gerd is making in his economic
contributions is that very often seq.el is overkill.

For example, in its current form should _not_ be used when
there is certainty that the sequences are Lisp listsm because as I
showed elsewhere here, seq-some is significantly slower than
cl-some and cl-loop when processing lots of small lists.

Also, I know you don't like cl-loop but in terms of efficiency
it's sometimes a very good choice.  For example, processing
maintain list in order without it often requires a nreverse
at the end or even more code.  And the fact that cl-loop doesn't
need to do dispatching on different sequence types means it's
faster.  And less polymorphic of course, but sometimes you
simply don't _need_ polymorphism.

I do agree fully that writing

  (cl-loop for i in l do (foo i))

is kind of uncalled _in any situation_ if you can just write

  (dolist (i l) (foo j))

But when you need to process plists, for example, it becomes
very handy.  This construct, which does require knowing a bit
of the mini-language that is cl-loop,

  (cl-loop for (k v) on plist by #'cddr collect (cons k v))

is still the most concise way IMHO of processing a plist
for example.  Of course that is debatable.  Maybe you
prefer a raw while loop with some auxiliary variables, but
while that is more universal, it doesn't necessarily mean
it's more readable.  These are two different things that
people are mixing in here.  I would be on the fence if this example
would justify introducing cl-lib.el into a file or not.
Depends on how often this plist-processing business is needed.
Or whether there is a pair-sequence-processing util in seq.el
AND performance is not at stake.

And certainly we should not immediately shoot down a contribution
to the core without "batting an eyelid", as someone suggested
here, just because it doesn't fit a stylistic preference.

By the way, I also think Michael's idea of splitting up
cl-lib.el further into smaller bits is very good.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> But when you need to process plists, for example, it becomes
> very handy.  This construct, which does require knowing a bit
> of the mini-language that is cl-loop,
>
>   (cl-loop for (k v) on plist by #'cddr collect (cons k v))
>
> is still the most concise way IMHO of processing a plist
> for example.  Of course that is debatable.  Maybe you
> prefer a raw while loop with some auxiliary variables, but
> while that is more universal, it doesn't necessarily mean
> it's more readable.  These are two different things that
> people are mixing in here.  I would be on the fence if this example
> would justify introducing cl-lib.el into a file or not.
> Depends on how often this plist-processing business is needed.
> Or whether there is a pair-sequence-processing util in seq.el
> AND performance is not at stake.

It might be pithy, but I don't know what "processing" is being done.  So
I cannot really judge it in terms of readability to the cl-loop
initiated.

Would everyone who posts passages of code using cl-loop please also
affix their macroexpansions, as, for want of a better word, a
reconciliatory gesture acknowledging the existence of those who are not
completely au fait with such matters?  HHOS...

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Sat, Nov 11, 2023 at 1:17 PM Po Lu <luangruo@yahoo.com> wrote:

> It might be pithy, but I don't know what "processing" is being done.  So
> I cannot really judge it in terms of readability to the cl-loop
> initiated.

It's converting a plist into an alist.  And I used it precisely
an example of non-universal code, that is  nevertheless concise.

No one asked you to "judge it".  You may if you so desire, write
your own while loop version (check your results this time!) and
then we can compare if that universal, primitive-exclusive code,
that results is more readable than the cl-loop version.  If you
weren't generally averse to "balkanization", we could even
collaborate in designing a good flexible, well-named utility for
this common operation (not just plist-alist conversion, but
more types of plist processing) to add to the Emacs core.

But alas, my dear dictionary-loving friend, this would imply
we have to leave this spirited yet vacuous exchange of
pompous elucubrations behind us.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> It's converting a plist into an alist.  And I used it precisely
> an example of non-universal code, that is  nevertheless concise.
>
> No one asked you to "judge it".

You're arguing in favor of it, and to repudiate such arguments the
measure that is being used to compare it with other postulated
alternatives must also be applied to these alternatives, correct?

In view of the incidence of the word "conciseness" within both your
follow-ups, that is the measure in question.  I think the existence of a
minimum of one person who does not quite follow the code at hand is
enough to outweigh any advantages it might hold, when this is the
alternative in question:

(let ((list '(:a b :c d)) (other nil))
  (while list
    (push (cons (car list) (cadr list)) other)
    (setq list (cddr list)))
  (nreverse other))

Using cl-lib for this kind of job, and expecting others to understand
it, is like a philosopher upholding the Tractatus Logico-Philosophicus
as a universal cynosure for reasoning and exhorting laymen to hold their
thought to it.  Bizarrely enough, I have been subject to both these
kinds of exhortations.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Sat, Nov 11, 2023 at 2:07 PM Po Lu <luangruo@yahoo.com> wrote:

> > we could even collaborate in designing [...]
> > But alas, my dear dictionary-loving friend, this would imply
> > we have to leave this spirited yet vacuous exchange of
> > pompous elucubrations behind us.

> Using cl-lib for this kind of job, and expecting others to understand
> it, is like a philosopher upholding the Tractatus Logico-Philosophicus
> as a universal cynosure for reasoning and exhorting laymen to hold their
> thought to it.  Bizarrely enough, I have been subject to both these
> kinds of exhortations.

I guess that answers my question!

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Sat, 11 Nov 2023 13:09:11 +0000
> Cc: Michael Heerdegen <michael_heerdegen@web.de>, emacs-devel@gnu.org
> 
> On Sat, Nov 11, 2023 at 7:56 AM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > >  - keep cl-lib available to it's current extent for end users
> > >
> > >  - try to get unnecessary cl-lib uses out of the code base
> > >
> > >  - make useful cl ideas available in the core language
> >
> > These are exactly our goals.  We've been doing this for years: that's
> > how stuff like 'when', 'unless', 'push', 'pop', and others got into
> > the core.
> >
> > IOW, the way to make some cl-lib functionality freely available in
> > core is to have a similar or identical implementation in the likes of
> > subr.el or seq.el, using our naming convention instead of cl-FOO.
> 
> I generally agree with the goal of bringing in cl-lib.el ideas into
> core, but it must be said here that seq.el, despite being a sequence
> processing library, should NOT be viewed as a drop-in replacement
> for cl-lib.el counterparts

That (and the rest of your message) completely misses my point, and
doesn't even belong to this discussion, to tell the truth.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Sat, Nov 11, 2023 at 1:54 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > I generally agree with the goal of bringing in cl-lib.el ideas into
> > core, but it must be said here that seq.el, despite being a sequence
> > processing library, should NOT be viewed as a drop-in replacement
> > for cl-lib.el counterparts
>
> That (and the rest of your message) completely misses my point, and
> doesn't even belong to this discussion, to tell the truth.

I think addressed your suggestion to have "similar or identical
implementation [to cl-lib.el] in the likes of [..] seq.el" quite
directly.  So I don't understand your criticism of my contribution,
but fair enough, I think I don't have much more to say on the matter
anyway.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Sat, 11 Nov 2023 14:01:24 +0000
> Cc: michael_heerdegen@web.de, emacs-devel@gnu.org
> 
> On Sat, Nov 11, 2023 at 1:54 PM Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > > I generally agree with the goal of bringing in cl-lib.el ideas into
> > > core, but it must be said here that seq.el, despite being a sequence
> > > processing library, should NOT be viewed as a drop-in replacement
> > > for cl-lib.el counterparts
> >
> > That (and the rest of your message) completely misses my point, and
> > doesn't even belong to this discussion, to tell the truth.
> 
> I think addressed your suggestion to have "similar or identical
> implementation [to cl-lib.el] in the likes of [..] seq.el" quite
> directly.

In my suggestion seq.el was just one of the examples not the only, nor
the main, package for such additions.  So making it the only one, and
then objecting only to that, is a kind of a strawman, and completely
misses my point.

I included seq.el because I see nothing wrong with adding
cl-defmethods to seq.el with more efficient implementations for
specific types of sequences.  Whether it does or doesn't make sense
depends on the specifics, which are not on the table at this time.  I
certainly don't agree with rejecting the idea of adding specific
implementations to seq.el, let alone its wholesale rejection, and I
likewise agree that in some cases adding a specific implementation to
subr.el or somesuch could be a better idea.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1289 bytes --]

On Sat, Nov 11, 2023, 14:09 Eli Zaretskii <eliz@gnu.org> wrote:

> In my suggestion seq.el was just one of the examples not the only, nor
> the main, package for such additions.  So making it the only one, and
> then objecting only to that, is a kind of a strawman, and completely
> misses my point.
>

I objected to that part because I generally agree with the rest!!! I even
said so early! Certainly pointing out a detail I think needs clarification
can't be seen as a strawman. Only if I was trying to magnify it
unreasonably to bring down your whole plan, which I am obviously not.

I included seq.el because I see nothing wrong with adding
> cl-defmethods to seq.el with more efficient implementations for
> specific types of sequences.  Whether it does or doesn't make sense
> depends on the specifics, which are not on the table at this time.  I
> certainly don't agree with rejecting the idea of adding specific
> implementations to seq.el, let alone its wholesale rejection,


I don't reject it either. I just wrote that seq.el can not be "viewed as a
drop-in replacement", which in common programming talk means we have to
look at each "similar implemention" case by case and be attentive to where
we use each, or to how we recommend its usage.

João

[-- Attachment #2: Type: text/html, Size: 1951 bytes --]

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Sat, 11 Nov 2023 14:22:19 +0000
> Cc: Michael Heerdegen <michael_heerdegen@web.de>, emacs-devel <emacs-devel@gnu.org>
> 
>  I included seq.el because I see nothing wrong with adding
>  cl-defmethods to seq.el with more efficient implementations for
>  specific types of sequences.  Whether it does or doesn't make sense
>  depends on the specifics, which are not on the table at this time.  I
>  certainly don't agree with rejecting the idea of adding specific
>  implementations to seq.el, let alone its wholesale rejection,
> 
> I don't reject it either. I just wrote that seq.el can not be "viewed as a drop-in replacement", which in
> common programming talk means we have to look at each "similar implemention" case by case and
> be attentive to where we use each, or to how we recommend its usage.

No one said that seq.el is a "drop-in replacement" for anything.  We
prefer it to be used where that makes sense, because seq.el is
preloaded.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Sat, Nov 11, 2023 at 2:25 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Sat, 11 Nov 2023 14:22:19 +0000
> > Cc: Michael Heerdegen <michael_heerdegen@web.de>, emacs-devel <emacs-devel@gnu.org>
> >
> >  I included seq.el because I see nothing wrong with adding
> >  cl-defmethods to seq.el with more efficient implementations for
> >  specific types of sequences.  Whether it does or doesn't make sense
> >  depends on the specifics, which are not on the table at this time.  I
> >  certainly don't agree with rejecting the idea of adding specific
> >  implementations to seq.el, let alone its wholesale rejection,
> >
> > I don't reject it either. I just wrote that seq.el can not be "viewed as a drop-in replacement", which in
> > common programming talk means we have to look at each "similar implemention" case by case and
> > be attentive to where we use each, or to how we recommend its usage.
>
> No one said that seq.el is a "drop-in replacement" for anything.

I didn't say you did!!  I just contributed that perspective of mine.

> We prefer it to be used where that makes sense, because seq.el is
> preloaded.

Very well, and I contributed the opinion that the reason it is
preloaded is not sufficient in itself.  It must, as you now more
clearly underline, "make sense".  And IMHO it makes _more_
sense the greater the likelihood that custom sequence types
-- a core seq.el design tenet -- may be be involved.  I pointed
out performance considerations and the self-documenting
nature of that usage to be two _additional_ factors to take
in consideration.

I think this is logical and doesn't really contradict anything
you or Michael said.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

We have heard from some programmers and maintainers that they
see the complexity increase as a problem. We have to believe
them, they think like that. And not everyone else has
to agree.

But I have not heard a lot what should be done about it.
Not generally, how to not have that tendency in the future.

And also not what to do in the particular case being
discussed, namely cl-lib, which is already in use to a huge
extent - indeed, that is exactly what is, according to them,
what has created the huge increase in complexity.

I have only heard improve the docstrings, and I think everyone
agree that is a good idea.

But what more? In general, and with respect to cl-lib in
particular? What is the plan, from the people who think it is
a problem?

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

João Távora <joaotavora@gmail.com> writes:

> I think the argument that Gerd is making in his economic
> contributions is that very often seq.el is overkill.

Sorry for being economic. I've never been very talkative.

About seq (and map): (a) to each his own, (b) I find it personally
useless, for me, and (c) I find it a mistake to make it something like a
preferred interface.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Michael Heerdegen
>  <michael_heerdegen@web.de>,  emacs-devel@gnu.org
> Date: Sat, 11 Nov 2023 15:53:31 +0100
> 
> I find it a mistake to make it something like a preferred interface.

You, and everyone else, are entitled to their opinions, but we made
this decision intentionally, and it wasn't made lightly.  Before we
declare it a mistake, I'd expect us to wait a bit, until some
unequivocal data to support the opinions is available.

It isn't like we as a project never made any mistakes, but there's no
reason to assume this decision is one of them until we have ample
evidence.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

> It isn't like we as a project never made any mistakes, but there's no
> reason to assume this decision is one of them until we have ample
> evidence.

Seq is 10 years in Emacs, its polymorphism is unused in the tree. Joao
showed that it's slow. Dmitry mentioned that it is not a full
replacement for what is in cl-lib. Po Lu mentioned the poor doc strings.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: joaotavora@gmail.com,  michael_heerdegen@web.de,  emacs-devel@gnu.org
> Date: Sat, 11 Nov 2023 19:07:59 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > It isn't like we as a project never made any mistakes, but there's no
> > reason to assume this decision is one of them until we have ample
> > evidence.
> 
> Seq is 10 years in Emacs

It is preloaded only since a little more than a year ago.

> its polymorphism is unused in the tree.

Searching for seq-* in the tree brings more than 590 hits in more than
170 Lisp files.

> Joao showed that it's slow.

No, he didn't.

> Dmitry mentioned that it is not a full replacement for what is in
> cl-lib.

Indeed, it is not a full replacement, and was never meant to be.  I
believe its name says that much.

> Po Lu mentioned the poor doc strings.

That can be said about a lot of functions in Emacs, unfortunately.
Including cl-lib, btw.  Which doesn't preclude us from using any of
those poorly-documented functions.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Eli Zaretskii wrote:

>>> It isn't like we as a project never made any mistakes, but
>>> there's no reason to assume this decision is one of them
>>> until we have ample evidence.
>> 
>> Seq is 10 years in Emacs
>
> It is preloaded only since a little more than a year ago.

Why is it important again if something is preloaded or not?

What does it mean in terms of technology?

And apart from technology, is it also some kind of
political signal?

I must admit, both points are vague to me. We understand that
the increase in complexity of Elisp because of the use of
cl-lib is unrelated to when cl-lib is loaded.

Because one would have to assume it is loaded when used.
And if it isn't used, it cannot really be a cause of
increased complexity.

Are you keeping it out of some preloaded subset of Elisp?
Is that "the last stand" vs Common Lisp that Elisp is making?

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Sat, 11 Nov 2023 21:53:32 +0100
> 
> Eli Zaretskii wrote:
> 
> >>> It isn't like we as a project never made any mistakes, but
> >>> there's no reason to assume this decision is one of them
> >>> until we have ample evidence.
> >> 
> >> Seq is 10 years in Emacs
> >
> > It is preloaded only since a little more than a year ago.
> 
> Why is it important again if something is preloaded or not?
> 
> What does it mean in terms of technology?
> 
> And apart from technology, is it also some kind of
> political signal?

It means that we consider it useful and important enough to have it
always available, including in the code that gets dumped into the bare
vanilla un-customized Emacs.

> I must admit, both points are vague to me. We understand that
> the increase in complexity of Elisp because of the use of
> cl-lib is unrelated to when cl-lib is loaded.

You don't have to understand it.  The maintainers do, but you don't.
If needed, we will point out the implications if they are relevant to
the changes you submit to Emacs.

> Are you keeping it out of some preloaded subset of Elisp?
> Is that "the last stand" vs Common Lisp that Elisp is making?

Don't be ridiculous.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> I must admit, both points are vague to me. We understand that
>> the increase in complexity of Elisp because of the use of
>> cl-lib is unrelated to when cl-lib is loaded.
>
> You don't have to understand it.  The maintainers do, but you don't.
> If needed, we will point out the implications if they are relevant to
> the changes you submit to Emacs.

That is so arrogant, that I can't believe you wrote it.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Alan Mackenzie]

Hello, Gerd.

On Sun, Nov 12, 2023 at 08:22:27 +0100, Gerd Möllmann wrote:
> Eli Zaretskii <eliz@gnu.org> writes:

> >> I must admit, both points are vague to me. We understand that
> >> the increase in complexity of Elisp because of the use of
> >> cl-lib is unrelated to when cl-lib is loaded.

> > You don't have to understand it.  The maintainers do, but you don't.
> > If needed, we will point out the implications if they are relevant to
> > the changes you submit to Emacs.

> That is so arrogant, that I can't believe you wrote it.

You have to understand that Emanuel is not a regular Emacs hacker, and
many of the things he's written in this thread are not conducive to
reaching a conclusion.

In fact, he's been questioning how Eli's been doing his job.  There is
no need for this.  I think Eli is doing an excellent job in difficult
circumstances.

Also, there's a history here.  In the medium past, Emanuel posted quite
a few posts that were less than unhelpful, and he was sanctioned for
this.

So I think it's understandable that Eli finally lost patience with
Emanuel.  I think I would have done the same, only much sooner.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > So I think it's understandable that Eli finally lost patience with
  > Emanuel.  I think I would have done the same, only much sooner.

I managed to control my temper, partly because I came to the conclusion
Berg was trolling us.  One sign of this is that he challenges us
to convince him, but we don't need his agreement for what we do.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>> Cc: joaotavora@gmail.com,  michael_heerdegen@web.de,  emacs-devel@gnu.org
>> Date: Sat, 11 Nov 2023 19:07:59 +0100
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > It isn't like we as a project never made any mistakes, but there's no
>> > reason to assume this decision is one of them until we have ample
>> > evidence.
>> 
>> Seq is 10 years in Emacs
>
> It is preloaded only since a little more than a year ago.

You're saying that the real promotion of seq is only a year old?
And that things will "improve" once the promotion picks up speed?

>> its polymorphism is unused in the tree.
>
> Searching for seq-* in the tree brings more than 590 hits in more than
> 170 Lisp files.

And? The polymorphism isn't used.

>> Joao showed that it's slow.
>
> No, he didn't.

Aha.

>> Dmitry mentioned that it is not a full replacement for what is in
>> cl-lib.
>
> Indeed, it is not a full replacement, and was never meant to be.  I
> believe its name says that much.
>
>> Po Lu mentioned the poor doc strings.
>
> That can be said about a lot of functions in Emacs, unfortunately.
> Including cl-lib, btw.  Which doesn't preclude us from using any of
> those poorly-documented functions.

I guess people like Alan will like that. Not only will he have to cope
with another set of poor doc strings, he will also have to cope more and
more with a new interface, while the old one won't go away.

Anyway, I'm not fighting against seq/map, should someone get that
impression. Live and let live.

But until someone can be bothered to make concrete arguments why generic
functions at this level make sense at all, and especially at this basic
level of the language, I think this is all useless and a mistake.

(And yes this is intentially provocative :-). I've lost hope to hear
something concrete a while ago.)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: joaotavora@gmail.com,  michael_heerdegen@web.de,  emacs-devel@gnu.org
> Date: Sun, 12 Nov 2023 07:59:13 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Seq is 10 years in Emacs
> >
> > It is preloaded only since a little more than a year ago.
> 
> You're saying that the real promotion of seq is only a year old?

Yes.

> And that things will "improve" once the promotion picks up speed?

No.  I'm saying that we are still very far from a point where we have
enough data to decide whether that year-old decision was wrong or not.
Whether we see an improvement or not, time will say.

> >> its polymorphism is unused in the tree.
> >
> > Searching for seq-* in the tree brings more than 590 hits in more than
> > 170 Lisp files.
> 
> And? The polymorphism isn't used.

Then I guess I don't understand what you mean by "polymorphism".

> >> Joao showed that it's slow.
> >
> > No, he didn't.
> 
> Aha.

No, really.  What he showed is that seq.el is in most (though not all)
cases _slower_ than the corresponding cl-lib functions.  Sometimes
much slower, sometimes slightly slower (and in at least one case
faster).  But that doesn't mean seq.el is "slow", enough to make its
use nonsensical.  Because why should we care that some call takes 2
usec instead of just 0.5 usec? both are negligible.  The difference
will only become visible if someone needs to call these in a very
tight loop with a very large number of iterations.

IOW, "slower" is not the same as "slow".  If we cared about "slow", we
would have implemented everything in C.  We didn't because we have
"other considerations", which are important to us and outweigh "slow"
when "slow" is "fast enough".  Exactly like in the case of seq.el vs
cl-lib.  Those "other considerations" in the latter case were
abundantly described and explained up-thread, so I'm sure you know
what they are, even though you disagree.

> I've lost hope to hear something concrete a while ago.

Same here, sadly.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>> Cc: joaotavora@gmail.com,  michael_heerdegen@web.de,  emacs-devel@gnu.org
>> Date: Sun, 12 Nov 2023 07:59:13 +0100
>>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>> >> Seq is 10 years in Emacs
>> >
>> > It is preloaded only since a little more than a year ago.
>>
>> You're saying that the real promotion of seq is only a year old?
>
> Yes.
>
>> And that things will "improve" once the promotion picks up speed?
>
> No.  I'm saying that we are still very far from a point where we have
> enough data to decide whether that year-old decision was wrong or not.
> Whether we see an improvement or not, time will say.

I think here's once difference in our thinking: One of my main critiques
does not depend on time. Please see below

>> >> its polymorphism is unused in the tree.
>> >
>> > Searching for seq-* in the tree brings more than 590 hits in more than
>> > 170 Lisp files.
>>
>> And? The polymorphism isn't used.
>
> Then I guess I don't understand what you mean by "polymorphism".

C'mon :-)

This sub-thread started, when I told why I'm not using seq and will not.
https://lists.gnu.org/archive/html/emacs-devel/2023-11/msg00122.html
I said this (this is all I wrote):

  The reason I will not use seq.el or map.el in the forseeable future is
  quite simple: I haven't ever needed an abstraction over sequence types
  using generic functions, and I never have CPU cycles to give away for
  free.

Neither you nor Richard ever addressed the question why this
polymorphism is needed or even a Good Thing. Richard even read seq.el
and even had his first exposure to pcase etc., but more than pondering
if one could make generic function calls faster was not the result.

>> >> Joao showed that it's slow.
>> >
>> > No, he didn't.
>>
>> Aha.
>
> No, really.  What he showed is that seq.el is in most (though not all)
> cases _slower_ than the corresponding cl-lib functions.  Sometimes
> much slower, sometimes slightly slower (and in at least one case
> faster).  But that doesn't mean seq.el is "slow", enough to make its
> use nonsensical.  Because why should we care that some call takes 2
> usec instead of just 0.5 usec? both are negligible.  The difference
> will only become visible if someone needs to call these in a very
> tight loop with a very large number of iterations.
>
> IOW, "slower" is not the same as "slow".  If we cared about "slow", we
> would have implemented everything in C.  We didn't because we have
> "other considerations", which are important to us and outweigh "slow"
> when "slow" is "fast enough".  Exactly like in the case of seq.el vs
> cl-lib.  Those "other considerations" in the latter case were
> abundantly described and explained up-thread, so I'm sure you know
> what they are, even though you disagree.

I actually got your intention when I wrote aha.

>
>> I've lost hope to hear something concrete a while ago.
>
> Same here, sadly.

Aha.

I doubt that your, and Richard's, intention is to really communicate
over these issues. It has already been decided by Richard and you,
right? The rest is rabulistic. Slow is fast enough, time will show,
maintainers think this or that, it's preloaded, it's not concrete, what
is polymorphism, and so on, and so on. Conspiracy theory is still missing.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: joaotavora@gmail.com,  michael_heerdegen@web.de,  emacs-devel@gnu.org
> Date: Mon, 13 Nov 2023 07:28:54 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> And that things will "improve" once the promotion picks up speed?
> >
> > No.  I'm saying that we are still very far from a point where we have
> > enough data to decide whether that year-old decision was wrong or not.
> > Whether we see an improvement or not, time will say.
> 
> I think here's once difference in our thinking: One of my main critiques
> does not depend on time. Please see below

It is perfectly fine to apply different strategies of assessing Emacs
features, especially since no one can have expert knowledge of every
part in Emacs, and we probably differ in which parts of Emacs are more
familiar to each one of us.  E.g., if we were talking about changes in
bidi.c, I would probably be able to offer definitive opinions and
critiques right there and then.

> >> > Searching for seq-* in the tree brings more than 590 hits in more than
> >> > 170 Lisp files.
> >>
> >> And? The polymorphism isn't used.
> >
> > Then I guess I don't understand what you mean by "polymorphism".
> 
> C'mon :-)
> 
> This sub-thread started, when I told why I'm not using seq and will not.
> https://lists.gnu.org/archive/html/emacs-devel/2023-11/msg00122.html
> I said this (this is all I wrote):
> 
>   The reason I will not use seq.el or map.el in the forseeable future is
>   quite simple: I haven't ever needed an abstraction over sequence types
>   using generic functions, and I never have CPU cycles to give away for
>   free.

This doesn't explain why you say that polymorphism isn't used, it just
states your reasons for not using seq.el.  So I'm still in the dark
here, sorry.

> Neither you nor Richard ever addressed the question why this
> polymorphism is needed or even a Good Thing.

Maybe it isn't strictly necessary, but I don't see why it should be
rejected, once it is there.  Several contributors to Emacs whose
opinions I respect liked the way seq.el is implemented, and wanted to
use it, which is one reason why it is preloaded.

> I doubt that your, and Richard's, intention is to really communicate
> over these issues. It has already been decided by Richard and you,
> right? The rest is rabulistic. Slow is fast enough, time will show,
> maintainers think this or that, it's preloaded, it's not concrete, what
> is polymorphism, and so on, and so on. Conspiracy theory is still missing.

"Rabulistic"? really?  With anyone else I'd take offense.  With you,
I'm just infinitely puzzled, not to say astonished.  Whatever did I do
or say to deserve such denigration?  I do have an opinion on this
issue, but since when is it deemed improper to have an opinion, and
why sticking to that opinion as long as no argument is brought up that
makes me change my mind is considered a vice?  I provided technical
arguments which explain my position; you can disagree, of course, but
please don't treat them as something unworthy, let alone dishonest.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> > Then I guess I don't understand what you mean by "polymorphism".
>> 
>> C'mon :-)
>> 
>> This sub-thread started, when I told why I'm not using seq and will not.
>> https://lists.gnu.org/archive/html/emacs-devel/2023-11/msg00122.html
>> I said this (this is all I wrote):
>> 
>>   The reason I will not use seq.el or map.el in the forseeable future is
>>   quite simple: I haven't ever needed an abstraction over sequence types
>>   using generic functions, and I never have CPU cycles to give away for
>>   free.
>
> This doesn't explain why you say that polymorphism isn't used, it just
> states your reasons for not using seq.el.  So I'm still in the dark
> here, sorry.

Been there already when it was lighter::
  https://lists.gnu.org/archive/html/emacs-devel/2023-11/msg00123.html

You said:
  What prevents us from implementing in seq.el methods for specific
  types of sequences?  AFAIU, the only reason we didn't is because no
  one has yet came up with important situations where the generics in
  seq.el are not performant enough.

>> Neither you nor Richard ever addressed the question why this
>> polymorphism is needed or even a Good Thing.
>
> Maybe it isn't strictly necessary, but I don't see why it should be
> rejected, once it is there.  Several contributors to Emacs whose
> opinions I respect liked the way seq.el is implemented, and wanted to
> use it, which is one reason why it is preloaded.

I never talked about rejecting anything. I said at least twice something
to the effect of "to each his own".

>
>> I doubt that your, and Richard's, intention is to really communicate
>> over these issues. It has already been decided by Richard and you,
>> right? The rest is rabulistic. Slow is fast enough, time will show,
>> maintainers think this or that, it's preloaded, it's not concrete, what
>> is polymorphism, and so on, and so on. Conspiracy theory is still missing.
>
> "Rabulistic"? really? With anyone else I'd take offense. With you, I'm
> just infinitely puzzled, not to say astonished. Whatever did I do or
> say to deserve such denigration?

No personal offence intended. If I caused that, I apologize.

> I do have an opinion on this
> issue, but since when is it deemed improper to have an opinion, and
> why sticking to that opinion as long as no argument is brought up that
> makes me change my mind is considered a vice?  I provided technical
> arguments which explain my position; you can disagree, of course, but
> please don't treat them as something unworthy, let alone dishonest.

As far as I am noncerned, everyone is entitled to his opinion. I can't
stress enough: live and let live. Would saying you're like a piece of
soap under the shower for me be better?

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Neither you nor Richard ever addressed the question why this
  > polymorphism is needed or even a Good Thing.

I have no opinion on that question.  I don't want to argue that we
_need_ seq.el.  (I don't think we _need_ CL functions either.)

Rather, I am comparing these constructs about how much they complicate
or burden the Lisp language.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 11/11/2023 20:07, Gerd Möllmann wrote:
> Seq is 10 years in Emacs, its polymorphism is unused in the tree. Joao
> showed that it's slow. Dmitry mentioned that it is not a full
> replacement for what is in cl-lib.

FWIW, it doesn't seem to me that seq's dynamic dispatch is the main part 
of what makes it "slow" in a number of cases. cl-lib is not that 
well-optimized either.

Just recently (https://debbugs.gnu.org/66806#23) we tested both 
cl-set-difference and seq-difference, and the hand-rolled version was 
much faster.

I think both of these libs could use more performance-oriented 
contributions, and for most other uses (data structures smaller than in 
that example) their speed is largely comparable.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 2658 bytes --]

On Sat, Nov 11, 2023 at 9:49 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
>
> On 11/11/2023 20:07, Gerd Möllmann wrote:
> > Seq is 10 years in Emacs, its polymorphism is unused in the tree. Joao
> > showed that it's slow. Dmitry mentioned that it is not a full
> > replacement for what is in cl-lib.
>
> FWIW, it doesn't seem to me that seq's dynamic dispatch is the main part
> of what makes it "slow" in a number of cases.

Indeed I don't know if dynamic dispatch is the main culprit, but
my measurements show that calling cl-some lots of times on
small lists specifically yields much worse performance than cl-some.

In larger lists, the difference is attenuated.  So that's why
I conjectured that dynamic dispatch played a role, but I
didn't check.

Now that you mention set difference, I find this:

(when nil
  (let ((l2 '(4 5 6 7)))
    (benchmark-run 100000
      (let ((l (list 1 2 3 4 8 9 10 11 12 13 14)))
        (cl-set-difference l l2))))(0.5964884789999999 20 0.38926870999999996)


  (let ((l2 '(4 5 6 7)))
    (benchmark-run 100000
      (let ((l (list 1 2 3 4 8 9 10 11 12 13 14)))
        (cl-nset-difference l l2))))(0.594109028 20 0.37972229299999993)


  (let ((l2 '(4 5 6 7)))
    (benchmark-run 100000
      (let ((l (list 1 2 3 4 8 9 10 11 12 13 14)))
        (seq-difference l l2))))(3.087290361 102 2.020063915)

Eli thinks this doesn't really show seq.el is slower than cl-lib.el
and that may be half-true: it just shows how it is slower in this
micro-benchmark.

But you seem to have a real-world measurable case, which is
of course better, so I'm curious.

> cl-lib is not that well-optimized either.

That's certainly true.  I've been looking at the code and there
are some serious optimization opportunities in cl-lib.el.  The
one you found right away, destructive versions delegating to
overconsing non-destructive ones, but also other opportunities.

I attach a patch that I hope you can try with that bug report.
Very lightly tested here, but seems to show a measurable
improvement.

(when nil
  (let ((l2 '(4 5 6 7)))
    (benchmark-run 100000
      (let ((l (list 1 2 3 4 8 9 10 11 12 13 14)))
        (cl-set-difference l l2))));; (0.480944603 8 0.3380962310000015)

  (let ((l2 '(4 5 6 7)))
    (benchmark-run 100000
      (let ((l (list 1 2 3 4 8 9 10 11 12 13 14)))
        (cl-nset-difference l l2))));; (0.31953939800000003 5
0.21426147399999707)

  (let ((l2 '(4 5 6 7)))
    (benchmark-run 100000
      (let ((l (list 1 2 3 4 8 9 10 11 12 13 14)))
        (seq-difference l l2)))) ;; (2.3330953689999996 41 1.8175730390000027)
  )

João

[-- Attachment #2: 0001-WIP-optimize-cl-nset-difference.patch --]
[-- Type: text/x-patch, Size: 6288 bytes --]

From 205c7e1d9972645cdbdc2dcc85b0a235f337503e Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Jo=C3=A3o=20T=C3=A1vora?= <joaotavora@gmail.com>
Date: Sun, 12 Nov 2023 00:52:18 +0000
Subject: [PATCH] WIP: optimize cl-nset-difference

---
 lisp/emacs-lisp/bytecomp.el          |  2 +-
 lisp/emacs-lisp/cl-seq.el            | 58 ++++++++++++++++++++++++++--
 test/lisp/emacs-lisp/cl-lib-tests.el | 45 ++++++++++++++++++++-
 3 files changed, 100 insertions(+), 5 deletions(-)

diff --git a/lisp/emacs-lisp/bytecomp.el b/lisp/emacs-lisp/bytecomp.el
index cc68db73c9f..e1a11ff9ba3 100644
--- a/lisp/emacs-lisp/bytecomp.el
+++ b/lisp/emacs-lisp/bytecomp.el
@@ -3601,7 +3601,7 @@ byte-compile-form
          (cl-nsubst 3) (cl-nsubst-if 3) (cl-nsubst-if-not 3)
          (cl-nsubstitute 3) (cl-nsubstitute-if 3) (cl-nsubstitute-if-not 3)
          (cl-nsublis 2)
-         (cl-nunion 1 2) (cl-nintersection 1 2) (cl-nset-difference 1 2)
+         (cl-nunion 1 2) (cl-nintersection 1 2) (cl-nset-difference 1)
          (cl-nset-exclusive-or 1 2)
          (cl-nreconc 1)
          (cl-sort 1) (cl-stable-sort 1) (cl-merge 2 3)
diff --git a/lisp/emacs-lisp/cl-seq.el b/lisp/emacs-lisp/cl-seq.el
index 74ca74db679..55c9926d916 100644
--- a/lisp/emacs-lisp/cl-seq.el
+++ b/lisp/emacs-lisp/cl-seq.el
@@ -961,16 +961,68 @@ cl-set-difference
 	  (pop cl-list1))
         (nreverse cl-res)))))
 
+(defmacro cl--list-member-with-keys (e l key test test-not)
+  "Checks if E is in L with standard CL KEY, TEST and TEST-NOT.
+Anaphoric macro! requires cl-test, cl-test-not and cl-key to be bound"
+  `(let ((e ,e) (l ,l) (key ,key) (test ,test) (test-not ,test-not))
+     (cond (test-not
+            (cond (key
+                   (setq e (funcall key e))
+                   (catch 'done
+                     (while l
+                       (unless (funcall test-not e (funcall key (car l)))
+                         (throw 'done t))
+                       (setq l (cdr l)))))
+                  (t
+                   (catch 'done
+                     (while l
+                       (unless (funcall test-not e (car l))
+                         (throw 'done t))
+                       (setq l (cdr l)))))))
+           (test
+            (cond (key
+                   (setq e (funcall key e))
+                   (catch 'done
+                     (while l
+                       (when (funcall test e (funcall key (car l)))
+                         (throw 'done t))
+                       (setq l (cdr l)))))
+                  (t
+                   (catch 'done
+                     (while l
+                       (when (funcall test e (car l))
+                         (throw 'done t))
+                       (setq l (cdr l)))))))
+           (key
+            (setq e (funcall key e))
+            (catch 'done
+              (while l
+                (when (eq e (funcall key (car l)))
+                  (throw 'done t))
+                (setq l (cdr l)))))
+           (t (memql e l)))))
+
 ;;;###autoload
-(defun cl-nset-difference (cl-list1 cl-list2 &rest cl-keys)
+(defun cl-nset-difference (list1 list2 &rest cl-keys)
   "Combine LIST1 and LIST2 using a set-difference operation.
 The resulting list contains all items that appear in LIST1 but not LIST2.
 This is a destructive function; it reuses the storage of LIST1 and LIST2
 whenever possible.
 \nKeywords supported:  :test :test-not :key
 \n(fn LIST1 LIST2 [KEYWORD VALUE]...)"
-  (if (or (null cl-list1) (null cl-list2)) cl-list1
-    (apply 'cl-set-difference cl-list1 cl-list2 cl-keys)))
+  (if (or (null list1) (null list2)) list1
+    (cl--parsing-keywords (:key :test :test-not) ()
+      (let ((res nil))
+	(while (consp list1)
+	  (if ;; (if (or cl-keys (numberp (car list1)))
+	      ;;     (apply 'cl-member (cl--check-key (car list1))
+	      ;;   	 list2 cl-keys)
+	      ;;   (memq (car list1) list2))
+              (cl--list-member-with-keys (car list1)
+                                         list2 cl-key cl-test cl-test-not)
+              (setf list1 (cdr list1))
+	      (cl-shiftf list1 (cdr list1) res list1)))
+        res))))
 
 ;;;###autoload
 (defun cl-set-exclusive-or (cl-list1 cl-list2 &rest cl-keys)
diff --git a/test/lisp/emacs-lisp/cl-lib-tests.el b/test/lisp/emacs-lisp/cl-lib-tests.el
index 0995e71db4e..2a55d982c42 100644
--- a/test/lisp/emacs-lisp/cl-lib-tests.el
+++ b/test/lisp/emacs-lisp/cl-lib-tests.el
@@ -558,5 +558,48 @@ cl-constantly
   (should (equal (mapcar (cl-constantly 3) '(a b c d))
                  '(3 3 3 3))))
 
-
+(ert-deftest cl-set-difference ()
+  ;; our set-difference preserves order, though it is not required to
+  ;; by cl standards, better keep that invariant
+  (should (equal (cl-set-difference '(1 2 3 4) '(3 4 5 6))
+                 '(1 2))))
+
+(ert-deftest cl-nset-difference ()
+  ;; our nset-difference doesn't
+  (let* ((l1 (list 1 2 3 4)) (l2 '(3 4 5 6))
+         (diff (cl-nset-difference l1 l2)))
+    (should (memq 1 diff))
+    (should (memq 2 diff))
+    (should (= (length diff) 2))
+    (should (equal l2 '(3 4 5 6))))
+  (let* ((l1 (list '(a . 1) '(b . 2) '(c . 3) '(d . 4)))
+         (l2 (list '(c . 3) '(d . 4) '(e . 5) '(f . 6)))
+         (diff (cl-nset-difference l1 l2 :key #'car)))
+    (should (member '(a . 1) diff))
+    (should (member '(b . 2) diff))
+    (should (= (length diff) 2)))
+  (let* ((l1 (list '("a" . 1) '("b" . 2) '("c" . 3) '("d" . 4)))
+         (l2 (list '("c" . 3) '("d" . 4) '("e" . 5) '("f" . 6)))
+         (diff (cl-nset-difference l1 l2 :key #'car :test #'string=)))
+    (should (member '("a" . 1) diff))
+    (should (member '("b" . 2) diff))
+    (should (= (length diff) 2))))
+
+
+(when nil
+  (let ((l2 '(4 5 6 7)))
+    (benchmark-run 100000
+      (let ((l (list 1 2 3 4 8 9 10 11 12 13 14)))
+        (cl-set-difference l l2))));; (0.480944603 8 0.3380962310000015)
+
+  (let ((l2 '(4 5 6 7)))
+    (benchmark-run 100000
+      (let ((l (list 1 2 3 4 8 9 10 11 12 13 14)))
+        (cl-nset-difference l l2))));; (0.31953939800000003 5 0.21426147399999707)
+
+  (let ((l2 '(4 5 6 7)))
+    (benchmark-run 100000
+      (let ((l (list 1 2 3 4 8 9 10 11 12 13 14)))
+        (seq-difference l l2)))) ;; (2.3330953689999996 41 1.8175730390000027)
+  )
 ;;; cl-lib-tests.el ends here
-- 
2.39.2

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Sun, 12 Nov 2023 02:36:54 +0000
> Cc: Gerd Möllmann <gerd.moellmann@gmail.com>, 
> 	Eli Zaretskii <eliz@gnu.org>, michael_heerdegen@web.de, emacs-devel@gnu.org
> 
> Eli thinks this doesn't really show seq.el is slower than cl-lib.el

That's not what I said.  Please don't mis-quote me.  I sometimes make
mistakes of writing inaccurate statements, but not in this case.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Sun, Nov 12, 2023 at 6:27 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Sun, 12 Nov 2023 02:36:54 +0000
> > Cc: Gerd Möllmann <gerd.moellmann@gmail.com>,
> >       Eli Zaretskii <eliz@gnu.org>, michael_heerdegen@web.de, emacs-devel@gnu.org
> >
> > Eli thinks this doesn't really show seq.el is slower than cl-lib.el
>
> That's not what I said.  Please don't mis-quote me.  I sometimes make
> mistakes of writing inaccurate statements, but not in this case.

1. I posted some micro-benchmarks about cl-some and seq-some.
2. Gerd wrote "Joao showed that it's slow.".
3. You replied, directly below  "No, he didn't.".

Now you say I misquoted you.  Hope to have rectified that.

So you _do_ think this shows seq.el is slower than cl-lib.el?

Because, to be perfectly honest, I don't!  Not with such little data,
at least.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Sun, 12 Nov 2023 10:21:00 +0000
> Cc: dmitry@gutov.dev, gerd.moellmann@gmail.com, michael_heerdegen@web.de, 
> 	emacs-devel@gnu.org
> 
> On Sun, Nov 12, 2023 at 6:27 AM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > > Eli thinks this doesn't really show seq.el is slower than cl-lib.el
> >
> > That's not what I said.  Please don't mis-quote me.  I sometimes make
> > mistakes of writing inaccurate statements, but not in this case.
> 
> 1. I posted some micro-benchmarks about cl-some and seq-some.
> 2. Gerd wrote "Joao showed that it's slow.".
> 3. You replied, directly below  "No, he didn't.".
> 
> Now you say I misquoted you.  Hope to have rectified that.
> 
> So you _do_ think this shows seq.el is slower than cl-lib.el?

I believe I explained the issue in another message.  It has to do with
the (significant in this case) difference between "slower" and "slow".
I responded to "slow", whereas you said I disagreed with "slower".

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Sun, Nov 12, 2023 at 11:22 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > 1. I posted some micro-benchmarks about cl-some and seq-some.
> > 2. Gerd wrote "Joao showed that it's slow.".
> > 3. You replied, directly below  "No, he didn't.".
> >
> > Now you say I misquoted you.  Hope to have rectified that.
> >
> > So you _do_ think this shows seq.el is slower than cl-lib.el?
>
> I believe I explained the issue in another message.  It has to do with
> the (significant in this case) difference between "slower" and "slow".
> I responded to "slow", whereas you said I disagreed with "slower".

That's clear now, thanks.  I read that email as well.

I agree things can never be taken in absolute terms, and one
has to consider specific jobs.

However, I think it is even more important to note that to
have any useful statement regarding the efficiency  -- relative
or absolute -- of any library, that statement should be falsifiable.
For statements like "micro-benchmark X doesn't matter in the common
case" to be useful we need to characterize the "common case"
well, because if we don't they're not falsifiable.

So I hope that in this discussion we can eventually come up with
some characterization of this "common case". so that we can
design experiments around it. I think performance, despite not being
the sole one, is an important consideration in this discussion.

IME sequence-processing ability and efficiency does play a very
significant role in determining the performance of certain
Lisp programs (including Common Lisp, of course).  This is so
especially when garbage generation is involved.

In certain Common Lisp implementations with more advanced
multi-generational garbage collectors, the results are sometimes
counter-intuitive: generating lots of fast-to-scavenge garbage
ends up being better than long-lived slow-to-scavenge
garbage).  But in Elisp at least, it seems garbage is always bad.

Destructive versions of sequence-processing functions are
in that respect very useful.  cl-lib.el has them but it seems
many of them are missing low-hanging optimization opportunities.
However I worked on that in another patch and it seems to have
provided a good speedup (in micro-benchmarks).  Let's see what
Dmitry comes up with when he takes them to a real-world case.

> Sometimes much slower, sometimes slightly slower (and in at least
> one case faster).

Anyway, I took the "cl-some vs seq-some" measurements again, after
applying this simple patch that I attach after my sig.  The summary:

Small lists:
  cl-some fastest
  seq-some 5.7 times slower, garbage collects

Big lists
  cl-some fastest
  seq-some 1.6 times slower

Small vectors:
  cl-some fastest
  seq-some 5.13 times slower, garbage collects

Big vectors
  cl-some fastest
  seq-some 1.27 times slower

The "in one case faster" is gone.  Also interesting is the
garbage generation footprint of seq.el functions in their current
form.

I'm not sure we can easily dismiss the "small seqs/tight loop" 5x
slowdown that seq.el exhibits in this microbenchmark, because as you
can see in the benchmark detail there's a lot of garbage generation
involved.  So even if the tight loop were to go away, that garbage
would pile up.  I also think that tight sequence-processing loops
are not all that uncommon in Elisp programs.  Finally, a program
may not use these tight loops but do lots of different sequence
processing functions: if it selects slow abstractions for all of
them, it ends up being just as if there was a tight loop.

But as I wrote above  it is best if we find real-world applications
to benchmark the two approaches.  Maybe someone else reading this
can help find one or two.

João


First the benchmarks in full:

(require 'cl-lib)

(defun bench-seq-some (seq)
  (seq-some #'identity seq))

(defun bench-cl-some (seq)
  (cl-some #'identity seq))

(defun bench-cl-loop-list (l)
  ;; checks for some types of improper lists
  (cl-loop for e in l thereis (identity e)))

(defun bench-cl-loop-vec (v)
  (cl-loop for e across v thereis (identity e)))


(when nil
  ;; Small lists
  (let ((l (list nil nil nil nil t)))
    ;; FASTEST (0.23516409500000002 0 0.0)
    (benchmark-run 1000000 (bench-cl-some l)))


  (let ((l (list nil nil nil nil t)))
     ;; 5.7x SLOWER (1.338184149 16 0.8307866220000051)
    (benchmark-run 1000000 (bench-seq-some l)))


  (let ((l (list nil nil nil nil t)))
     ;; 1.14x SLOWER (0.26885113699999996 0 0.0)
    (benchmark-run 1000000 (bench-cl-loop-list l)))


  ;; Big lists
  (let ((l (make-list 10000000 nil)))
    ;; FASTEST (0.266716895 0 0.0)
    (benchmark-run 1 (bench-cl-some l)))


  (let ((l (make-list 10000000 nil)))
    ;; 1.6x SLOWER (0.428996694 0 0.0)
    (benchmark-run 1 (bench-seq-some l)))


  (let ((l (make-list 10000000 nil)))
    ;; 1.05x SLOWER (0.279309231 0 0.0)
    (benchmark-run 1 (bench-cl-loop-list l)))


  ;; Small vectors
  (let ((v (vector nil nil nil nil t)))
    ;; FASTEST (0.257238335 0 0.0)
    (benchmark-run 1000000 (bench-cl-some v)))


  (let ((v (vector nil nil nil nil t)))
    ;; 5.13x SLOWER (1.317641304 16 0.8448574659999935)
    (benchmark-run 1000000 (bench-seq-some v)))


  (let ((v (vector nil nil nil nil t)))
    ;; 1.14x SLOWER (0.29413928100000003 0 0.0)
    (benchmark-run 1000000 (bench-cl-loop-vec v)))


  ;; Big vectors
  (let ((v (make-vector 10000000 nil)))
    ;; FASTEST (0.316211001 0 0.0)
    (benchmark-run 1 (bench-cl-some v)))


  (let ((v (make-vector 10000000 nil)))
    ;; 1.27x SLOWER (0.40362057500000004 0 0.0)
    (benchmark-run 1 (bench-seq-some v)))


  (let ((v (make-vector 10000000 nil)))
    ;; 1.11x SLOWER
    (benchmark-run 1 (bench-cl-loop-vec v)))

  )

And here's the patch I used to make cl-some faster for
the common case of only one non-list sequence passed.

diff --git a/lisp/emacs-lisp/cl-extra.el b/lisp/emacs-lisp/cl-extra.el
index 2ca2d03170a..7c09328eda5 100644
--- a/lisp/emacs-lisp/cl-extra.el
+++ b/lisp/emacs-lisp/cl-extra.el
@@ -206,16 +206,26 @@ cl-some
 non-nil value.

 \n(fn PREDICATE SEQ...)"
-  (if (or cl-rest (nlistp cl-seq))
-      (catch 'cl-some
-        (apply #'cl-map nil
-               (lambda (&rest cl-x)
-                 (let ((cl-res (apply cl-pred cl-x)))
-                   (if cl-res (throw 'cl-some cl-res))))
-       cl-seq cl-rest) nil)
-    (let ((cl-x nil))
-      (while (and cl-seq (not (setq cl-x (funcall cl-pred (pop cl-seq))))))
-      cl-x)))
+  (cond
+   (cl-rest
+    (catch 'cl-some
+      (apply #'cl-map nil
+             (lambda (&rest cl-x)
+               (let ((cl-res (apply cl-pred cl-x)))
+                 (if cl-res (throw 'cl-some cl-res))))
+             cl-seq cl-rest) nil))
+   ((nlistp cl-seq)
+    (let ((cl-x nil)
+          (l (length cl-seq))
+          (i 0))
+      (while (and (< i l)
+                  (prog1 (not (setq cl-x (funcall cl-pred
+                                                  (aref cl-seq i))))
+                    (cl-incf i))))
+      cl-x))
+   (t (let ((cl-x nil))
+        (while (and cl-seq (not (setq cl-x (funcall cl-pred (pop cl-seq))))))
+        cl-x))))

 ;;;###autoload
 (defun cl-every (cl-pred cl-seq &rest cl-rest)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

João Távora <joaotavora@gmail.com> writes:

> In larger lists, the difference is attenuated.  So that's why
> I conjectured that dynamic dispatch played a role, but I
> didn't check.

Also interesting could be how/if performance changes once methods are
defined on generic functions. That would make a difference in PCL, for
instance, because the presence of more than one method would make a
dicriminating function necessary. That's the function dispatching to
methods based on argument types, for example.

Or it could be, of course, that Emacs's doesn't do optimizations on that
level at all.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 12/11/2023 04:36, João Távora wrote:
>> cl-lib is not that well-optimized either.
> That's certainly true.  I've been looking at the code and there
> are some serious optimization opportunities in cl-lib.el.  The
> one you found right away, destructive versions delegating to
> overconsing non-destructive ones, but also other opportunities.
> 
> I attach a patch that I hope you can try with that bug report.
> Very lightly tested here, but seems to show a measurable
> improvement.

Looks like a solid improvement:

(setq cc (all-completions "" obarray))
;; (length cc) => 46002

(defvar list2 '("abc" "abc" "abc" "abc" "abc" "abc"))

(benchmark-run 10 (cl-set-difference cc list2 :test 'equal))
;; => 0.430

(benchmark-run 10 (setq cc (cl-nset-difference cc list2 :test 'equal)))
;; => 0.137 (!)

(benchmark-run 10 (my/set-difference cc list2))
;; => 0.144

(benchmark-run 10 (seq-difference cc list2))
;; => 0.814

(defun my/set-difference (list1 list2)
   (delq 'wrong
         (mapcar
          (lambda (c) (if (member c list2)
                     'wrong
                   c))
          list1)))

So it seems on par with the hand-written version (byte-compiled, too) 
that hardcodes the test function.

Curiously, though, if I make list2 longer (e.g. 12 elements rather than 
6), then my/set-difference starts to win noticeably (0.166 vs 0.200).

I'll keep the real-world case as-is (code in project--vc-list-files) 
both because the result with cl-nset-difference is still a little 
slower, and because it's slower in all released emacsen, but it 
definitely becomes an option.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Sun, Nov 12, 2023 at 10:09 PM Dmitry Gutov <dmitry@gutov.dev> wrote:

> So it seems on par with the hand-written version (byte-compiled, too)
> that hardcodes the test function.
>
> Curiously, though, if I make list2 longer (e.g. 12 elements rather than
> 6), then my/set-difference starts to win noticeably (0.166 vs 0.200).

I have a suspicion, I'll see what I can do.

More importantly, there's an important takeaway here.  Your
results seem to show that regardless of the alternative (cl-lib
or hand-rolled) the solution with Emacs's current recommended
sequence processing library is nearly 6 times slower in a
real-world use-case.

Maybe seq.el can be made faster too?  Who knows, but it seems
difficult without breaking at least some of its defgeneric-based
contract.

> I'll keep the real-world case as-is (code in project--vc-list-files)
> both because the result with cl-nset-difference is still a little
> slower, and because it's slower in all released emacsen, but it
> definitely becomes an option.

Not that I think it's worth it just for this, but isn't cl-lib.el a
:core package?  Theoretically, we could bump the version and depend
on that in project.el, as we do with other such things when one
dependency of a core package is enhanced.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 13/11/2023 01:14, João Távora wrote:
> On Sun, Nov 12, 2023 at 10:09 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
> 
>> So it seems on par with the hand-written version (byte-compiled, too)
>> that hardcodes the test function.
>>
>> Curiously, though, if I make list2 longer (e.g. 12 elements rather than
>> 6), then my/set-difference starts to win noticeably (0.166 vs 0.200).
> 
> I have a suspicion, I'll see what I can do.
> 
> More importantly, there's an important takeaway here.  Your
> results seem to show that regardless of the alternative (cl-lib
> or hand-rolled) the solution with Emacs's current recommended
> sequence processing library is nearly 6 times slower in a
> real-world use-case.

Note that it's still the difference for the case where the "business 
logic" (the filtering predicate or whatever) doesn't do anything. 
Although certain order-of-magnitude differences are worrying.

> Maybe seq.el can be made faster too?  Who knows, but it seems
> difficult without breaking at least some of its defgeneric-based
> contract.

I was wondering whether you tried looking into seq.el's performance 
problems. It being slower on shorter lists is quite expected: if the 
type dispatch has non-negligible overhead, that should be most 
noticeable when the rest of the work is small.

The case with longer lists and other data structures should be possible 
to improve, though. As long as the type dispatch only happens once per 
sequence, and not for each element.

>> I'll keep the real-world case as-is (code in project--vc-list-files)
>> both because the result with cl-nset-difference is still a little
>> slower, and because it's slower in all released emacsen, but it
>> definitely becomes an option.
> 
> Not that I think it's worth it just for this, but isn't cl-lib.el a
> :core package?

It's not: the version in ELPA is a small backward-compatibility shim for 
older emacsen.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

Dmitry Gutov <dmitry@gutov.dev> writes:

>> More importantly, there's an important takeaway here.  Your
>> results seem to show that regardless of the alternative (cl-lib
>> or hand-rolled) the solution with Emacs's current recommended
>> sequence processing library is nearly 6 times slower in a
>> real-world use-case.
>
> Note that it's still the difference for the case where the "business
> logic" (the filtering predicate or whatever) doesn't do
> anything.

OK.  So can you provide an even more realistic case?

> Although certain order-of-magnitude differences are worrying.

You can say that again...  I optimized cl-nset-difference considerably
more.  In the process also found your handrolled version can be sped
up considerably.  Have a look at this:

   (defun joaot/handrolled-nset-difference (list1 list2)
     (if (or (null list1) (null list2)) list1
       (let ((res nil))
         (while (consp list1)
           (if (funcall #'member (car list1) list2)
               (setf list1 (cdr list1))
             (cl-shiftf list1 (cdr list1) res list1)))
         res)))
    
   (defun dmitry/set-difference (list1 list2)
      (delq 'wrong
            (mapcar
             (lambda (c) (if (member c list2)
                        'wrong
                      c))
             list1)))
    
   (setq cc (all-completions "" obarray))
   (setq list2 (make-list 12 "shooveedoowaa"))
    
   (when nil
     ;; FASTEST (0.074594068 0 0.0)
     (benchmark-run 10 (setq cc (joaot/handrolled-nset-difference cc list2)))
    
     ;; FASTEST (0.070370948 0 0.0)
     (benchmark-run 10 (setq cc (cl-nset-difference cc list2 :test #'equal)))
    
     ;; 1.8x SLOWER (0.138797637 1 0.06212087500000507)
     (benchmark-run 10 (cl-set-difference cc list2 :test #'equal))
    
     ;; 3.2x SLOWER (0.22628817199999998 2 0.13694317399999534)
     (benchmark-run 10 (dmitry/set-difference cc list2))
    
     ;; 18x SLOWER  (1.29373404 12 0.7763814810000014) 
     (benchmark-run 10 (seq-difference cc list2 #'equal))
    
     )

Yes, that's _eighteen_.

All my work, including the docstring overhauls Alan requested and some
new tests, now in branch feature/cl-lib-improvements.

>> Maybe seq.el can be made faster too?  Who knows, but it seems
>> difficult without breaking at least some of its defgeneric-based
>> contract.
>
> I was wondering whether you tried looking into seq.el's performance
> problems. It being slower on shorter lists is quite expected: if the
> type dispatch has non-negligible overhead, that should be most
> noticeable when the rest of the work is small.
>
> The case with longer lists and other data structures should be
> possible to improve, though. As long as the type dispatch only happens
> once per sequence, and not for each element.

Maybe it's possible.  But there are two things here: first, you need
non-destructive versions of things in seq.el, because consing is always
a killer.  Second, the generic function interface means the typical
shortcuts I applied in cl-lib.el are difficult.  Maybe even impossible
without breaking the current contract?  I don't know the contract well
enough to tell.  At any rate seems like non-trivial work, but I'm happy
if someone can give it a shot.

Another thing I noticed is that recently cl-lib.el started depending on
seql.el, in its implementation.  Given what I've been seeing, this tells
me there's more low-hanging fruit to optimize in cl-lib.el.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 13/11/2023 03:03, João Távora wrote:
> Dmitry Gutov <dmitry@gutov.dev> writes:
> 
>>> More importantly, there's an important takeaway here.  Your
>>> results seem to show that regardless of the alternative (cl-lib
>>> or hand-rolled) the solution with Emacs's current recommended
>>> sequence processing library is nearly 6 times slower in a
>>> real-world use-case.
>>
>> Note that it's still the difference for the case where the "business
>> logic" (the filtering predicate or whatever) doesn't do
>> anything.
> 
> OK.  So can you provide an even more realistic case?

Probably not. Not sure which big users of cl-lib we can find in the 
core: comp.el uses it, but mostly for the 'loop' macro.

Anyway, my point was these benchmarks are very good for improving each 
of the libs, but not necessarily for making the ultimate choice between 
them (if we really had to).

>> Although certain order-of-magnitude differences are worrying.
> 
> You can say that again...  I optimized cl-nset-difference considerably
> more.  In the process also found your handrolled version can be sped
> up considerably.  Have a look at this:
> 
>     (defun joaot/handrolled-nset-difference (list1 list2)
>       (if (or (null list1) (null list2)) list1
>         (let ((res nil))
>           (while (consp list1)
>             (if (funcall #'member (car list1) list2)
>                 (setf list1 (cdr list1))
>               (cl-shiftf list1 (cdr list1) res list1)))
>           res)))
>      
>     (defun dmitry/set-difference (list1 list2)
>        (delq 'wrong
>              (mapcar
>               (lambda (c) (if (member c list2)
>                          'wrong
>                        c))
>               list1)))
>      
>     (setq cc (all-completions "" obarray))
>     (setq list2 (make-list 12 "shooveedoowaa"))
>      
>     (when nil
>       ;; FASTEST (0.074594068 0 0.0)
>       (benchmark-run 10 (setq cc (joaot/handrolled-nset-difference cc list2)))
>      
>       ;; FASTEST (0.070370948 0 0.0)
>       (benchmark-run 10 (setq cc (cl-nset-difference cc list2 :test #'equal)))
>      
>       ;; 1.8x SLOWER (0.138797637 1 0.06212087500000507)
>       (benchmark-run 10 (cl-set-difference cc list2 :test #'equal))
>      
>       ;; 3.2x SLOWER (0.22628817199999998 2 0.13694317399999534)
>       (benchmark-run 10 (dmitry/set-difference cc list2))
>      
>       ;; 18x SLOWER  (1.29373404 12 0.7763814810000014)
>       (benchmark-run 10 (seq-difference cc list2 #'equal))
>      
>       )

All right, time to roll out the big guns. For your attention, ladies and 
gentlemen, a version in pure, unadulterated Elisp, extracted from my 
unused patch of two weeks ago (https://debbugs.gnu.org/66806#17):

(defun dmitry/set-difference-nocons (list1 list2)
   (let (ref)
     (while (member (car list1) list2)
       (setq list1 (cdr list1)))
     (setq ref list1)
     (while ref
       (if (member (cadr ref) list2)
           (setcdr ref (cddr ref))
         (setq ref (cdr ref))))
     list1))

And the benchmarks (we're so fast, 100 iterations for stable numbers):

(when nil
   ;; (0.38175291299999997 0 0.0) 1.1X SLOWER
   (benchmark-run 100 (setq cc (joaot/handrolled-nset-difference cc list2)))

   ;; (0.9393577359999999 16 0.5063504589999965) NO COMMENTS
   (benchmark-run 100 (dmitry/set-difference cc list2))

   ;; (0.345876673 0 0.0) FASTEST (of course)
   (benchmark-run 100 (dmitry/set-difference-nocons cc list2))
   )

Anyway, it's pretty cool to have cl-nset-difference so "down to the metal".

Although I'm not sure if we should be worried that the funcall overhead 
makes basically no difference (e.g. if I inline the funcall in 
joaot/handrolled-nset-difference). I think it's been said that our 
funcalls are relatively slow.

> All my work, including the docstring overhauls Alan requested and some
> new tests, now in branch feature/cl-lib-improvements.

Nice.

>>> Maybe seq.el can be made faster too?  Who knows, but it seems
>>> difficult without breaking at least some of its defgeneric-based
>>> contract.
>>
>> I was wondering whether you tried looking into seq.el's performance
>> problems. It being slower on shorter lists is quite expected: if the
>> type dispatch has non-negligible overhead, that should be most
>> noticeable when the rest of the work is small.
>>
>> The case with longer lists and other data structures should be
>> possible to improve, though. As long as the type dispatch only happens
>> once per sequence, and not for each element.
> 
> Maybe it's possible.  But there are two things here: first, you need
> non-destructive versions of things in seq.el, because consing is always
> a killer.  Second, the generic function interface means the typical
> shortcuts I applied in cl-lib.el are difficult.  Maybe even impossible
> without breaking the current contract?  I don't know the contract well
> enough to tell.  At any rate seems like non-trivial work, but I'm happy
> if someone can give it a shot.

I hope someone will. And agree about destructive versions.

> Another thing I noticed is that recently cl-lib.el started depending on
> seql.el, in its implementation.  Given what I've been seeing, this tells
> me there's more low-hanging fruit to optimize in cl-lib.el.

Stefan moved a bunch of code there in 2019 (0e4dd67aae8b1003).

Good example to try and see if this actually made anything slower.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 4516 bytes --]

On 13/11/2023 04:43, Dmitry Gutov wrote:
>>> The case with longer lists and other data structures should be
>>> possible to improve, though. As long as the type dispatch only happens
>>> once per sequence, and not for each element.
>>
>> Maybe it's possible.  But there are two things here: first, you need
>> non-destructive versions of things in seq.el, because consing is always
>> a killer.  Second, the generic function interface means the typical
>> shortcuts I applied in cl-lib.el are difficult.  Maybe even impossible
>> without breaking the current contract?  I don't know the contract well
>> enough to tell.  At any rate seems like non-trivial work, but I'm happy
>> if someone can give it a shot.
> 
> I hope someone will. And agree about destructive versions.

Here's an experimental patch that makes seq-difference about as fast as 
your new improved non-destructive cl-set-difference. And some notes.

First of all, the type dispatch _does_ happen more than once per 
sequence in the current master. That doesn't seem to hurt much while the 
method is not specialized (only has the default implementation), but has 
impact as soon as the function gets additional method definitions.

Second, the seq-reduce implementations for the set functions don't seem 
optimal. So, check out the attached with the below continued benchmark:

(defun joaot/handrolled-nset-difference (list1 list2)
   (if (or (null list1) (null list2)) list1
     (let ((res nil))
       (while (consp list1)
         (if (funcall #'member (car list1) list2)
             (setf list1 (cdr list1))
           (cl-shiftf list1 (cdr list1) res list1)))
       res)))

(defun dmitry/set-difference-nocons (list1 list2)
   (let (ref)
     (while (member (car list1) list2)
       (setq list1 (cdr list1)))
     (setq ref list1)
     (while ref
       (if (member (cadr ref) list2)
           (setcdr ref (cddr ref))
         (setq ref (cdr ref))))
     list1))

(setq cc (all-completions "" obarray))
(setq list2 (make-list 12 "shooveedoowaa"))

(when nil
   ;; (0.38175291299999997 0 0.0)
   (benchmark-run 100 (setq cc (joaot/handrolled-nset-difference cc list2)))

   ;; (0.345876673 0 0.0)
   (benchmark-run 100 (dmitry/set-difference-nocons cc list2))

   ;; (1.2209711170000002 38 0.7669010760000001)
   (benchmark-run 100 (cl-set-difference cc list2 :test #'equal))

   ;; (2.10207541 67 1.410268502) NOT THE FASTEST
   (benchmark-run 100 (seq-difference cc list2))

   ;; (1.3434452970000001 33 0.7025866390000006)
   (benchmark-run 100 (seq-difference-2 cc list2))

   ;; (1.243865238 34 0.7060731869999994)
   (benchmark-run 100 (seq-difference-3 cc list2))
   )

seq-difference is the original implementation based on seq-reduce. It's 
much faster here, though, because of the change to seq-contains-p which 
teaches it to use 'member' when it can.

seq-difference-2 is an implementation that just switched to using 
seq-filter.

And seq-difference-3 is the one that makes sure the type dispatch 
happens only once (or twice), and not for every element in SEQUENCE1. 
For that, I defined a new generic seq-contains-pred which returns a 
function.

seq-difference-3 is the fastest among the last three and is about the 
speed of the cl-lib's variant.

The plot twist is that when I tried to extract the sequence type check 
into a separate method (see the commented out "cl-defmethod 
seq-contains-p" line), the performance of seq-difference and 
seq-difference-2 fell by an order of a magnitude (2s -> 9s). So it seems 
like using the new seq-contains-pred is the one way that would keep 
decent performance while supporting generic extensions.

The latter also means all current uses of seq-contains-p inside seq.el 
should be rewritten using seq-contains-pred.

As for seq-some, 1.27x or 1.6x slower for the identity predicate doesn't 
look as bad in comparison. Especially for an implementation this short 
and generic. It incurs an extra funcall in Lisp through the use in 
'seq-doseq', so that might be the cost. It should be easy to add a 
specialization for lists while still keeping the code shorter than 
cl-some, if one were so inclined. Would be cooler to find a more generic 
bottleneck like in the case above, but so far, no luck.

And in other interesting functions, cl-remove-if-not is about 4x faster 
than seq-filter in the best case (e.g. the list is not modified), but 
about the same in the worst case (when the last link is removed). There 
must be some shortcut there too which could be reproduced.

[-- Attachment #2: seq-difference.diff --]
[-- Type: text/x-patch, Size: 2979 bytes --]

diff --git a/lisp/emacs-lisp/seq.el b/lisp/emacs-lisp/seq.el
index 346250c1d35..6869f498e43 100644
--- a/lisp/emacs-lisp/seq.el
+++ b/lisp/emacs-lisp/seq.el
@@ -440,12 +440,43 @@ seq-contains
 (cl-defgeneric seq-contains-p (sequence elt &optional testfn)
   "Return non-nil if SEQUENCE contains an element \"equal\" to ELT.
 \"Equality\" is defined by the function TESTFN, which defaults to `equal'."
+  (cond
+   ((and (listp sequence) (or (null testfn) (eq testfn 'equal)))
+    (member elt sequence))
+   ((and (listp sequence) (eq testfn 'eq))
+    (memq elt sequence))
+   (t
     (catch 'seq--break
-      (seq-doseq (e sequence)
-        (let ((r (funcall (or testfn #'equal) e elt)))
-          (when r
-            (throw 'seq--break r))))
-      nil))
+    (seq-doseq (e sequence)
+      (let ((r (funcall (or testfn #'equal) e elt)))
+        (when r
+          (throw 'seq--break r))))
+    nil)))
+  )
+
+;; (cl-defmethod seq-contains-p ((sequence list) elt &optional testfn)
+;;   (cond
+;;    ((or (null testfn) (eq testfn 'equal))
+;;     (member elt sequence))
+;;    ((eq testfn 'eq)
+;;     (memq elt sequence))
+;;    (t
+;;     (cl-call-next-method))))
+
+(cl-defgeneric seq-contains-pred (_sequence &optional testfn)
+  (cond
+   ((or (null testfn) (eq testfn 'equal))
+    #'member)
+   ((eq testfn 'eq)
+    #'memq)
+   (t
+    (lambda (elt sequence)
+      (catch 'seq--break
+        (seq-doseq (e sequence)
+          (let ((r (funcall testfn e elt)))
+            (when r
+              (throw 'seq--break r))))
+        nil)))))
 
 (cl-defgeneric seq-set-equal-p (sequence1 sequence2 &optional testfn)
   "Return non-nil if SEQUENCE1 and SEQUENCE2 contain the same elements.
@@ -488,9 +519,10 @@ seq-positions
 (cl-defgeneric seq-uniq (sequence &optional testfn)
   "Return a list of the elements of SEQUENCE with duplicates removed.
 TESTFN is used to compare elements, and defaults to `equal'."
-  (let ((result '()))
+  (let ((result '())
+        (pred (seq-contains-pred testfn)))
     (seq-doseq (elt sequence)
-      (unless (seq-contains-p result elt testfn)
+      (unless (funcall pred elt result)
         (setq result (cons elt result))))
     (nreverse result)))
 
@@ -574,6 +606,20 @@ seq-difference
               (seq-reverse sequence1)
               '()))
 
+(cl-defgeneric seq-difference-2 (sequence1 sequence2 &optional testfn)
+  "Return list of all the elements that appear in SEQUENCE1 but not in SEQUENCE2.
+\"Equality\" of elements is defined by the function TESTFN, which
+defaults to `equal'."
+  (seq-filter
+   (lambda (elt) (not (seq-contains-p sequence2 elt testfn)))
+   sequence1))
+
+(cl-defgeneric seq-difference-3 (sequence1 sequence2 &optional testfn)
+  (let ((pred (seq-contains-pred sequence2 testfn)))
+    (seq-filter
+     (lambda (elt) (not (funcall pred elt sequence2)))
+     sequence1)))
+
 ;;;###autoload
 (cl-defgeneric seq-group-by (function sequence)
   "Apply FUNCTION to each element of SEQUENCE.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 3640 bytes --]

On Tue, Nov 14, 2023 at 12:41 AM Dmitry Gutov <dmitry@gutov.dev> wrote:

> On 13/11/2023 04:43, Dmitry Gutov wrote:
> >>> The case with longer lists and other data structures should be
> >>> possible to improve, though. As long as the type dispatch only happens
> >>> once per sequence, and not for each element.
> >>
> >> Maybe it's possible.  But there are two things here: first, you need
> >> non-destructive versions of things in seq.el, because consing is always
> >> a killer.  Second, the generic function interface means the typical
> >> shortcuts I applied in cl-lib.el are difficult.  Maybe even impossible
> >> without breaking the current contract?  I don't know the contract well
> >> enough to tell.  At any rate seems like non-trivial work, but I'm happy
> >> if someone can give it a shot.
> >
> > I hope someone will. And agree about destructive versions.
>
> Here's an experimental patch that makes seq-difference about as fast as
> your new improved non-destructive cl-set-difference. And some notes.


This is all interesting, until one ponders what happens if an existing
seq.el user somewhere has:

(cl-defmethod seq-contains-p ((seq my-voodoo-seq)
                              (elt (eql :secret-voodoo)) &optional _tesfn)
  (invoke-voodoo-priests seq))

making use of seq.el's support for abstract polymorphic sequences.

With seq.el 2.24 a seq-difference operation would consider this user's
method, with seq.el 2.24.dmitry (i.e. your fast seq-difference-3) it
simply won't.  This user's code is clearly broken.

But was the user allowed to do that in the first place?  If not,
why is seq-contains-p a public generic function?

Generic function protocols aren't just a bunch of generic functions
thrown together, they're also precise documentation as to how/when
the generics are called by the framework and what the user may
add to them, desirably making common things easy, difficult things
possible, and mistakes hard.  Normally the framework that calls
into generics isn't exposed to the user customizations, i.e.
it is made of regular defuns.  Of course, you seem to know this
as xref.el has such a practice.  But in seq.el, almost everything
is a generic function, even the entry points, somewhat bizarrely.

What I meant before is that these are non-trivial questions that
must be answered when embarking on these optimizations of seq.el.

So when I say it's non-trivial to optimize, it's not because of
figuring out if seq-filter or seq-reduce is better, or which shortcut
is faster, or if dispatch takes time, it's because if you have made
most everything public and customizable, you have offered an
extremely rich contract to users, so taking any shortcuts is
much more likely break it.

cl-lib.el is easy to optimize because the contract it offers
is mostly clear (it's straight out of the CL standard).

The case in the CL world against generic functions' performance
is often not that the dispatch is slow, but that
employing them too liberally makes framework optimization
hard, because you restrict yourself from optimization
opportunities.

seq.el is 10 years old.  Are people making custom sequences?
To what degree?  In any case, I think a tight contract should
clearly be written down as soon as possible, preferably with all
optimization opportunities like the ones you're making
planned out ahead first.  Maybe demote a bunch of these generic
functions to defuns, and make them internal.

Also as advice future set.el designers, do not just make
everything a generic fucntion, think in terms of protocols.

João

[-- Attachment #2: Type: text/html, Size: 4839 bytes --]

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 14/11/2023 04:27, João Távora wrote:
> 
> On Tue, Nov 14, 2023 at 12:41 AM Dmitry Gutov <dmitry@gutov.dev 
> <mailto:dmitry@gutov.dev>> wrote:
> 
>     On 13/11/2023 04:43, Dmitry Gutov wrote:
>      >>> The case with longer lists and other data structures should be
>      >>> possible to improve, though. As long as the type dispatch only
>     happens
>      >>> once per sequence, and not for each element.
>      >>
>      >> Maybe it's possible.  But there are two things here: first, you need
>      >> non-destructive versions of things in seq.el, because consing is
>     always
>      >> a killer.  Second, the generic function interface means the typical
>      >> shortcuts I applied in cl-lib.el are difficult.  Maybe even
>     impossible
>      >> without breaking the current contract?  I don't know the
>     contract well
>      >> enough to tell.  At any rate seems like non-trivial work, but
>     I'm happy
>      >> if someone can give it a shot.
>      >
>      > I hope someone will. And agree about destructive versions.
> 
>     Here's an experimental patch that makes seq-difference about as fast as
>     your new improved non-destructive cl-set-difference. And some notes.
> 
> 
> This is all interesting, until one ponders what happens if an existing
> seq.el user somewhere has:
> 
> (cl-defmethod seq-contains-p ((seq my-voodoo-seq)
>                                (elt (eql :secret-voodoo)) &optional _tesfn)
>    (invoke-voodoo-priests seq))
> 
> making use of seq.el's support for abstract polymorphic sequences.
> 
> With seq.el 2.24 a seq-difference operation would consider this user's
> method, with seq.el 2.24.dmitry (i.e. your fast seq-difference-3) it
> simply won't.  This user's code is clearly broken.
> 
> But was the user allowed to do that in the first place?  If not,
> why is seq-contains-p a public generic function?

It was allowed, and yes, indeed, it's a breaking change. So we should at 
least examine the existing public code out there and see whether such 
overrides exist.

> Generic function protocols aren't just a bunch of generic functions
> thrown together, they're also precise documentation as to how/when
> the generics are called by the framework and what the user may
> add to them, desirably making common things easy, difficult things
> possible, and mistakes hard.  Normally the framework that calls
> into generics isn't exposed to the user customizations, i.e.
> it is made of regular defuns.  Of course, you seem to know this
> as xref.el has such a practice.  But in seq.el, almost everything
> is a generic function, even the entry points, somewhat bizarrely.

I'm not sure that is a problem in itself, except in the cases like I've 
described, where the type dispatch ends up happening N times instead of 
just once.

> What I meant before is that these are non-trivial questions that
> must be answered when embarking on these optimizations of seq.el.
> 
> So when I say it's non-trivial to optimize, it's not because of
> figuring out if seq-filter or seq-reduce is better, or which shortcut
> is faster, or if dispatch takes time, it's because if you have made
> most everything public and customizable, you have offered an
> extremely rich contract to users, so taking any shortcuts is
> much more likely break it.
> 
> cl-lib.el is easy to optimize because the contract it offers
> is mostly clear (it's straight out of the CL standard).

cl-lib is pretty complex and alien from somebody with zero experience in 
it (and CL), and it's a lot more code, with different macros. So I'm not 
sure I'd call cl-lib easier overall.

> The case in the CL world against generic functions' performance
> is often not that the dispatch is slow, but that
> employing them too liberally makes framework optimization
> hard, because you restrict yourself from optimization
> opportunities.

I don't know if that's a major issue: just like cl-some has different 
branches, seq-some could have three different definitions. The 
approaches to optimization seem transferable, more or less.

> seq.el is 10 years old.  Are people making custom sequences?
> To what degree?

Personally I'd call it a code organization approach: instead of having 
type checks inside functions, you get the cl-generic's infrastructure to 
do it for you. There are some existing cl-defmethod's inside seq.el 
already. And their callers are taking advantage of specialized 
optimizations (e.g. seq-map -> mapcar, seq-do -> mapc). Specializations 
using 'member' and 'mapc' were missing, though (aside from 'seq-uniq').

seq-contains-pred should also have a defmethod for lists and the default 
impl for the rest.

> In any case, I think a tight contract should
> clearly be written down as soon as possible, preferably with all
> optimization opportunities like the ones you're making
> planned out ahead first.

If someone were to sit down and try to figure out the next optimization 
opportunity, the result could imply additional restrictions. Otherwise, 
I don't think we're going to be rewriting it from the ground up.

> Maybe demote a bunch of these generic
> functions to defuns, and make them internal.

I believe the actual value it provides is a list of implementations that 
are quite compact and as such easy to fix/extend/modify. And if cl-lib 
is handy for someone with CL experience, I'll guess that seq.el rings 
more familiar to the Clojure users.

Note that I'm not arguing in favor in one against the other, and the 
latter's still missing destructive versions. Which might also be harder 
to fit in a similar design even if one wanted to (though I'll welcome 
such attempts).

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 7585 bytes --]

[Replying to both of your mails here]

On Tue, Nov 14, 2023 Dmitry Gutov <dmitry@gutov.dev> wrote:

> And if the list has just 1 element (or zero?), the gap would be even
> larger.
>
> Only seeing a 1.6x difference here is a nice surprise, actually.

These two statements together seem contradictory.  Why would
you be happy to see a particular factor for a given
list length if you know the factor is unbounded in general
for small lists?

And it's only in the small lists case.  If I pass my own predicate to
both cl-set-difference and your best seq-difference-3

(defun myequal (a b)
  (equal a b))

(setq cc (make-list 12 "blabla"))
(setq list2 (make-list 12 "shooveedoowaa"))

(when nil
  ;; (0.106080265 4 0.03849865399999963)
  (benchmark-run 10000 (cl-set-difference cc list2 :test #'myequal))
  ;; (0.5504704210000001 39 0.394207416)
  (benchmark-run 10000 (seq-difference-3 cc list2 #'myequal))

   )

I get the 5.5x slowdown again (in one experiment, not shown,
I got a whopping 200x slowdown, but I can't reproduce it right now
from Emacs -Q, maybe I had some method on some crucial seq.el generic)

> > This is all interesting, until one ponders what happens if an existing
> > seq.el user somewhere has:
> >
> > (cl-defmethod seq-contains-p ((seq my-voodoo-seq)
> >                                (elt (eql :secret-voodoo)) &optional
_tesfn)
> >    (invoke-voodoo-priests seq))
> >
> > making use of seq.el's support for abstract polymorphic sequences.
> >
> > With seq.el 2.24 a seq-difference operation would consider this user's
> > method, with seq.el 2.24.dmitry (i.e. your fast seq-difference-3) it
> > simply won't.  This user's code is clearly broken.
> >
> > But was the user allowed to do that in the first place?  If not,
> > why is seq-contains-p a public generic function?
>
> It was allowed, and yes, indeed, it's a breaking change. So we should at
> least examine the existing public code out there and see whether such
> overrides exist.

Not so easy, I'm afraid.  seq-contains-p is not the only such generic,
it's just one I happened to spot first.  seq-do is a generic and in that
group mentioned in the sparse seq.el documentation about mandatory
customization, meaning presumably it is essential for custom sequences.

See https://github.com/search?q=%22%28cl-defmethod+seq-do%22&type=code for
a list of custom sequences that use it.

Your seq-difference-3 doesn't call seq-do like the original seq-difference
does, so the set difference operations for those custom sequences might
well be broken.  What's even more problematic is that it skips seq-do
entirely for certain predicates and not entirely for certain other
predicates.

And this is the type of headache that make-everything-a-gf designs
like seq.el's brings to developers trying to optimize it.

> > Generic function protocols aren't just a bunch of generic functions
> > thrown together, they're also precise documentation as to how/when
> > the generics are called by the framework and what the user may
> > add to them, desirably making common things easy, difficult things
> > possible, and mistakes hard.  Normally the framework that calls
> > into generics isn't exposed to the user customizations, i.e.
> > it is made of regular defuns.  Of course, you seem to know this
> > as xref.el has such a practice.  But in seq.el, almost everything
> > is a generic function, even the entry points, somewhat bizarrely.
>
> I'm not sure that is a problem in itself, except in the cases like I've
> described, where the type dispatch ends up happening N times instead of
> just once.

It's still bizarre.  Here's how I think: if an application is customizing
the entry point directly, why even call the entry point?  OK, so what if
a library is customizing the entry point?   It's presumably because that
library provides a data type for applications to instantiate and use.  But
if the library does that, all calling guarantees of other generic
functions are lost for, say, user method for subtypes of that data type
(or :around, or :after, etc).  So it's also nonsensical.  The only think
where it _could_ make a shred of sense is in a kind of "private library"
where
the application controls both the data type and knows exactly the shortcuts
taken.  But then "private library" is an oxymoron.

> > What I meant before is that these are non-trivial questions that
> > must be answered when embarking on these optimizations of seq.el.
> >
> > So when I say it's non-trivial to optimize, it's not because of
> > figuring out if seq-filter or seq-reduce is better, or which shortcut
> > is faster, or if dispatch takes time, it's because if you have made
> > most everything public and customizable, you have offered an
> > extremely rich contract to users, so taking any shortcuts is
> > much more likely break it.
> >
> > cl-lib.el is easy to optimize because the contract it offers
> > is mostly clear (it's straight out of the CL standard).
>
> cl-lib is pretty complex and alien from somebody with zero experience in
> it (and CL), and it's a lot more code, with different macros. So I'm not
> sure I'd call cl-lib easier overall.

Oh, don't get me wrong: cl-lib.el's implementation details are not pretty
and not easy, likely typical library code (though i've seen much much
worse).  What's fundamentally easier about optimizing cl-lib is that the
contract it offers is much, much more well specified.

Possibly because it comes as a result of careful design in committees of
very capable programmers in the 90's, before the CL winter, that were
already on to this class of difficulties and wanted to make a common
interface.

The interface they made for sequences is not fully generic, but there
_are_ a lot of different implementations for that interface, each CL
compiler has  one.  Most of the tricks, like what I did to optimize
cl-{n}set-difference  are standard stuff in the CL world.   There are even
reference LOOP implementations (likely much more performant than ours,
though possibly not compatible with some non-standard edge cases our
cl-loop has of which I know a couple).

> > The case in the CL world against generic functions' performance
> > is often not that the dispatch is slow, but that
> > employing them too liberally makes framework optimization
> > hard, because you restrict yourself from optimization
> > opportunities.
>
> I don't know if that's a major issue: just like cl-some has different
> branches, seq-some could have three different definitions. The
> approaches to optimization seem transferable, more or less.

Seq-some also calls into a lot of user customizable code, so it'll
suffer from the same class of problems.  Say, are you going to skip
the seq-do generic sometimes there as well? Maybe.  Depends on what
contract we want to uphold, and we have no idea.  At least I don't.

> I believe the actual value it provides is a list of implementations that
> are quite compact and as such easy to fix/extend/modify. And if cl-lib
> is handy for someone with CL experience, I'll guess that seq.el rings
> more familiar to the Clojure users.

Maybe in the superficial user interface, because of names etc.  Or
are you saying seq.el interface is extracted from Clojure's standard much
like cl-lib is from CL standard?  (Do Clojure generic functions work like
ours?)  Then by all means we should rush to study that contract closely,
to know what we can and can't do as optimizers.

João

[-- Attachment #2: Type: text/html, Size: 9378 bytes --]

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 14/11/2023 12:34, João Távora wrote:
> [Replying to both of your mails here]
> 
> On Tue, Nov 14, 2023 Dmitry Gutov <dmitry@gutov.dev 
> <mailto:dmitry@gutov.dev>> wrote:
> 
>  > And if the list has just 1 element (or zero?), the gap would be even
>  > larger.
>  >
>  > Only seeing a 1.6x difference here is a nice surprise, actually.
> 
> These two statements together seem contradictory.  Why would
> you be happy to see a particular factor for a given
> list length if you know the factor is unbounded in general
> for small lists?

I don't think it's unbounded, just high. And 12-element list is a nice 
size for this particular comparison. Programs using tiny list would in 
their majority be tiny as well, so it wouldn't matter.

Though of course there are exceptions: compilers, for example.

> And it's only in the small lists case.  If I pass my own predicate to
> both cl-set-difference and your best seq-difference-3
> 
> (defun myequal (a b)
>    (equal a b))
> 
> (setq cc (make-list 12 "blabla"))
> (setq list2 (make-list 12 "shooveedoowaa"))
> 
> (when nil
>    ;; (0.106080265 4 0.03849865399999963)
>    (benchmark-run 10000 (cl-set-difference cc list2 :test #'myequal))
>    ;; (0.5504704210000001 39 0.394207416)
>    (benchmark-run 10000 (seq-difference-3 cc list2 #'myequal))
> 
>     )
> 
> I get the 5.5x slowdown again (in one experiment, not shown,

Right. This is still for the small lists case. Here we suffer the 
dynamic dispatch at least twice, along with funcall indirection. Again, 
would be great to get these ratios down, but the main scenarios where I 
*did* worry about the performance of a sequence primitive, have always 
been large lists.

> I got a whopping 200x slowdown, but I can't reproduce it right now
> from Emacs -Q, maybe I had some method on some crucial seq.el generic)

That would be helpful to know as well.

>  > > This is all interesting, until one ponders what happens if an existing
>  > > seq.el user somewhere has:
>  > >
>  > > (cl-defmethod seq-contains-p ((seq my-voodoo-seq)
>  > >                                (elt (eql :secret-voodoo)) &optional 
> _tesfn)
>  > >    (invoke-voodoo-priests seq))
>  > >
>  > > making use of seq.el's support for abstract polymorphic sequences.
>  > >
>  > > With seq.el 2.24 a seq-difference operation would consider this user's
>  > > method, with seq.el 2.24.dmitry (i.e. your fast seq-difference-3) it
>  > > simply won't.  This user's code is clearly broken.
>  > >
>  > > But was the user allowed to do that in the first place?  If not,
>  > > why is seq-contains-p a public generic function?
>  >
>  > It was allowed, and yes, indeed, it's a breaking change. So we should at
>  > least examine the existing public code out there and see whether such
>  > overrides exist.
> 
> Not so easy, I'm afraid.  seq-contains-p is not the only such generic,
> it's just one I happened to spot first.  seq-do is a generic and in that
> group mentioned in the sparse seq.el documentation about mandatory
> customization, meaning presumably it is essential for custom sequences.
> 
> See https://github.com/search?q=%22%28cl-defmethod+seq-do%22&type=code 
> <https://github.com/search?q=%22%28cl-defmethod+seq-do%22&type=code> for
> a list of custom sequences that use it.
> 
> Your seq-difference-3 doesn't call seq-do like the original seq-difference
> does, so the set difference operations for those custom sequences might
> well be broken.  What's even more problematic is that it skips seq-do
> entirely for certain predicates and not entirely for certain other
> predicates.

The way I understand this, is any new sequence type has to implement 
seq-do. As soon as that happens, a lot of (probably all) sequence 
functions in seq.el start working on that type.

But the author is also free to provide specialized implementations for 
individual functions (usually for better performance) -- and those 
implementations don't have to use seq-do. Indeed, in most cases the 
optimization would involve cutting out the overhead that seq-do brings 
in the general case.

> It's still bizarre.  Here's how I think: if an application is customizing
> the entry point directly, why even call the entry point?  OK, so what if
> a library is customizing the entry point?   It's presumably because that
> library provides a data type for applications to instantiate and use.  But
> if the library does that, all calling guarantees of other generic
> functions are lost for, say, user method for subtypes of that data type
> (or :around, or :after, etc).  So it's also nonsensical.  The only think
> where it _could_ make a shred of sense is in a kind of "private library" 
> where
> the application controls both the data type and knows exactly the shortcuts
> taken.  But then "private library" is an oxymoron.

An application shouldn't be customizing separate entry points. Separate 
libs (providing new data types) can do that. E.g. if we have a lib 
providing lazy sequences, it can provide its own set of implementations 
for seq.el. And when doing that, make sure the resulting behavior is 
consistent between functions.

I haven't though much about the interaction with :around, :after, etc. 
Maybe they'll be fine, or maybe it would be better to avoid it. Look 
fine on the superficial level, though.

> Oh, don't get me wrong: cl-lib.el's implementation details are not pretty
> and not easy, likely typical library code (though i've seen much much
> worse).  What's fundamentally easier about optimizing cl-lib is that the
> contract it offers is much, much more well specified.
> 
> Possibly because it comes as a result of careful design in committees of
> very capable programmers in the 90's, before the CL winter, that were
> already on to this class of difficulties and wanted to make a common
> interface.
> > The interface they made for sequences is not fully generic, but there
> _are_ a lot of different implementations for that interface, each CL
> compiler has  one.  Most of the tricks, like what I did to optimize
> cl-{n}set-difference  are standard stuff in the CL world.   There are even
> reference LOOP implementations (likely much more performant than ours,
> though possibly not compatible with some non-standard edge cases our
> cl-loop has of which I know a couple).

I do believe it's helpful to have it around.

>  > > The case in the CL world against generic functions' performance
>  > > is often not that the dispatch is slow, but that
>  > > employing them too liberally makes framework optimization
>  > > hard, because you restrict yourself from optimization
>  > > opportunities.
>  >
>  > I don't know if that's a major issue: just like cl-some has different
>  > branches, seq-some could have three different definitions. The
>  > approaches to optimization seem transferable, more or less.
> 
> Seq-some also calls into a lot of user customizable code, so it'll
> suffer from the same class of problems.  Say, are you going to skip
> the seq-do generic sometimes there as well?

I believe so.

>  > I believe the actual value it provides is a list of implementations that
>  > are quite compact and as such easy to fix/extend/modify. And if cl-lib
>  > is handy for someone with CL experience, I'll guess that seq.el rings
>  > more familiar to the Clojure users.
> 
> Maybe in the superficial user interface, because of names etc.  Or
> are you saying seq.el interface is extracted from Clojure's standard much
> like cl-lib is from CL standard?  (Do Clojure generic functions work like
> ours?)  Then by all means we should rush to study that contract closely,
> to know what we can and can't do as optimizers.

It looks very much inspired, both in naming and in the implementation 
approach. But it's not a carbon copy (much farther from it than in 
cl-lib's example), and our VM is also quite different from JVM in 
performance characteristics.

If someone were to write up a direct comparison, that would be great, of 
course.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Tue, Nov 14, 2023 at 11:47 AM Dmitry Gutov <dmitry@gutov.dev> wrote:

> I don't think it's unbounded, just high. And 12-element list is a nice
> size for this particular comparison. Programs using tiny list would in
> their majority be tiny as well, so it wouldn't matter.
>
> Though of course there are exceptions: compilers, for example.

Yes, of course, of which there are many in Lisp: every macro
is one.  And most of the uses for set difference operations that I
see in core seem to be for relatively small lists.  Just because
for large sets (at least for large list2), lists are inherently
not good anyway because quadratic ;-)

> > (defun myequal (a b)
> >    (equal a b))
> >
> > (setq cc (make-list 12 "blabla"))
> > (setq list2 (make-list 12 "shooveedoowaa"))
> >
> > (when nil
> >    ;; (0.106080265 4 0.03849865399999963)
> >    (benchmark-run 10000 (cl-set-difference cc list2 :test #'myequal))
> >    ;; (0.5504704210000001 39 0.394207416)
> >    (benchmark-run 10000 (seq-difference-3 cc list2 #'myequal))
> >
> >     )
> >
> > I get the 5.5x slowdown again (in one experiment, not shown,
>
> Right. This is still for the small lists case.

Not sure.  I don't have time to test now, but the custom #'myequal
predicates negates your seq-do shortcut (which is probably illegal anyway,
but let's pretend it isn't).  If that shortcut is negated, things
become slower than in the pred=#'equal case, no?

> > Your seq-difference-3 doesn't call seq-do like the original seq-difference
> > does, so the set difference operations for those custom sequences might
> > well be broken.  What's even more problematic is that it skips seq-do
> > entirely for certain predicates and not entirely for certain other
> > predicates.
>
> The way I understand this, is any new sequence type has to implement
> seq-do. As soon as that happens, a lot of (probably all) sequence
> functions in seq.el start working on that type.

Right.  But seq-difference-3 doesn't call into seq-do anymore,
at least not always and not in the same way.  So today, _before_
your seq-difference "skips-the-seq-do" optimization, a given person's
set difference operations would work just fine for their custom
data type.  The person is happy and doesn't care about seq.el's
performance.

One day they upgrade seq.el which comes with your optimization to
seq-difference, and that code is mysteriously broken.  Worse  for some
equality predicates it is broken  in a given way, for other equality
predicates it's a different error.

Can't you see the fundamental problem?

> > the application controls both the data type and knows exactly the shortcuts
> > taken.  But then "private library" is an oxymoron.
>
> An application shouldn't be customizing separate entry points.

Right.  We agree.  It makes no sense.

> Separate
> libs (providing new data types) can do that. E.g. if we have a lib
> providing lazy sequences, it can provide its own set of implementations
> for seq.el. And when doing that, make sure the resulting behavior is
> consistent between functions.

Right, but then if they fail the customization of the entry point
generic like not calling the right generics at the right time, then
_other_  customizations of _other_ generics won't be called, or will
be called too much.  Just like in your seq-difference-3 optimization
the user code breaks.

> > cl-{n}set-difference  are standard stuff in the CL world.   There are even
> > reference LOOP implementations (likely much more performant than ours,
> > though possibly not compatible with some non-standard edge cases our
> > cl-loop has of which I know a couple).
>
> I do believe it's helpful to have it around.

What is?

> > Seq-some also calls into a lot of user customizable code, so it'll
> > suffer from the same class of problems.  Say, are you going to skip
> > the seq-do generic sometimes there as well?
>
> I believe so.

Then my seq-do methods will just stop being called.

> It looks very much inspired, both in naming and in the implementation
> approach. But it's not a carbon copy (much farther from it than in
> cl-lib's example), and our VM is also quite different from JVM in
> performance characteristics.

OK, but naming and implementation are exactly the two least
important things that don't matter much here.  What matters are the
interfaces, the contracts for the implementor's interface and user's
interface: which would specify exactly how those generic functions
are expected to cooperate with each other.  This is what has to be
written down in a generic protocol.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 14/11/2023 14:14, João Távora wrote:
>> The way I understand this, is any new sequence type has to implement
>> seq-do. As soon as that happens, a lot of (probably all) sequence
>> functions in seq.el start working on that type.
> Right.  But seq-difference-3 doesn't call into seq-do anymore,
> at least not always and not in the same way.  So today,_before_
> your seq-difference "skips-the-seq-do" optimization, a given person's
> set difference operations would work just fine for their custom
> data type.  The person is happy and doesn't care about seq.el's
> performance.

seq-difference-3 calls seq-filter which calls seq-map, which in turn 
delegates to seq-do (or a specialized implementation).

There is a bug in my prototype patch -- the absence of the type check 
inside seq-contains-pred. I think it should be a specialized impl for 
seq-contains-pred anyway (for when sequence is a list). Since 
seq-contains-pred is only called once, that shouldn't have any 
measurable effect on performance, aside from very tiny lists.

> One day they upgrade seq.el which comes with your optimization to
> seq-difference, and that code is mysteriously broken.  Worse  for some
> equality predicates it is broken  in a given way, for other equality
> predicates it's a different error.
> 
> Can't you see the fundamental problem?
> 
> Right, but then if they fail the customization of the entry point
> generic like not calling the right generics at the right time, then
> _other_   customizations of_other_  generics won't be called, or will
> be called too much.  Just like in your seq-difference-3 optimization
> the user code breaks.
>>> cl-{n}set-difference  are standard stuff in the CL world.   There are even
>>> reference LOOP implementations (likely much more performant than ours,
>>> though possibly not compatible with some non-standard edge cases our
>>> cl-loop has of which I know a couple).
>> I do believe it's helpful to have it around.
> What is?

cl-lib.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Tue, Nov 14, 2023 at 12:20 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
>
> On 14/11/2023 14:14, João Távora wrote:
> >> The way I understand this, is any new sequence type has to implement
> >> seq-do. As soon as that happens, a lot of (probably all) sequence
> >> functions in seq.el start working on that type.
> > Right.  But seq-difference-3 doesn't call into seq-do anymore,
> > at least not always and not in the same way.  So today,_before_
> > your seq-difference "skips-the-seq-do" optimization, a given person's
> > set difference operations would work just fine for their custom
> > data type.  The person is happy and doesn't care about seq.el's
> > performance.
>
> seq-difference-3 calls seq-filter which calls seq-map, which in turn
> delegates to seq-do

OK, but not always in the same ways as seq-difference did,
and not always in the same way for different predicates.
And and not for sequences of type sequence and list, so if I have
special method on seq-do for my FUNCTION argument, it will
not be called.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 793 bytes --]

On Tue, Nov 14, 2023 at 2:27 AM João Távora <joaotavora@gmail.com> wrote:

>
>
> The case in the CL world against generic functions' performance
> is often not that the dispatch is slow, but that
>


That said, it still matters, of course.  Here's your non-destructive
seq-difference-3 but with many small lists:

(setq cc (make-list 12 "blabla"))
(setq list2 (make-list 12 "shooveedoowaa"))

;; (4.225398191 31 2.3103362619999928)
(benchmark-run 1000000 (cl-set-difference cc list2 :test #'equal))

;; (6.959332908 56 4.235225837999991)
(benchmark-run 1000000 (seq-difference-3 cc list2))

And of course the destructive version still wins by 10x

;; (0.676608019 4 0.26997238299998116)
(benchmark-run 1000000 (cl-nset-difference cc list2 :test #'equal))

João

[-- Attachment #2: Type: text/html, Size: 1367 bytes --]

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 14/11/2023 04:44, João Távora wrote:
> 
> On Tue, Nov 14, 2023 at 2:27 AM João Távora <joaotavora@gmail.com 
> <mailto:joaotavora@gmail.com>> wrote:
> 
> 
> 
>     The case in the CL world against generic functions' performance
>     is often not that the dispatch is slow, but that
> 
> 
> That said, it still matters, of course.  Here's your non-destructive
> seq-difference-3 but with many small lists:
> 
> (setq cc (make-list 12 "blabla"))
> (setq list2 (make-list 12 "shooveedoowaa"))

> ;; (4.225398191 31 2.3103362619999928)
> (benchmark-run 1000000 (cl-set-difference cc list2 :test #'equal))
> 
> ;; (6.959332908 56 4.235225837999991)
> (benchmark-run 1000000 (seq-difference-3 cc list2))

And if the list has just 1 element (or zero?), the gap would be even 
larger. Yes, generic dispatch has a cost.

Only seeing a 1.6x difference here is a nice surprise, actually.

> And of course the destructive version still wins by 10x
> 
> ;; (0.676608019 4 0.26997238299998116)
> (benchmark-run 1000000 (cl-nset-difference cc list2 :test #'equal))

No doubt.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Michael Heerdegen]

João Távora <joaotavora@gmail.com> writes:

> This is all interesting, until one ponders what happens if an existing
> seq.el user somewhere has:
>
> (cl-defmethod seq-contains-p ((seq my-voodoo-seq)
>                               (elt (eql :secret-voodoo)) &optional _tesfn)
>   (invoke-voodoo-priests seq))
>
> making use of seq.el's support for abstract polymorphic sequences.
>
> With seq.el 2.24 a seq-difference operation would consider this user's
> method, with seq.el 2.24.dmitry (i.e. your fast seq-difference-3) it 
> simply won't.  This user's code is clearly broken.

This is something to keep in mind indeed.  But this problem only appears
when a new generic is added to seq.el.  Then the programmers defining
implementations of the generics needs to update definitions to include a
definition for the new generic - s?he probably wants to do that anyway.

We will surely not have to add a new generic very often.

> But was the user allowed to do that in the first place?  If not,
> why is seq-contains-p a public generic function?

We have lost our way a little more, because cl-lib would support your
my-voodoo-seq not very well.

Michael.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 14/11/2023 04:27, João Távora wrote:
> This is all interesting, until one ponders what happens if an existing
> seq.el user somewhere has:
> 
> (cl-defmethod seq-contains-p ((seq my-voodoo-seq)
>                                (elt (eql :secret-voodoo)) &optional _tesfn)
>    (invoke-voodoo-priests seq))
> 
> making use of seq.el's support for abstract polymorphic sequences.
> 
> With seq.el 2.24 a seq-difference operation would consider this user's
> method, with seq.el 2.24.dmitry (i.e. your fast seq-difference-3) it
> simply won't.  This user's code is clearly broken.
> 
> But was the user allowed to do that in the first place?  If not,
> why is seq-contains-p a public generic function?

Allow me to try to answer this again: special voodoo is probably not 
allowed. What the specialized methods are supposed to do is implement 
the same logic (described in the generic function's docstring) but in a 
more efficient manner.

So the author of the new data type (who works on the implementation of 
seq-do and specialized methods) is responsible for making sure the new 
definition is equivalent but faster.

This is not documented, but seems to hold for all the code in the 
package. If someone found a case where violating the above rule would be 
helpful for a certain practical purpose, we could discuss it.

Note that all the functions in seq.el are currently non-destructive or 
perhaps even "pure", so it's not like there could be something special 
done to the input data behind the scenes, aside from what's in the 
contract (data in + data out).

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Tue, Nov 14, 2023 at 11:11 PM Dmitry Gutov <dmitry@gutov.dev> wrote:

> Allow me to try to answer this again: special voodoo is probably not
> allowed. What the specialized methods are supposed to do is implement
> the same logic (described in the generic function's docstring) but in a
> more efficient manner.

But isn't my method just that? Let's say the methods's is simpler,
like this:

(cl-defmethod seq-contains-p ((seq my-voodoo-seq)
                                (elt (eql :secret-voodoo)) &optional _tesfn)
    ;; my-voodoo-seq always has :secret-voodoo.
    t)

Hard to be more efficient than that, right?  Then I could use

(seq-difference '(1 2 3 :secret-voodoo 5) (make-a-voodoo-seq))

And get any voodoo out.

> This is not documented,

You don't say? :-)  i think we should get started on documenting all
these ad-hoc rules (sorry but in this case they are quite literally
ad-hoc).  Better than nothing, I guess, since as to actually enforcing
them, I think that train has left the station a long time ago and is
now in a voodoo swamp in monkey island.

Anyway, I'll be back with more custom sequences soon.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 15/11/2023 01:52, João Távora wrote:
> On Tue, Nov 14, 2023 at 11:11 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
> 
>> Allow me to try to answer this again: special voodoo is probably not
>> allowed. What the specialized methods are supposed to do is implement
>> the same logic (described in the generic function's docstring) but in a
>> more efficient manner.
> 
> But isn't my method just that?

It's a good style, but any specific project's requirements might differ, 
and libraries often have to get defensive against bad practices, too.

> Let's say the methods's is simpler,
> like this:
> 
> (cl-defmethod seq-contains-p ((seq my-voodoo-seq)
>                                  (elt (eql :secret-voodoo)) &optional _tesfn)
>      ;; my-voodoo-seq always has :secret-voodoo.
>      t)
> 
> Hard to be more efficient than that, right?  Then I could use
> 
> (seq-difference '(1 2 3 :secret-voodoo 5) (make-a-voodoo-seq))
> 
> And get any voodoo out.

Okay... so a voodoo seq contains the :secret-voodoo keyword (maybe once, 
maybe several times, but not any other element).

Then it must have a seq-do implementation that yields :secret-voodoo one 
or several times. And then the seq-contains-pred implementation will do 
right by it (the default one, not the specialization for lists).

>> This is not documented,
> 
> You don't say? :-)  i think we should get started on documenting all
> these ad-hoc rules (sorry but in this case they are quite literally
> ad-hoc).  Better than nothing, I guess, since as to actually enforcing
> them, I think that train has left the station a long time ago and is
> now in a voodoo swamp in monkey island.

Not sure "enforcing them" is the right phrase: I wonder if there's ever 
been a change proposed to seq.el which would break them.

Anyway, documenting them sounds best. As soon as we can arrive on a 
phrasing that sounds clear to all interested parties.

> Anyway, I'll be back with more custom sequences soon.

Please do.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Wed, Nov 15, 2023 at 12:46 AM Dmitry Gutov <dmitry@gutov.dev> wrote:

> Then it must have a seq-do implementation that yields :secret-voodoo one
> or several times.

Not really.  Only if I use my seq in contexts where I need that
generic, and being second argument to seq-difference doesn't count.
So why should I bother implementing that when my code is working
fine today?

But say I did that seq-do, then what is the seq-contains-p generic good
for then?  Why some many redundant generics that the user learns the
hard way have to be consistent with each other?

> And then the seq-contains-pred implementation will do
> right by it (the default one, not the specialization for lists).
>
> >> This is not documented,
> >
> > You don't say? :-)  i think we should get started on documenting all
> > these ad-hoc rules (sorry but in this case they are quite literally
> > ad-hoc).  Better than nothing, I guess, since as to actually enforcing
> > them, I think that train has left the station a long time ago and is
> > now in a voodoo swamp in monkey island.
>
> Not sure "enforcing them" is the right phrase: I wonder if there's ever
> been a change proposed to seq.el which would break them.

I think it's pretty easy to do, especially if these optimziations
want going to go all the way to cl-lib's easier-to-get optimal
performance.  What about this?  Is this toy implementation of
compressed space-saving lists legal?

(cl-defmethod seq-do (function (l (head :m6-sparse)))
  (mapc (lambda (e)
          (if (eq e :double-o-seven)
              (progn (funcall function 0)
                     (funcall function 0)
                     (funcall function 7))
            (funcall function e)))
        (cdr l)))

(cl-defmethod seq-reverse ((l (head :m6-sparse)))
  (let (res)
    (seq-do (lambda (e) (push e res)) l)
    res))

(seq-difference '(:m6-sparse 1 2 :double-o-seven 4) '(7)) ;; => (1 2 0 0 4)

Again, not implementing all the generics, just the ones I need
to get my desired effect, which in this case is passing a
custom sequence type as a first argument to seq-difference.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Wed, Nov 15, 2023 at 1:07 AM João Távora <joaotavora@gmail.com> wrote:

> Not really.  Only if I use my seq in contexts where I need that
> generic, and being second argument to seq-difference doesn't count.
> So why should I bother implementing that when my code is working
> fine today?
>
> But say I did that seq-do, then what is the seq-contains-p generic good
> for then?

Let me answer that for you... It could be, maybe even likely, that the
code was trying to get out of seq-do for some types of operation, because in
the case of lazy lists, seq-do would mandate the list be completely expanded.

So, while the code works, you seq-difference-3 now calls seq-do on the
second argument, and kills my laziness, naughty naughty.

Actually for this point to make sense, it doesn't even have to be
a bizarre voodoo list.  The lazy list may just be expanded enough already
for me to know in seq-contains-p that there's a given element
there without having to call the generator for more list elements.

So seq-contains-p was indeed an optimization, in more or less
the sense you described, which is now lost when you tried to optimize
for bare Lisp sequences.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 15/11/2023 03:17, João Távora wrote:
> On Wed, Nov 15, 2023 at 1:07 AM João Távora <joaotavora@gmail.com> wrote:
> 
>> Not really.  Only if I use my seq in contexts where I need that
>> generic, and being second argument to seq-difference doesn't count.
>> So why should I bother implementing that when my code is working
>> fine today?
>>
>> But say I did that seq-do, then what is the seq-contains-p generic good
>> for then?
> 
> Let me answer that for you... It could be, maybe even likely, that the
> code was trying to get out of seq-do for some types of operation, because in
> the case of lazy lists, seq-do would mandate the list be completely expanded.
> 
> So, while the code works, you seq-difference-3 now calls seq-do on the
> second argument, and kills my laziness, naughty naughty.

Indeed, it might as well be that a type had a specialized fast 
implementation for seq-contains-p and doesn't have one for 
seq-contains-pred, and the switchover would tank the performance of its 
callers.

As long as the observable behavior stays the same (aside from 
performance), that seems fine. The type's author will need to implement 
seq-contains-pred. Hopefully such migrations will remain rare enough.

> Actually for this point to make sense, it doesn't even have to be
> a bizarre voodoo list.  The lazy list may just be expanded enough already
> for me to know in seq-contains-p that there's a given element
> there without having to call the generator for more list elements.
> 
> So seq-contains-p was indeed an optimization, in more or less
> the sense you described, which is now lost when you tried to optimize
> for bare Lisp sequences.

Right. The author will now need to optimize seq-contains-pred, getting 
even better performance than previously in return.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 15/11/2023 03:07, João Távora wrote:
> On Wed, Nov 15, 2023 at 12:46 AM Dmitry Gutov <dmitry@gutov.dev> wrote:
> 
>> Then it must have a seq-do implementation that yields :secret-voodoo one
>> or several times.
> 
> Not really.  Only if I use my seq in contexts where I need that
> generic, and being second argument to seq-difference doesn't count.
> So why should I bother implementing that when my code is working
> fine today?

Commentary at the top of seq.el says:

;; seq.el can be extended to support new type of sequences.  Here are
;; the generic functions that must be implemented by new seq types:
;; - `seq-elt'
;; - `seq-length'
;; - `seq-do'
;; - `seqp'
;; - `seq-subseq'
;; - `seq-into-sequence'
;; - `seq-copy'
;; - `seq-into'

> But say I did that seq-do, then what is the seq-contains-p generic good
> for then?  Why some many redundant generics that the user learns the
> hard way have to be consistent with each other?

So any new type gets a lot of functions implemented very easily, and 
could add specialized implementations for performance.

>> And then the seq-contains-pred implementation will do
>> right by it (the default one, not the specialization for lists).
>>
>>>> This is not documented,
>>>
>>> You don't say? :-)  i think we should get started on documenting all
>>> these ad-hoc rules (sorry but in this case they are quite literally
>>> ad-hoc).  Better than nothing, I guess, since as to actually enforcing
>>> them, I think that train has left the station a long time ago and is
>>> now in a voodoo swamp in monkey island.
>>
>> Not sure "enforcing them" is the right phrase: I wonder if there's ever
>> been a change proposed to seq.el which would break them.
> 
> I think it's pretty easy to do, especially if these optimziations
> want going to go all the way to cl-lib's easier-to-get optimal
> performance.  What about this?  Is this toy implementation of
> compressed space-saving lists legal?
> 
> (cl-defmethod seq-do (function (l (head :m6-sparse)))
>    (mapc (lambda (e)
>            (if (eq e :double-o-seven)
>                (progn (funcall function 0)
>                       (funcall function 0)
>                       (funcall function 7))
>              (funcall function e)))
>          (cdr l)))
> 
> (cl-defmethod seq-reverse ((l (head :m6-sparse)))
>    (let (res)
>      (seq-do (lambda (e) (push e res)) l)
>      res))
> 
> (seq-difference '(:m6-sparse 1 2 :double-o-seven 4) '(7)) ;; => (1 2 0 0 4)
> 
> Again, not implementing all the generics, just the ones I need
> to get my desired effect, which in this case is passing a
> custom sequence type as a first argument to seq-difference.

Very clever. I'm not thinking too well this time of night, but the first 
part of the reply in this email probably answers the question (there is 
a certain set of generics that must be implemeneted).

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

[replying to both subthreads here]

On Wed, Nov 15, 2023 at 1:28 AM Dmitry Gutov <dmitry@gutov.dev> wrote:

> > But say I did that seq-do, then what is the seq-contains-p generic good
> > for then?  Why some many redundant generics that the user learns the
> > hard way have to be consistent with each other?
> So any new type gets a lot of functions implemented very easily, and
> could add specialized implementations for performance.

Fair enough.  Except, as you noted, those optimizations can be
completely nullified overnight, when trying to optimize something
else.

> the switchover would tank the performance of its callers.
> As long as the observable behavior stays the same (aside from
> performance), that seems fine.

I don't think this is "fine" at all to tank user code performance,
not from any library, much less such a basic foundation library.

> > (seq-difference '(:m6-sparse 1 2 :double-o-seven 4) '(7)) ;; => (1 2 0 0 4)

> Very clever. I'm not thinking too well this time of night, but the first
> part of the reply in this email probably answers the question (there is
> a certain set of generics that must be implemeneted).

Alright, if you must have them, they're all very trivial and I don't
think it'll add much to your analysis.  Here's the full
compressed/sparse/lazy list impl:

(cl-defmethod seq-do (function (l (head :m6-sparse)))
  (mapc (lambda (e)
          (if (eq e :double-o-seven) (mapc function '(0 0 7))
            (funcall function e)))
        (cdr l)))

(cl-defmethod seq-reverse ((l (head :m6-sparse)))
  (let (res) (seq-do (lambda (e) (push e res)) l) res))

(cl-defmethod seq-elt ((l (head :m6-sparse)) n) (elt (nreverse
(seq-reverse l)) n))
(cl-defmethod seq-length ((l (head :m6-sparse))) (length (nreverse
(seq-reverse l))))
(cl-defmethod seqp ((_l (head :m6-sparse))) t)
(cl-defmethod seq-into-sequence ((l (head :m6-sparse))) (nreverse
(seq-reverse l)))
(cl-defmethod seq-subseq ((l (head :m6-sparse)) start &optional end)
  (cl-subseq (nreverse (seq-reverse l)) start end))
(cl-defmethod seq-copy ((l (head :m6-sparse))) (nreverse (seq-reverse l)))
(cl-defmethod seq-into ((l (head :m6-sparse)) type)
  (cl-call-next-method (nreverse (seq-reverse l)) type))

(defvar my-compressed-list '(:m6-sparse 1 2 :double-o-seven 4
:double-o-seven 9 10 11))

(seq-difference my-compressed-list '(7)) ;; => (1 2 0 0 4 0 0 9 10 11)
(seq-elt my-compressed-list 0)           ;; => 1
(seq-elt my-compressed-list 4)           ;; => 7
(seq-elt my-compressed-list 5)           ;; => 4
(seq-elt my-compressed-list 10)          ;; => 10
(seq-elt my-compressed-list 1)           ;; => 2
(seq-elt my-compressed-list 3)           ;; => 0
(seq-elt my-compressed-list 13)          ;; => nil
(seq-length my-compressed-list)          ;; => 12
(seqp my-compressed-list)                ;; => t
(seq-subseq my-compressed-list 2 5)      ;; => (0 0 7)
(seq-copy my-compressed-list)            ;; => (1 2 0 0 7 4 0 0 7 9 10 11)
(seq-into my-compressed-list 'vector)    ;; => [1 2 0 0 7 4 0 0 7 9 10 11]
(seq-into-sequence my-compressed-list)   ;; => (1 2 0 0 7 4 0 0 7 9 10 11)


;; non consing version
(cl-defmethod seq-elt ((l (head :m6-sparse)) n)
  (cl-loop for e in (cdr l)
           for diff = (- n i)
           while (cl-plusp diff)
           sum (if (eq e :double-o-seven) 3 1) into i
           finally return
           (cond ((eq e :double-o-seven) 0)
                 ((cl-minusp diff) (elt '(7 0 0) (- (1+ diff))))
                 ((zerop diff) e)
                 (t nil))))
;; non consing version
(cl-defmethod seq-length ((l (head :m6-sparse)))
  (cl-loop for e in (cdr l)
           sum (if (eq e :double-o-seven) 3 1)))

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 15/11/2023 14:14, João Távora wrote:
>>> But say I did that seq-do, then what is the seq-contains-p generic good
>>> for then?  Why some many redundant generics that the user learns the
>>> hard way have to be consistent with each other?
>> So any new type gets a lot of functions implemented very easily, and
>> could add specialized implementations for performance.
> Fair enough.  Except, as you noted, those optimizations can be
> completely nullified overnight, when trying to optimize something
> else.

I don't see where they would be "completely nullified". The gains seem 
consistent, even if they don't extend as much to certain scenarios, like 
custom test-fn. Even that one should be improved, though.

More importantly, I'm not seeing any meaningful regressions, in any 
related or different scenario.

>> the switchover would tank the performance of its callers.
>> As long as the observable behavior stays the same (aside from
>> performance), that seems fine.
> I don't think this is "fine" at all to tank user code performance,
> not from any library, much less such a basic foundation library.

If that is the only drawback we can find, it is an unavoidable one.

But that is the price one has to pay for correcting a design mistake. 
We're not going to do this every Tuesday.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Wed, Nov 15, 2023 at 2:05 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
>
> On 15/11/2023 14:14, João Távora wrote:
> >>> But say I did that seq-do, then what is the seq-contains-p generic good
> >>> for then?  Why some many redundant generics that the user learns the
> >>> hard way have to be consistent with each other?
> >> So any new type gets a lot of functions implemented very easily, and
> >> could add specialized implementations for performance.
> > Fair enough.  Except, as you noted, those optimizations can be
> > completely nullified overnight, when trying to optimize something
> > else.
>
> I don't see where they would be "completely nullified".

You wrote "tank" :-)  I was talking about those kinds of optimisations,
not the ones you're making.  What I meant is in order to optimize the
list/sequence/#equal/#eql/#eq case you tank the user's optimizations
to custom sequences.

> The gains seem
> consistent, even if they don't extend as much to certain scenarios, like
> custom test-fn.

In that particular corner, both your (the library dev's) optimisations
and the user optimisations are tanked, yes.

> Even that one should be improved, though.

The only way you are eventually going to get to equal performance
with cl-lib's fastest variants (but only the non-destructive, of course)
is if you start every seq.el entry point with sth like

   (if (sequence-p seq) (call-equivalent-cl-lib-version seq) ...)

But then I don't see how that fits in with the cl-defgeneric,
which does dispatching before that if.  The only way I see this
is to invent a new kind of specializer in cl-generic.el like
'definitely-not-sequence'.  There could be some other solution,
maybe Michael as a clue.

Anyway we do it, it's much easier than trying to wrangle entry-point
by entry-point by adding more generics and finding pitfalls like this
one about seq-contains-p.  And this I think, is what Michael is
suggesting: fair enough.

But of course we will be outlawing every extension like the :m6-sparse
example I gave.

And also, IMHO, we end up with a much poorer protocol in terms
of versatility (no find-by-key :key, no :from-end, no :start/:end).
But this part is not about performance, rather interface, and that's
not the topic of this subthread.

> If that is the only drawback we can find, it is an unavoidable one.

There is still the other example I gave you.  I think your only
way out is to outlaw my :m6-sparse extension.  You basically have
to outlaw everything (:around, :before, specializers) that
cl-defgeneric gives the user when it comes to the sequence types
for which you want to have acceptable performance.

And even outlaw more stuff.  For example, these generics even have
specializers on all mandatory arguments?

For example, why does seq-do have FUNCTION as a required argument???
It should be &optional or a &key, with a default value.  Which
cl-degeneric obviously supports. That way specializations on FUNCTION
become impossible,  or at least become much harder and there's less
risk of tanking user code.  Design mistake IMO.

> But that is the price one has to pay for correcting a design mistake.
> We're not going to do this every Tuesday.

Sure, but that price increases manyfold if we start suggesting
seq.el as a replacement for all your sequence processing needs.
We just shouldn't.

Why working on the :m6-sparse extension, I noticed Emacs becomes
noticeably slower, and I suspect that's because while I was editing,
all the seq  functions I was writing where being probed for
applicability, while core things like completion, buffer-switching
etc are calling seq-foo generics.

I find this performance aspect very bad.  Maybe it can be obviated,
but only if you drop the '(if (sequence-p seq)' bomb into seq.el
somehow.  I don't see how we can avoid that one.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

Sorry for all the typos, there's probably more but I found these:

> maybe Michael as a clue.
                ^^
"has"

> Why working on the :m6-sparse extension, I noticed Emacs becomes
  ^^^
"While"

> noticeably slower, and I suspect that's because while I was editing,
> all the seq  functions I was writing where being probed for
                                       ^^^^^
"were also"

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 15/11/2023 17:02, João Távora wrote:
> On Wed, Nov 15, 2023 at 2:05 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
>>
>> On 15/11/2023 14:14, João Távora wrote:
>>>>> But say I did that seq-do, then what is the seq-contains-p generic good
>>>>> for then?  Why some many redundant generics that the user learns the
>>>>> hard way have to be consistent with each other?
>>>> So any new type gets a lot of functions implemented very easily, and
>>>> could add specialized implementations for performance.
>>> Fair enough.  Except, as you noted, those optimizations can be
>>> completely nullified overnight, when trying to optimize something
>>> else.
>>
>> I don't see where they would be "completely nullified".
> 
> You wrote "tank" :-) I was talking about those kinds of optimisations,
> not the ones you're making.  What I meant is in order to optimize the
> list/sequence/#equal/#eql/#eq case you tank the user's optimizations
> to custom sequences.

Only until the author of the third-party data type updates their code to 
include the method seq-contains-pred.

And if that code is not inside Emacs, its time-to-market measures in 
days, not months or years.

>> Even that one should be improved, though.
> 
> The only way you are eventually going to get to equal performance
> with cl-lib's fastest variants (but only the non-destructive, of course)
> is if you start every seq.el entry point with sth like
> 
>     (if (sequence-p seq) (call-equivalent-cl-lib-version seq) ...)
> 
> But then I don't see how that fits in with the cl-defgeneric,
> which does dispatching before that if.  The only way I see this
> is to invent a new kind of specializer in cl-generic.el like
> 'definitely-not-sequence'.  There could be some other solution,
> maybe Michael as a clue.
> 
> Anyway we do it, it's much easier than trying to wrangle entry-point
> by entry-point by adding more generics and finding pitfalls like this
> one about seq-contains-p.  And this I think, is what Michael is
> suggesting: fair enough.

I don't immediately see that new generics are needed. But if someone 
finds a workable approach (with hard numbers), good.

> But of course we will be outlawing every extension like the :m6-sparse
> example I gave.
> 
> And also, IMHO, we end up with a much poorer protocol in terms
> of versatility (no find-by-key :key, no :from-end, no :start/:end).
> But this part is not about performance, rather interface, and that's
> not the topic of this subthread.

cl-lib is more flexible in one aspect (its additional keywords 
vocabulary which basically multiplies the provided interface by 100x), 
but it's more rigid in the data it works with.

>> If that is the only drawback we can find, it is an unavoidable one.
> 
> There is still the other example I gave you.  I think your only
> way out is to outlaw my :m6-sparse extension.  You basically have
> to outlaw everything (:around, :before, specializers) that
> cl-defgeneric gives the user when it comes to the sequence types
> for which you want to have acceptable performance.

Could you try to explain what I should find in the second example? What 
do you mean by "outlaw"? Does causing worse performance for a short time 
constitute "outlawing" something?

> And even outlaw more stuff.  For example, these generics even have
> specializers on all mandatory arguments?
> 
> For example, why does seq-do have FUNCTION as a required argument???

Because FUNCTION is applied to SEQUENCE?

> It should be &optional or a &key, with a default value.  Which
> cl-degeneric obviously supports. That way specializations on FUNCTION
> become impossible,  or at least become much harder and there's less
> risk of tanking user code.  Design mistake IMO.

I'm reasonably sure nobody expects function to be anything but a 
straight function (symbol or a lambda), because that's how 'seq-do' is 
used throughout the code.

>> But that is the price one has to pay for correcting a design mistake.
>> We're not going to do this every Tuesday.
> 
> Sure, but that price increases manyfold if we start suggesting
> seq.el as a replacement for all your sequence processing needs.

We can first fix the mistake and then go on to continue "suggesting it 
as a replacement". Or not.

I don't exactly see it that way, though. And you give an impression of 
arguing for the opposite: toward never using it at all.

> Why working on the :m6-sparse extension, I noticed Emacs becomes
> noticeably slower, and I suspect that's because while I was editing,
> all the seq  functions I was writing where being probed for
> applicability, while core things like completion, buffer-switching
> etc are calling seq-foo generics.

It could be helpful to do some profiling and see where the slowdown came 
from. Could it come exactly from the set operations?

> I find this performance aspect very bad.  Maybe it can be obviated,
> but only if you drop the '(if (sequence-p seq)' bomb into seq.el
> somehow.  I don't see how we can avoid that one.

I don't quite see the need. And it's unlikely to be of reliable help: my 
observations say that method dispatch simply becomes slower as soon as a 
generic function gets a second implementation. And that implementation 
might arrive from any third-party code.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Dmitry Gutov <dmitry@gutov.dev> writes:

> my observations say that method dispatch simply becomes slower as soon
> as a generic function gets a second implementation. And that
> implementation might arrive from any third-party code.

To me that observation makes a lot of sense, extrapolating from PCL. It
means that Emacs does an optimization that PCL also does. PCL being the
CLOS implementationof CMUCL/SBCL.

In the most general case, the function definition of a generic function
is a discriminating function which dispatches to effective method
functions based on actual argument types etc. Each effective method
function then calls the applicable method functions.

In the case that there is only 1 method, Emacs seems to arrange for that
method function to be used as function definition of the generic
function (I'm a bit simplifying here). So it avoids both using a
discrimating functions, and an effective method function.

In the case of more than one method, using a discrimating function
cannot possibly be avoided, but I'd expect that Emacs is still able to
avoid effective method functions. The emfs cannot possibly be avoided
when there is more than 1 applicable method.

Pretty impressive, actually. Hats off to Stefan!

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 15/11/2023 21:56, Gerd Möllmann wrote:
> In the case that there is only 1 method, Emacs seems to arrange for that
> method function to be used as function definition of the generic
> function (I'm a bit simplifying here). So it avoids both using a
> discrimating functions, and an effective method function.

That makes a lot of sense indeed: since the function slot can be 
modified directly (and wouldn't break any inline caches, for example), 
why not do that when there's only one definition.

The only downside is that the performance characteristics of callers can 
change unexpectedly as more definitions arrive (which the authors of the 
caller code might not have tested for).

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Wed, Nov 15, 2023 at 11:25 PM Dmitry Gutov <dmitry@gutov.dev> wrote:

> The only downside is that the performance characteristics of callers can
> change unexpectedly as more definitions arrive (which the authors of the
> caller code might not have tested for).

Yes, I think so too.  Expect more benchmarks soon.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

João Távora <joaotavora@gmail.com> writes:

> On Wed, Nov 15, 2023 at 11:25 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
>
>> The only downside is that the performance characteristics of callers can
>> change unexpectedly as more definitions arrive (which the authors of the
>> caller code might not have tested for).
>
> Yes, I think so too.  Expect more benchmarks soon.

It's probably not important, but since you mention benchmarks, let me
just quickly add that, in PCL, the way discriminating functions are
implemented, the algorithm for dispatching, can also vary, depending on
various things. I don't have the details in my head anymore.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 9237 bytes --]

Dmitry Gutov <dmitry@gutov.dev> writes:

> I don't immediately see that new generics are needed. But if someone
> finds a workable approach (with hard numbers), good.

See benchmarks.

>> But of course we will be outlawing every extension like the :m6-sparse
>> example I gave.
>> And also, IMHO, we end up with a much poorer protocol in terms
>> of versatility (no find-by-key :key, no :from-end, no :start/:end).
>> But this part is not about performance, rather interface, and that's
>> not the topic of this subthread.
>
> cl-lib is more flexible in one aspect (its additional keywords
> vocabulary which basically multiplies the provided interface by 100x),
> but it's more rigid in the data it works with.

Exactly.  It's the CL tradeoff, and it give you superior performance.

CL was never meant for custom sequences.  One should use seq.el for
that, and IMO, _only_ for that.

The seq.el tradeoff gives you inferior performance and poorer interface.

But it's not only a matter of performance.  Take the
'seq-remove-at-position' generic.  Presumably someone thought that
operationq was common enough to merit a separate entry point. In CL,
this very operation can be done (probably faster) with cl-remove using
keyword arguments.

But say you want to remove two elements at two consecutive positions?
Two calls with two slowdowns?  What about n positions?  cl-remove can do
all of that in the same call with :COUNT and :START for example.

If you want to cut that time down in seq.el you either have to introduce
'seq-remove-at-consecutive-positions' generic or pay what is probably a
tremedous price by using `seq-drop-while` which calls the seq-elt
generic in a hot loop.

And so, in general, the seq.el interface either bloats up, or is slow,
or is inherently inflexible.  I think seq.el can obviate any one of
these problems at best, but not two or three simultaneously.

cl-lib obviates all three of them at the cost of one drawback: custom
sequence support.

So in general, different tools, different jobs.

>>> If that is the only drawback we can find, it is an unavoidable one.

> Could you try to explain what I should find in the second example?
> What do you mean by "outlaw"? 

What I meant is that the only way to get the same performance out of
seq.el is to have early #'sequencep checks that bypass the generics
completely, and this makes custom sequences based on sequences
impossible.

> Does causing worse performance for a short time constitute "outlawing"
> something?

But you should also find in that "m6sparse" example that the logic is
broken -- not only in terms of performance -- until its author
implements seq-contains-pred.  So this is pure "Incompatible Lisp
changes" material (which I also think tanked performance should be btw.)

But in fact is is already broken by all the "list optimzations" in
seq.el. Optimizations, before yours, that caused whatever the contract
was to be violated.  To be able to use `seq-drop-while` for my m6sparse
sequence, I have to add implementations to all those generic entry
points, which is just akward.

>> And even outlaw more stuff.  For example, these generics even have
>> specializers on all mandatory arguments?
>> For example, why does seq-do have FUNCTION as a required argument???
>
> Because FUNCTION is applied to SEQUENCE?

>
>> It should be &optional or a &key, with a default value.  Which
>> cl-degeneric obviously supports. That way specializations on FUNCTION
>> become impossible,  or at least become much harder and there's less
>> risk of tanking user code.  Design mistake IMO.
>
> I'm reasonably sure nobody expects function to be anything but a
> straight function (symbol or a lambda), because that's how 'seq-do' is
> used throughout the code.

Yes, but putting as a required argument in the arglist means users can
specialize for it, and that isn't needed.  Not sure anyone does it or
even if that makes the generic even slower.

>>> But that is the price one has to pay for correcting a design mistake.
>>> We're not going to do this every Tuesday.
>> Sure, but that price increases manyfold if we start suggesting
>> seq.el as a replacement for all your sequence processing needs.
>
> We can first fix the mistake and then go on to continue "suggesting it
> as a replacement". Or not.
>
> I don't exactly see it that way, though. And you give an impression of
> arguing for the opposite: toward never using it at all.

Not at all.  Maybe you missed some of my previous messages.  I think
seq.el's support for polymorphic sequences, though a little bit flawed
in some respects, is very useful.  For example, I've been pondering
using it in eglot.el to try to speed up JSON parsing.  If some kind of
seq-plist-get and seq-plist-destructuring-bind can be designed, I might
be able to skip consing much of the useless elements of a gigantic JSON
blob and parse just the parts I need.  Of course, not easy, but I think
seq.el is the tool for that.

>> Why working on the :m6-sparse extension, I noticed Emacs becomes
>> noticeably slower, and I suspect that's because while I was editing,
>> all the seq  functions I was writing where being probed for
>> applicability, while core things like completion, buffer-switching
>> etc are calling seq-foo generics.
>
> It could be helpful to do some profiling and see where the slowdown
> came from. Could it come exactly from the set operations?

Not sure.  It might have come from tracing seq.el functions, for
example.  You might say that it's my fault I was tracing them, but
should I be punished in Emacs usability just for trying to use Emacs to
iteratively develop a seq.el extension?

Anyway tracing basic staples such as seq-do and seq-let gives some
insight as to where they are used and what shape of arguments they are
called with in your normal programming activities.  Small lists seem to
appear a lot more often.  But expect a massive slowdown while tracing:
even with modest seq.el usage in current core, these generics are called
a lot already.

>> I find this performance aspect very bad.  Maybe it can be obviated,
>> but only if you drop the '(if (sequence-p seq)' bomb into seq.el
>> somehow.  I don't see how we can avoid that one.
>
> I don't quite see the need. And it's unlikely to be of reliable help:
> my observations say that method dispatch simply becomes slower as soon
> as a generic function gets a second implementation. And that
> implementation might arrive from any third-party code.

Exactly.  The entry point generics probably can never be avoided.  I
think we agreed that noone -- user or library -- should add
implementations to them.  That's why I think not making them defuns was
another design mistake.  But other intermediary generics _can_ be
skipped and would bring a performance boost to seq.el.

Alright.  That all said, here's the latest results, which I gathered
using the attached sequence-benchmarks.el are also attached in
results.txt.

I gathered each set of timings by running these two things

   src/emacs -Q -nw sequence-benchmarks.el -f emacs-lisp-byte-compile-and-load

and then

  src/emacs -Q -nw -l m6sparse.el sequence-benchmarks.el -f emacs-lisp-byte-compile-and-load

And I also attach m6sparse.el.

The branch I used was feature/cl-lib-improvements where I also pushed
your seq-difference-3 patch.

This email is long enough, so take your conclusions.  I don't think any
results are exactly flattering to seq.el, especially -- but not only --
when you compare to the destructive versions of some utils that cl-lib
offers.

But this one stands out.  I hope you can read this macro more or less.

   (joaot/with-benchmark-group "destructuring"
     ((list1 . (make-list 3 0)))
     100000
     (joaot/bench (pcase-let ((`(,a ,b ,c) list1)) (+ a b c)))
     (joaot/bench (cl-destructuring-bind (a b c) list1 (+ a b c)))
     (joaot/bench (seq-let (a b c) list1 (+ a b c))))

Running in Emacs -Q:

  ("destructuring"
    (cl-destructuring-bind "FASTEST" 0.010702393 0 0.0)
    (pcase-let "1.3x SLOWER" 0.014360937999999998 0 0.0)
    (seq-let "3.5x (rel 2.6x) SLOWER" 0.03706726 0 0.0))

Where after loading m6sparse.el we go to this:

  (("destructuring" (cl-destructuring-bind "FASTEST" 0.010157632 0 0.0)
    (pcase-let "1.3x SLOWER" 0.013152518000000002 0 0.0)
    (seq-let "14.8x (rel 11.4x) SLOWER" 0.15057331499999999
      6 0.04785139399999849))

You may try to optimize this, but you'll probably have to introduce
yet another generic.

And this seq-let thing should also remind us that this should never only
be about "fast enough".  Eli was suggesting seq-let as an alternative to
pcase-let the other day.  Let-like forms do appear in tight loops (and
tight loops aplenty do exist), I expect a let-like form on a list to
expand to little more than some car, cadr, etc calls, not some immensely
slow generic call by comparison.

So, in summary.  YES to seq.el for custom sequences, we need more of
those (probably in core even) and NO to seq.el as a drop-in
general-purpose sequence processing library.

João


[-- Attachment #2: m6sparse.el --]
[-- Type: text/plain, Size: 3086 bytes --]

;;; m6-sparse -*- lexical-binding: t -*-
(cl-defmethod seq-do (function (l (head :m6-sparse)))
  (mapc (lambda (e)
          (if (eq e :double-o-seven) (mapc function '(0 0 7))
            (funcall function e)))
        (cdr l)))

(cl-defmethod seq-contains-p ((l (head :m6-sparse)) elt &optional test-fn)
  (cl-member elt (nreverse (seq-reverse l)) :test test-fn))

(cl-defmethod seq-contains-pred ((_l (head :m6-sparse)) &optional test-fn)
  (lambda (elt sequence)
    (cl-member elt (nreverse (seq-reverse sequence)) :test test-fn)))

(cl-defmethod seq-reverse ((l (head :m6-sparse)))
  (let (res) (seq-do (lambda (e) (push e res)) l) res))

(cl-defmethod seq-elt ((l (head :m6-sparse)) n) (elt (nreverse (seq-reverse l)) n))

(cl-defmethod seq-length ((l (head :m6-sparse))) (length (nreverse (seq-reverse l))))
(cl-defmethod seqp ((_l (head :m6-sparse))) t)
(cl-defmethod seq-into-sequence ((l (head :m6-sparse))) (nreverse (seq-reverse l)))
(cl-defmethod seq-subseq ((l (head :m6-sparse)) start &optional end) (cl-subseq (nreverse (seq-reverse l)) start end))
(cl-defmethod seq-copy ((l (head :m6-sparse))) (nreverse (seq-reverse l)))
(cl-defmethod seq-into ((l (head :m6-sparse)) type)
  (cl-call-next-method (nreverse (seq-reverse l)) type))
(cl-defmethod seq-map (function (l (head :m6-sparse)))
  (let ((res))
    (seq-do (lambda (e)
              (push (funcall function e) res))
            l)
    (nreverse res)))

(defvar my-compressed-list '(:m6-sparse 1 2 :double-o-seven 4
                                        :double-o-seven 9 10 11))

(seq-difference my-compressed-list '(7)) ;; => (1 2 0 0 4 0 0 9 10 11)
(seq-difference  '(42 42 42 0 0 7 42 42) my-compressed-list) ;; => (42 42 42 42 42)
(seq-difference-3  '(42 42 42 0 0 7 42 42) my-compressed-list) ;; => (42 42 42 42 42)
(seq-elt my-compressed-list 0)           ;; => 1
(seq-elt my-compressed-list 4)           ;; => 7
(seq-elt my-compressed-list 5)           ;; => 4
(seq-elt my-compressed-list 10)          ;; => 10
(seq-elt my-compressed-list 1)           ;; => 2
(seq-elt my-compressed-list 3)           ;; => 0
(seq-elt my-compressed-list 13)          ;; => nil
(seq-length my-compressed-list)          ;; => 12
(seqp my-compressed-list)                ;; => t
(seq-subseq my-compressed-list 2 5)      ;; => (0 0 7)
(seq-copy my-compressed-list)            ;; => (1 2 0 0 7 4 0 0 7 9 10 11)
(seq-into my-compressed-list 'vector)    ;; => [1 2 0 0 7 4 0 0 7 9 10 11]
(seq-into-sequence my-compressed-list)   ;; => (1 2 0 0 7 4 0 0 7 9 10 11)

;; non consing version
(cl-defmethod seq-elt ((l (head :m6-sparse)) n)
  (cl-loop for e in (cdr l)
           for diff = (- n i)
           while (cl-plusp diff)
           sum (if (eq e :double-o-seven) 3 1) into i
           finally return
           (cond ((eq e :double-o-seven) 0)
                 ((cl-minusp diff) (elt '(7 0 0) (- (1+ diff))))
                 ((zerop diff) e)
                 (t nil))))
;; non consing version
(cl-defmethod seq-length ((l (head :m6-sparse)))
  (cl-loop for e in (cdr l)
           sum (if (eq e :double-o-seven) 3 1)))

[-- Attachment #3: results.txt --]
[-- Type: text/plain, Size: 5906 bytes --]

BEFORE LOADING m6sparse

(("destructuring" (cl-destructuring-bind "FASTEST" 0.010702393 0 0.0)
  (pcase-let "1.3x SLOWER" 0.014360937999999998 0 0.0)
  (seq-let "3.5x (rel 2.6x) SLOWER" 0.03706726 0 0.0))
 ("\"some\" operation, small lists"
  (cl-some "FASTEST" 0.034695811 0 0.0)
  (seq-some "3.9x SLOWER" 0.135977155 10 0.0755767589999996))
 ("\"some\" operation, big lists"
  (cl-some "FASTEST" 0.29491865700000003 0 0.0)
  (seq-some "1.5x SLOWER" 0.44417032 0 0.0))
 ("set difference, small lists, custom pred"
  (cl-nset-difference "FASTEST" 0.19550686 15 0.11099579500000001)
  (cl-set-difference "5.1x SLOWER" 1.000326248 39 0.2885333050000005)
  (seq-difference-3 "13.2x (rel 2.6x) SLOWER" 2.575436995 192
		    1.4284019790000002)
  (seq-difference "14.4x (rel 1.1x) SLOWER" 2.8172440610000002 212
		  1.5752595139999999))
 ("set difference, big lists, custom pred"
  (cl-nset-difference "FASTEST" 0.011390392 0 0.0)
  (cl-set-difference "5.5x SLOWER" 0.062250377999999995 1
		     0.007866447000000054)
  (seq-difference-3 "15.9x (rel 2.9x) SLOWER" 0.18127607499999998 12
		    0.09376674499999993)
  (seq-difference "19.0x (rel 1.2x) SLOWER" 0.21642625100000001 15
		  0.1219803650000002))
 ("set difference, small lists, #'eql pred"
  (cl-nset-difference "FASTEST" 0.025416642 0 0.0)
  (cl-set-difference "10.6x SLOWER" 0.268309301 24 0.17514854700000004)
  (seq-difference-3 "19.3x (rel 1.8x) SLOWER" 0.491795223 47
		    0.3400537589999999)
  (seq-difference "101.8x (rel 5.3x) SLOWER" 2.5877423690000003 212
		  1.5652519980000001))
 ("set difference, big lists, #'eql pred"
  (cl-nset-difference "FASTEST" 0.001086087 0 0.0)
  (cl-set-difference "31.2x SLOWER" 0.033846905999999996 3
		     0.022589776999999867)
  (seq-difference-3 "38.0x (rel 1.2x) SLOWER" 0.041256314 3
		    0.024156587999999868)
  (seq-difference "363.2x (rel 9.6x) SLOWER" 0.39448685299999997 30
		  0.24322218400000017))
 ("set difference, small lists, #'equal pred"
  (joaot/handrolled-nset-difference "FASTEST" 0.015404288 0 0.0)
  (cl-nset-difference "3.9x SLOWER" 0.060033165 4 0.027628342)
  (dmitry/set-difference-nocons "7.8x (rel 2.0x) SLOWER" 0.120780652 0
				0.0)
  (cl-set-difference "23.6x (rel 3.0x) SLOWER" 0.362814037 28
		     0.201481361)
  (seq-difference-3 "36.5x (rel 1.6x) SLOWER" 0.5629666289999999 47
		    0.33870848)
  (seq-difference "48.9x (rel 1.3x) SLOWER" 0.753127336 68
		  0.47948227499999996))
 ("set difference, big lists, #'equal pred"
  (joaot/handrolled-nset-difference "FASTEST" 0.002013658 0 0.0)
  (cl-nset-difference "1.7x SLOWER" 0.0033535120000000003 0 0.0)
  (dmitry/set-difference-nocons "8.1x (rel 4.9x) SLOWER" 0.016377504 0
				0.0)
  (cl-set-difference "21.6x (rel 2.7x) SLOWER" 0.043477418 3
		     0.021467216000000004)
  (seq-difference-3 "25.2x (rel 1.2x) SLOWER" 0.050788065 3
		    0.022892674000000002)
  (seq-difference "39.0x (rel 1.5x) SLOWER" 0.07861690199999999 6
		  0.04483213700000001)))

AFTER loading it

(("destructuring" (cl-destructuring-bind "FASTEST" 0.010157632 0 0.0)
  (pcase-let "1.3x SLOWER" 0.013152518000000002 0 0.0)
  (seq-let "14.8x (rel 11.4x) SLOWER" 0.15057331499999999
    6 0.04785139399999849))
 ("\"some\" operation, small lists"
  (cl-some "FASTEST" 0.030627893999999996 0 0.0)
  (seq-some "5.1x SLOWER" 0.156528706 10 0.08218940500000116))
 ("\"some\" operation, big lists"
  (cl-some "FASTEST" 0.253589884 0 0.0)
  (seq-some "1.7x SLOWER" 0.42605076199999997 0 0.0))
 ("set difference, small lists, custom pred"
  (cl-nset-difference "FASTEST" 0.201229483 15 0.11990391199999983)
  (cl-set-difference "5.0x SLOWER" 1.002034638 39 0.31673883500000066)
  (seq-difference-3 "15.6x (rel 3.1x) SLOWER" 3.136544874 194
		    1.7686539850000003)
  (seq-difference "18.9x (rel 1.2x) SLOWER" 3.795912411 260
		  2.140581182000002))
 ("set difference, big lists, custom pred"
  (cl-nset-difference "FASTEST" 0.011600632 0 0.0)
  (cl-set-difference "5.4x SLOWER" 0.062588482 1 0.008457714000000394)
  (seq-difference-3 "18.1x (rel 3.4x) SLOWER" 0.21003662699999998 12
		    0.10525734000000142)
  (seq-difference "25.5x (rel 1.4x) SLOWER" 0.29535328699999996 19
		  0.16559472099999972))
 ("set difference, small lists, #'eql pred"
  (cl-nset-difference "FASTEST" 0.023517136 0 0.0)
  (cl-set-difference "11.5x SLOWER" 0.271050456 24 0.18550450299999888)
  (seq-difference-3 "24.1x (rel 2.1x) SLOWER" 0.567758752 50
		    0.39358252499999935)
  (seq-difference "148.5x (rel 6.1x) SLOWER" 3.491236757 260
		  2.072315871999999))
 ("set difference, big lists, #'eql pred"
  (cl-nset-difference "FASTEST" 0.0010804710000000002 0 0.0)
  (cl-set-difference "33.0x SLOWER" 0.035650159 3 0.024719922999999255)
  (seq-difference-3 "39.0x (rel 1.2x) SLOWER" 0.042105057 3
		    0.025471970999999982)
  (seq-difference "520.3x (rel 13.4x) SLOWER" 0.5621415289999999 39
		  0.34187674300000026))
 ("set difference, small lists, #'equal pred"
  (joaot/handrolled-nset-difference "FASTEST" 0.014848083 0 0.0)
  (cl-nset-difference "4.4x SLOWER" 0.064930546 4 0.03191499300000089)
  (dmitry/set-difference-nocons "8.2x (rel 1.9x) SLOWER"
				0.12118037799999999 0 0.0)
  (cl-set-difference "25.4x (rel 3.1x) SLOWER" 0.377759915 28
		     0.22155792999999946)
  (seq-difference-3 "42.5x (rel 1.7x) SLOWER" 0.630364124 50
		    0.3947469649999995)
  (seq-difference "93.6x (rel 2.2x) SLOWER" 1.390502927 116
		  0.9019907159999985))
 ("set difference, big lists, #'equal pred"
  (cl-nset-difference "FASTEST" 0.002104955 0 0.0)
  (joaot/handrolled-nset-difference "1.2x SLOWER" 0.002503964 0 0.0)
  (dmitry/set-difference-nocons "7.9x (rel 6.6x) SLOWER"
				0.016558740000000002 0 0.0)
  (cl-set-difference "22.2x (rel 2.8x) SLOWER" 0.046786407 3
		     0.02427869799999982)
  (seq-difference-3 "26.0x (rel 1.2x) SLOWER" 0.054804263 3
		    0.02563775999999862)
  (seq-difference "93.5x (rel 3.6x) SLOWER" 0.196867405 15
		  0.12648469499999848)))




[-- Attachment #4: sequence-benchmarks.el --]
[-- Type: text/plain, Size: 6036 bytes --]

;;; -*- lexical-binding: t -*-

(defun joaot/handrolled-nset-difference (list1 list2)
   (if (or (null list1) (null list2)) list1
     (let ((res nil))
       (while (consp list1)
         (if (funcall #'member (car list1) list2)
             (setf list1 (cdr list1))
           (cl-shiftf list1 (cdr list1) res list1)))
       res)))

(defun dmitry/set-difference-nocons (list1 list2)
   (let (ref)
     (while (member (car list1) list2)
       (setq list1 (cdr list1)))
     (setq ref list1)
     (while ref
       (if (member (cadr ref) list2)
           (setcdr ref (cddr ref))
         (setq ref (cdr ref))))
     list1))

(defvar joaot/bench-group-name nil)
(defvar joaot/setup-form nil)
(defvar joaot/timings-alist nil)
(defvar joaot/repetitions nil)

(defmacro joaot/with-benchmark-group (bench-name
                                      setup-form
                                      repetitions
                                      &rest body)
  (declare (indent 1))
  (let ((f `(alist-get ,bench-name joaot/timings-alist nil nil #'equal)))
    `(let ((joaot/bench-group-name ,bench-name)
           (joaot/setup-form ',setup-form)
           (joaot/repetitions ,repetitions))
       (setf ,f (list))
       ,@body
       (cl-loop with sorted = (setf ,f (cl-sort ,f #'< :key #'cadr))
                with fastest = (cadr (car sorted))
                for i from 0
                for res in sorted
                for prev-time = time
                for (_n time . more) = res
                do (setcdr res
                           (cl-list* (if (eq time fastest) "FASTEST"
                                       (format "%2.1fx %sSLOWER"
                                               (/ time fastest)
                                               (if (= i 1) ""
                                                 (format "(rel %2.1fx) "
                                                         (/ time prev-time)))))
                                     time
                                     more)))
       joaot/timings-alist)))

(defmacro joaot/bench (form)
  `(cl-progv
       (mapcar #'car joaot/setup-form)
       (mapcar #'eval (mapcar #'cdr joaot/setup-form))
     (let ((res
            (benchmark-call (prog1
                                (,(if (native-comp-available-p)
                                      'native-compile
                                    'byte-compile)
                                 '(lambda () ,form))
                              (garbage-collect))
                            joaot/repetitions))
           (group-name joaot/bench-group-name)
           (bench-name ',(car form)))
       (push (cons bench-name res)
             (alist-get group-name joaot/timings-alist nil nil #'equal)))))

(joaot/with-benchmark-group "set difference, big lists, #'equal pred"
  ((list1 . (all-completions "" obarray))
   (list2 . (make-list 12 "shooveedoowaa")))
  10
  (joaot/bench (joaot/handrolled-nset-difference list1 list2))
  (joaot/bench (dmitry/set-difference-nocons list1 list2))
  (joaot/bench (cl-nset-difference list1 list2 :test #'equal))
  (joaot/bench (cl-set-difference list1 list2 :test #'equal))
  (joaot/bench (seq-difference list1 list2))
  (joaot/bench (seq-difference-3 list1 list2)))

(joaot/with-benchmark-group "set difference, small lists, #'equal pred"
  ((list1 . (make-list 12 "bla"))
   (list2 . (make-list 12 "shooveedoowaa")))
  100000
  (joaot/bench (joaot/handrolled-nset-difference list1 list2))
  (joaot/bench (dmitry/set-difference-nocons list1 list2))
  (joaot/bench (cl-nset-difference list1 list2 :test #'equal))
  (joaot/bench (cl-set-difference list1 list2 :test #'equal))
  (joaot/bench (seq-difference list1 list2))
  (joaot/bench (seq-difference-3 list1 list2)))

(joaot/with-benchmark-group "set difference, big lists, #'eql pred"
  ((list1 . (all-completions "" obarray))
   (list2 . (make-list 12 "shooveedoowaa")))
  10
  (joaot/bench (cl-nset-difference list1 list2))
  (joaot/bench (cl-set-difference list1 list2))
  (joaot/bench (seq-difference list1 list2 #'eql))
  (joaot/bench (seq-difference-3 list1 list2 #'eql)))

(joaot/with-benchmark-group "set difference, small lists, #'eql pred"
  ((list1 . (make-list 12 "bla"))
   (list2 . (make-list 12 "shooveedoowaa")))
  100000
  (joaot/bench (cl-nset-difference list1 list2))
  (joaot/bench (cl-set-difference list1 list2))
  (joaot/bench (seq-difference list1 list2 #'eql))
  (joaot/bench (seq-difference-3 list1 list2 #'eql)))

(joaot/with-benchmark-group "set difference, big lists, custom pred"
  ((list1 . (all-completions "" obarray))
   (list2 . (make-list 12 "shooveedoowaa")))
  5
  (byte-compile (defun myequal (a b) (equal a b)))
  (joaot/bench (cl-nset-difference list1 list2 :test #'myequal))
  (joaot/bench (cl-set-difference list1 list2 :test #'myequal))
  (joaot/bench (seq-difference-3 list1 list2 #'myequal))
  (joaot/bench (seq-difference list1 list2 #'myequal)))

(joaot/with-benchmark-group "set difference, small lists, custom pred"
  ((list1 . (make-list 12 "bla"))
   (list2 . (make-list 12 "shooveedoowaa")))
  100000
  (byte-compile (defun myequal (a b) (equal a b)))
  (joaot/bench (cl-nset-difference list1 list2 :test #'myequal))
  (joaot/bench (cl-set-difference list1 list2 :test #'myequal))
  (joaot/bench (seq-difference-3 list1 list2 #'myequal))
  (joaot/bench (seq-difference list1 list2 #'myequal)))

(joaot/with-benchmark-group "\"some\" operation, big lists"
  ((list1 . (make-list 100000 nil)))
  100
  (joaot/bench (cl-some #'identity list1))
  (joaot/bench (seq-some #'identity list1)))

(joaot/with-benchmark-group "\"some\" operation, small lists"
  ((list1 . (make-list 10 nil)))
  100000
  (joaot/bench (cl-some #'identity list1))
  (joaot/bench (seq-some #'identity list1)))

(joaot/with-benchmark-group "destructuring"
  ((list1 . (make-list 3 0)))
  100000
  (joaot/bench (pcase-let ((`(,a ,b ,c) list1)) (+ a b c)))
  (joaot/bench (cl-destructuring-bind (a b c) list1 (+ a b c)))
  (joaot/bench (seq-let (a b c) list1 (+ a b c))))

Re: What's missing in ELisp that makes people want to use cl-lib?

[Michael Heerdegen]

João Távora <joaotavora@gmail.com> writes:

> But this one stands out.  I hope you can read this macro more or less.
>
>    (joaot/with-benchmark-group "destructuring"
>      ((list1 . (make-list 3 0)))
>      100000
>      (joaot/bench (pcase-let ((`(,a ,b ,c) list1)) (+ a b c)))
>      (joaot/bench (cl-destructuring-bind (a b c) list1 (+ a b c)))
>      (joaot/bench (seq-let (a b c) list1 (+ a b c))))
>
> Running in Emacs -Q:
>
>   ("destructuring"
>     (cl-destructuring-bind "FASTEST" 0.010702393 0 0.0)
>     (pcase-let "1.3x SLOWER" 0.014360937999999998 0 0.0)
>     (seq-let "3.5x (rel 2.6x) SLOWER" 0.03706726 0 0.0))
>
> Where after loading m6sparse.el we go to this:
>
>   (("destructuring" (cl-destructuring-bind "FASTEST" 0.010157632 0 0.0)
>     (pcase-let "1.3x SLOWER" 0.013152518000000002 0 0.0)
>     (seq-let "14.8x (rel 11.4x) SLOWER" 0.15057331499999999
>       6 0.04785139399999849))

Isn't seq-let directly translated into a pcase-let call (it's a
simplistic macro)?  How is this possible?

Michael.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Thu, Nov 16, 2023 at 4:02 PM Michael Heerdegen
<michael_heerdegen@web.de> wrote:
>
> João Távora <joaotavora@gmail.com> writes:

> >
> >   (("destructuring" (cl-destructuring-bind "FASTEST" 0.010157632 0 0.0)
> >     (pcase-let "1.3x SLOWER" 0.013152518000000002 0 0.0)
> >     (seq-let "14.8x (rel 11.4x) SLOWER" 0.15057331499999999
> >       6 0.04785139399999849))
>
> Isn't seq-let directly translated into a pcase-let call (it's a
> simplistic macro)?  How is this possible?

Yes, but look closely at the expansion ;-)

And you can also think.  How can that pcase-let call know at at
runtime whether it's dealing with a bare list, a vector/string,
or a custom sequence.  Only through runtime indirection
obviously!  Based on what?  Generic functions no less, seq-elt
in this case, which become slower when added to.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

João Távora <joaotavora@gmail.com> writes:

> AFTER loading it
>
> ...

The magnitude of the difference when additional methods are defined I
find surprising. I take it as a strong indicator that cl-generic.el
indeed works completely differently than PCL. Assuming it is not a bug
af some sort. That the manual nowhere uses the term "discriminating
function" might also be a hint.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Thu, Nov 16, 2023 at 5:44 PM Gerd Möllmann <gerd.moellmann@gmail.com> wrote:
>
> João Távora <joaotavora@gmail.com> writes:
>
> > AFTER loading it
> >
> > ...
>
> The magnitude of the difference when additional methods are defined I
> find surprising.

Speaking of that, be sure to reproduce my results.   These are results
from an Emacs -Q run, and fortunately they are consistent across runs
in the same machine.  In my normal work session, I sometimes get much
more dramatic slowdowns with these very same benchmarks, and I don't
know why, maybe some garbage collection effect?

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

João Távora <joaotavora@gmail.com> writes:

> On Thu, Nov 16, 2023 at 5:44 PM Gerd Möllmann <gerd.moellmann@gmail.com> wrote:
>>
>> João Távora <joaotavora@gmail.com> writes:
>>
>> > AFTER loading it
>> >
>> > ...
>>
>> The magnitude of the difference when additional methods are defined I
>> find surprising.
>
> Speaking of that, be sure to reproduce my results.   These are results
> from an Emacs -Q run, and fortunately they are consistent across runs
> in the same machine.  In my normal work session, I sometimes get much
> more dramatic slowdowns with these very same benchmarks, and I don't
> know why, maybe some garbage collection effect?

Yeah, please see my other mail, I've looked into cl-generics a bit. Not
that I could explain anything now :-)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> João Távora <joaotavora@gmail.com> writes:
>
>> On Thu, Nov 16, 2023 at 5:44 PM Gerd Möllmann <gerd.moellmann@gmail.com> wrote:
>>>
>>> João Távora <joaotavora@gmail.com> writes:
>>>
>>> > AFTER loading it
>>> >
>>> > ...
>>>
>>> The magnitude of the difference when additional methods are defined I
>>> find surprising.
>>
>> Speaking of that, be sure to reproduce my results.   These are results
>> from an Emacs -Q run, and fortunately they are consistent across runs
>> in the same machine.  In my normal work session, I sometimes get much
>> more dramatic slowdowns with these very same benchmarks, and I don't
>> know why, maybe some garbage collection effect?
>
> Yeah, please see my other mail, I've looked into cl-generics a bit. Not
> that I could explain anything now :-)

Hm, that's actually not true.

One thing that one can conclude, I think, is that the very first
execution of a generic function with a given argument type combination
(i.e. an effective method) does not count, so to say, because that
computes the effective memthod and memoizes it.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> João Távora <joaotavora@gmail.com> writes:
>
>> AFTER loading it
>>
>> ...
>
> The magnitude of the difference when additional methods are defined I
> find surprising. I take it as a strong indicator that cl-generic.el
> indeed works completely differently than PCL. Assuming it is not a bug
> af some sort. That the manual nowhere uses the term "discriminating
> function" might also be a hint.

Looking a bit at cl-generics.el, at least the discriminating functions
part is indeed like nothing in PCL. And discriminating functions are not
in the manual, because there are none.

Disclaimer: I do not know cl-generics.el, and I've just looked enough to
get an impression.

So: I would describe the difference between PCL and cl-generic something
like static vs. dynamic, perhaps.

PCL goes to great lengths computing applicable methods etc. constructing
an as optimal as possible discriminating function. Once this has all
been done, nothing more has to be done at runtime. When methods are
changed, added etc. the whole thing is done from scratch again.

Cl-generics, in contrast, I'd say relies on runtime computation of
applicable methods and so on, plus memoization. If someone wants to see
what a generic function looks like, see cl-generic-define ->
cl--generic-make-function -> cl--generic-make-next-function ->
cl--generic-get-dispatcher. There we see

      (funcall
       cl--generic-compiler
       `(lambda (generic dispatches-left methods)
          (let ((method-cache (make-hash-table :test #'eql)))
            (lambda (,@fixedargs &rest args)
              (let ,bindings
                (apply (with-memoization
                           (gethash ,tag-exp method-cache)
                         (cl--generic-cache-miss
                          generic ',dispatch-arg dispatches-left methods
                          ,(if (cdr typescodes)
                               `(append ,@typescodes) (car typescodes))))
                       ,@fixedargs args)))))))))

The hash-table is a cache, the inner lambda is the function definition
of the gf, the apply is the execution of an effective method, AFAIU. If
we hit an argument combination not in the cache, we compute applicable
methods at runtime, I believe.

A bit strange is that cl--generic-next-function seems to be called
recursively in the process, which I think could create another such
hash-table. Or I'm reading that simply wrong, as I mentioned I just
wanted to see if cl-generic is so different, so I didn't spend much time
on this.

The question how that leads to such-and-such performance effects I can't
answer. I haven't seen such an implementation before.

And I'm not saying it's bad! There are very very very (did I say very
enough?) good reasons to try and avoid the incredible complexity of
something like PCL.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

Thanks, Gerd.

I think Stefan is not subscribed to this list anymore, so I'll Cc him in 
case he has something to add (or considers worth improving).

But cl-defmethod with just one method resulting in a simple function 
definition in an experimental fact (evaluate (cl-defmethod abc () 345); 
then (cl-defmethod abc () 345) returns (lambda nil (progn 345))), so 
that must be faster, without any computation of applicable methods.

The comment above the code you quoted mentions "generic functions with a 
single method"; maybe it was written before the above optimization was made.

On 16/11/2023 22:09, Gerd Möllmann wrote:
> Gerd Möllmann<gerd.moellmann@gmail.com>  writes:
> 
>> João Távora<joaotavora@gmail.com>  writes:
>>
>>> AFTER loading it
>>>
>>> ...
>> The magnitude of the difference when additional methods are defined I
>> find surprising. I take it as a strong indicator that cl-generic.el
>> indeed works completely differently than PCL. Assuming it is not a bug
>> af some sort. That the manual nowhere uses the term "discriminating
>> function" might also be a hint.
> Looking a bit at cl-generics.el, at least the discriminating functions
> part is indeed like nothing in PCL. And discriminating functions are not
> in the manual, because there are none.
> 
> Disclaimer: I do not know cl-generics.el, and I've just looked enough to
> get an impression.
> 
> So: I would describe the difference between PCL and cl-generic something
> like static vs. dynamic, perhaps.
> 
> PCL goes to great lengths computing applicable methods etc. constructing
> an as optimal as possible discriminating function. Once this has all
> been done, nothing more has to be done at runtime. When methods are
> changed, added etc. the whole thing is done from scratch again.
> 
> Cl-generics, in contrast, I'd say relies on runtime computation of
> applicable methods and so on, plus memoization. If someone wants to see
> what a generic function looks like, see cl-generic-define ->
> cl--generic-make-function -> cl--generic-make-next-function ->
> cl--generic-get-dispatcher. There we see
> 
>        (funcall
>         cl--generic-compiler
>         `(lambda (generic dispatches-left methods)
>            (let ((method-cache (make-hash-table :test #'eql)))
>              (lambda (,@fixedargs &rest args)
>                (let ,bindings
>                  (apply (with-memoization
>                             (gethash ,tag-exp method-cache)
>                           (cl--generic-cache-miss
>                            generic ',dispatch-arg dispatches-left methods
>                            ,(if (cdr typescodes)
>                                 `(append ,@typescodes) (car typescodes))))
>                         ,@fixedargs args)))))))))
> 
> The hash-table is a cache, the inner lambda is the function definition
> of the gf, the apply is the execution of an effective method, AFAIU. If
> we hit an argument combination not in the cache, we compute applicable
> methods at runtime, I believe.
> 
> A bit strange is that cl--generic-next-function seems to be called
> recursively in the process, which I think could create another such
> hash-table. Or I'm reading that simply wrong, as I mentioned I just
> wanted to see if cl-generic is so different, so I didn't spend much time
> on this.
> 
> The question how that leads to such-and-such performance effects I can't
> answer. I haven't seen such an implementation before.
> 
> And I'm not saying it's bad! There are very very very (did I say very
> enough?) good reasons to try and avoid the incredible complexity of
> something like PCL.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Stefan Monnier]

> But cl-defmethod with just one method resulting in a simple function
>  definition in an experimental fact (evaluate (cl-defmethod abc () 345);
>  then (cl-defmethod abc () 345) returns (lambda nil (progn 345))), so that
> must be faster, without any computation of applicable methods.

"one method" is necessary but not sufficient for this optimization.
This one method has to have no specializer (aka only the `t` specializer).

> The comment above the code you quoted mentions "generic functions with
> a single method"; maybe it was written before the above optimization
> was made.

No, that comment refers to the case where there's a single method but
with a non-t specializer, so we can't just always call that one method
without first testing that the arg matches the specializer.


        Stefan

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 17/11/2023 04:44, Stefan Monnier wrote:
>> But cl-defmethod with just one method resulting in a simple function
>>   definition in an experimental fact (evaluate (cl-defmethod abc () 345);
>>   then (cl-defmethod abc () 345) returns (lambda nil (progn 345))), so that
>> must be faster, without any computation of applicable methods.
> "one method" is necessary but not sufficient for this optimization.
> This one method has to have no specializer (aka only the `t` specializer).
> 
>> The comment above the code you quoted mentions "generic functions with
>> a single method"; maybe it was written before the above optimization
>> was made.
> No, that comment refers to the case where there's a single method but
> with a non-t specializer, so we can't just always call that one method
> without first testing that the arg matches the specializer.

Makes sense. Thanks!

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> Date: Fri, 17 Nov 2023 04:47:41 +0200
> Cc: Gerd Möllmann <gerd.moellmann@gmail.com>,
>  João Távora <joaotavora@gmail.com>,
>  Eli Zaretskii <eliz@gnu.org>, michael_heerdegen@web.de, emacs-devel@gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 17/11/2023 04:44, Stefan Monnier wrote:
> >> But cl-defmethod with just one method resulting in a simple function
> >>   definition in an experimental fact (evaluate (cl-defmethod abc () 345);
> >>   then (cl-defmethod abc () 345) returns (lambda nil (progn 345))), so that
> >> must be faster, without any computation of applicable methods.
> > "one method" is necessary but not sufficient for this optimization.
> > This one method has to have no specializer (aka only the `t` specializer).
> > 
> >> The comment above the code you quoted mentions "generic functions with
> >> a single method"; maybe it was written before the above optimization
> >> was made.
> > No, that comment refers to the case where there's a single method but
> > with a non-t specializer, so we can't just always call that one method
> > without first testing that the arg matches the specializer.
> 
> Makes sense. Thanks!

If this warrants some improvements of the commentary, please feel free
to install such improvements.  TIA.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Dmitry Gutov <dmitry@gutov.dev> writes:

> I think Stefan is not subscribed to this list anymore, so I'll Cc him
> in case he has something to add (or considers worth improving).

Oops, I didn't realize that, sorry. Hi Stefan :-).

> But cl-defmethod with just one method resulting in a simple function
> definition in an experimental fact (evaluate (cl-defmethod abc ()
> 345); then (cl-defmethod abc () 345) returns (lambda nil (progn
> 345))), so that must be faster, without any computation of applicable
> methods.
>
> The comment above the code you quoted mentions "generic functions with
> a single method"; maybe it was written before the above optimization
> was made.

That might be a slight misunderstanding. I was not wondering why there
is a performance difference between the 1 method case and the rest. I
was wondering why it is so large, or can be so large, which I didn't
expect from a PCL-like implementation. And the reason for that was my
false assumption that it is PCL-like. Sorry for leading you and Joao on
wrong tracks by extrapolating from PCL, BTW.

Anyway - as I already mentioned I have absolutely no intuition about the
performance characteristics of cl-generic's dispatch mechanism.

Maybe, Stefan, you can help with that? Possibly in the context of
Dmitry's and Joao's case of using seq.el's generic function interface,
how do I say it, to replace the sequences part of cl-lib in Emacs?

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: João Távora <joaotavora@gmail.com>,  Stefan Monnier
>  <monnier@IRO.UMontreal.CA>,  Eli Zaretskii <eliz@gnu.org>,
>   michael_heerdegen@web.de,  emacs-devel@gnu.org
> Date: Fri, 17 Nov 2023 07:08:19 +0100
> 
> Dmitry Gutov <dmitry@gutov.dev> writes:
> 
> > I think Stefan is not subscribed to this list anymore, so I'll Cc him
> > in case he has something to add (or considers worth improving).
> 
> Oops, I didn't realize that, sorry. Hi Stefan :-).

You didn't have to: Dmitry misremembered -- Stefan _is_ subscribed to
emacs-devel.  It is bug-gnu-emacs that Stefan isn't subscribed to.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Stefan Monnier]

> Anyway - as I already mentioned I have absolutely no intuition about the
> performance characteristics of cl-generic's dispatch mechanism.

I don't have a good understanding of it either.
I did investigate a performance issue in `cl-print.el` where it seemed
clear that a significant part of the overhead comes from memory
allocations (and associated GC load) coming from `&rest`.

For this reason, I've been toying (in my mind only, so far) with how we
could have something like `&rest` but which doesn't allocate memory
from the heap for the specific sub-case of

    (lambda (.. &rest args)
      ...
      (apply ... args)
      ...)


-- Stefan

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Anyway - as I already mentioned I have absolutely no intuition about the
>> performance characteristics of cl-generic's dispatch mechanism.
>
> I don't have a good understanding of it either.

Thanks, Stefan. Then I guess no-one knows.

And of course we're not a millimeter closer to answering the question
why seq.el should replace cl-lib.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Augusto Stoffel]

On Sat, 18 Nov 2023 at 06:58, Gerd Möllmann wrote:

> And of course we're not a millimeter closer to answering the question
> why seq.el should replace cl-lib.

I think the original question was, why do you need an extra library of
basic list processing functions in a LISt Processing language...  Also,
assuming performance is indeed crucial, shouldn't they be implemented in
C?

The vocabulary missing from the built-ins is pretty small.  The
following are the fundamental ones IMO:

- filter aka remove-if-not
- some

(reduce is also academically fundamental, but I guess dolist is the more
idiomatic replacement.)

The following I would count as nice-to-haves:

- drop (already exists under a funny name)
- drop-while (basically like `member' with a different TESTFN)
- take-while
- perhaps find and position (those are small variations on `some')

The rest of seq I would count as "sporadically handy" (seq-partition,
seq-mapn, etc) or "pretty much unnecessary" (seq-difference,
seq-intersection, etc).

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
> A bit strange is that cl--generic-next-function seems to be called
> recursively in the process, which I think could create another such
> hash-table. Or I'm reading that simply wrong, as I mentioned I just
> wanted to see if cl-generic is so different, so I didn't spend much time
> on this.

Perhaps someone could verify if I'm reading this wrong?

Start in cl--generic-get-dispatcher again:

      (funcall
       cl--generic-compiler
       `(lambda (generic dispatches-left methods)
          (let ((method-cache (make-hash-table :test #'eql)))
            (lambda (,@fixedargs &rest args)
              (let ,bindings
                (apply (with-memoization
                           (gethash ,tag-exp method-cache)
                         (cl--generic-cache-miss
                          generic ',dispatch-arg dispatches-left methods
                          ,(if (cdr typescodes)
                               `(append ,@typescodes) (car typescodes))))
                       ,@fixedargs args)))))))))

The hash table is the cache, and so on, as before.

We have a cache miss, so -> cl--generic-cache-miss ->
cl--generic-make-next-function -> cl--generic-get-dispatcher, and we
procuce a function like the above, with a a method-cache of its own?

I suspect that I'm taking a wrong turn here, somewhere.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 16/11/2023 16:30, João Távora wrote:

> But it's not only a matter of performance.  Take the
> 'seq-remove-at-position' generic.  Presumably someone thought that
> operationq was common enough to merit a separate entry point. In CL,
> this very operation can be done (probably faster) with cl-remove using
> keyword arguments.

Probably. Like I said, it looks more like Clojure's stdlib, but due to 
the absence of efficient "persistent data structures" the performance 
profile is different.

>>>> If that is the only drawback we can find, it is an unavoidable one.
> 
>> Could you try to explain what I should find in the second example?
>> What do you mean by "outlaw"?
> 
> What I meant is that the only way to get the same performance out of
> seq.el is to have early #'sequencep checks that bypass the generics
> completely, and this makes custom sequences based on sequences
> impossible.

If we optimize for short lists -- maybe. But for longer lists the 
dynamic dispatch (if performed once) shouldn't be a problem.

>> Does causing worse performance for a short time constitute "outlawing"
>> something?
> 
> But you should also find in that "m6sparse" example that the logic is
> broken -- not only in terms of performance -- until its author
> implements seq-contains-pred.  So this is pure "Incompatible Lisp
> changes" material (which I also think tanked performance should be btw.)
> 
> But in fact is is already broken by all the "list optimzations" in
> seq.el. Optimizations, before yours, that caused whatever the contract
> was to be violated.  To be able to use `seq-drop-while` for my m6sparse
> sequence, I have to add implementations to all those generic entry
> points, which is just akward.

Were those the 'list' type specializations?

>>> And even outlaw more stuff.  For example, these generics even have
>>> specializers on all mandatory arguments?
>>> For example, why does seq-do have FUNCTION as a required argument???
>>
>> Because FUNCTION is applied to SEQUENCE?
> 
>>
>>> It should be &optional or a &key, with a default value.  Which
>>> cl-degeneric obviously supports. That way specializations on FUNCTION
>>> become impossible,  or at least become much harder and there's less
>>> risk of tanking user code.  Design mistake IMO.
>>
>> I'm reasonably sure nobody expects function to be anything but a
>> straight function (symbol or a lambda), because that's how 'seq-do' is
>> used throughout the code.
> 
> Yes, but putting as a required argument in the arglist means users can
> specialize for it, and that isn't needed.  Not sure anyone does it or
> even if that makes the generic even slower.

And what would happen if FUNCTION were optional and omitted by the caller?

>>>> But that is the price one has to pay for correcting a design mistake.
>>>> We're not going to do this every Tuesday.
>>> Sure, but that price increases manyfold if we start suggesting
>>> seq.el as a replacement for all your sequence processing needs.
>>
>> We can first fix the mistake and then go on to continue "suggesting it
>> as a replacement". Or not.
>>
>> I don't exactly see it that way, though. And you give an impression of
>> arguing for the opposite: toward never using it at all.
> 
> Not at all.  Maybe you missed some of my previous messages.  I think
> seq.el's support for polymorphic sequences, though a little bit flawed
> in some respects, is very useful.  For example, I've been pondering
> using it in eglot.el to try to speed up JSON parsing.  If some kind of
> seq-plist-get and seq-plist-destructuring-bind can be designed, I might
> be able to skip consing much of the useless elements of a gigantic JSON
> blob and parse just the parts I need.  Of course, not easy, but I think
> seq.el is the tool for that.

plist like Elisp plist? It's difficult to write a type predicate for.

>>> Why working on the :m6-sparse extension, I noticed Emacs becomes
>>> noticeably slower, and I suspect that's because while I was editing,
>>> all the seq  functions I was writing where being probed for
>>> applicability, while core things like completion, buffer-switching
>>> etc are calling seq-foo generics.
>>
>> It could be helpful to do some profiling and see where the slowdown
>> came from. Could it come exactly from the set operations?
> 
> Not sure.  It might have come from tracing seq.el functions, for
> example.  You might say that it's my fault I was tracing them, but
> should I be punished in Emacs usability just for trying to use Emacs to
> iteratively develop a seq.el extension?

I'm not sure I understand. If you added tracing to 'car', it would also 
slow Emacs down, wouldn't it? Is that a problem with the design of 'car'?

> Anyway tracing basic staples such as seq-do and seq-let gives some
> insight as to where they are used and what shape of arguments they are
> called with in your normal programming activities.  Small lists seem to
> appear a lot more often.  But expect a massive slowdown while tracing:
> even with modest seq.el usage in current core, these generics are called
> a lot already.

That both sounds like a compliment and a pressing motivation to iron out 
the most apparent performance pitfalls.

>>> I find this performance aspect very bad.  Maybe it can be obviated,
>>> but only if you drop the '(if (sequence-p seq)' bomb into seq.el
>>> somehow.  I don't see how we can avoid that one.
>>
>> I don't quite see the need. And it's unlikely to be of reliable help:
>> my observations say that method dispatch simply becomes slower as soon
>> as a generic function gets a second implementation. And that
>> implementation might arrive from any third-party code.
> 
> Exactly.  The entry point generics probably can never be avoided.  I
> think we agreed that noone -- user or library -- should add
> implementations to them.

Um, no. I'm saying that seq.el function should be written in a way that 
causes the dynamic dispatch to happen only once (or perhaps a few 
times), as opposed to doing it N times (for every element of a sequence) 
or more.

seq-let might suffer for a similar problem.

Having overrides for entry points, OTOH, should be fine enough, and 
whatever added cost the dispatch itself incurs, it still happens once, 
and can be made up for by the more efficient specialized implementation 
of the function's logic.

> That's why I think not making them defuns was
> another design mistake.  But other intermediary generics _can_ be
> skipped and would bring a performance boost to seq.el.

Other intermediate generics, in general, can be skipped by defining 
different methods at earlier points.

> Alright.  That all said, here's the latest results, which I gathered
> using the attached sequence-benchmarks.el are also attached in
> results.txt.
> 
> I gathered each set of timings by running these two things
> 
>     src/emacs -Q -nw sequence-benchmarks.el -f emacs-lisp-byte-compile-and-load

First of all, when I try to run the above command (on your branch, with 
your attachments from the last email), it ends with:

Compiling file 
/home/dgutov/examples/cl-lib-vs-seq/sequence-benchmarks.el at Fri Nov 17 
03:57:31 2023
sequence-benchmarks.el:73:2: Error: Wrong type argument: listp, more

So before I proceed further, could you make sure that you ran the tests 
with these exact files? Superficially, it looks like just (require 
'cl-lib) is missing, but maybe something else inside was also different?

I also tried to produce some pretty-printed comparisons, but the above 
command (even with "fixed" require statement) doesn't produce anything 
to *Messages* or stdout when ran with --batch.

I can evaluate individual joaot/with-benchmark-group forms, and they 
print -- how'd you call it -- minified Lisp data to messages, but it's 
not formatted the same way as in your results.txt, nor is it easy to get 
out of the -nw session because the clipboard is naturally not shared. Is 
-nw needed? I expected it'd be used to print something to stdout.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 1477 bytes --]

On 17/11/2023 04:09, Dmitry Gutov wrote:
> I can evaluate individual joaot/with-benchmark-group forms, and they 
> print -- how'd you call it -- minified Lisp data to messages, but it's 
> not formatted the same way as in your results.txt, nor is it easy to get 
> out of the -nw session because the clipboard is naturally not shared. Is 
> -nw needed? I expected it'd be used to print something to stdout.

OK, I used just that, as well as 'benchmark-run-compiled' inside 
'joaot/bench' instead of the complex expression which chose whether to 
compile a form.

Here are some results.

0) "mine" - just running the tests on your branch.

1) "mine-proper" - is me splitting of the typecheck into a separate 
method. Not an optimization, just better code.

2) "mine-proper+astoff" - including Augusto's optimization as well. 
Notably, seq-some gets ahead on big lists by about 1.2x. Not sure how 
reliable this number is, though.

3) "astoff" - just Augusto's patch. In some cases it's faster than 2 
because of no extra dispatch (or multimorphic methods, I guess), but the 
order-of-magnitude difference is still there in many set-difference 
examples.

Loading m6sparse.el (using (load (expand-file-name "../m6sparse.el" 
load-file-name)) at the top) didn't seem to affect the results of (2) 
too much, though it brought -- predictably -- the performance of 
seq-difference (the original version) further down. Though the results 
fluctuate a lot, so I might have missed some.

[-- Attachment #2: results-astoff.txt --]
[-- Type: text/plain, Size: 2278 bytes --]

(("destructuring"
  (cl-destructuring-bind "FASTEST" 0.008716133000000001 0 0.0)
  (pcase-let "1.2x SLOWER" 0.010243486 0 0.0)
  (seq-let "3.1x (rel 2.7x) SLOWER" 0.027214243 0 0.0))
 ("\"some\" operation, small lists"
  (seq-some "FASTEST" 0.02428174 0 0.0)
  (cl-some "1.0x SLOWER" 0.0244231 0 0.0))
 ("\"some\" operation, big lists"
  (seq-some "FASTEST" 0.191672023 0 0.0)
  (cl-some "1.0x SLOWER" 0.197579632 0 0.0))
 ("set difference, small lists, custom pred"
  (cl-nset-difference "FASTEST" 0.102954774 5 0.03422299700000003)
  (cl-set-difference "6.3x SLOWER" 0.650314872 14 0.08941152499999994)
  (seq-difference "7.9x (rel 1.2x) SLOWER" 0.8126014229999999 22 0.13833590900000003))
 ("set difference, big lists, custom pred"
  (cl-nset-difference "FASTEST" 0.009105017 0 0.0)
  (cl-set-difference "4.9x SLOWER" 0.044808576999999995 0 0.0)
  (seq-difference "6.3x (rel 1.3x) SLOWER" 0.057444879000000004 1 0.006419620000000015))
 ("set difference, small lists, #'eql pred"
  (cl-nset-difference "FASTEST" 0.019827222 0 0.0)
  (cl-set-difference "6.4x SLOWER" 0.12714725799999999 9 0.05405195400000001)
  (seq-difference "31.7x (rel 4.9x) SLOWER" 0.62896921 21 0.130310427))
 ("set difference, big lists, #'eql pred"
  (cl-nset-difference "FASTEST" 0.000826303 0 0.0)
  (cl-set-difference "18.8x SLOWER" 0.015559618999999998 1 0.0064666000000000445)
  (seq-difference "98.1x (rel 5.2x) SLOWER" 0.081062122 1 0.0074823319999999804))
 ("set difference, small lists, #'equal pred"
  (joaot/handrolled-nset-difference "FASTEST" 0.011843948 0 0.0)
  (cl-nset-difference "3.3x SLOWER" 0.039604176 2 0.013306504999999996)
  (dmitry/set-difference-nocons "8.0x (rel 2.4x) SLOWER" 0.09519276600000001 0 0.0)
  (cl-set-difference "15.4x (rel 1.9x) SLOWER" 0.182937625 10 0.061539529999999995)
  (seq-difference "57.1x (rel 3.7x) SLOWER" 0.676095982 21 0.129525192))
 ("set difference, big lists, #'equal pred"
  (cl-nset-difference "FASTEST" 0.0030955889999999997 0 0.0)
  (joaot/handrolled-nset-difference "2.1x SLOWER" 0.006550018 0 0.0)
  (dmitry/set-difference-nocons "4.9x (rel 2.3x) SLOWER" 0.015023027000000001 0 0.0)
  (cl-set-difference "8.2x (rel 1.7x) SLOWER" 0.025435355 1 0.007049931999999995)
  (seq-difference "28.2x (rel 3.4x) SLOWER" 0.087276368 1 0.006623073)))

[-- Attachment #3: results-mine.txt --]
[-- Type: text/plain, Size: 2772 bytes --]

(("destructuring" (cl-destructuring-bind "FASTEST" 0.007508548 0 0.0)
  (pcase-let "1.3x SLOWER" 0.010040264 0 0.0)
  (seq-let "3.3x (rel 2.5x) SLOWER" 0.025065294000000002 0 0.0)
 ("\"some\" operation, small lists"
  (cl-some "FASTEST" 0.023633029 0 0.0)
  (seq-some "2.2x SLOWER" 0.052465596 1 0.007674551000000029)
 ("\"some\" operation, big lists"
  (cl-some "FASTEST" 0.195993925 0 0.0)
  (seq-some "1.6x SLOWER" 0.315841508 0 0.0)
 ("set difference, small lists, custom pred"
  (cl-nset-difference "FASTEST" 0.100909502 5 0.03441776400000007)
  (cl-set-difference "6.4x SLOWER" 0.6473928059999999 14 0.090176123)
  (seq-difference-3 "13.6x (rel 2.1x) SLOWER" 1.3704889370000002 68 0.42959640599999993)
  (seq-difference "14.1x (rel 1.0x) SLOWER" 1.42464227 75 0.4613546180000001))
 ("set difference, big lists, custom pred"
  (cl-nset-difference "FASTEST" 0.008583963 0 0.0)
  (cl-set-difference "6.7x SLOWER" 0.057137517 1 0.007286566000000105)
  (seq-difference-3 "10.3x (rel 1.6x) SLOWER" 0.088811108 3 0.02027387699999994)
  (seq-difference "10.9x (rel 1.1x) SLOWER" 0.09363935100000001 3 0.020642897000000104))
 ("set difference, small lists, #'eql pred"
  (cl-nset-difference "FASTEST" 0.019733991 0 0.0)
  (cl-set-difference "6.2x SLOWER" 0.122059607 8 0.04862236800000003)
  (seq-difference-3 "11.5x (rel 1.9x) SLOWER" 0.22615767 17 0.10414085099999992)
  (seq-difference "63.1x (rel 5.5x) SLOWER" 1.2446914809999998 75 0.44716219399999996))
 ("set difference, big lists, #'eql pred"
  (cl-nset-difference "FASTEST" 0.000827584 0 0.0)
  (cl-set-difference "18.8x SLOWER" 0.015517249 1 0.006894151000000015)
  (seq-difference-3 "23.9x (rel 1.3x) SLOWER" 0.019756616 1 0.006745060999999997)
  (seq-difference "200.8x (rel 8.4x) SLOWER" 0.16621615199999998 7 0.04840139199999999))
 ("set difference, small lists, #'equal pred"
  (joaot/handrolled-nset-difference "FASTEST" 0.01179042 0 0.0)
  (cl-nset-difference "2.8x SLOWER" 0.032970384 1 0.006517913)
  (dmitry/set-difference-nocons "7.9x (rel 2.8x) SLOWER" 0.09332390900000001 0 0.0)
  (cl-set-difference "15.8x (rel 2.0x) SLOWER" 0.18612888100000002 10 0.061874161)
  (seq-difference-3 "23.5x (rel 1.5x) SLOWER" 0.276987626 17 0.10811353499999998)
  (seq-difference "32.0x (rel 1.4x) SLOWER" 0.376752902 25 0.152489869))
 ("set difference, big lists, #'equal pred"
  (joaot/handrolled-nset-difference "FASTEST" 0.003132761 0 0.0)
  (cl-nset-difference "1.2x SLOWER" 0.003833 0 0.0)
  (dmitry/set-difference-nocons "5.6x (rel 4.6x) SLOWER" 0.017451134 0 0.0)
  (cl-set-difference "8.3x (rel 1.5x) SLOWER" 0.025889626 1 0.006706274999999998)
  (seq-difference-3 "9.2x (rel 1.1x) SLOWER" 0.028935888 1 0.006479378000000001)
  (seq-difference "11.4x (rel 1.2x) SLOWER" 0.035722551 1 0.0070541059999999975))
 )

[-- Attachment #4: results-mine-proper.txt --]
[-- Type: text/plain, Size: 2817 bytes --]

(("destructuring"
  (cl-destructuring-bind "FASTEST" 0.008374391 0 0.0)
  (pcase-let "1.3x SLOWER" 0.0104752 0 0.0)
  (seq-let "3.1x (rel 2.5x) SLOWER" 0.026114335 0 0.0))
 ("\"some\" operation, small lists"
  (cl-some "FASTEST" 0.023994479 0 0.0)
  (seq-some "2.9x SLOWER" 0.07007991599999999 3 0.02121758500000004))
 ("\"some\" operation, big lists"
  (cl-some "FASTEST" 0.20129617100000002 0 0.0)
  (seq-some "1.7x SLOWER" 0.350107408 0 0.0))
 ("set difference, small lists, custom pred"
  (cl-nset-difference "FASTEST" 0.103420572 5 0.03438092000000026)
  (cl-set-difference "6.6x SLOWER" 0.677639723 17 0.11333318999999986)
  (seq-difference "15.5x (rel 2.4x) SLOWER" 1.6034870689999998 91 0.6029158240000001)
  (seq-difference-3 "15.6x (rel 1.0x) SLOWER" 1.610416625 88 0.6408485689999999))
 ("set difference, big lists, custom pred"
  (cl-nset-difference "FASTEST" 0.008586518999999999 0 0.0)
  (cl-set-difference "5.7x SLOWER" 0.049010411999999996 0 0.0)
  (seq-difference-3 "12.0x (rel 2.1x) SLOWER" 0.103116948 4 0.028720732999999887)
  (seq-difference "12.4x (rel 1.0x) SLOWER" 0.106261259 4 0.029437464999999996))
 ("set difference, small lists, #'eql pred"
  (cl-nset-difference "FASTEST" 0.019229662 0 0.0)
  (cl-set-difference "7.6x SLOWER" 0.145943298 11 0.07134210400000007)
  (seq-difference-3 "16.9x (rel 2.2x) SLOWER" 0.32409309399999997 27 0.17366687899999977)
  (seq-difference "74.4x (rel 4.4x) SLOWER" 1.430963487 90 0.610517504))
 ("set difference, big lists, #'eql pred"
  (cl-nset-difference "FASTEST" 0.0008175820000000001 0 0.0)
  (cl-set-difference "18.7x SLOWER" 0.01528772 1 0.0060815740000000895)
  (seq-difference-3 "24.4x (rel 1.3x) SLOWER" 0.019946676 1 0.006836014000000001)
  (seq-difference "208.7x (rel 8.6x) SLOWER" 0.170651892 7 0.05150274999999993))
 ("set difference, small lists, #'equal pred"
  (joaot/handrolled-nset-difference "FASTEST" 0.013188440999999999 0 0.0)
  (cl-nset-difference "3.0x SLOWER" 0.039793513 2 0.013435085999999999)
  (dmitry/set-difference-nocons "7.2x (rel 2.4x) SLOWER" 0.09443695199999999 0 0.0)
  (cl-set-difference "16.1x (rel 2.2x) SLOWER" 0.21211317799999999 12 0.08097277)
  (seq-difference-3 "29.4x (rel 1.8x) SLOWER" 0.38735803599999996 28 0.18473822899999998)
  (seq-difference "32.3x (rel 1.1x) SLOWER" 0.426180881 29 0.19561955699999997))
 ("set difference, big lists, #'equal pred"
  (cl-nset-difference "FASTEST" 0.002728576 0 0.0)
  (joaot/handrolled-nset-difference "5.0x SLOWER" 0.013517241 1 0.008937456)
  (dmitry/set-difference-nocons "8.7x (rel 1.8x) SLOWER" 0.023723545999999998 0 0.0)
  (seq-difference-3 "10.4x (rel 1.2x) SLOWER" 0.028304957 1 0.006144501000000004)
  (cl-set-difference "10.4x (rel 1.0x) SLOWER" 0.028483389 1 0.007728865)
  (seq-difference "15.9x (rel 1.5x) SLOWER" 0.043508337999999994 2 0.014429581)))

[-- Attachment #5: results-mine-proper+astoff.txt --]
[-- Type: text/plain, Size: 2829 bytes --]

(("destructuring"
  (cl-destructuring-bind "FASTEST" 0.008805193 0 0.0)
  (pcase-let "1.2x SLOWER" 0.010528567 0 0.0)
  (seq-let "3.0x (rel 2.5x) SLOWER" 0.026212175 0 0.0))
 ("\"some\" operation, small lists"
  (cl-some "FASTEST" 0.024282236999999998 0 0.0)
  (seq-some "1.1x SLOWER" 0.025573062 0 0.0))
 ("\"some\" operation, big lists"
  (seq-some "FASTEST" 0.197265253 0 0.0)
  (cl-some "1.2x SLOWER" 0.23777484799999998 0 0.0))
 ("set difference, small lists, custom pred"
  (cl-nset-difference "FASTEST" 0.108415098 5 0.03888447500000003)
  (cl-set-difference "6.3x SLOWER" 0.681953283 17 0.10882586699999997)
  (seq-difference "8.2x (rel 1.3x) SLOWER" 0.889311564 26 0.1774222139999999)
  (seq-difference-3 "8.8x (rel 1.1x) SLOWER" 0.958568638 33 0.2262132990000001))
 ("set difference, big lists, custom pred"
  (cl-nset-difference "FASTEST" 0.008485322 0 0.0)
  (cl-set-difference "6.8x SLOWER" 0.057843906 1 0.008032880999999992)
  (seq-difference-3 "7.0x (rel 1.0x) SLOWER" 0.059558123 1 0.006597587000000016)
  (seq-difference "7.3x (rel 1.0x) SLOWER" 0.061867936 1 0.007283522000000042))
 ("set difference, small lists, #'eql pred"
  (cl-nset-difference "FASTEST" 0.019067578999999998 0 0.0)
  (cl-set-difference "6.9x SLOWER" 0.131699003 10 0.05825310500000003)
  (seq-difference-3 "16.2x (rel 2.3x) SLOWER" 0.308024268 27 0.15978121099999998)
  (seq-difference "35.4x (rel 2.2x) SLOWER" 0.6750600449999999 26 0.15592904399999996))
 ("set difference, big lists, #'eql pred"
  (cl-nset-difference "FASTEST" 0.000829391 0 0.0)
  (cl-set-difference "17.9x SLOWER" 0.014861217 1 0.00603653799999998)
  (seq-difference-3 "23.6x (rel 1.3x) SLOWER" 0.019564613999999998 1 0.00658552200000001)
  (seq-difference "106.0x (rel 4.5x) SLOWER" 0.087925567 2 0.012859830999999988))
 ("set difference, small lists, #'equal pred"
  (joaot/handrolled-nset-difference "FASTEST" 0.012052723000000001 0 0.0)
  (cl-nset-difference "3.2x SLOWER" 0.038597212000000006 2 0.012354418999999998)
  (dmitry/set-difference-nocons "7.7x (rel 2.4x) SLOWER" 0.09330716600000001 0 0.0)
  (cl-set-difference "16.0x (rel 2.1x) SLOWER" 0.192478013 12 0.069325267)
  (seq-difference "28.6x (rel 1.8x) SLOWER" 0.344323734 26 0.152115906)
  (seq-difference-3 "30.6x (rel 1.1x) SLOWER" 0.369063511 27 0.16940377700000003))
 ("set difference, big lists, #'equal pred"
  (cl-nset-difference "FASTEST" 0.0027295080000000003 0 0.0)
  (dmitry/set-difference-nocons "5.2x SLOWER" 0.014315372 0 0.0)
  (joaot/handrolled-nset-difference "6.1x (rel 1.2x) SLOWER" 0.016597400000000002 1 0.010671791)
  (cl-set-difference "8.8x (rel 1.5x) SLOWER" 0.024141194999999997 1 0.0063431120000000014)
  (seq-difference-3 "10.2x (rel 1.1x) SLOWER" 0.027760892 1 0.006199146000000003)
  (seq-difference "14.0x (rel 1.4x) SLOWER" 0.038263214000000004 2 0.012877529999999998)))

Re: What's missing in ELisp that makes people want to use cl-lib?

[Michael Heerdegen]

João Távora <joaotavora@gmail.com> writes:

> The only way you are eventually going to get to equal performance
> with cl-lib's fastest variants (but only the non-destructive, of course)
> is if you start every seq.el entry point with sth like
>
>    (if (sequence-p seq) (call-equivalent-cl-lib-version seq) ...)
>
> But then I don't see how that fits in with the cl-defgeneric,
> which does dispatching before that if.  The only way I see this
> is to invent a new kind of specializer in cl-generic.el like
> 'definitely-not-sequence'.  There could be some other solution,
> maybe Michael as a clue.

I'm a bit lost I must admit.  I don't understand the use of your
`sequence-p' test (all we use seq.el for are sequences...?).

Sorry if I missed something obvious,

Michael.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1339 bytes --]

On Thu, Nov 16, 2023 at 1:23 PM Michael Heerdegen <michael_heerdegen@web.de>
wrote:

> João Távora <joaotavora@gmail.com> writes:
>
> > The only way you are eventually going to get to equal performance
> > with cl-lib's fastest variants (but only the non-destructive, of course)
> > is if you start every seq.el entry point with sth like
> >
> >    (if (sequence-p seq) (call-equivalent-cl-lib-version seq) ...)
> >
> > But then I don't see how that fits in with the cl-defgeneric,
> > which does dispatching before that if.  The only way I see this
> > is to invent a new kind of specializer in cl-generic.el like
> > 'definitely-not-sequence'.  There could be some other solution,
> > maybe Michael as a clue.
>
> I'm a bit lost I must admit.  I don't understand the use of your
> `sequence-p' test (all we use seq.el for are sequences...?).


Yes, but custom sequences don't respond t to sequencep.  It is a
C built-in that only returns t for lists and arrays and strings.
And those are the types that the sequence functions in cl-lib.el
works with, and why it is faster than seq.el, because it only
dispatches between them with ifs.  If seq.el wants to be just
as fast, it must to that much simpler kind of dispatch, and
this has of course has implications for its generic function
interface.

João

[-- Attachment #2: Type: text/html, Size: 1927 bytes --]

Re: What's missing in ELisp that makes people want to use cl-lib?

[Michael Heerdegen]

João Távora <joaotavora@gmail.com> writes:

>>  I'm a bit lost I must admit.  I don't understand the use of your
>>  `sequence-p' test (all we use seq.el for are sequences...?).
>
> Yes, but custom sequences don't respond t to sequencep.

Ok, I got confused by the hyphen you had added to the predicate name.

If adding new methods slows down the generic functions significantly
this is something to be investigated.  Maybe there is a problem, AFAIR
parts of cl-generic are relatively new.

Michael.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Thu, Nov 16, 2023 at 3:24 PM Michael Heerdegen
<michael_heerdegen@web.de> wrote:
>
> João Távora <joaotavora@gmail.com> writes:
>
> >>  I'm a bit lost I must admit.  I don't understand the use of your
> >>  `sequence-p' test (all we use seq.el for are sequences...?).
> >
> > Yes, but custom sequences don't respond t to sequencep.
>
> Ok, I got confused by the hyphen you had added to the predicate name.

Yeah, my bad, sorry.

> If adding new methods slows down the generic functions significantly
> this is something to be investigated.  Maybe there is a problem, AFAIR
> parts of cl-generic are relatively new.

I think cl-generic is already pretty awesomely optimized.  But maybe
who knows?

Re: What's missing in ELisp that makes people want to use cl-lib?

[Michael Heerdegen]

João Távora <joaotavora@gmail.com> writes:

> This is all interesting, until one ponders what happens if an existing
> seq.el user somewhere has:
>
> (cl-defmethod seq-contains-p ((seq my-voodoo-seq)
>                               (elt (eql :secret-voodoo)) &optional _tesfn)
>   (invoke-voodoo-priests seq))
>
> making use of seq.el's support for abstract polymorphic sequences.
>
> With seq.el 2.24 a seq-difference operation would consider this user's
> method, with seq.el 2.24.dmitry (i.e. your fast seq-difference-3) it 
> simply won't.  This user's code is clearly broken.

It is debatable whether the result of such a seq difference is even well
defined with the information you provided to Elisp:

  - the difference of two voodoo sequences should be a voodoo sequence
    so it will contain :secret-voodoo
  - since both voodoo sequences contain :secret-voodoo, the result
    should _not_ contain it.

AFAIR it was not among the design ideas of seq.el to provide an
interface for tweaking seq.el implementations for special cases.

That you can define new implementations doesn't mean you have to, like
the fact that you can advice any functions doesn't mean it's a good idea
to do that.

The functions are generics mainly for the purpose to allow the support
new sequence types.  When you want to add a new type, it's expected
that you provide a consistent set of implementations for all generics.

In some cases, as you mentioned, some arguments will probably never be
specialized on (sorry if this is not correct English...).  This could be
avoided but one of the main goals was to keep the code simple and
developers didn't expect people to shoot themselves in the foot on
purpose.

But yes, a bit more documentation about how this lib is supposed to be
used would definitely not harm.  I can't even tell whether it was a good
idea to use generic functions but the author and Stefan M. did so I
think that this had been thought about.

Michael.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> João Távora <joaotavora@gmail.com> writes:
>
>> This is all interesting, until one ponders what happens if an existing
>> seq.el user somewhere has:
>>
>> (cl-defmethod seq-contains-p ((seq my-voodoo-seq)
>>                               (elt (eql :secret-voodoo)) &optional _tesfn)
>>   (invoke-voodoo-priests seq))
>>
>> making use of seq.el's support for abstract polymorphic sequences.
>>
>> With seq.el 2.24 a seq-difference operation would consider this user's
>> method, with seq.el 2.24.dmitry (i.e. your fast seq-difference-3) it 
>> simply won't.  This user's code is clearly broken.
>
> It is debatable whether the result of such a seq difference is even well
> defined with the information you provided to Elisp:

Yes, of course: that's why I called it "voodoo".  It was just for the
sake of argument, and made Dmitry aware of a serious performance bug
that happens in much more legitimate* cases. The fix to that bug, while
it exists to a great extent (it's seq-difference-3 in Dmitry's patch or
branch feature/cl-lib-improvements), requires incompatible lisp changes,
hence proving my point that a bunch of generics thrown together does not
a good library make.

Anyway, read my latest mail to Dmitry.

João

[*] For some meaning of "legitimate", of course, but I think sparse
sequences are a good legitimate, for example.  seq.el's documentation is
where?  I don't see it.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Michael Heerdegen]

João Távora <joaotavora@gmail.com> writes:

> seq.el's documentation is where?  I don't see it.

It's currently the file header and the rest of the file (the code).

Michael.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Thu, Nov 16, 2023 at 3:16 PM Michael Heerdegen
<michael_heerdegen@web.de> wrote:
>
> João Távora <joaotavora@gmail.com> writes:
>
> > seq.el's documentation is where?  I don't see it.
>
> It's currently the file header and the rest of the file (the code).

Precisely, as I learned the hard way, much of that code already
means that the custom sequence idea expressed very sparsely
at the top isn't quite so.  IMO it'd be very nice to have a contract
we can stick to for a long time.  At least for such a fundamental
thing in a Lisp language.  It'll be a lot of work to go
through the corner cases and write that contract, but better
late than never, I guess.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Date: Thu, 16 Nov 2023 16:16:34 +0100
> 
> João Távora <joaotavora@gmail.com> writes:
> 
> > seq.el's documentation is where?  I don't see it.
> 
> It's currently the file header and the rest of the file (the code).

?? There's a whole section in the ELisp manual called "Sequence
Functions", which documents the seq-* functions.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> > > seq.el's documentation is where?  I don't see it.
> > 
> > It's currently the file header and the rest of the file (the code).
>
> ?? There's a whole section in the ELisp manual called "Sequence
> Functions", which documents the seq-* functions.

Argh, I should have checked before writing that.  I'm sorry for my
nonsense reply.

Michael.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1657 bytes --]

On Thu, Nov 16, 2023 at 4:42 PM Eli Zaretskii <eliz@gnu.org> wrote:

> ?? There's a whole section in the ELisp manual called "Sequence
> Functions", which documents the seq-* functions.

Yes, but from a custom sequence implementors perspective,
the "best" documentation is still that bit in the header,
which describes "you must implement these to get a working
custom sequence".

So while the Elisp manual addresses custom sequences
(very briefly -- it just tells the user to look up generic
functions) it spends most time describing the generic
functions one by one, and while that's important, it's
just not enough.

It doesn't talk about the guarantees that the framework
offers with respect to when they are called, or if they
are called at all. For example, while I didn't read the
whole thing, I don't think it is stated that custom sequences
based on Lisp lists need many more gf implementations
than others, else they simply won't work (because of
list-specific shortcutting optimizations only found
when reading the seq.el code).  Nor is it stated that, for
any representation,  :around methods are probably a bad
idea. Nor is it stated that if you just implement the bare
minimum (which is described in seq.el's header) you will
likely get very poor performance.

Such a manual is where one would find sentences like
"to make a working custom sequence that is accepted by
any seq-* call, the user must add implementations to
the following N generics.  To get good performance
you must also add implementations to these other M
generics".  (Dmitry's proposed seq-contains-pred would
be in this set).

João

[-- Attachment #2: Type: text/html, Size: 2616 bytes --]

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Thu, 16 Nov 2023 21:58:51 +0000
> Cc: Michael Heerdegen <michael_heerdegen@web.de>, emacs-devel <emacs-devel@gnu.org>
> 
> On Thu, Nov 16, 2023 at 4:42 PM Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > ?? There's a whole section in the ELisp manual called "Sequence
> > Functions", which documents the seq-* functions.
> 
> Yes, but from a custom sequence implementors perspective, 
> the "best" documentation is still that bit in the header, 
> which describes "you must implement these to get a working 
> custom sequence".
> 
> So while the Elisp manual addresses custom sequences
> (very briefly -- it just tells the user to look up generic
> functions) it spends most time describing the generic 
> functions one by one, and while that's important, it's 
> just not enough.

Feel free to suggest improvements and additions to what we have.  Such
improvements are always welcome here.  I just answered your original
question, viz.:

> > seq.el's documentation is where?  I don't see it.

which seemed to imply that you didn't know at all that we have seq.el
functions described in the manual.

> It doesn't talk about the guarantees that the framework 
> offers with respect to when they are called, or if they 
> are called at all. For example, while I didn't read the 
> whole thing, I don't think it is stated that custom sequences
> based on Lisp lists need many more gf implementations 
> than others, else they simply won't work (because of 
> list-specific shortcutting optimizations only found 
> when reading the seq.el code).  Nor is it stated that, for 
> any representation,  :around methods are probably a bad
> idea. Nor is it stated that if you just implement the bare
> minimum (which is described in seq.el's header) you will 
> likely get very poor performance.

It says this:

     All functions defined in this library are free of side-effects; i.e.,
  they do not modify any sequence (list, vector, or string) that you pass
  as an argument.  Unless otherwise stated, the result is a sequence of
  the same type as the input.  For those functions that take a predicate,
  this should be a function of one argument.

     The ‘seq.el’ library can be extended to work with additional types of
  sequential data-structures.  For that purpose, all functions are defined
  using ‘cl-defgeneric’.  *Note Generic Functions::, for more details
  about using ‘cl-defgeneric’ for adding extensions.

If this is not enough, once again, please feel free to suggest
improvements and extensions.

> Such a manual is where one would find sentences like 
> "to make a working custom sequence that is accepted by
> any seq-* call, the user must add implementations to 
> the following N generics.  To get good performance 
> you must also add implementations to these other M 
> generics".

Do you see something like that in other Lisp-related manuals we have?
Does cl.info has this information regarding cl-lib?

I think the answer is NO, because we generally don't say such things
in our manuals, except in rare exceptional cases (which the seq.el one
isn't, IMNSHO).  Perhaps we should, but then we have a lot of work to
do on our manuals, and that work is not only the necessary writeups,
but also non-trivial research and analysis.  Look how much effort was
needed to perform a similar analysis on seq.el.  Volunteers are
welcome to do the same in other places, and suggest additions to the
manuals for any significant findings.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

Eli Zaretskii <eliz@gnu.org> writes:

> Such improvements are always welcome here.  I just answered your
> original question, viz.:
>
>> > seq.el's documentation is where?  I don't see it.
>
> which seemed to imply that you didn't know at all that we have seq.el
> functions described in the manual.

Sorry, I should have been clearer.  I was talking for some while in this
thread about API implementor's documentation, a description on how to
_extend_ seq.el to define new sequence types, not how to _use_ it in
programs to process sequences.

> Feel free to suggest improvements and additions to what we have.

I've been doing that.  But it still feels like some people are not
convinced that that documentation is essential at all (you seem to be
one of them, reading further).  And before this manual is written down,
we have to come to an agreement of exactly what guarantees seq.el gives
the user who wants to extend it for new sequence types.

> Do you see something like that in other Lisp-related manuals we have?

Yes, for my part see Flymake, Eglot and JSONrpc manuals.  They're not
great, but they each have a section on how to extend the library to
provide new diagnostic backends, new request handlers and new JSONRPC
transports, respectively.

> Does cl.info has this information regarding cl-lib? I think the answer
> is NO

Of course, and that's obvious.  cl-lib is not extensible.

>, because we generally don't say such things in our manuals,

Of course we do.  See 21.6.7

    * Programmed Completion::  Writing your own completion function.

It tells you how to _write_ a completion backend.  The instructions on
how to _use_ completion functions to complete things are elsewhere.

I use this part of the manual all the time, and so does everyone else
writing a backend.  We've been upholding that contract very well for a
number of Emacs releases so, we don't break backends willy-nilly.  I
recently added an optimization (completion-lazy-hilit) that doesn't
break anything (AFAI and other reviewers could tell).

> except in rare exceptional cases

Huh, is Flymake rare and exceptional?  I see a lot of backends popping
up that (correctly) take inspiration from the manual describing how to
extend Flymake.

> (which the seq.el one isn't, IMNSHO).

As far as I could gather from this discussion, seq.el is to be
recommended to users as the preferred way to process sequences in their
Lisp programs, most any program, including basic facilities in the core.

I have shown that performance aspects of this library, even when applied
to primitive data types such as lists and vectors, vary in accordance
with the presence or absence of other custom sequences, because of the
GF-based nature of seq.el.  The nature of these custom sequences changes
the magnitude of the impact.  So IMO it is wise to explain to users
exactly what custom sequences are and aren't supported/recommended.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

Eli Zaretskii <eliz@gnu.org> writes:

> Such improvements are always welcome here.  I just answered your
> original question, viz.:
>
>> > seq.el's documentation is where?  I don't see it.
>
> which seemed to imply that you didn't know at all that we have seq.el
> functions described in the manual.

Sorry, I should have been clearer.  I was talking for some while in this
thread about API implementor's documentation, a description on how to
_extend_ seq.el to define new sequence types, not how to _use_ it in
programs to process sequences.

> Feel free to suggest improvements and additions to what we have.

I've been doing that.  But it still feels like some people are not
convinced that that documentation is essential at all (you seem to be
one of them, reading further).  And before this manual is written down,
we have to come to an agreement of exactly what guarantees seq.el gives
the user who wants to extend it for new sequence types.

> Do you see something like that in other Lisp-related manuals we have?

Yes, for my part see Flymake, Eglot and JSONrpc manuals.  They're not
great, but they each have a section on how to extend the library to
provide new diagnostic backends, new request handlers and new JSONRPC
transports, respectively.

> Does cl.info has this information regarding cl-lib? I think the answer
> is NO

Of course, and that's obvious.  cl-lib is not extensible.

>, because we generally don't say such things in our manuals,

Of course we do.  See 21.6.7

    * Programmed Completion::  Writing your own completion function.

It tells you how to _write_ a completion backend.  The instructions on
how to _use_ completion functions to complete things are elsewhere.

I use this part of the manual all the time, and so does everyone else
writing a backend.  We've been upholding that contract very well for a
number of Emacs releases so, we don't break backends willy-nilly.  I
recently added an optimization (completion-lazy-hilit) that doesn't
break anything (AFAI and other reviewers could tell).

> except in rare exceptional cases

Huh, is Flymake rare and exceptional?  I see a lot of backends popping
up that (correctly) take inspiration from the manual describing how to
extend Flymake.

> (which the seq.el one isn't, IMNSHO).

As far as I could gather from this discussion, seq.el is to be
recommended to users as the preferred way to process sequences in their
Lisp programs, most any program, including basic facilities in the core.

I have shown that performance aspects of this library, even when applied
to primitive data types such as lists and vectors, vary in accordance
with the presence or absence of other custom sequences, because of the
GF-based nature of seq.el.  The nature of these custom sequences changes
the magnitude of the impact.  So IMO it is wise to explain to users
exactly what custom sequences are and aren't supported/recommended.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Cc: michael_heerdegen@web.de,  emacs-devel@gnu.org
> Date: Fri, 17 Nov 2023 10:47:29 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Feel free to suggest improvements and additions to what we have.
> 
> I've been doing that.  But it still feels like some people are not
> convinced that that documentation is essential at all (you seem to be
> one of them, reading further).

Show me the docs changes you think should be installed, and let's take
it from there.  Until then, this kind of hand-waving and vague
accusations are not useful.

In general, I suggest to be less "combative" in this and similar
discussions.  It doesn't help arriving to a consensus.  But I digress.

> And before this manual is written down, we have to come to an
> agreement of exactly what guarantees seq.el gives the user who wants
> to extend it for new sequence types.

The code is there, so AFAIU any "agreements" boil down to
understanding what it does and what it does not.  In which case,
perhaps bringing the author on board of this discussion would be
helpful?

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 17, 2023 at 12:23 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > But it still feels like some people are not
> > convinced that that documentation is essential at all (you seem to be
> > one of them, reading further).
> Show me the docs changes you think should be installed, and let's take
> it from there.  Until then, this kind of hand-waving and vague
> accusations are not useful.

You can't make this stuff up.  Didn't you just write 30 minutes ago

  because we generally don't say such things
  in our manuals, except in rare exceptional cases (which the
  seq.el one isn't, IMNSHO)

?? Is this not crystal clear evidence that IYNSHO this type
of documentation for seq.el isn't needed?  Are you sure _I'm_ the
vague accuser??

I hope that by sniping the rest of the message you at least
acknowledge that we _do_ say such things in our Elisp manual.

> In general, I suggest to be less "combative" in this and similar
> discussions.  It doesn't help arriving to a consensus.  But I digress.

Indeed.  And likewise.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Fri, 17 Nov 2023 13:14:11 +0000
> Cc: michael_heerdegen@web.de, emacs-devel@gnu.org
> 
> On Fri, Nov 17, 2023 at 12:23 PM Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > > But it still feels like some people are not
> > > convinced that that documentation is essential at all (you seem to be
> > > one of them, reading further).
> > Show me the docs changes you think should be installed, and let's take
> > it from there.  Until then, this kind of hand-waving and vague
> > accusations are not useful.
> 
> You can't make this stuff up.  Didn't you just write 30 minutes ago
> 
>   because we generally don't say such things
>   in our manuals, except in rare exceptional cases (which the
>   seq.el one isn't, IMNSHO)
> 
> ?? Is this not crystal clear evidence that IYNSHO this type
> of documentation for seq.el isn't needed?

No, it states the facts as I know them.  It doesn't mean I necessarily
think we should keep doing that.

> Are you sure _I'm_ the vague accuser??

Show me that I was wrong by submitting the docs you think should be
added, and let's take it from there.

> I hope that by sniping the rest of the message you at least
> acknowledge that we _do_ say such things in our Elisp manual.

No, it just means that I have nothing useful to say about that, and
prefer to say nothing.

> > In general, I suggest to be less "combative" in this and similar
> > discussions.  It doesn't help arriving to a consensus.  But I digress.
> 
> Indeed.  And likewise.

Yeah, right.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 17, 2023 at 3:01 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > You can't make this stuff up.  Didn't you just write 30 minutes ago
> >
> >   because we generally don't say such things
> >   in our manuals, except in rare exceptional cases (which the
> >   seq.el one isn't, IMNSHO)
> >
> > ?? Is this not crystal clear evidence that IYNSHO this type
> > of documentation for seq.el isn't needed?
>
> No, it states the facts as I know them.  It doesn't mean I necessarily
> think we should keep doing that.

This gives me serious vibes of Monty Python's "Not necessarily: I could be
arguing in my spare time".

> > Are you sure _I'm_ the vague accuser??
>
> Show me that I was wrong by submitting the docs you think should be
> added, and let's take it from there.

OK.  Noting that there's no documentation of a particular kind for a
given library without producing such documentation immediately merits
the super-friendly epithet of "handwaver" and "vague accuser".  So
now I know.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

João Távora <joaotavora@gmail.com> writes:

> On Fri, Nov 17, 2023 at 3:01 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
>> > You can't make this stuff up.  Didn't you just write 30 minutes ago
>> >
>> >   because we generally don't say such things
>> >   in our manuals, except in rare exceptional cases (which the
>> >   seq.el one isn't, IMNSHO)
>> >
>> > ?? Is this not crystal clear evidence that IYNSHO this type
>> > of documentation for seq.el isn't needed?
>>
>> No, it states the facts as I know them.  It doesn't mean I necessarily
>> think we should keep doing that.
>
> This gives me serious vibes of Monty Python's "Not necessarily: I could be
> arguing in my spare time".

You made me spill my coffee :-) :-) :-).
Thanks for reminding me of that.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> In which case, perhaps bringing the author on board of this discussion
> would be helpful?

In the past Nicolas had been quite busy most of the time and responded
only sporadically.  I'm not even sure his email address is still valid.

The co-author had been Stefan M, and to some small amount, I had been
involved, among others.

Michael.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Date: Fri, 17 Nov 2023 15:49:16 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > In which case, perhaps bringing the author on board of this discussion
> > would be helpful?
> 
> In the past Nicolas had been quite busy most of the time and responded
> only sporadically.  I'm not even sure his email address is still valid.

That's fine, but the worst that could happen here is that Nicolas will
not respond, in which case we lose nothing.  But maybe he will respond...

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > ?? There's a whole section in the ELisp manual called "Sequence
  > > Functions", which documents the seq-* functions.

  > Yes, but from a custom sequence implementors perspective,
  > the "best" documentation is still that bit in the header,
  > which describes "you must implement these to get a working
  > custom sequence".

  > So while the Elisp manual addresses custom sequences
  > (very briefly -- it just tells the user to look up generic
  > functions) it spends most time describing the generic
  > functions one by one, and while that's important, it's
  > just not enough.

We need both of these two kinds of documentation.
The Emacs Lisp Reference Manual should say how to _use_ the seq functions,
and comments in seq.el should say how to understand and maintain it.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Dmitry Gutov <dmitry@gutov.dev> writes:

> First of all, the type dispatch _does_ happen more than once per
> sequence in the current master. That doesn't seem to hurt much while
> the method is not specialized (only has the default implementation),
> but has impact as soon as the function gets additional method
> definitions.

Thanks for taking a look az the effect of having more than one method.
It's pretty much what is to expect, I guess. It's not bad though.

Next hit is then probably in the case where effective method functions
have to be constructed. Emfs being the result of combining applicable
methods. I think in Emacs that boils down to combining a main method
with around, after, before methods.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Michael Heerdegen]

João Távora <joaotavora@gmail.com> writes:

> Maybe seq.el can be made faster too?  Who knows, but it seems
> difficult without breaking at least some of its defgeneric-based
> contract.

Of course can it, and it is not difficult in most cases.

For example, `cl-some' has two code paths, while `seq-some' has only one
that corresponds to the more general and slower one in `cl-some'.
Nothing forbids us to add the same optimization to the `seq-some'
algorithm.

These kinds of benchmarks are more or less irrelevant.  seq.el is new
while cl-lib has been tuned and optimized for decades.  Nobody has so
far ever decidedly tried to optimize seq.el with respect to efficiency.
Not because it would be hard (it isn't) - just because nobody did the
job yet.  seq.el is in a state where most functions work reliably and
most bugs of the initial code have been fixed, but improvements are
still low relatively hanging fruits.  Unless we decide that seq.el
should intentionally have a more simple design.  But then, this would
have to be discussed again, it is a minor question compared to this one
(about cl-lib usage), IMO.

So please let's avoid such benchmarking contests here unless you want to
work on seq.el (then a bug report is a better place for discussion).
Because it has little relevance here.

Michael.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Mon, Nov 13, 2023 at 8:34 AM Michael Heerdegen
<michael_heerdegen@web.de> wrote:
>
> João Távora <joaotavora@gmail.com> writes:
>
> > Maybe seq.el can be made faster too?  Who knows, but it seems
> > difficult without breaking at least some of its defgeneric-based
> > contract.
>
> Of course can it, and it is not difficult in most cases.

Go right ahead, presuming you understand the pros and cons.

> For example, `cl-some' has two code paths, while `seq-some' has only one
> that corresponds to the more general and slower one in `cl-some'.
> Nothing forbids us to add the same optimization to the `seq-some'
> algorithm.

Sure you could have seq.el require 'cl-lib.el' ;-)

Now really, sure?  What about those generic functions that you
presumably have to shortcut?  Won't it break an existing contract to users
that all of the sudden you're not calling them anymore for certain
types of sequence?  And how are you going to do this without introducing
non-destructive versions of the seq.el utils?  Isn't non-mutability
in the contract?  And wasn't it a goal here to  end up with a smaller
dictionary of such utils?  Won't it bloat up comparibly to cl-lib while
still being less versatile?  These questions might have answers, of
course, but they're not answers as trivial as you make them sound.

> These kinds of benchmarks are more or less irrelevant.

I beg to differ, I think a recommended sequence processing
function that needlessly conses and takes orders of magnitude more
time to do its job is really quite relevant.  Elisp is still a
"LISt Processing" language by and large.

But also, I have to ask: what _would_ be a relevant benchmark
to you?

Just for cl-set-{n}difference, I'm seeing significant usage
in core right now, direct _and_ indirect.  I'm not sure all
the authors, let alone users, of these packages would like
to learn that these parts of their code have suddenly become
a full order of magnitude slower, let alone users using such
things, by virtue of some cl-lib.el -> seq.el operation
as is being suggested here.

At the very least, if this idea is to go ahead, answering
all those questions about seq.el's performance and measuring
everything twice should be a prerequisite.

> seq.el is new
> while cl-lib has been tuned and optimized for decades.

No, the point is that it hasn't.  Some of the destructive versions
weren't even destructive at all!  if you take a look at my code you'll
notice I optimized cl-lib considerably in very few cases.
There's a lot more that can be optimized there.

Also it  seems that the optional keyword arguments in these
functions are  not a  cause for performance concerns at all,
at least in these cases.  So just for learning that, these
benchmarks are useful.

> Nobody has so
> far ever decidedly tried to optimize seq.el with respect to efficiency.

The seq.el file has "optimized implementation for lists section" by
tweaking the generics for lists.  I find plausible the designer of
seq.el noticed that it is still much slower to do this but wanted to
keep a specific generic function contract anyway.

IOW, there's nothing wrong with declaring "library X is slow, but
powerful".  It's just that IMO it's bad to tell people to use just
that library all the time for a class of jobs.

> So please let's avoid such benchmarking contests here unless you want to
> work on seq.el (then a bug report is a better place for discussion).
> Because it has little relevance here.

Last I checked, the title of this thread is still "What's missing
in Elisp that makes people want to use cl-lib?".

One answer among several: fast sequence functions.

So, uncomfortable as the results may be (for now at least)
for proponents of "seq.el only" or "seq.el first", as long as
there's talk of deprecating cl-lib in favour of these
alternatives, I think concrete evidence and hard numbers very
much belong to this discussion.  Surely much more useful than
many ideological considerations about what complexity is.

If they lead to an improvement of seq.el in terms of speed,
so much the better.  If they lead to improvement of seq.el in
terms of versatility (like allowing backward traversal, key functions),
even better.

Finally, please note I'm not trying to bash seq.el.  I think it
has its place -- polymorphic sequences are great, if I ever
need them -- , but its use should _also_ be discriminate, not
indiscriminate.  Same for your set.el idea: IMHO it's very welcome,
but I think it will suffer (and gain, of course) from the same
kind of tradeoffs as seq.el.


João Távora

Re: What's missing in ELisp that makes people want to use cl-lib?

[Michael Heerdegen]

João Távora <joaotavora@gmail.com> writes:

> Now really, sure?  What about those generic functions that you
> presumably have to shortcut?  Won't it break an existing contract to
> users that all of the sudden you're not calling them anymore for
> certain types of sequence?

If you don't make a mess, no.  It's of course possible to choose
different implementations (algorithms) for different types of sequences.
There a few technical reasons why seq.el should be slower.

> And how are you going to do this without introducing non-destructive
> versions of the seq.el utils?

That's a thing we probably can't do, yes.  But this shouldn't lead to
algorithms reaching a different complexity class, or running slower by a
factor N with large N.

Do we have places in the Emacs Elisp sources where this would make a
significant difference?

>  And wasn't it a goal here to end up with a smaller dictionary of such
> utils?  Won't it bloat up comparibly to cl-lib while still being less
> versatile?

The result will still be much smaller than cl-lib.  The idea is of
course not to use nonsense algorithms just to spare some dozens of
Bytes.

> > These kinds of benchmarks are more or less irrelevant.
>
> I beg to differ, I think a recommended sequence processing
> function that needlessly conses and takes orders of magnitude more
> time to do its job is really quite relevant.  Elisp is still a
> "LISt Processing" language by and large.
>
> But also, I have to ask: what _would_ be a relevant benchmark
> to you?

You misunderstood.  Of course do these benchmarks matter very much, but
as I said, please, this thread is not a seq.el bug report.  What I said
is that benchmarks do not much tell us about how our future treatment of
cl-lib should look like as long as seq.el can be improved and sped up
and you argument with seq.el efficiency.

> Just for cl-set-{n}difference, I'm seeing significant usage in core
> right now, direct _and_ indirect.  I'm not sure all the authors, let
> alone users, of these packages would like to learn that these parts of
> their code have suddenly become a full order of magnitude slower, let
> alone users using such things, by virtue of some cl-lib.el -> seq.el
> operation as is being suggested here.

These actually do not really fit in seq.el but would rather be a natural
part of set.el.

> No, the point is that it hasn't.  Some of the destructive versions
> weren't even destructive at all!  if you take a look at my code you'll
> notice I optimized cl-lib considerably in very few cases.
> There's a lot more that can be optimized there.

Other parts were optimized in the time, of course.

> The seq.el file has "optimized implementation for lists section" by
> tweaking the generics for lists.  I find plausible the designer of
> seq.el noticed that it is still much slower to do this but wanted to
> keep a specific generic function contract anyway.

Again, such a contract does _not_ per se have implications about
efficiency.  A seq.el implementation of a generic function for lists can
find the type is 'list' and then use exactly the same algorithm as
cl-lib.  All you have to do additionally is one check (`listp').
This is currently not the case everywhere in seq.el, some algorithms
perform repeated, strictly speaking redundant, tests, but that can (and
should be, where it matters) avoided.

> Finally, please note I'm not trying to bash seq.el.  I think it
> has its place -- polymorphic sequences are great, if I ever
> need them -- , but its use should _also_ be discriminate, not
> indiscriminate.  Same for your set.el idea: IMHO it's very welcome,
> but I think it will suffer (and gain, of course) from the same
> kind of tradeoffs as seq.el.

The only relevant question for me in this message is: is an improved
seq.el good enough to replace the sequence functions in cl-lib?  I think
it is.

To convince Eli that it is not, he would need something like a real-life
case where the seq.el functions make some program significantly slower,
and where we know that this can't be improved in seq.el (something like
sorting a million three-element lists in a row is not a good example).


Michael.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> The only relevant question for me in this message is: is an improved
> seq.el good enough to replace the sequence functions in cl-lib?  I think
> it is.

The real issue is hidden in there: why do "we" need to replace cl-lib
again? IOW, I think the "seq.el question" is only part of something
else going on.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Tue, Nov 14, 2023 at 2:46 PM Michael Heerdegen
<michael_heerdegen@web.de> wrote:
>
> João Távora <joaotavora@gmail.com> writes:
>
> > Now really, sure?  What about those generic functions that you
> > presumably have to shortcut?  Won't it break an existing contract to
> > users that all of the sudden you're not calling them anymore for
> > certain types of sequence?
>
> If you don't make a mess, no.  It's of course possible to choose
> different implementations (algorithms) for different types of sequences.
> There a few technical reasons why seq.el should be slower.

Beg to differ.  See my latest benchmarks sent to Dmitry and his
updated code in feature/cl-lib-improvements.  Sure, it can be argues
that you can make it not as horribly slow as it is in some cases,
but never quite as fast.  And you'll be breaking compatibility all
the way, the further you go.

> > And how are you going to do this without introducing non-destructive
> > versions of the seq.el utils?
>
> That's a thing we probably can't do, yes.  But this shouldn't lead to
> algorithms reaching a different complexity class, or running slower by a
> factor N with large N.

Of course, algorithms don't change complexity :-) That would have
been too terrible.  But when you compare non-destructive versions of
some seq operations to cl-lib destructive operations "that we
probably can't do", you will see large N.  A snippet from my results
(again, see message to Dmitry with these files).

("set difference, small lists, custom pred"
  (cl-nset-difference "FASTEST" 0.201229483 15 0.11990391199999983)
  (cl-set-difference "5.0x SLOWER" 1.002034638 39 0.31673883500000066)
  (seq-difference-3 "15.6x (rel 3.1x) SLOWER" 3.136544874 194
   1.7686539850000003)
  (seq-difference "18.9x (rel 1.2x) SLOWER" 3.795912411 260
 2.140581182000002))
 ("set difference, big lists, custom pred"
  (cl-nset-difference "FASTEST" 0.011600632 0 0.0)
  (cl-set-difference "5.4x SLOWER" 0.062588482 1 0.008457714000000394)
  (seq-difference-3 "18.1x (rel 3.4x) SLOWER" 0.21003662699999998 12
   0.10525734000000142)
  (seq-difference "25.5x (rel 1.4x) SLOWER" 0.29535328699999996 19
 0.16559472099999972))

Where seq-difference-3's is what I think is Dmitry's best attempt
so far.

> Do we have places in the Emacs Elisp sources where this would make a
> significant difference?
>
> >  And wasn't it a goal here to end up with a smaller dictionary of such
> > utils?  Won't it bloat up comparibly to cl-lib while still being less
> > versatile?
>
> The result will still be much smaller than cl-lib.

It's completely unfair to compare seq.el to cl-lib.el in general.
You should be comparing to the cl-lib's sequence dictionary.

This is such a common (and annoying I must say) misconception

Here's the sequences dictionary summarized directly from the
CL Hyperspec.

   https://www.lispworks.com/documentation/lw70/CLHS/Body/c_sequen.htm

Notice how neatly organized it is.  Compare it to the seq.el shortdoc.
I don't see a big difference in size, do you?  When compared directly
to the cl-lib.el versions seq.el are almost always less versatile
and often greatly less versatile (not to mention slow as heck at least
int the current form).

Sure it has interesting things like seq-group, but it is missing
other interesting things like the "mismatch" operations, and
super-important things like destructive versions of things.

> These actually do not really fit in seq.el but would rather be a natural
> part of set.el.

But they're there.  Are they bloat?  I think it's pretty good to
have these "set difference" operations on lists, even if it obviously
the naming is off (as it is in mnay other languages by the way).

Anyway naming aside sometimes it's just what you want.  Of course,
if you also want fast, use cl-lib.el.

> > No, the point is that it hasn't.  Some of the destructive versions
> > weren't even destructive at all!  if you take a look at my code you'll
> > notice I optimized cl-lib considerably in very few cases.
> > There's a lot more that can be optimized there.
>
> Other parts were optimized in the time, of course.

And so were seq's

>
> > The seq.el file has "optimized implementation for lists section" by
> > tweaking the generics for lists.  I find plausible the designer of
> > seq.el noticed that it is still much slower to do this but wanted to
> > keep a specific generic function contract anyway.
>
> Again, such a contract does _not_ per se have implications about
> efficiency.

Of course it does.  The "list optimizations" I pointed to, are
designed for efficiency, to state the obvious.  And they already break
the contract laid out in the very spare seq.el documentation
that I only need to implement those 6 or 7 generics to have
a fully functioning custom seq.  I need only make something based
on lists or arrays to be bitten in the tail by those optimizations.
And even without this "based on lists" thing, the seq-contains-p
performance hog had to be circumvented by changing the contract,
changing the imaginary lines where it said

   "the user agrees to implement seq-contains-p"

to

   "the user agrees to implement seq-contains-pred"

> > Finally, please note I'm not trying to bash seq.el.  I think it
> > has its place -- polymorphic sequences are great, if I ever
> > need them -- , but its use should _also_ be discriminate, not
> > indiscriminate.  Same for your set.el idea: IMHO it's very welcome,
> > but I think it will suffer (and gain, of course) from the same
> > kind of tradeoffs as seq.el.
>
> The only relevant question for me in this message is: is an improved
> seq.el good enough to replace the sequence functions in cl-lib?  I think
> it is.

An "improved" seq.el?  Sure, maybe.  As we say in my country,
"if my grandmother had wheels whe should be school bus".  Let's see
what wheels you put on seq.el.

Currently, please no.  DO use seq.el for custom sequences, or when
there is good suspicion that custom sequences might appear.  Do NOT
use seq.el for macro expansions, bread and butter list processing, etc.
Do NOT use seq-let for example in anything that might eventually
be called many times.

> To convince Eli that it is not, he would need something like a real-life
> case where the seq.el functions make some program significantly slower,
> and where we know that this can't be improved in seq.el (something like
> sorting a million three-element lists in a row is not a good example).

I think we should be cautious instead, and take those benchmarks
seriously (like Dmitry is taking them).

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > But when you need to process plists, for example, it becomes
  > very handy.  This construct, which does require knowing a bit
  > of the mini-language that is cl-loop,

  >   (cl-loop for (k v) on plist by #'cddr collect (cons k v))

It is certainly concise.  If I saw that code I would not understand it,
but if you said what job it does I could see how it might do that.

That's because cl-loop defines its own little language, which I don't know.

The question of readability here is a combination of two questions
that work against each other.  First, how readable are the programs ou
can write in that little language?  Second, how readable are they to
those who don't know the little language?

This is a rather extreme example of how the frequent use of the CL
constructs, so that they become part of typical Emacs Lisp code, adds
to what people need to know in order to understand typical Emacs Lisp
code.
-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Sat, 11 Nov 2023 21:57:02 -0500
> 
>   > But when you need to process plists, for example, it becomes
>   > very handy.  This construct, which does require knowing a bit
>   > of the mini-language that is cl-loop,
> 
>   >   (cl-loop for (k v) on plist by #'cddr collect (cons k v))
> 
> It is certainly concise.  If I saw that code I would not understand it,
> but if you said what job it does I could see how it might do that.
> 
> That's because cl-loop defines its own little language, which I don't know.

Btw, the above is a very simple use of cl-loop.  We have quite a few
of much more complex ones.  For example:

  (cl-loop
   with comp-ctxt = (make-comp-cstr-ctxt)
   with h = (make-hash-table :test #'eq)
   for (f type-spec) in comp-known-type-specifiers
   for cstr = (comp-type-spec-to-cstr type-spec)
   do (puthash f cstr h)
   finally return h)

or this:

            (cl-loop for i = 0 then (+ i (if (eq (nth i list) ?\\) 4 1))
                     for var = (nth i list)
                     while (< i size)
                     if (eq var ?\\)
                     collect (string-to-number
                              (concat (cl-subseq list (+ i 1) (+ i 4))) 8)
                     else
                     collect var))

or this:

  (cl-loop
   named loop
   with above
   for lane in (comp-cstr-ctxt-typeof-types comp-ctxt)
   do (let ((x (memq type lane)))
        (cond
         ((null x) nil)
         ((eq x lane) (cl-return-from loop x)) ;A base type: easy case.
         (t (setq above
                  (if above (comp--intersection x above) x)))))
   finally return above))

It isn't an accident that the reference documentation of cl-loop in
cl.info takes a whopping 700(!) lines.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Btw, the above is a very simple use of cl-loop.  We have quite a few
  > of much more complex ones.  For example:

  >   (cl-loop
  >    with comp-ctxt = (make-comp-cstr-ctxt)
  >    with h = (make-hash-table :test #'eq)
  >    for (f type-spec) in comp-known-type-specifiers
  >    for cstr = (comp-type-spec-to-cstr type-spec)
  >    do (puthash f cstr h)
  >    finally return h)

Boy that is hard to understand.  I suggest we rewrite those cl-loop calls
into ordinary Lisp, avoiding use of cl-loop.

I looked at the manual for the iter construct ("iterate").  The added
parentheses make it fit better into Lisp.

However, I have to ask how the complexity of the usage of iter.
Does iter offer a corresponding construct for each cl-loop construct?

Are the iter constructs in 1-1 correspondence with cl-loop constructs?
If so, replacing c-loop with iter might not reduce the complexity that
either one adds to Emacs Lisp.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Sebastián Monía]

Richard Stallman <rms@gnu.org> writes:

> Boy that is hard to understand.  I suggest we rewrite those cl-loop calls
> into ordinary Lisp, avoiding use of cl-loop.

I find that loop very readable. I am not an expert Lisper. Beginner at
CL, maybe mid-level if being generous. I did write a couple Emacs
packages.

I have been following this thread somewhat closely. My opinion maybe
isn't worth much because I am not a regular contributor. But maybe it
helps as one of the angles discussed earlier was related to the size of
Elisp and complexity for newcomers.

1. This kind of thread is more "scary" to potential Emacs contributors
than any degree of complexity in cl-lib. I can confirm that.

2. There's a lot of vague arguments for and against the library that, as
an semi-outsider, read more like stylistic preference. Sometimes
familiarity, or lack of it. Few objective reasons for/against.

3. Anyone interested in writing code for Emacs has to make the leap to a
Lisp (elisp, in this case) which is already quite alien to a large
population. For non-programmers, the problem is indistinguishable from
learning any other language. Even for most people familiar with software
development, cl-lib is the least of their difficulties.

I hope that was in some way helpful.
Thank you,
Sebastián

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Sebastián Monía <code@sebasmonia.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  joaotavora@gmail.com,  emacs-devel@gnu.org
> Date: Mon, 13 Nov 2023 22:31:59 -0500
> 
> 1. This kind of thread is more "scary" to potential Emacs contributors
> than any degree of complexity in cl-lib. I can confirm that.

There's nothing to be scared of.  People express their opinions,
sometimes in strong and even harsh terms.  Nothing special, it happens
here all the time.  People are people.  Show me another development
forum with many very different participants that doesn't exhibit such
behavior from time to time.

> 2. There's a lot of vague arguments for and against the library that, as
> an semi-outsider, read more like stylistic preference. Sometimes
> familiarity, or lack of it. Few objective reasons for/against.

This, too, is quite usual in forums that discuss coding and styles.
Many people don't possess basic abilities to stay focused and conduct
a rational, constructive, and meaningful discussion, and we are no
different here.  Just open any TV channel where a few people discuss
something non-trivial, and you will see it more often than not.

> 3. Anyone interested in writing code for Emacs has to make the leap to a
> Lisp (elisp, in this case) which is already quite alien to a large
> population. For non-programmers, the problem is indistinguishable from
> learning any other language.

I came to Emacs Lisp from very different languages, and don't remember
that as being a leap, just an adjustment of how I think about code and
data.

> Even for most people familiar with software development, cl-lib is
> the least of their difficulties.

Not my experience, FWIW.  Maybe not the most of my difficulties, but
it certainly isn't negligible.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Sebastián Monía wrote:

> 1. This kind of thread is more "scary" to potential Emacs
>    contributors than any degree of complexity in cl-lib.
>    I can confirm that.

Very true! It is alienating to newcomers but also to
experienced people who have solved problem with cl-lib in
Emacs and saw no problem with that, on the contrary, they used
what they thought was the best way to do it.

> 2. There's a lot of vague arguments for and against the
>    library that, as an semi-outsider, read more like
>    stylistic preference. Sometimes familiarity, or lack of
>    it. Few objective reasons for/against.

Again, true. In particular, what is lacking is whatever the
alternative would be. If cl-lib has increased the general
complexity, fine, maybe that is a problem to some. But surely
it is totally impractical to kick it out and for the vast
majority of the places where it is used we can't make the case
not using it will improve the quality of that code on
a case-by-case basis.

> 3. Anyone interested in writing code for Emacs has to make
>    the leap to a Lisp (elisp, in this case) which is already
>    quite alien to a large population. For non-programmers,
>    the problem is indistinguishable from learning any other
>    language. Even for most people familiar with software
>    development, cl-lib is the least of their difficulties.

Very true, there is no huge leap from Elisp not including
cl-lib to Elisp including cl-lib. If used well, cl-lib will on
the contrary reduce the complexity and make the code cleaner.
And in very complicated cases, as we have seen with some
examples of `cl-loop', it is not clear in any way that
re-writing that in any other style would reduce complexity.
On the contrary `cl-loop' can be seen as a handy frame where
all that complexity, when encountered, is neatly organized
and accessible.

At the very least, the people who used cl-lib and cl-loop and
every thing else cl- that they used, they didn't agree that
use was detrimental to anything, style or functionality.
And they solved their problems! In general, we should trust
them, and in particular, we should absolutely not alienate
them or anyone else for no reason other than style issues that
other people have on their contributions to Emacs.

We shipped the library, people used it, that's the situation.

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[T.V Raman]

Emanuel Berg <incal@dataswamp.org> writes:

And to make things concrete, take a look for example at some of the good
uses of cl-loop  within the emacs codebase itself, eg eww.el.

Many of those idiomatic uses would I suspect be  not as elegant  (for
some measure of elegance) without cl-loop

> Sebastián Monía wrote:
>
>> 1. This kind of thread is more "scary" to potential Emacs
>>    contributors than any degree of complexity in cl-lib.
>>    I can confirm that.
>
> Very true! It is alienating to newcomers but also to
> experienced people who have solved problem with cl-lib in
> Emacs and saw no problem with that, on the contrary, they used
> what they thought was the best way to do it.
>
>> 2. There's a lot of vague arguments for and against the
>>    library that, as an semi-outsider, read more like
>>    stylistic preference. Sometimes familiarity, or lack of
>>    it. Few objective reasons for/against.
>
> Again, true. In particular, what is lacking is whatever the
> alternative would be. If cl-lib has increased the general
> complexity, fine, maybe that is a problem to some. But surely
> it is totally impractical to kick it out and for the vast
> majority of the places where it is used we can't make the case
> not using it will improve the quality of that code on
> a case-by-case basis.
>
>> 3. Anyone interested in writing code for Emacs has to make
>>    the leap to a Lisp (elisp, in this case) which is already
>>    quite alien to a large population. For non-programmers,
>>    the problem is indistinguishable from learning any other
>>    language. Even for most people familiar with software
>>    development, cl-lib is the least of their difficulties.
>
> Very true, there is no huge leap from Elisp not including
> cl-lib to Elisp including cl-lib. If used well, cl-lib will on
> the contrary reduce the complexity and make the code cleaner.
> And in very complicated cases, as we have seen with some
> examples of `cl-loop', it is not clear in any way that
> re-writing that in any other style would reduce complexity.
> On the contrary `cl-loop' can be seen as a handy frame where
> all that complexity, when encountered, is neatly organized
> and accessible.
>
> At the very least, the people who used cl-lib and cl-loop and
> every thing else cl- that they used, they didn't agree that
> use was detrimental to anything, style or functionality.
> And they solved their problems! In general, we should trust
> them, and in particular, we should absolutely not alienate
> them or anyone else for no reason other than style issues that
> other people have on their contributions to Emacs.
>
> We shipped the library, people used it, that's the situation.

-- 

Re: What's missing in ELisp that makes people want to use cl-lib?

[Tomas Hlavaty]

On Mon 13 Nov 2023 at 21:58, Richard Stallman <rms@gnu.org> wrote:
>   >   (cl-loop
>   >    with comp-ctxt = (make-comp-cstr-ctxt)
>   >    with h = (make-hash-table :test #'eq)
>   >    for (f type-spec) in comp-known-type-specifiers
>   >    for cstr = (comp-type-spec-to-cstr type-spec)
>   >    do (puthash f cstr h)
>   >    finally return h)
> Boy that is hard to understand.

It is ugly.

> I suggest we rewrite those cl-loop calls
> into ordinary Lisp, avoiding use of cl-loop.

That would be something along the lines of:

(let ((comp-ctxt (make-comp-cstr-ctxt))
      (h (make-hash-table :test #'eq))
      f)
  (while (setq f (pop comp-known-type-specifiers))
    (puthash f (comp-type-spec-to-cstr type-spec) h))
  h)

comp-ctxt does not seem to be used.
Does emacs-lisp compiler not warn about unused variables?

> I looked at the manual for the iter construct ("iterate").  The added
> parentheses make it fit better into Lisp.

I find the idea of iterate even uglier that cl-loop
and I think that it leads people completely astray.
It is not ordinary lisp code or macro, it is a compiler.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 327 bytes --]

On Tue, Nov 14, 2023 at 8:13 AM Tomas Hlavaty <tom@logand.com> wrote:

>
> I find the idea of iterate even uglier that cl-loop
> and I think that it leads people completely astray.
> It is not ordinary lisp code or macro, it is a compiler.
>

Every macro is a compiler: it translates arbitrary forms to Lisp
forms.

[-- Attachment #2: Type: text/html, Size: 625 bytes --]

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> Every macro is a compiler: it translates arbitrary forms to Lisp 
> forms.

That's quibbling.  The transformations carried out by cl-loop are by a
long shot more consequential to the structure of their input than those
by most any other macro there is, which is a demerit and, I think, is
what Thomas meant to communicate.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Tue, Nov 14, 2023 at 10:55 AM Po Lu <luangruo@yahoo.com> wrote:

> The transformations carried out by cl-loop are by a
> long shot more consequential to the structure of their input than those
> by most any other macro there is

That's true.

> , which is a demerit

That's not.

LOOP, pcase, etc, like any other macros, are mini-languages.  Like
any higher-level language, the more powerful they are, the more
transformations they will do on your input.  LOOP is pretty powerful,
and may or may not be suitable for the task at hand, like any other
language.

At any rate Tomas' translation of the loop is just plain
wrong.  Where is 'type-spec' being bound?  It needs to be
destructured: LOOP is a language which is very good
at expressing this fact concisely in a self-documenting way:

  for (f type-spec) in comp-known-type-specifiers

Here from this one line, without ever having looked at Andrea's
code I learn that comp-known-type-specifiers is a list of
2-element lists, where each such tuple is at least made of
an 'f' and a 'type-spec'.

It would take longer (for me!) to learn this fact if
'car' and 'cadr' had been used in subsequent forms away from
detached from a hypothetical (pop comp-known-type-specifiers).

But again, that's just for me, and that's because I have learned
the LOOP language, because I thought it was worth learning
just for reading other people's code.  And it sure was.

But if you don't want to learn it, for whatever personal reason,
open up the macroexpansion, or step through the function with
edebug.  To each his own.

Anyway Tomas' example, if it were to be corrected, would become
"ugly" perhaps "uglier" (as if "ugly" were even an objective
thing)

Here, I think dolist + pcase-let/cl-destructuring-case/seq-let
would also be appropriate.  Though I'm sure someone out there
would find it insufferably ugly or anti-Elisp as well.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> That's not.
>
> LOOP, pcase, etc, like any other macros, are mini-languages.  Like
> any higher-level language, the more powerful they are, the more
> transformations they will do on your input.  LOOP is pretty powerful,
> and may or may not be suitable for the task at hand, like any other
> language.

Four (by some counts five) individuals, all Emacs users, that is to say
the constituency who will ultimately judge these constructs, have taken
issue with these macros on the grounds that they are difficult to
understand.

When all's said and done, the factor that contributes to whether using a
feature is meritorious is not the amount of use that can be gotten out
of them alone.  Rather, it is that amount considered in tandem with the
complexity introduced into its users as perceived from "20,000
ft. views" and the extent to which its users cannot be recast in terms
of different, simpler constructs.  The examples that have been
circulated in support of cl-loop thus far have not demonstrated this
extent to be anything except nil, so it's fair to conclude that most
calls to cl-loop are frivolous, and its complexity is a great failing.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 690 bytes --]

On Tue, Nov 14, 2023 at 2:04 PM Po Lu <luangruo@yahoo.com> wrote:

>
> extent to be anything except nil, so it's fair to conclude that most
> calls to cl-loop are frivolous,


You want to believe that, obviously, what can I do? you're preaching
to your choir of converts.  IMO this just a very poor generatlization
on far-fetched  conceptions of 20.000 feet views and vague defition of
merits.  If you would show me code, I could tell you if I think a given
cl-loop  is frivolous or not like I did there.  But t would still be my
opinion, and we'd descend into this ridiculous thesaurus-fueled
rambling again.

So take care and learn loop or don't, I don't care.

João

[-- Attachment #2: Type: text/html, Size: 2430 bytes --]

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> You want to believe that, obviously, what can I do? you're preaching
> to your choir of converts.  IMO this just a very poor generatlization
> on far-fetched conceptions of 20.000 feet views and vague defition of
> merits.  If you would show me code, I could tell you if I think a
> given cl-loop is frivolous or not like I did there.  But t would still
> be my opinion, and we'd descend into this ridiculous thesaurus-fueled
> rambling again.
>
> So take care and learn loop or don't, I don't care.

Then there's no point in continuing, is there?  Because my intent was to
arrive at a consensus agreeable to me and others who resent the
pervasiveness of cl-loop, and that of like contrivances in general.

If you're insular enough to never accept anything in that vein, then no
consensus can be established, since I'm well aware I too am hidebound to
my own beliefs, formed and tempered by time, such that I will not
concede in this regard.  Thus instead of assenting to a series of
procedures that preclude the introduction of cryptic code, the stage
will be set for more bickering to erupt each time one of us chances upon
code he needlessly cannot understand, consequent on the refusal of
Common Lisp sectarians like you to write simple loops with constructs
native to *this* language.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Tue, Nov 14, 2023 at 3:03 PM Po Lu <luangruo@yahoo.com> wrote:
>
> João Távora <joaotavora@gmail.com> writes:

> > So take care and learn loop or don't, I don't care.
>
> Then there's no point in continuing, is there?  Because my intent was to
> arrive at a consensus agreeable to me and others who resent the
> pervasiveness of cl-loop, and that of like contrivances in general.
>
> If you're insular enough to never accept anything in that vein, then no
> consensus can be established,

If there's anything "frivolous" and "insular" here, it's this rococo arguing
of style matters, full of unfounded adjectivation, not even comparing to
any actual code or working alternatives. This is Lisp, it has macros.  Like C++
has templates.  I use both when I think there is an advantage.  C programmers
naturally don't like any of those, some of them think the preprocessor is truly
the chef's kiss.  I don't judge them, it's useless.  In the other thread even
the very idea of a sequence processing library was aberrant to you.  What can
I say?  Just that it's not for me to judge, not even worth my time

Again: take care and learn loop, cl-position, seq.el, pcase, or don't.
Though I don't use your code, I think you're doing important contributions
to Emacs, keep it up.  I won't be a jury in the silly code beauty pageant.
Really I won't, and neither should you if you want a piece of advice.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> If there's anything "frivolous" and "insular" here, it's this rococo
> arguing of style matters, full of unfounded adjectivation, not even
> comparing to any actual code or working alternatives. This is Lisp, it
> has macros.  Like C++ has templates.  I use both when I think there is
> an advantage.  C programmers naturally don't like any of those, some
> of them think the preprocessor is truly the chef's kiss.  I don't
> judge them, it's useless.  In the other thread even the very idea of a
> sequence processing library was aberrant to you.  What can I say?
> Just that it's not for me to judge, not even worth my time

Of course, that's your choice.  Up until it begins to affect _others_
who must read the code produced by such a choice, at which point their
views become the last word.

So use cl-lib if you want.  Just keep it out of Emacs.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Wed, Nov 15, 2023 at 12:13 AM Po Lu <luangruo@yahoo.com> wrote:

> So use cl-lib if you want.  Just keep it out of Emacs.

The day you become Emacs maintainer, you can issue that order, at
which point I will just leave the project, go back to GitHub or
whatever, no hard feelings.  Until then, stop the rhetorical
buffoonery and your bullying (or don't, but I will just ignore
you).

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

On November 15, 2023 12:28:44 PM GMT+02:00, "João Távora" <joaotavora@gmail.com> wrote:
>
>  Until then, stop the rhetorical
> buffoonery and your bullying

Please keep such personal attacks out of this forum.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Wed, Nov 15, 2023 at 10:41 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> On November 15, 2023 12:28:44 PM GMT+02:00, "João Távora" <joaotavora@gmail.com> wrote:
> >
> >  Until then, stop the rhetorical
> > buffoonery and your bullying
>
> Please keep such personal attacks out of this forum.

Sure thing, and if you want to second Po's order
(not bullish at all, never!) just say so.

In that case I will leave right now, no hard feelings.  Else,
I'll take your advice, step away from this subthread,
keep on using cl-lib.el when I feel it's fit, be open to
feedback about each and every specific use, but not to
mind-numbing religious theories.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Wed, 15 Nov 2023 10:56:01 +0000
> Cc: emacs-devel@gnu.org
> 
> > Please keep such personal attacks out of this forum.
> 
> Sure thing, and if you want to second Po's order
> (not bullish at all, never!) just say so.

My remark has nothing to do with the technical issue at hand, and
doesn't express any opinion on it.  It is only about the words you
used.  I'm asking you and others to please make a point of keeping the
discussions here civilized in tone and wording.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> The day you become Emacs maintainer, you can issue that order, at
> which point I will just leave the project, go back to GitHub or
> whatever, no hard feelings.  Until then, stop the rhetorical
> buffoonery and your bullying

In other words, "it's my way or the highway."  Since that's how you've
been behaving from the outset, I can't be taxed for responding in equal
measure.

> (or don't, but I will just ignore you).

I'll hold you to that.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Wed, Nov 15, 2023 at 1:15 PM Po Lu <luangruo@yahoo.com> wrote:
>
> João Távora <joaotavora@gmail.com> writes:
>
> > The day you become Emacs maintainer, you can issue that order, at
> > which point I will just leave the project, go back to GitHub or
> > whatever, no hard feelings.  Until then, stop the rhetorical
> > buffoonery and your bullying
>
> In other words, "it's my way or the highway."  Since that's how you've
> been behaving from the outset, I can't be taxed for responding in equal
> measure.

Nothing of the sort.  It's my highway, I'm not telling _you_ to go
anywhere or change anything.  My guess is you'll be quite happy
in a cl-lib.el free Emacs, so when you become maintainer, you'll
have it.

> > (or don't, but I will just ignore you).
>
> I'll hold you to that.

OK.  You're now in my MUA ignore list, direct to the spam folder.
It's sad I won't be reading your contributions to other matters,
but if you don't want to stop this little theater, so be it.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Tomas Hlavaty <tom@logand.com>,  rms@gnu.org,  Eli Zaretskii
>  <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Wed, 15 Nov 2023 21:15:01 +0800
> 
> João Távora <joaotavora@gmail.com> writes:
> 
> > The day you become Emacs maintainer, you can issue that order, at
> > which point I will just leave the project, go back to GitHub or
> > whatever, no hard feelings.  Until then, stop the rhetorical
> > buffoonery and your bullying
> 
> In other words, "it's my way or the highway."  Since that's how you've
> been behaving from the outset, I can't be taxed for responding in equal
> measure.

Yes, you can.  Please try not to respond unkindly when someone attacks
you.

> > (or don't, but I will just ignore you).
> 
> I'll hold you to that.

That was unnecessary.  Please try to avoid that in the future.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Tomas Hlavaty]

On Tue 14 Nov 2023 at 11:47, João Távora <joaotavora@gmail.com> wrote:
> On Tue, Nov 14, 2023 at 10:55 AM Po Lu <luangruo@yahoo.com> wrote:
> LOOP, pcase, etc, like any other macros, are mini-languages.  Like
> any higher-level language, the more powerful they are, the more
> transformations they will do on your input.  LOOP is pretty powerful,
> and may or may not be suitable for the task at hand, like any other
> language.

The question is if those are "nice" mini-languages or expressing the
code in simple emacs-lisp is nicer.

> At any rate Tomas' translation of the loop is just plain
> wrong.  Where is 'type-spec' being bound?  It needs to be
> destructured:

Thanks, you found a bug in the sketch.  It should have been:

    (puthash (pop f) (comp-type-spec-to-cstr (pop f)) h)

or maybe

    (puthash (car f) (comp-type-spec-to-cstr (cadr f)) h)

> LOOP is a language which is very good
> at expressing this fact concisely in a self-documenting way:
>
>   for (f type-spec) in comp-known-type-specifiers

or

(dolist (x comp-known-type-specifiers)
  (destructuring-bind (f type-spec) x ...))

if emacs-lisp had destructuring-bind.

Note that in this particular example, the variables are not needed at
all.

Also note that comp-known-type-specifiers contains unnecessary cons cell
for each element.

> Here from this one line, without ever having looked at Andrea's
> code I learn that comp-known-type-specifiers is a list of
> 2-element lists, where each such tuple is at least made of
> an 'f' and a 'type-spec'.

Even better feature of destructuring-bind is: when
comp-known-type-specifiers changes structure, destructuring-bind will
fail laudly.

If cl-loop supported keywords, it could be even more strict:

(destructuring-bind (&key f type-spec) comp-known-type-specifiers ...)

Or maybe comp-known-type-specifiers should be a structure?

How strict do we want to be?

> Here, I think dolist + pcase-let/cl-destructuring-case/seq-let
> would also be appropriate.  Though I'm sure someone out there
> would find it insufferably ugly or anti-Elisp as well.

I think dolist would be the right choice.

>   >   (cl-loop
>   >    with comp-ctxt = (make-comp-cstr-ctxt)
>   >    with h = (make-hash-table :test #'eq)
>   >    for (f type-spec) in comp-known-type-specifiers
>   >    for cstr = (comp-type-spec-to-cstr type-spec)
>   >    do (puthash f cstr h)
>   >    finally return h)

That would be something along the lines of:

(let ((h (make-hash-table :test #'eq)))
  (dolist (f comp-known-type-specifiers h)
    (puthash (pop f) (comp-type-spec-to-cstr (pop f)) h)))

which seems the most concise translation of the cl-loop code above

or

(let ((h (make-hash-table :test #'eq)))
  (dolist (f comp-known-type-specifiers h)
    (puthash (car f) (comp-type-spec-to-cstr (cadr f)) h)))

> Anyway Tomas' example, if it were to be corrected, would become
> "ugly" perhaps "uglier" (as if "ugly" were even an objective
> thing)

So I removed the unused variable comp-ctxt, pulled the hash-table out of
cl-loop, and puthash stuff for each element of
comp-known-type-specifiers.

The only ugly thing remaining is destructuring the elements of
comp-known-type-specifiers which plain emacs-lisp does not have a good
way of expressing:

(let ((h (make-hash-table :test #'eq)))
  (dolist (x comp-known-type-specifiers h)
    (destructuring-bind (f type-spec) x
      (puthash f (comp-type-spec-to-cstr type-spec) h))))

or (if emacs-lisp had &key)

(let ((h (make-hash-table :test #'eq)))
  (dolist (x comp-known-type-specifiers h)
    (destructuring-bind (&key f type-spec) x
      (puthash f (comp-type-spec-to-cstr type-spec) h))))

or in plain emacs-lisp:

(let ((h (make-hash-table :test #'eq)))
  (dolist (x comp-known-type-specifiers h)
    (apply (lambda (f type-spec)
             (puthash f (comp-type-spec-to-cstr type-spec) h))
           x)))

or

(defun mapcly (fn seq)
  (mapc (lambda (x) (apply fn x)) seq))

(let ((h (make-hash-table :test #'eq)))
  (mapcly (lambda (f type-spec)
            (puthash f (comp-type-spec-to-cstr type-spec) h))
          comp-known-type-specifiers)
  h)

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Tue, Nov 14, 2023 at 5:40 PM Tomas Hlavaty <tom@logand.com> wrote:
>
> On Tue 14 Nov 2023 at 11:47, João Távora <joaotavora@gmail.com> wrote:
> > On Tue, Nov 14, 2023 at 10:55 AM Po Lu <luangruo@yahoo.com> wrote:
> > LOOP, pcase, etc, like any other macros, are mini-languages.  Like
> > any higher-level language, the more powerful they are, the more
> > transformations they will do on your input.  LOOP is pretty powerful,
> > and may or may not be suitable for the task at hand, like any other
> > language.
>
> The question is if those are "nice" mini-languages or expressing the
> code in simple emacs-lisp is nicer.
>
> > At any rate Tomas' translation of the loop is just plain
> > wrong.  Where is 'type-spec' being bound?  It needs to be
> > destructured:
>
> Thanks, you found a bug in the sketch.  It should have been:
>
>     (puthash (pop f) (comp-type-spec-to-cstr (pop f)) h)
>
> or maybe
>
>     (puthash (car f) (comp-type-spec-to-cstr (cadr f)) h)

Yeah, this one's better.  The pop is just confusing.  As is
the setq in the while condition, which my common lisp priests
would not approve.  But the dolist+destructuring-bind is pretty
nice and solves all that.

> (dolist (x comp-known-type-specifiers)
>   (destructuring-bind (f type-spec) x ...))
>
> if emacs-lisp had destructuring-bind.

But it does!  in cl-lib.el no less :-)

> (let ((h (make-hash-table :test #'eq)))
>   (dolist (x comp-known-type-specifiers h)
>     (apply (lambda (f type-spec)
>              (puthash f (comp-type-spec-to-cstr type-spec) h))
>            x)))
>

This one is kinda hard to read for me, at least I couldn't tell
if it was correct right way, putting a lambda in there confuses
things (also if I not mistaken 'apply' instead of directly accessing
the elements is slower).

Anyway, I think

(cl-loop with h = (make-hash-table :test #'eq)
         for (f type-spec) in comp-known-type-specifiers
         do (puthash f (puthash f (comp-type-spec-to-cstr type-spec) h)
         finally return h)

is the comparable loop and I don't think it's so bad.
It's one line less and less complex sexp tree.
But all your solutions are nice too, so I guess I'm pretty
tolerant.  I don't know if I would write this loop myself
because while the destructuring is nice, the return h spoils
it, and the dolist gives you for free (if you know the
mini-language ;-) )

Loop is really good (IMHO!) for complex iteration with
repeat, counts, maximizations, early returns.  I like loop's
self-documenting properties.
Complex iteration happens in programs, in all languages! Some
things are hard regardless of the idiom, unless you learn the
problem domain and the code is documented.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Tomas Hlavaty]

On Tue 14 Nov 2023 at 18:55, Po Lu <luangruo@yahoo.com> wrote:
> João Távora <joaotavora@gmail.com> writes:
>
>> Every macro is a compiler: it translates arbitrary forms to Lisp 
>> forms.
>
> That's quibbling.  The transformations carried out by cl-loop are by a
> long shot more consequential to the structure of their input than those
> by most any other macro there is, which is a demerit and, I think, is
> what Thomas meant to communicate.

Yes, thanks, that was the idea.

Claiming "every macro is a compiler" is not useful for understanding,
why cl-loop or iterate are ugly.

cl-loop is ugly because it invents kind of flat (at the list level)
syntax and rewrites the top flat part.  But it does not change deeper
parts of the input.

iterate is another level of ugly.  It does not have flat syntax, but
rewrites the input all the way to the bottom of the form.  And it needs
to understand every form it walks.

I usually prefer using the simplest tool for the job.

In this example the choices could be:

   while|dolist|mapcar < cl-loop < iterate

I would use while or dolist or mapcar.
There is no point using cl-loop.  It is bearable if somebody insist.
iterate should not exist.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > I suggest we rewrite those cl-loop calls
  > > into ordinary Lisp, avoiding use of cl-loop.

  > That would be something along the lines of:

  > (let ((comp-ctxt (make-comp-cstr-ctxt))
  >       (h (make-hash-table :test #'eq))
  >       f)
  >   (while (setq f (pop comp-known-type-specifiers))
  >     (puthash f (comp-type-spec-to-cstr type-spec) h))
  >   h)

That is simpler than the cl-loop construct that you rewrote.
Though it will alter the ambient valu eof comp-type-spec-to-cstr,
which might be undesirable.

To avoid that, we could use

   (dolist (f comp-known-type-specifiers)
     (puthash f (comp-type-spec-to-cstr type-spec) h))

and avoid binding f in the let.  That would be one line less.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Plesae forgive my delay in getting to this.

  > Btw, the above is a very simple use of cl-loop.  We have quite a few
  > of much more complex ones.  For example:

  >   (cl-loop
  >    with comp-ctxt = (make-comp-cstr-ctxt)
  >    with h = (make-hash-table :test #'eq)
  >    for (f type-spec) in comp-known-type-specifiers
  >    for cstr = (comp-type-spec-to-cstr type-spec)
  >    do (puthash f cstr h)
  >    finally return h)

  ...

  > It isn't an accident that the reference documentation of cl-loop in
  > cl.info takes a whopping 700(!) lines.

This suggests an idea to me:
we can define something in the byte compiler to distinguish simple
uses of cl-loop from non-simle ones, and warn about the latter.

The distinction could be a matter of whether it uses cl-loop features
that are less frequently used.  Or perhaps how MANY cl-loop features
are used.  (There is no need to try to make this precise or quibble
about the details.)

We could enable the warning for source files in the Emacs distribution.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Wed, Jan 3, 2024 at 4:11 AM Richard Stallman <rms@gnu.org> wrote:

> The distinction could be a matter of whether it uses cl-loop features
> that are less frequently used.  Or perhaps how MANY cl-loop features
> are used.

Whatever becomes of this (sorry Richard, I hope not much), I do hope
this doesn't go into the byte-compiler proper, but some special
linter.  Give this special linter the ability to detect lisp sexps
trees deeper than a given threshold or longer than a certain number
of lines.  That's likely much simpler to agree on and implement.

Make this linter optional, like M-x checkdoc.

> (There is no need to try to make this precise or quibble
> about the details.)

Of course there is.  cl-loop users should be consulted about
which features they use to make something happen concisely
and readably.

What you're proposing amounts to making that linter mostly
ignorant about exactly what makes cl-loop useful. A linter that
flags a construct and can't suggest a universally accepted better
alternative -- like the byte-compiler does today -- is just
unneeded nuisance.

I'll be very clear: It's totally legitimate for the maintainers
to decide these kinds of things if they think it's better, and if
this were a paid job I might go along with it out of necessity,
but it isn't so i'd just stop pushing to any repo that starts
enforcing this kind of stuff, wish you all good luck and no hard
feelings.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Wed, 3 Jan 2024 11:55:52 +0000
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> 
> I'll be very clear: It's totally legitimate for the maintainers
> to decide these kinds of things if they think it's better, and if
> this were a paid job I might go along with it out of necessity,
> but it isn't so i'd just stop pushing to any repo that starts
> enforcing this kind of stuff, wish you all good luck and no hard
> feelings.

Noted.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: joaotavora@gmail.com, emacs-devel@gnu.org
> Date: Tue, 02 Jan 2024 23:11:27 -0500
> 
> This suggests an idea to me:
> we can define something in the byte compiler to distinguish simple
> uses of cl-loop from non-simle ones, and warn about the latter.
> 
> The distinction could be a matter of whether it uses cl-loop features
> that are less frequently used.  Or perhaps how MANY cl-loop features
> are used.  (There is no need to try to make this precise or quibble
> about the details.)
> 
> We could enable the warning for source files in the Emacs distribution.

If someone wants to work on this, it's fine by me.  The actual
decision needs to be taken only when and if patches to that effect are
submitted.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Michael Heerdegen]

João Távora <joaotavora@gmail.com> writes:

> Also, I know you don't like cl-loop but in terms of efficiency
> it's sometimes a very good choice.

The problem of `cl-loop' is more one of expressiveness and readability.
Even some Common Lisp people don't like it and have developed
alternatives - like "iterate" (https://iterate.common-lisp.dev/).

Dunno if something like that would be more acceptable to Po Lu, but it
could be an improvement compared to `cl-loop'.


Michael.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Michael Heerdegen wrote:

> The problem of `cl-loop' is more one of expressiveness
> and readability.

It depends how you use it, it can be used in a very clean and
clear way.

Because it has been used so much (2nd on the CL list in the
Emacs source, 1004 times in 192 files), doesn't that mean it
is either good, or the alternatives are worse?

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Augusto Stoffel]

On Sat, 11 Nov 2023 at 13:09, João Távora wrote:

> seq.el strength over cl-lib.el is that it is a generic sequence
> processing library.  IMO it _should_ be used if there is the suspicion
> that the sequences being in that data being processed by a given
> function can be custom seq.el sequences.  Or when there is good
> reason to suspect that one they might be such custom sequences.
>
> I think that seq.el _should_ be used in those situations because
> it expresses exactly this fact, so it becomes self-documenting.
>
> I think the argument that Gerd is making in his economic
> contributions is that very often seq.el is overkill.
>
> For example, in its current form should _not_ be used when
> there is certainty that the sequences are Lisp listsm because as I
> showed elsewhere here, seq-some is significantly slower than
> cl-some and cl-loop when processing lots of small lists.

FWIW, I don't really use seq for its polymorphism.  Rather, I just care
about miscellaneous operations such as seq-some, seq-filter, seq-uniq
where a list-only built-in is not available.

From what I see in other people's code, this is by far the most frequent
reason to use seq- functions.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Wed, Nov 15, 2023 at 2:41 PM Augusto Stoffel <arstoffel@gmail.com> wrote:

> FWIW, I don't really use seq for its polymorphism.

Few people do, in core 0.

> From what I see in other people's code, this is by far the most frequent
> reason to use seq- functions.

That and the fact they aren't aware they're anywhere from 5x to
10x slower, sometimes more (I measured 18x for decently realistic),
than what they could be.

Also the fact they don't know cl-seq.el provides all that with
a much more versatile interface, and frequently much better
performance.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Augusto Stoffel]

On Wed, 15 Nov 2023 at 14:51, João Távora wrote:

> That and the fact they aren't aware they're anywhere from 5x to
> 10x slower, sometimes more (I measured 18x for decently realistic),
> than what they could be.

seq can surely be optimized, and I'm skeptical the issue lies in the
generic method dispatch (which would be a fundamental problem).  It
seems more likely due to the fact that the default implementations wrap
predicate functions and macro bodies into lambdas.

I tried one of the your benchmarks, with results consistent with yours:

  (setq cc (all-completions "" obarray))
  (setq list2 (make-list 12 "shooveedoowaa"))

  (when nil
     (benchmark-run 100 (cl-set-difference cc list2 :test #'equal))
     ;(11.517403284999999 45 4.082223424999995)

     (benchmark-run 100 (seq-difference cc list2))
     ;(18.507667340999998 119 10.999239876000019)
     )

Then I tried the first optimization that comes to mind, which is to
redefine seq-doseq as

  (defmacro seq-doseq (spec &rest body)
    "Loop over a SEQUENCE, evaluating BODY with VAR bound to each of its elements.

  Similar to `dolist' but can be applied to lists, strings, and vectors.

  \(fn (VAR SEQUENCE) BODY...)"
    (declare (indent 1) (debug ((symbolp form &optional form) body)))
    `(let ((var1234 ,(cadr spec)))
       (cond
        ((listp var1234)
         (dolist ,spec ,@body))
        (t
         (seq-do (lambda (,(car spec))
                   ,@body)
                 var1234)))))

And now I get:

  (when nil
     (benchmark-run 100 (cl-set-difference cc list2 :test #'equal))
     ;(11.517403284999999 45 4.082223424999995)

     (benchmark-run 100 (seq-difference cc list2))
     ;(7.308574982 29 2.6960850209999876)
     )

> Also the fact they don't know cl-seq.el provides all that with
> a much more versatile interface, and frequently much better
> performance.

I for one like the simple aesthetics of seq.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Wed, Nov 15, 2023 at 9:21 PM Augusto Stoffel <arstoffel@gmail.com> wrote:
>
> On Wed, 15 Nov 2023 at 14:51, João Távora wrote:
>
> > That and the fact they aren't aware they're anywhere from 5x to
> > 10x slower, sometimes more (I measured 18x for decently realistic),
> > than what they could be.
>
> seq can surely be optimized, and I'm skeptical the issue lies in the
> generic method dispatch (which would be a fundamental problem).  It
> seems more likely due to the fact that the default implementations wrap
> predicate functions and macro bodies into lambdas.
>
> I tried one of the your benchmarks, with results consistent with yours:
>
>   (setq cc (all-completions "" obarray))
>   (setq list2 (make-list 12 "shooveedoowaa"))
>
>   (when nil
>      (benchmark-run 100 (cl-set-difference cc list2 :test #'equal))
>      ;(11.517403284999999 45 4.082223424999995)
>
>      (benchmark-run 100 (seq-difference cc list2))
>      ;(18.507667340999998 119 10.999239876000019)
>      )
>
> Then I tried the first optimization that comes to mind, which is to
> redefine seq-doseq as
>
>   (defmacro seq-doseq (spec &rest body)
>     "Loop over a SEQUENCE, evaluating BODY with VAR bound to each of its elements.
>
>   Similar to `dolist' but can be applied to lists, strings, and vectors.
>
>   \(fn (VAR SEQUENCE) BODY...)"
>     (declare (indent 1) (debug ((symbolp form &optional form) body)))
>     `(let ((var1234 ,(cadr spec)))
>        (cond
>         ((listp var1234)
>          (dolist ,spec ,@body))

I don't think you just skip the seq-do generic like that :-) That breaks
any sequence type based on lists.  This is exactly the problem I
was raising with too-many-generic interfaces, btw.

Try Dmitry's patch instead, the one containing seq-difference-3.  That's
more consequential (but still not fully consequential, though,
also breaks things).

> And now I get:
>
>   (when nil
>      (benchmark-run 100 (cl-set-difference cc list2 :test #'equal))
>      ;(11.517403284999999 45 4.082223424999995)
>
>      (benchmark-run 100 (seq-difference cc list2))
>      ;(7.308574982 29 2.6960850209999876)
>      )

Did you try my feature/cl-lib-improvements branch?
Give cl-nset-difference a try there.

> > Also the fact they don't know cl-seq.el provides all that with
> > a much more versatile interface, and frequently much better
> > performance.

> I for one like the simple aesthetics of seq.

Are you familiar with the cl-lib.el "aesthetics"?
The simple versions argumentless versions are no uglier than seq's,

cl-some vs seq-some
cl-every vs set-every-p
cl-set-difference vs seq-intersection

What specific thing do you like in seq.el that you don't
find in the sequence functions of cl-lib.el?

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Augusto Stoffel]

On Thu, 16 Nov 2023 at 00:28, João Távora wrote:

> I don't think you just skip the seq-do generic like that :-)

Of course you don't.  This _is not_ a patch submission :-).

What I'm pointing out is that the seq-doseq macro is probably the most
consequential bit regarding performance.  If you use a mapc and closures
where you could use a simple dolist, performance will obviously suffer.

There exist extensible macros in Emacs (such as pcase), so adding some
kind dynamic dispatch for seq-doseq should be possible.  Barring that,
one would have to define list-specific methods for 8 generic functions
that use seq-doseq, which is also not unfeasible.

> That breaks any sequence type based on lists.  This is exactly the
> problem I was raising with too-many-generic interfaces, btw.

Good point.  IMO you just can't have both lists and "types based on
lists" to work together properly; if we defined a method for
seq-difference on lists, the issue would be the same (I guess that's the
caveat you raised regarding Dmitri's patch).

I don't see much appeal in user-defined seq types (as opposed to dealing
uniformly with lists, vectors and strings, which is mildly useful).  In
any case, the user always has the options to wrap a struct or class
around their custom list types.  So "sequence types based on lists"
could just be disallowed (I suppose this was part of those long
discussions, which I admit I didn't follow closely.)

> Try Dmitry's patch instead, the one containing seq-difference-3.

That approach is fine, but it only addresses one function at a time.
(If we go that way, I suggest starting by the most useful ones:
seq-filter, seq-some and seq-reduce.)

>> I for one like the simple aesthetics of seq.
>
> Are you familiar with the cl-lib.el "aesthetics"?

Yes.

> The simple versions argumentless versions are no uglier than seq's,

As far as names are concerned, cl-remove-if-not is uglier than
seq-filter.

> cl-some vs seq-some
> cl-every vs set-every-p
> cl-set-difference vs seq-intersection

cl-set-difference is an operation on lists, not sets...

> What specific thing do you like in seq.el that you don't
> find in the sequence functions of cl-lib.el?

Like other people already said, it's more about what you _don't_ find in
seq.  The seq API is much more straightforward than cl-lib's.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Thu, Nov 16, 2023 at 2:36 PM Augusto Stoffel <arstoffel@gmail.com> wrote:
>
> On Thu, 16 Nov 2023 at 00:28, João Távora wrote:
>
> > I don't think you just skip the seq-do generic like that :-)
>
> Of course you don't.  This _is not_ a patch submission :-).
>
> What I'm pointing out is that the seq-doseq macro is probably the most
> consequential bit regarding performance.  If you use a mapc and closures
> where you could use a simple dolist, performance will obviously suffer.
>
> There exist extensible macros in Emacs (such as pcase), so adding some
> kind dynamic dispatch for seq-doseq should be possible.  Barring that,
> one would have to define list-specific methods for 8 generic functions
> that use seq-doseq, which is also not unfeasible.
>
> > That breaks any sequence type based on lists.  This is exactly the
> > problem I was raising with too-many-generic interfaces, btw.
>
> Good point.  IMO you just can't have both lists and "types based on
> lists" to work together properly; if we defined a method for
> seq-difference on lists, the issue would be the same (I guess that's the
> caveat you raised regarding Dmitri's patch).
>
> I don't see much appeal in user-defined seq types (as opposed to dealing
> uniformly with lists, vectors and strings, which is mildly useful).


User-defined sequences _are_ useful in certain problem domains.
Generators, streams, lazy sequences, these are all great things.

But in bread-and-butter Lisp (LISt Processing ;-) ) we don't
need them indeed.  And we should absolutely use the fastest
library we have.  seq.el can hardly  be it, even if you do that
shortcut (and effectively outlaw a big class of custom sequences,
presumably the thing seq.el is good at).

>   In
> any case, the user always has the options to wrap a struct or class
> around their custom list types.  So "sequence types based on lists"
> could just be disallowed (I suppose this was part of those long
> discussions, which I admit I didn't follow closely.)

Yes, exactly.  And sure yes, but in many cases the lingering
performance overhead of generic functions still remains, unless
you go all the way to the entry point and break compatibility
there too.  See my latest benchmarks i sent in the Dmitry thread.

> > Try Dmitry's patch instead, the one containing seq-difference-3.
>
> That approach is fine, but it only addresses one function at a time.
> (If we go that way, I suggest starting by the most useful ones:
> seq-filter, seq-some and seq-reduce.)

Feel free to start your branch from feature/cl-lib-improvements
and keep posting benchmarks.

> As far as names are concerned, cl-remove-if-not is uglier than
> seq-filter.

Ha, this one haunts me from the early 2000's

"filter" for me personally has, for as long as I remember, been
a pain in the brain.  I can never tell if is is meant to keep
things that verify a predicate or throw away such things.  I
still can't, I always have to look it up.

like: "filtering coffee" leaves the bad stuff behind.
"filtering Yukon river water" leaves the good stuff behind (gold).

So while I have to agree that cl-remove-if-not is a pain to type,
it's very clear as to what it does when you read it.   Have long
argued it should have been called "keep-if" though.

> > cl-some vs seq-some
> > cl-every vs set-every-p
> > cl-set-difference vs seq-intersection
>
> cl-set-difference is an operation on lists, not sets...

So is seq-intersection.  Also not on sets.

> > What specific thing do you like in seq.el that you don't
> > find in the sequence functions of cl-lib.el?
> Like other people already said, it's more about what you _don't_ find in
> seq.  The seq API is much more straightforward than cl-lib's.

See my message to Dmitry.  You think the super-special
seq-remove-at-position is a good replacement for
the much more versatile cl-remove?  Bah.

But hey, if it weren't for the performance aspect  I would
absolutely surely 1000% not mind you using seq.el in
your code or in Emacs.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 16/11/2023 16:36, Augusto Stoffel wrote:
>> That breaks any sequence type based on lists.  This is exactly the
>> problem I was raising with too-many-generic interfaces, btw.
> Good point.  IMO you just can't have both lists and "types based on
> lists" to work together properly; if we defined a method for
> seq-difference on lists, the issue would be the same (I guess that's the
> caveat you raised regarding Dmitri's patch).

Actually, we might not allow custom sequences based on lists to begin 
with. E.g. if there are two definitions

  (cl-defmethod seq-length ((l (head :m6-sparse)))

and

  (cl-defmethod seq-length ((l list))

...is there a well-defined ordering between them which would determine 
the one that will be used for (cons :m6-sparse nil)?

If not, the optimization proposed by Augusto should be quite valid. And 
whatever custom sequences are added later, should use cl-defstruct or 
similar.

>> Try Dmitry's patch instead, the one containing seq-difference-3.
> That approach is fine, but it only addresses one function at a time.
> (If we go that way, I suggest starting by the most useful ones:
> seq-filter, seq-some and seq-reduce.)

My patch should help all the set operations (difference, union, 
intersection). And seq-uniq too.

Anyway, there is no reason not to use both optimizations together.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Thu, Nov 16, 2023 at 3:43 PM Dmitry Gutov <dgutov@yandex.ru> wrote:
>
> On 16/11/2023 16:36, Augusto Stoffel wrote:
> >> That breaks any sequence type based on lists.  This is exactly the
> >> problem I was raising with too-many-generic interfaces, btw.
> > Good point.  IMO you just can't have both lists and "types based on
> > lists" to work together properly; if we defined a method for
> > seq-difference on lists, the issue would be the same (I guess that's the
> > caveat you raised regarding Dmitri's patch).
>
> Actually, we might not allow custom sequences based on lists to begin
> with. E.g. if there are two definitions
>
>   (cl-defmethod seq-length ((l (head :m6-sparse)))
>
> and
>
>   (cl-defmethod seq-length ((l list))
>
> ...is there a well-defined ordering between them which would determine
> the one that will be used for (cons :m6-sparse nil)?

The former is more specific and kicks in first.  So it's an implicit
interface already even if it not described anywhere (but it probably
is, I didn't check).

And then there's the :around, :before, etc methods.  Those are perfectly
well defined.

> If not, the optimization proposed by Augusto should be quite valid. And
> whatever custom sequences are added later, should use cl-defstruct or
> similar.

Looking forward to see benchmarks (and maybe some unit tests) based
on the latest branch.  If you can, please use my joaot/bench macro
it makes all this data easier to digest.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Eli Zaretskii wrote:

> We use keyword arguments when there are many of them, or
> when a few of them are obscure and rarely used. From where
> I stand, the dispute is not about banishing then -- that
> will not happen -- the dispute is about how _much_ to
> use them.

But doesn't that mean there has to be rules for every
individual library? And what kind of rule, for example for
cl-lib? "You are allowed to use it - but not too much"?
How are you going to formulate and enforce such rules?

Well, anyway, then yes - it is not the way I think about
programming and libraries. If there is a library, one should
be allowed to use it as much or little as is needed to improve
one's code.

If everyone is instead supposed to write their own things
instead of using the library solution then it means there will
be code everywhere doing the same thing, almost, and in
a slightly new way for every time it is implemented.

I see that as more complexity and more time spent trying to
understand other people's code, not less.

But yes, the Emacs maintainers have to be familiar with the
libraries being shipped with and included in Emacs. There is
no way around it.

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Sat, 11 Nov 2023 16:23:18 +0100
> 
> Eli Zaretskii wrote:
> 
> > We use keyword arguments when there are many of them, or
> > when a few of them are obscure and rarely used. From where
> > I stand, the dispute is not about banishing then -- that
> > will not happen -- the dispute is about how _much_ to
> > use them.
> 
> But doesn't that mean there has to be rules for every
> individual library? And what kind of rule, for example for
> cl-lib? "You are allowed to use it - but not too much"?
> How are you going to formulate and enforce such rules?

It's a judgment call, to be made when reviewing patches.  Like many
other similar issues.  Not everything could or should be codified.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Eli Zaretskii wrote:

>> But doesn't that mean there has to be rules for every
>> individual library? And what kind of rule, for example for
>> cl-lib? "You are allowed to use it - but not too much"?
>> How are you going to formulate and enforce such rules?
>
> It's a judgment call, to be made when reviewing patches.
> Like many other similar issues. Not everything could or
> should be codified.

Sure, but it will be a lot of work for you having to think
about it and doing that judgment call every time.

You also risk alienating people who do drive-by fixes and are
unaware of or do not agree with the approach to cl-lib.
Even explaining to them in the best of friendly terms will
not guarantee they will return with a new patch. So when they
don't, it'll be even more work for you if you intend to remove
the cl-lib from their code, so it can be used.

But okay, this is how you will keep it out then.

What will happen to the code already in? Are you gonna kick
that code out?

That will again be a lot of work for you, time that could have
been spent elsewhere.

But obviously it is up to you how to spend your time and what
efforts in Emacs you prioritize.

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Sat, 11 Nov 2023 17:41:20 +0100
> 
> But obviously it is up to you how to spend your time and what
> efforts in Emacs you prioritize.

Indeed.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Po Lu wrote:

>> So it's not true what you said originally that everytime
>> you find a cl-list* it can be cleanly replaced by nconc.
>> Not if you understand or even appreciate
>> function composition.
>
> I don't appreciate "function combination," whatever that is.

Composition - it is

  y = f(g(x))

or what is sometimes called set, aggregate or most often
perhaps higher-order functions, HOFs. [1]

Yes, those are indispensible - and I think present in C as
well, actually, by means of function pointers. But they are
much easier to handle in Lisp.

Writing such functions manually with loops etc is very much
not recommended whenever there are HOFs that do that already.

[1] https://en.wikipedia.org/wiki/Higher-order_function

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

Emanuel Berg <incal@dataswamp.org> writes:

> Yes, those are indispensible - and I think present in C as
> well, actually, by means of function pointers. But they are
> much easier to handle in Lisp.

And by virtue of being ungainly in C, they are absent from programs
written in that language.  Unequivocally a change for the better.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Sat, Nov 11, 2023 at 1:50 PM Po Lu <luangruo@yahoo.com> wrote:
>
> Emanuel Berg <incal@dataswamp.org> writes:
>
> > Yes, those are indispensible - and I think present in C as
> > well, actually, by means of function pointers. But they are
> > much easier to handle in Lisp.
>
> And by virtue of being ungainly in C, they are absent from programs
> written in that language.

Wooot?  I don't dispute they're "ungainly", but clearly, you've
never used qsort of stdlib.h or so, so many other examples.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> Wooot?  I don't dispute they're "ungainly", but clearly, you've
> never used qsort of stdlib.h or so, so many other examples.

qsort doesn't chain multiple functions together or rewrite the natural
control flow of an expression, it accepts a single callback used to
compare two elements of an array.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Po Lu wrote:

> There is no reason to require a 21 KB file for a six line
> function. None.

Yes, one: If that six line function is useful and is defined
in a readily available library of 21 KB.

Sure, one can break up that library do decrease its modules'
sizes, but this is up to the people at the library's side, not
the person writing code and using the library, because s/he is
using it as intended.

Especially if the library is readily available, like cl-lib,
there is no hesitation. If it has to be downloaded from the
web or by means of some package manager etc then the practical
reasons not to do it appear, so maybe it makes sense in one
sense, but isn't reasonable practically to do, for just
six lines.

But again, that isn't the case with cl-lib. It is already
there and it is very easy to use to improve your code.
Even for a single file and a single use case, what would be
the reason no to do it?

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

João Távora wrote:

> But I can agree it's not worth requiring cl-lib.el just for
> cl-list* alone.

Some people will say one should write as good code as
possible, and `require' whatever necessary for that.

I can agree bringing in external packages and such should be
avoided if one can alter just some detail so to not have to do
that, but cl-lib.el is not external in any way. In this
directory, where the cl-lib files are

  /usr/local/share/emacs/30.0.50/lisp/emacs-lisp

are 102 .el files.

Are we gonna have individual rules for 102 files? What files
are to be used and what aren't?

I think there should be only one rule, write as good code you
can and 'require' any file you need to achieve this.
These files are there because they are useful and there is no
harm interconnecting them.

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> Are _you_ the suit?  I missed the point of most of your very
> elegant text, but isn't policy-making exactly what you (and I) are
> trying to do, i.e. discourage some things and encourage others?

I'm not advocating partitioning Elisp into compartments, and to give
countenance to some of them on the grounds of the stature of their
designers.  That's your proposal.  My synthesis is roughly as follows:

  1. cl-lib is a comparatively large file.

  2. cl-lib is frequently loaded to provide trivial functions
     which could effortlessly be replaced by built-in constructs.

  3. cl-lib also defines a number of functions imported from a different
     programming language, the beliefs underlying whose design don't
     align with those of Emacs.

Therefore, we should discourage cl-lib from being introduced in the
first place, because its proliferation will render programmers more
eager to use it without considering its implications.

> What's the gripe with compartments?  We don't have all of Elisp in
> a single library?  I was talking about namespaces by the way,
> if you didn't catch the reference (admittedly obscure)

Because Elisp isn't naturally partitioned into "compartments."  The
basic unit into which our code is divided is the file, and all
considerations must be framed on that basis.

> Also, is really cl-set-difference to take the difference
> of two sets worse than writing a "hallowed" multi-line
> construct??  Maybe I'm not seeing what the construct is, so
> what is your preferred Elisp hallowed way to take the
> difference of two sets, say make a list  that contains
> all items that appear in LIST1 but not LIST2?  Or to find
> the index of a certain element of a sequence...

dolist with a catch/throw, or dolist, member and push?

> These comparisons, taken one by one, are the hard evidence of that
> allows us to determine what is "alien" or not, not theological
> consecrations of personal tastes of a given group, _any_ group.
> For example, I don't think your 'nconc' form is more readable
> than 'cl-list*'.  It's slightly less readable IMO.  We could alias
> `list*` to nconc though, even better.  But I can agree it's not
> worth requiring cl-lib.el just for cl-list* alone.
>
> Much like you prefer setcar to rplacd. Which I also do, btw.  But
> I prefer (setf (car ...))  even more.  But you'll frequently see
> me using setcar and setcdr (or even rplacwahtevercannevertypeit)
> if that's the style of the file I'm working on.  Because I too
> have my personal tastes, but contrary to others I can let go of
> them without too much fuss.

My point was that comparing rplaca to cl-lib is much the same as
comparing apples to oranges, since rplaca is an _alias_ to setcar.

> Stefan M's pcase in particular is great.  Its docstring is tough,
> I agree, and could use some examples (I always end up grepping
> for examples).  But that macro is a lifesaver.
>
> The way you talk about seq.el and pcase.el makes me think you
> want to ban _all the things_, not just cl-lib.  I'm confused about
> what you want Elisp code in core to look like.

Well, touch-screen.el is a good example of that.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 10, 2023 at 12:32 AM Po Lu <luangruo@yahoo.com> wrote:
>
> João Távora <joaotavora@gmail.com> writes:
>
> > Are _you_ the suit?  I missed the point of most of your very
> > elegant text, but isn't policy-making exactly what you (and I) are
> > trying to do, i.e. discourage some things and encourage others?
>
> I'm not advocating partitioning Elisp into compartments, and to give
> countenance to some of them on the grounds of the stature of their
> designers.  That's your proposal.

What the heck?  What I said, is that if Elisp _had_ better
ways of making compartments, such as namespaces, managing which
compartments are preferred in certain parts of code would be easier.
I stated multiple times that I tend to trust developers to pick
the tools they find most appropriate for the job.  I simply stated
that it's funny and ironic that the language whose influence you're
trying to exorcize is precisely the one with one of the best
tools to do that exorcizing.

Maybe you didn't understand this because you're not familiar with
CL packages, so I can explain.  In CL you can actually write a program
in a package that doesn't ':USE' the :COMMON-LISP package at all, or
that only imports a handful of symbols from it (maybe the sacred
Elisp subset).  Or you can make your own packages with just the
symbols you want and maybe use that.  Meaning you'd be effectively
insulated from CL or at least the parts you don't like and could
write your perfect purist Lisp program.   Come on, isn't that funny?

>   3. cl-lib also defines a number of functions imported from a different
>      programming language, the beliefs underlying whose design don't
>      align with those of Emacs.

There are no "beliefs" in Common Lisp's, neither should there be
in Elisp.  I don't think it helps your case to repeatedly evoke
religion.

> Therefore, we should discourage cl-lib from being introduced in the
> first place, because its proliferation will render programmers more
> eager to use it without considering its implications.

Every other choice also has "implications".  Most people favourable
to cl-lib in this discussion know why they chose to use its functions.
I wouldn't be so condescending.

> > Also, is really cl-set-difference to take the difference
> > of two sets worse than writing a "hallowed" multi-line
> > construct??  Maybe I'm not seeing what the construct is, so
> > what is your preferred Elisp hallowed way to take the
> > difference of two sets, say make a list  that contains
> > all items that appear in LIST1 but not LIST2?  Or to find
> > the index of a certain element of a sequence...
>
> dolist with a catch/throw, or dolist, member and push?

Hmmm, a bit vague, no?  Humor me: if it's really so easy and so
readable please write out your preferred equivalent of, say

   (cl-set-difference l1 l2) ; hopefully obvious, like l1 - l2 in python

and

   ;; iterate through someseq and find index of first element
   ;; whose car matches probe according to 'probe-equals'.
   (cl-position probe someseq :key #'car :test #'probe-equals)

Shouldn't take long.  I can try myself, of course, but I think it's
fair to give the proponent of the alternate approach a shot before
comparing the two approaches.

> My point was that comparing rplaca to cl-lib is much the same as
> comparing apples to oranges, since rplaca is an _alias_ to setcar.

Some get confused by cl-labels, some by archaic things like rplaca,
some by new stuff like static-if.  Doesn't matter what is alias to
what, when you see both these things  in code you don't know what
they do until you look them up.  And many times you'll think "there's
a much better way".

> Well, touch-screen.el is a good example of that.

OK.  So, basically, your code :-) '(if x (progn ...))' instead
of 'cond', and lots of handrolled catch/throw with while t. OK fine.

I have a feeling I think you'd love cl-loop, maybe even too much :-)

Also quite some long functions there.  300 lines? Maybe typing
"touch-screen" each time to define and call functions
discourages you from modularizing more?  Or maybe it doesn't
bother you at all, so fine.

Don't get me wrong, I am very thankful for your contributions,
I really am.  Write Elisp code in the manner you find most
comfortable.  But there are programmers working on problem domains
in the core other than touch-screen.el, maybe with different
non-C backgrounds.  Many of them are historical contributors to
Emacs.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> What the heck?  What I said, is that if Elisp _had_ better
> ways of making compartments, such as namespaces, managing which
> compartments are preferred in certain parts of code would be easier.
> I stated multiple times that I tend to trust developers to pick
> the tools they find most appropriate for the job.  I simply stated
> that it's funny and ironic that the language whose influence you're
> trying to exorcize is precisely the one with one of the best
> tools to do that exorcizing.

Which is tantamount to, or at least comes off as, arguing in favor of
that language's influence.

> Maybe you didn't understand this because you're not familiar with
> CL packages, so I can explain.  In CL you can actually write a program
> in a package that doesn't ':USE' the :COMMON-LISP package at all, or
> that only imports a handful of symbols from it (maybe the sacred
> Elisp subset).  Or you can make your own packages with just the
> symbols you want and maybe use that.  Meaning you'd be effectively
> insulated from CL or at least the parts you don't like and could
> write your perfect purist Lisp program.   Come on, isn't that funny?

I know how packages function under Common Lisp, but they are not
salvation.  My objective is not to write code free of Common Lisp
influence; this I already do.  It is for everyone else to write them, or
provide a 95% effective guarantee that I (and many others) will never
have to read them, which is easier said than done; a leitmotif in Emacs
development is for active developers to vanish overnight, leaving a
small corps of developers with reponsibility for their upkeep in the
years that follow.

Granted, the foregoing is something of an overstatement.  But the
alacrity with which programmers seize at opportunities to employ cl-lib
does call for one.

> There are no "beliefs" in Common Lisp's, neither should there be
> in Elisp.  I don't think it helps your case to repeatedly evoke
> religion.

Really?  Here is one example to the contrary: keyword arguments.  No
Emacs Lisp built-in takes them, yet it is scarcely possible to name a
Common Lisp list manipulation function that does not.

> Every other choice also has "implications".  Most people favourable
> to cl-lib in this discussion know why they chose to use its functions.
> I wouldn't be so condescending.

Yet few of these people consider the impact their decision makes to
others, who will in due course have the responsibility for reading this
code pawned off them.

> Hmmm, a bit vague, no?  Humor me: if it's really so easy and so
> readable please write out your preferred equivalent of, say
>
>    (cl-set-difference l1 l2) ; hopefully obvious, like l1 - l2 in python

(let ((list nil))
  (dolist (x l1)
    (unless (member x l2)
      (push x list)))
  (dolist (x l2)
    (unless (member x l1)
      (push x list)))
  list)

> and
>
>    ;; iterate through someseq and find index of first element
>    ;; whose car matches probe according to 'probe-equals'.
>    (cl-position probe someseq :key #'car :test #'probe-equals)
>
> Shouldn't take long.  I can try myself, of course, but I think it's
> fair to give the proponent of the alternate approach a shot before
> comparing the two approaches.

(catch 'tag
  (let ((index 0))
    (dolist (tem someseq)
      (when (eq (car tem) probe)
        (throw 'tag index))
      (setq index (1+ index)))))

> Some get confused by cl-labels, some by archaic things like rplaca,
> some by new stuff like static-if.  Doesn't matter what is alias to
> what, when you see both these things  in code you don't know what
> they do until you look them up.  And many times you'll think "there's
> a much better way".

There's not much of a point in arguing against such patent absurdities
as considering the operation of setcar as complex as that of cl-labels,
is there?

> OK.  So, basically, your code :-) '(if x (progn ...))' instead
> of 'cond', and lots of handrolled catch/throw with while t. OK fine.
>
> I have a feeling I think you'd love cl-loop, maybe even too much :-)

I think not.

> Also quite some long functions there.  300 lines? Maybe typing
> "touch-screen" each time to define and call functions
> discourages you from modularizing more?  Or maybe it doesn't
> bother you at all, so fine.

It doesn't bother me that functions for whom there are no reasons to use
their constituents elsewhere have yet to be balkanized, no.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 10, 2023 at 3:17 AM Po Lu <luangruo@yahoo.com> wrote:
>
> João Távora <joaotavora@gmail.com> writes:
>
> > What the heck?  What I said, is that if Elisp _had_ better
> > ways of making compartments, such as namespaces, managing which
> > compartments are preferred in certain parts of code would be easier.
> > I stated multiple times that I tend to trust developers to pick
> > the tools they find most appropriate for the job.  I simply stated
> > that it's funny and ironic that the language whose influence you're
> > trying to exorcize is precisely the one with one of the best
> > tools to do that exorcizing.
>
> Which is tantamount to, or at least comes off as, arguing in favor of
> that language's influence.

No, not mounting any tantas, I was just making a joke.  But yes, of course
I like Common Lisp.  I like C++ too (not as much, tho) and it pisses off
C people, too.

> Granted, the foregoing is something of an overstatement.  But the
> alacrity with which programmers seize at opportunities to employ cl-lib
> does call for one.

But that leitmotif is everywhere (and us much less strong in open-source,
btw).  Surely if you've ever worked at a large enough company you'll see
bizarre code from people not working there anymore.  And that goes for
everything.  You think everyone will forever be grateful to read your
300-line inline-all-the-things functions in touch-screen.el when you're
not around to explain what they do?  They won't.  It makes no sense
to single out any given library for this evil, especially not cl-lib.el
which is a well-understood stable piece of kit.  In Emacs core and
just as strongly outside it.

> > There are no "beliefs" in Common Lisp's, neither should there be
> > in Elisp.  I don't think it helps your case to repeatedly evoke
> > religion.
>
> Really?  Here is one example to the contrary: keyword arguments.  No
> Emacs Lisp built-in takes them, yet it is scarcely possible to name a
> Common Lisp list manipulation function that does not.

Many examples of Emacs Lisp functions take them.  Even Richard seems
to have come around to their usefulness.  CL functions take them in
consistent ways, but you can use them without keyword arguments if
you're stubborn enough.  Anyway, it's not a "belief", it's just a
good way to provide versatility for functions.

> > Every other choice also has "implications".  Most people favourable
> > to cl-lib in this discussion know why they chose to use its functions.
> > I wouldn't be so condescending.
>
> Yet few of these people consider the impact their decision makes to
> others, who will in due course have the responsibility for reading this
> code pawned off them.

Why do you exempt yourself from this irresponsibility?  Can't you see
this is all subjective and it seems a bit arrogant to say "my code is
so responsible"??  All code is bad.

> > Hmmm, a bit vague, no?  Humor me: if it's really so easy and so
> > readable please write out your preferred equivalent of, say
> >
> >    (cl-set-difference l1 l2) ; hopefully obvious, like l1 - l2 in python
>
> (let ((list nil))
>   (dolist (x l1)
>     (unless (member x l2)
>       (push x list)))
>   (dolist (x l2)
>     (unless (member x l1)
>       (push x list)))
>   list)

>
> (catch 'tag
>   (let ((index 0))
>     (dolist (tem someseq)
>       (when (eq (car tem) probe)
>         (throw 'tag index))
>       (setq index (1+ index)))))

Nuff said :-)  .  So if multiple set difference operations or multiple
index-finding operations are needed you write these choo-choo trains
again and again.  That sure explains the 300 lines.

> > Some get confused by cl-labels, some by archaic things like rplaca,
> > some by new stuff like static-if.  Doesn't matter what is alias to
> > what, when you see both these things  in code you don't know what
> > they do until you look them up.  And many times you'll think "there's
> > a much better way".
>
> There's not much of a point in arguing against such patent absurdities
> as considering the operation of setcar as complex as that of cl-labels,
> is there?

I guess not for human versions of --finline-functions who think,
or rather "believe" that such common practices for code reuse
as subroutine encapsulation is "balkanization".  It's probably "patently absurd"
for them yes.

João Távora

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> But that leitmotif is everywhere (and us much less strong in open-source,
> btw).  Surely if you've ever worked at a large enough company you'll see
> bizarre code from people not working there anymore.

Where I work, our practices and policies prevent these difficulties from
ever materializing.  Think a standardized coding style, mandatory quotas
for documentation length, expositions written and, hmm, anthologized
independently of the code itself, and review by individuals who have,
needless to say, never seen the code before in their lives, and will not
so much as bat an eyelid before striking down code they cannot easily
understand.  The first and third are what's being proposed here for
Emacs, unless I'm greatly mistaken.

> And that goes for everything.  You think everyone will forever be
> grateful to read your 300-line inline-all-the-things functions in
> touch-screen.el when you're not around to explain what they do?  They
> won't.

It's well-documented and in line with other Emacs Lisp code, so I think
they will.  Indeed one individual is already implementing the
recognition of panning and zoom gestures with it as a basis, without any
counsel from me besides referring him to the file itself.

> It makes no sense to single out any given library for this evil,
> especially not cl-lib.el which is a well-understood stable piece of
> kit.  In Emacs core and just as strongly outside it.

See what I explained earlier.

> Many examples of Emacs Lisp functions take them.

Like?  Which list manipulation functions do?
Contrast cl-member to member.

> Even Richard seems to have come around to their usefulness.  CL
> functions take them in consistent ways, but you can use them without
> keyword arguments if you're stubborn enough.  Anyway, it's not a
> "belief", it's just a good way to provide versatility for functions.

A "belief" in this sense is a model or general scheme that is adhered to
when designing a function, of course, so let's not descend into
quibbling.

> Why do you exempt yourself from this irresponsibility?  Can't you see
> this is all subjective and it seems a bit arrogant to say "my code is
> so responsible"??

Thus far there have been no complaints that code _without_ cl-lib,
pcase, seq or elt is unreadable.  Most of our code (this time measured
by lines) falls squarely within that category.

> All code is bad.

With that outlook, there's not much of a purpose in writing any code for
Emacs, is there?

>> > Hmmm, a bit vague, no?  Humor me: if it's really so easy and so
>> > readable please write out your preferred equivalent of, say
>> >
>> >    (cl-set-difference l1 l2) ; hopefully obvious, like l1 - l2 in python
>>
>> (let ((list nil))
>>   (dolist (x l1)
>>     (unless (member x l2)
>>       (push x list)))
>>   (dolist (x l2)
>>     (unless (member x l1)
>>       (push x list)))
>>   list)
>
>>
>> (catch 'tag
>>   (let ((index 0))
>>     (dolist (tem someseq)
>>       (when (eq (car tem) probe)
>>         (throw 'tag index))
>>       (setq index (1+ index)))))
>
> Nuff said :-)  .  So if multiple set difference operations or multiple
> index-finding operations are needed you write these choo-choo trains
> again and again.  That sure explains the 300 lines.

Sure, but catch, let, dolist, when, throw, setq and member are
elementary operations under Emacs Lisp everyone learns.  They are also
few in number.

> I guess not for human versions of --finline-functions who think, or
> rather "believe" that such common practices for code reuse as
> subroutine encapsulation is "balkanization".  It's probably "patently
> absurd" for them yes.

But the Emacs Lisp compiler doesn't inline functions, does it?  At any
rate you have conflated one of my statements with the other, so read
again.

Thanks.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 10, 2023 at 1:14 PM Po Lu <luangruo@yahoo.com> wrote:

> needless to say, never seen the code before in their lives, and will not
> so much as bat an eyelid before striking down code they cannot easily
> understand.

Wow, sounds like a fantastic healthy work environment.  Do people
ever leave the company?  Like, alive?

> > Many examples of Emacs Lisp functions take them.
>
> Like?  Which list manipulation functions do?
> Contrast cl-member to member.

You didn't say "list manipulation function" . You wrote "No Emacs
built-in takes them".  That's false.  Take make-hash-table.  Oh noes,
looks just like CL.  Guess you're going to hand-roll hash tables
now, right? :-)

Take make-process or json-serialize.  There are more.  Keyword
arguments are in other Non-lisp languages too, as you probably
know, they're called named parameters sometimes.  Heck C++
has been pining for them for so long it's sad.

> > All code is bad.
>
> With that outlook, there's not much of a purpose in writing any code for
> Emacs, is there?

There is, because bad as it is, code gets jobs done that humans can't do.

> But the Emacs Lisp compiler doesn't inline functions, does it?

No, but you do, in actual source code!  That was my point!  You
have these fine abstractions to common operations like intersecting
sets, finding indices, and so much more, and you prefer to handroll
your expanded versions each time instead of going through a function
of an established library.

Very rarely, IME, are there for doing this manual inlining
even more rarely in Elisp which has a compiler-macro mechanism.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> Wow, sounds like a fantastic healthy work environment.  Do people
> ever leave the company?  Like, alive?

Sarcasm aside, most people have no trouble assenting to these
procedures.  Not when they yield good code whose meaning remains clear
for generations to come.

Dismissing the intrinsic merit to readable code should be a no-starter
hereabouts.

> You didn't say "list manipulation function" . You wrote "No Emacs
> built-in takes them".  That's false.  Take make-hash-table.  Oh noes,
> looks just like CL.  Guess you're going to hand-roll hash tables
> now, right? :-)

Hash tables don't count, because they were designed to be compatible
with the old CL hash table functions, just all that faster.

> Take make-process or json-serialize.  There are more.  Keyword
> arguments are in other Non-lisp languages too, as you probably
> know, they're called named parameters sometimes.  Heck C++
> has been pining for them for so long it's sad.

I don't know why make-process was introduced, but it enjoys the excuse
of accepting plenty of arguments.  json-serialize does not: its two
optional parameters don't even encounter the 7-argument limit for C
DEFUNs.

It is lamentable that the keyword argument illness has spread into C,
but that doesn't palliate its deficiencies.

> There is, because bad as it is, code gets jobs done that humans can't
> do.

I don't think anyone seriously believes what you claim here, in essence
that it is OK to install dismal code while being well aware of its
nature as such.

> No, but you do, in actual source code!  That was my point!  You
> have these fine abstractions to common operations like intersecting
> sets, finding indices, and so much more, and you prefer to handroll
> your expanded versions each time instead of going through a function
> of an established library.

These "fine abstractions" are also abstruse, while the set of primitives
you never hesitate to impugn have carried Emacs through its entire
existence.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Sat, Nov 11, 2023 at 12:38 AM Po Lu <luangruo@yahoo.com> wrote:

> > No, but you do, in actual source code!  That was my point!  You
> > have these fine abstractions to common operations like intersecting
> > sets, finding indices, and so much more, and you prefer to handroll
> > your expanded versions each time instead of going through a function
> > of an established library.
>
> These "fine abstractions" are also abstruse, while the set of primitives
> you never hesitate to impugn have carried Emacs through its entire
> existence.

Where in heck did I "impugn" Emacs's Lisp primitives?? I use them
all the time!!  I just think it's a really really bad idea to
use them exclusively, almost religiously, to inline existing utils
in write-only code.  When given the opportunity to demonstrate the
superiority of this approach to substitute cl-set-difference, you
failed.  It's just my opinion, of course, but that hard patent fact
that is on display in this thread, along with the hard patent fact
that I simply failed to notice your mistake -- because reading that
code takes much more effort -- is a much better demonstration
of the relative "abstrusity" of the practice you so energetically
recommend.

Like Dmitry, I don't think it's because you don't have
coding ability or intelligence -- which you most clearly do --
just that your practice lends itself to these problems _more often
than_ other more eclectic and pondered practices such as using
more programming techniques.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> Where in heck did I "impugn" Emacs's Lisp primitives?? I use them
> all the time!!  I just think it's a really really bad idea to
> use them exclusively, almost religiously, to inline existing utils
> in write-only code.  When given the opportunity to demonstrate the
> superiority of this approach to substitute cl-set-difference, you
> failed.  It's just my opinion, of course, but that hard patent fact
> that is on display in this thread, along with the hard patent fact
> that I simply failed to notice your mistake -- because reading that
> code takes much more effort -- is a much better demonstration
> of the relative "abstrusity" of the practice you so energetically
> recommend.

That failure belies the alleged advantages of cl-lib, and vindicates all
Alan and Richard have said thus far: for all their augustness, cl-lib's
designers did not select an unambiguous English word for a comparatively
elementary set operation.  Instead devolving responsibility for second
guessing their intent to such as I and they.

> Like Dmitry, I don't think it's because you don't have
> coding ability or intelligence -- which you most clearly do --
> just that your practice lends itself to these problems _more often
> than_ other more eclectic and pondered practices such as using
> more programming techniques.

Actually, a problem would have arisen had I used cl-set-difference while
under the impression that it calculated the disjunctive union of LIST1
with LIST2, as the signification of the word "difference" suggests.

Writing such code with only primitives sidesteps such pitfalls
altogether, since what is thought is directly set down.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Sat, Nov 11, 2023 at 1:04 PM Po Lu <luangruo@yahoo.com> wrote:

> That failure belies the alleged advantages of cl-lib, and vindicates all
> Alan and Richard have said thus far: for all their augustness, cl-lib's
> designers did not select an unambiguous English word for a comparatively
> elementary set operation.  Instead devolving responsibility for second
> guessing their intent to such as I and they.

First, that's such a weak pillar which remains on which to base all your
cl-lib.el fury.   Hair-splitting over a word and and bringing up a
dictionary definition written more than 100 years ago when programming
was basically non-existent.

And did you see my reply to Tomas where I listed that at least 5 other
programming languages use the same word (or at least the '-'
operator) for the same operation?  There is indeed such a thing
as set theory in mathematics and the literature uses that term
to refer to that operation.  What a bad idea it would be for Elisp to
stray from that practice just so your argument can stay afloat.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> First, that's such a weak pillar which remains on which to base all your
> cl-lib.el fury.   Hair-splitting over a word and and bringing up a
> dictionary definition written more than 100 years ago when programming
> was basically non-existent.
>
> And did you see my reply to Tomas where I listed that at least 5 other
> programming languages use the same word (or at least the '-'
> operator) for the same operation?  There is indeed such a thing
> as set theory in mathematics and the literature uses that term
> to refer to that operation.  What a bad idea it would be for Elisp to
> stray from that practice just so your argument can stay afloat.

That's one example out of many, scarcely the "only pillar on which to
lash out at cl-lib."

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Po Lu wrote:

> These "fine abstractions" are also abstruse, while the set
> of primitives you never hesitate to impugn have carried
> Emacs through its entire existence.

The abstractions elevate Elisp to new levels, and from there,
they make it easier for everyone else writing code not having
to do on the level of very basic building blocks. Relying on
wizards to juggle and combine basics like crazy to achieve
wonders isn't the way computer technology has been moving
forward, on the contrary that has been done by moving
complexity into modules and providing neat interfaces so
people only have to master the interfaces, not the details
what happens below that to carry out the desired task.

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

Emanuel Berg <incal@dataswamp.org> writes:

> The abstractions elevate Elisp to new levels, and from there,
> they make it easier for everyone else writing code not having
> to do on the level of very basic building blocks. Relying on
> wizards to juggle and combine basics like crazy to achieve
> wonders isn't the way computer technology has been moving
> forward, on the contrary that has been done by moving
> complexity into modules and providing neat interfaces so
> people only have to master the interfaces, not the details
> what happens below that to carry out the desired task.

When such interfaces number greater than the finer details, that proves
so much more challenging than expressing these details with the
elementary operations which constitute them.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Po Lu wrote:

>> The abstractions elevate Elisp to new levels, and from
>> there, they make it easier for everyone else writing code
>> not having to do on the level of very basic building
>> blocks. Relying on wizards to juggle and combine basics
>> like crazy to achieve wonders isn't the way computer
>> technology has been moving forward, on the contrary that
>> has been done by moving complexity into modules and
>> providing neat interfaces so people only have to master the
>> interfaces, not the details what happens below that to
>> carry out the desired task.
>
> When such interfaces number greater than the finer details,
> that proves so much more challenging than expressing these
> details with the elementary operations which
> constitute them.

As we have seen here and in the stats I provided earlier in
the thread how much cl-lib is used, it follows a lot of people
have been able to put it into good use. It is not too
complicated for anyone, and often on the contrary makes the
code shorter and easier to read. If a bunch of guys take on
the entire Emacs source and for that reason want a more simple
Elisp, then that's their opinion based on their particular
point of view. But cl-lib is still awesome and makes the code
better, not worse.

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

Emanuel Berg <incal@dataswamp.org> writes:

> As we have seen here and in the stats I provided earlier in
> the thread how much cl-lib is used, it follows a lot of people
> have been able to put it into good use. It is not too
> complicated for anyone, and often on the contrary makes the
> code shorter and easier to read.

For you.  Not for Alan, Richard, Eli, or me, or for scores of others.

If four active Emacs developers have expressed their discontent at an
undoubtedly popular feature, there there must be more users scratching
their heads in the face of cl-lib code among the ranks of the
disenfranchised Emacs users deterred from voicing their concerns as to
the Emacs development process by our allegedly unfamiliar development
practices, or so the frequently-cited truism goes.  The self-anointed
warriors who generally take every opportunity to fight for their
representation are uncharacteristically taciturn in this regard.  But
that is by the by, since ultimately the practices we adopt are subject
to the concerns of people who must read this code; and alas, very few of
our users do, for largely unrelated reasons.

> If a bunch of guys take on the entire Emacs source and for that reason
> want a more simple Elisp, then that's their opinion based on their
> particular point of view. But cl-lib is still awesome and makes the
> code better, not worse.

You are entitled to that view, but when this "bunch of guys" have to
tread through such code, they are, in like fashion, entitled to a say in
how it is written.  Correct?

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Po Lu wrote:

> You are entitled to that view, but when this "bunch of guys"
> have to tread through such code, they are, in like fashion,
> entitled to a say in how it is written. Correct?

Yes, but all the people who have used it then, whose code are
now all over Emacs? Don't their view count as well?

cl-lib is included with Emacs. They have used a library that
is shipped with Emacs because they thought that was the best
way to solve whatever situation they had. Now those solutions
are in place, working.

This speaks very strongly in favor of cl-lib.

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

Emanuel Berg <incal@dataswamp.org> writes:

> Yes, but all the people who have used it then, whose code are
> now all over Emacs? Don't their view count as well?

Nobody said theirs doesn't, but it is within their power not to write
such code as well.  Thus, it is not reasonable to tax us for attempting
to exact concessions they can easily grant, and will ultimately make
life easier for us.

> cl-lib is included with Emacs. They have used a library that
> is shipped with Emacs because they thought that was the best
> way to solve whatever situation they had. Now those solutions
> are in place, working.
>
> This speaks very strongly in favor of cl-lib.

It does not.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Po Lu wrote:

>> cl-lib is included with Emacs. They have used a library
>> that is shipped with Emacs because they thought that was
>> the best way to solve whatever situation they had.
>> Now those solutions are in place, working.
>>
>> This speaks very strongly in favor of cl-lib.
>
> It does not.

It does and any anti cl-lib policy will just alienate people
using it for now reason.

Telling people it is better to hand-roll with basic Elisp than
to use libraries is bad technology and a very unwise
alienating move for no reason.

Especially since there is nothing to do about it, the
situation is the way it is. So even less reason.

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

João Távora wrote:

> No, but you do, in actual source code! That was my point!
> You have these fine abstractions to common operations like
> intersecting sets, finding indices, and so much more, and
> you prefer to handroll your expanded versions each time
> instead of going through a function of an
> established library.

Yes, this is exactly what will happen if one doesn't rely on
libraries. Everyone will handroll their versions of the same
thing. Just looking at that it is easy to say "nice code,
I know all these functions, that code is easy to maintain" but
actually all those versions aren't quite the same, and they
will start to drift even more afar. Some dude will think, "oh,
I understand this from somewhere else already" but it isn't
exactly the same, just something close to what he previously
saw. Replace all those with one library function - yes,
everyone will have to learn that function but it will be much
better code and also easier to maintain since it is much more
reliable, a neat module that always does what it does.

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]



  > You didn't say "list manipulation function" . You wrote "No Emacs
  > built-in takes them".  That's false.  Take make-hash-table.  Oh noes,
  > looks just like CL.  Guess you're going to hand-roll hash tables
  > now, right? :-)

The tone of your message is harsh and contemptuous.  You can disagree
with him, nut please disagree respectfully.

It is not unusual for people who have basically valid points to
overstate them and thus make an incorrect assertion.  We all make such
minor mistakes like that.  To keep the discussion civil, and aim for a
useful conclusion, we should not exaggerate the importance of those
minor mistakes.

The usefuk goal is to see what the real difference is -- even if it
is not a simple absolute rule without exceptions.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 10/11/2023 15:14, Po Lu wrote:
>> Why do you exempt yourself from this irresponsibility?  Can't you see
>> this is all subjective and it seems a bit arrogant to say "my code is
>> so responsible"??
> Thus far there have been no complaints that code_without_  cl-lib,
> pcase, seq or elt is unreadable.  Most of our code (this time measured
> by lines) falls squarely within that category.

Without having a specific enemy to blame, when the function is long, 
people just either admit out loud they find the code difficult or 
silently avoid changing it.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

Dmitry Gutov <dmitry@gutov.dev> writes:

> Without having a specific enemy to blame, when the function is long,
> people just either admit out loud they find the code difficult or
> silently avoid changing it.

The length of the function plays a far lesser role in one's perception
of a function's complexity than the quantity (or scarcity) of commentary
and the number of unfamiliar constructs it employs do.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Dmitry Gutov wrote:

>>> Why do you exempt yourself from this irresponsibility?
>>> Can't you see this is all subjective and it seems a bit
>>> arrogant to say "my code is so responsible"??
>>
>> Thus far there have been no complaints that code_without_
>> cl-lib, pcase, seq or elt is unreadable. Most of our code
>> (this time measured by lines) falls squarely within
>> that category.
>
> Without having a specific enemy to blame, when the function
> is long, people just either admit out loud they find the
> code difficult or silently avoid changing it.

The way people have been writing Elisp in functions that are so
long it is insane always puzzled me. I mean, not because they
were so long - just look at the Gnus source for example - but
why they were written like that to begin with?

Also the files themselves are so long? (Maybe because they are
made up of so long functions LOL.)

In my own Elisp I never had that problem of almost endless
functions. But I'm unsure why, maybe people solved much more
difficult problems then? *impressed* *envious*

But I think cl-lib and pcase should make the code more compact
(i.e., shorter lines for the same job).

So yeah, it is less complexity in the sense fewer lines and
more standard library usage.

But also more complexity in the sense people have to learn the
basics of cl-lib and pcase, at the very least, including their
mini-languages (e.g. `cl-loop'), and further not be frustrated
by what they can't learn by heart the way they once mastered
Elisp but must look up in docstrings when encountered.

One can also hope that the most common use cases of cl-lib and
pcase will in time be idiomatic as well - so even tho you
don't know the whole library or construct by heart, the way
they are usually applied, that will be familiar enough to the
trained eyes of Emacs maintainers and contributors.

For example, I'm sure many people are fond of their scientific
calculators. And are quite fluent with those. But no one knows
what all these functions and buttons, every single one,
actually do, right?

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 10/11/2023 05:17, Po Lu wrote:
>> Hmmm, a bit vague, no?  Humor me: if it's really so easy and so
>> readable please write out your preferred equivalent of, say
>>
>>     (cl-set-difference l1 l2) ; hopefully obvious, like l1 - l2 in python
> (let ((list nil))
>    (dolist (x l1)
>      (unless (member x l2)
>        (push x list)))
>    (dolist (x l2)
>      (unless (member x l1)
>        (push x list)))
>    list)

Just to take this example, there are about 2 major bugs: 1) the list 
will come out reversed, 2) the code does XOR rather than subtraction.

To be clear, this is not to take a dig at your coding ability (which I'm 
convinced is quite high), but rather to make a point about the potential 
for problems when one uses lower-level tools and has to reinvent the wheel.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

Dmitry Gutov <dmitry@gutov.dev> writes:

> Just to take this example, there are about 2 major bugs: 1) the list
> will come out reversed, 2) the code does XOR rather than subtraction.

Hmm, reading its doc string, I think a more illustrative name for
cl-set-difference would be cl-set-complement.  In this case, the second
loop can be removed.

If the order is significant (it seldom is), the list can be nreversed
afterwards.  Or alternatively a local variable can be maintained holding
the last cdr of the list, and insertion performed on that instead.

Re: What's missing in ELisp that makes people want to use cl-lib?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 569 bytes --]

On Sat, Nov 11, 2023 at 09:23:26AM +0800, Po Lu wrote:
> Dmitry Gutov <dmitry@gutov.dev> writes:
> 
> > Just to take this example, there are about 2 major bugs: 1) the list
> > will come out reversed, 2) the code does XOR rather than subtraction.
> 
> Hmm, reading its doc string, I think a more illustrative name for
> cl-set-difference would be cl-set-complement.  In this case, the second
> loop can be removed.

Yes, "difference" is, surprisingly, ambiguous. Your implementation is
usually called "symmetric difference" to avoid that.

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Sat, Nov 11, 2023 at 6:12 AM <tomas@tuxteam.de> wrote:
>
> On Sat, Nov 11, 2023 at 09:23:26AM +0800, Po Lu wrote:
> > Dmitry Gutov <dmitry@gutov.dev> writes:
> >
> > > Just to take this example, there are about 2 major bugs: 1) the list
> > > will come out reversed, 2) the code does XOR rather than subtraction.
> >
> > Hmm, reading its doc string, I think a more illustrative name for
> > cl-set-difference would be cl-set-complement.  In this case, the second
> > loop can be removed.
>
> Yes, "difference" is, surprisingly, ambiguous.

I wouldn't say "ambiguous", but this page has a brief review of
different ways the operation is referred to in literature.

   https://proofwiki.org/wiki/Definition:Set_Difference/Also_known_as

In programming languages "set difference" seems to be the
overwhelmingly established term

  https://cplusplus.com/reference/algorithm/set_difference/
  http://clhs.lisp.se/Body/f_set_di.htm
  https://docs.python.org/3/library/stdtypes.html#set
  https://ruby-doc.org/core-1.9.3/Array.html#method-i-2D
  https://doc.rust-lang.org/std/collections/struct.HashSet.html#method.difference

> Your implementation is
> usually called "symmetric difference" to avoid that.

Correct.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Michael Heerdegen]

Po Lu <luangruo@yahoo.com> writes:

> Dmitry Gutov <dmitry@gutov.dev> writes:
>
> > Just to take this example, there are about 2 major bugs: 1) the list
> > will come out reversed, 2) the code does XOR rather than subtraction.
>
> Hmm, reading its doc string, I think a more illustrative name for
> cl-set-difference would be cl-set-complement.  In this case, the second
> loop can be removed.

BTW, a "set.el" library is on our wishlist, complementing seq.el and
map.el.

Michael.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Michael Heerdegen wrote:

>> Hmm, reading its doc string, I think a more illustrative
>> name for cl-set-difference would be cl-set-complement.
>> In this case, the second loop can be removed.
>
> BTW, a "set.el" library is on our wishlist, complementing
> seq.el and map.el.

Good idea - but are you allowed to use cl-lib from it?

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Michael Heerdegen]

Emanuel Berg <incal@dataswamp.org> writes:

> > BTW, a "set.el" library is on our wishlist, complementing
> > seq.el and map.el.
>
> Good idea - but are you allowed to use cl-lib from it?

That would not make much sense.

Michael.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Michael Heerdegen wrote:

>>> BTW, a "set.el" library is on our wishlist, complementing
>>> seq.el and map.el.
>>
>> Good idea - but are you allowed to use cl-lib from it?
>
> That would not make much sense.

Why not? Because then cl-lib would have to be loaded and we
are not allowed to do that?

But even so, tell me then, how will additional functions
dealing with sets, supposedly added to set.el, how would they
not contribute to the increased complexity of Emacs Lisp the
same way cl-lib has?

A computer program is a bunch of files, this whole strategy is
putting a limit how good an individual file can be, because
that increases the overall complexity over a large amount of
such files, that a few maintainers have taken on to maintain.

That is a problem on the human side. And not all humans side,
even, as all the people who have used cl-lib in their source
and in their source for Emacs are, of course, humans as well.
So there are two different human perspectives.

On the technology side alone, if a file can be improved by
using a library, how is that not the right thing?

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Michael Heerdegen]

Emanuel Berg <incal@dataswamp.org> writes:

> >> Good idea - but are you allowed to use cl-lib from it?
> >
> > That would not make much sense.
>
> Why not? Because then cl-lib would have to be loaded and we
> are not allowed to do that?

cl-lib has not really good support for set operations.  I has some
limited support for performing set operations on lists when interpreting
these as set specifications.  There is little sense in requiring this
large suite just for these three functions.

> But even so, tell me then, how will additional functions
> dealing with sets, supposedly added to set.el, how would they
> not contribute to the increased complexity of Emacs Lisp the
> same way cl-lib has?

cl-lib only implements a small subset on what is needed.  Sets are
useful mostly to provide a test "is some thing in the set?".  A natural
way to implement this feature efficiently is using hash tables.  cl-lib
has literally nothing to offer here.  There are only the functions for
lists.

And "set" can be a useful abstraction to improve readability, so having
a dedicated library providing efficient implementations would be a good
thing.


Michael.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Mon, Nov 13, 2023 at 8:53 AM Michael Heerdegen
<michael_heerdegen@web.de> wrote:
>
> Emanuel Berg <incal@dataswamp.org> writes:
>
> > >> Good idea - but are you allowed to use cl-lib from it?
> > >
> > > That would not make much sense.
> >
> > Why not? Because then cl-lib would have to be loaded and we
> > are not allowed to do that?
>
> cl-lib has not really good support for set operations.  I has some
> limited support for performing set operations on lists when interpreting
> these as set specifications.  There is little sense in requiring this
> large suite just for these three functions.

C.f. my benchmarks.  But I agree it would be a good thing to split it up.
into, say cl-list-set.el, I've been working on this code, it's not particularly
difficult.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Michael Heerdegen wrote:

> There is little sense in requiring this large suite just for
> these three functions.

The sense is: then you don't have to write these three
functions again, locally :)

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Sat, Nov 11, 2023 at 1:23 AM Po Lu <luangruo@yahoo.com> wrote:
>
> Dmitry Gutov <dmitry@gutov.dev> writes:
>
> > Just to take this example, there are about 2 major bugs: 1) the list
> > will come out reversed, 2) the code does XOR rather than subtraction.
>
> Hmm, reading its doc string, I think a more illustrative name for
> cl-set-difference would be cl-set-complement.

No, it's called "set difference" in CL and in other languages for a reason:

   https://proofwiki.org/wiki/Definition:Set_Difference

> If the order is significant (it seldom is),

You don't know that, of course.

> Or alternatively a local variable can be maintained holding
> the last cdr of the list, and insertion performed on that instead.

choo-choo!

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> No, it's called "set difference" in CL and in other languages for a reason:
>
>    https://proofwiki.org/wiki/Definition:Set_Difference

With Emacs, Webster's 1913 calls the shots to a far greater extent than
that does:

   1. The act of differing; the state or measure of being
      different or unlike; distinction; dissimilarity;
      unlikeness; variation; as, a difference of quality in
      paper; a difference in degrees of heat, or of light; what
      is the difference between the innocent and the guilty?
      [1913 Webster]

The difference between the innocent and the guilty is the culpability of
the latter.

   (cl-set-difference '(innocent) '(innocent culpable))

> You don't know that, of course.

Yes, I do: I've written much Emacs Lisp over the years.

> choo-choo!

Sure, if that's your preferred sobriquet for the customary practice as
regards the creation of linked lists.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

Po Lu <luangruo@yahoo.com> writes:

> Respect for volunteers is no doubt laudable, but it is very much an
> position to be considered alone which everything you've presented above
> does not impact and nobody takes exception to.  From it, and even
> supposing for the sake of argument that everything else you've said is
> incontrovertible fact, it doesn't really follow that we should not seek
> to reduce the incidence of cl-lib usage in core code.  Now I can't be
> certain that is the object of your militating, if you will, though
> promoting cl-set-difference and cl-position as alternatives to hallowed
> Emacs Lisp constructs does suggest so, but the principle argument
                                                 ---------

Principal, sorry.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Po Lu <luangruo@yahoo.com> writes:

> Respect for volunteers is no doubt laudable, but it is very much an
> position to be considered alone which everything you've presented above
> does not impact and nobody takes exception to.  From it, and even
> supposing for the sake of argument that everything else you've said is
> incontrovertible fact, it doesn't really follow that we should not seek
> to reduce the incidence of cl-lib usage in core code.  Now I can't be
> certain that is the object of your militating, if you will, though
> promoting cl-set-difference and cl-position as alternatives to hallowed
> Emacs Lisp constructs does suggest so, but the principle argument
> against these constructs is that they are alien to the small group of
> individuals charged with trudging through code where people want to make
> use of them.  So no contentions resting on their virtues and
> distinctions rather than their prevalence among us can hold water...

(Sorry if I don't cite the whole mail. Please re-add what you find
important.)

Could you perhaps formulate a plan for what should be done? I personally
find something concrete much easier to handle than a back-and-forth of
arguments.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Could you perhaps formulate a plan for what should be done? I personally
> find something concrete much easier to handle than a back-and-forth of
> arguments.

First off, I think people should make it clear that new code saturated
with calls to cl-lib will not be accepted.  Once people are coaxed out
of this habit, it will be possible to start removing calls to cl-lib
from existing code.

This will also afford us opportunities to move frequently used cl-lib
constructs that aren't all that adventitious to subr.el.  If all goes
well, it will be feasible to use Emacs without loading cl-lib at all.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 10, 2023 at 10:42 AM Po Lu <luangruo@yahoo.com> wrote:
>
> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
> > Could you perhaps formulate a plan for what should be done? I personally
> > find something concrete much easier to handle than a back-and-forth of
> > arguments.
>
> First off, I think people should make it clear that new code saturated
> with calls to cl-lib will not be accepted.  Once people are coaxed out
> of this habit, it will be possible to start removing calls to cl-lib
> from existing code.
>
> This will also afford us opportunities to move frequently used cl-lib
> constructs that aren't all that adventitious to subr.el.  If all goes
> well, it will be feasible to use Emacs without loading cl-lib at all.

Just a quick question.  Does all of this go for seq.el, map.el and
pcase.el as well, or just cl-lib?

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> Just a quick question.  Does all of this go for seq.el, map.el and
> pcase.el as well, or just cl-lib?

It's not a requirement for pcase to be loaded in code which makes use of
it, and the latter two are (I believe) preloaded.  So only cl-lib.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Po Lu <luangruo@yahoo.com> writes:

> João Távora <joaotavora@gmail.com> writes:
>
>> Just a quick question.  Does all of this go for seq.el, map.el and
>> pcase.el as well, or just cl-lib?
>
> It's not a requirement for pcase to be loaded in code which makes use of
> it, and the latter two are (I believe) preloaded.  So only cl-lib.

I don't understand that. If cl-lib were preloaded, it would also be no
problem? 

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> I don't understand that. If cl-lib were preloaded, it would also be no
> problem? 

That's an invitation to use it, which will lead to more cryptic cl-lib
code.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Po Lu <luangruo@yahoo.com> writes:

> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
>> I don't understand that. If cl-lib were preloaded, it would also be no
>> problem? 
>
> That's an invitation to use it, which will lead to more cryptic cl-lib
> code.

Aha, but seq/map are okay because they are preloaded.
I guess I don't have to understand everything.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Aha, but seq/map are okay because they are preloaded.
> I guess I don't have to understand everything.

I don't think they are, but their incursion into our code is
accomplished fact.  So we can only bow to fate...

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: João Távora <joaotavora@gmail.com>,  Alan Mackenzie
>  <acm@muc.de>,
>  Dmitry Gutov <dmitry@gutov.dev>,  Björn Bidar
>  <bjorn.bidar@thaodan.de>, emacs-devel <emacs-devel@gnu.org>
> Date: Sat, 11 Nov 2023 07:34:49 +0100
> 
> Po Lu <luangruo@yahoo.com> writes:
> 
> > Gerd Möllmann <gerd.moellmann@gmail.com> writes:
> >
> >> I don't understand that. If cl-lib were preloaded, it would also be no
> >> problem? 
> >
> > That's an invitation to use it, which will lead to more cryptic cl-lib
> > code.
> 
> Aha, but seq/map are okay because they are preloaded.

map.el isn't preloaded.  Only seq.el is.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Augusto Stoffel]

On Sat, 11 Nov 2023 at 10:01, Eli Zaretskii wrote:

> map.el isn't preloaded.  Only seq.el is.

By the way, map.el has some serious flaws that should be fixed before it
becomes more widespread (I've filed a couple of bugs some time ago).

The main issue is that it treats maps as mutable objects rather than
values, so it only feels natural when used on hash tables.  It's
cumbersome when operating on alists and plists.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Just a quick question.  Does all of this go for seq.el, map.el and
  > pcase.el as well, or just cl-lib?

Those cases have similarities, but also differences.  The cl functions
cover a much bigger range of activities than seq.e or map.el.

pcase is just one construct -- but it is a very confusing one that I
can't grasp from its doc string.  The basic idea is a natural thing to
want, but the details are so unnatural that I can't figure them out.

I am sure we can find a cleaner way to express the idea of trying a
series of patterns against one or more objects until one matches.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Po Lu <luangruo@yahoo.com> writes:

> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
>> Could you perhaps formulate a plan for what should be done? I personally
>> find something concrete much easier to handle than a back-and-forth of
>> arguments.
>
> First off, I think people should make it clear that new code saturated
> with calls to cl-lib will not be accepted.  Once people are coaxed out
> of this habit, it will be possible to start removing calls to cl-lib
> from existing code.
>
> This will also afford us opportunities to move frequently used cl-lib
> constructs that aren't all that adventitious to subr.el.  If all goes
> well, it will be feasible to use Emacs without loading cl-lib at all.

Thanks!

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

João Távora <joaotavora@gmail.com> writes:

> On Thu, Nov 9, 2023 at 10:05 AM Alan Mackenzie <acm@muc.de> wrote:
>
>> >    If it needed any confirmation, I too like cl-lib and I too help
>> >    maintain other people's code in the Emacs core tree as well as
>> >    maintaining a number of libraries I have authored.
>>
>> There's a difference between liking cl-lib and advocating its
>> indiscriminate use.  I don't think you've done the latter in this (and
>> related) threads.
>
> Yes, you're right.  Indeed I don't' advocate for its indiscriminate use,
> just as I don't advocate for indiscriminate use of anything, except
> perhaps drinking water and brushing teeth.
>
>> Nobody who likes cl-lib has yet addressed the point made by Richard and
>> (less eloquently) by me, namely that the incorporation and use of cl-lib
>> swells the size and complexity of Emacs Lisp to the point of making
>> maintenance difficult.  What is your view on this important point?
>
> That it doesn't make maintenance any more difficult than any other
> Elisp construct, be it very old and curiously named like 'rplacd' or
> much, much newer like `seq-do` or `pcase-lambda`.
>
> My specific view on cl-lib.el is that it brings us a small part of
> the results of  non-trivial design work put in when important figures
> in the Lisp world met  regularly for many years to deliver what has
> proved to become excellent,  battle-tested, widely understood and
> impeccably documented programming abstractions.
>
> What I'm reading so far in this long discussion is that the argument
> of its detractors isn't really that cl-lib isn't good, but that
> it is superfluous and that learning it is a burden on maintainers.
> Well, it's just as superfluous as all of Elisp apart from two handfuls
> of primitives, I guess.  Or any programming language for that matter, if
> you know enough machine code.  Or any other programming abstraction I
> happen not to be familiar with.
>
> Also I seem to hear that Elisp has some kind of ideal hard-to-define
> identity or fingerprint and that it shouldn't try to become anything
> else.  But this argument is very strange given Elisp is, like any
> decent language, easy to extend.  Not to mention I struggle to see
> the technical advantage in uniqueness for uniqueness sake.  A good
> part of Elisp is about a special purpose function: editing buffers,
> and an the equally important part is about doing things with bits in
> memory, there's no reason to advocate for singularity here.
>
> I also hear Elisp shouldn't become Common Lisp, but not only is the
> use of cl-lib.el nowhere a step in that direction, but also -- somewhat
> ironically -- if Elisp _were_ Common Lisp, then that hard-to-define
> identity would be much easier to define and language extension would be
> much easier to organize into compartments to facilitate policy-making.
>
> Again, the only thing that has brought Elisp any closer to Common
> Lisp significantly, was the introduction of lexical binding some 10
> years ago.  Elisp looks a lot different now than it did in the 90's.
> Closures everywhere, higher-order functions!  Shudder!
>
> There's even talk of a continuation-passing style library, to
> be called future.el or promise.el or something.  Oh dear, what
> will be of us that we will have to evolve like any other language
> in the world!
>
> So I propose we let programmers use their judgement.  Really
> respect people who write code for no money and give the copyright
> away to the FSF.  Maybe suggest guidelines such as not introduce
> cl-loop where a dolist would do the job just as economically and
> elegantly. Don't use higher-order functions just because it looks
> cool. But maybe do suggest to use cl-position with a :key and a
> :test instead of a complex while loop with an auxiliary variable.
> Or cl-set-difference instead of nested loops.  Suggest to express
> intent, use abstractions.   Suggest to be consistent, be scrutinous,
> be "discriminate", be mindful of the style of the current area
> you are working on.
>
> But don't suggest anything too hard, especially if it's not
> modifying code you have authored.  Don't use arguments of authority
> when you can point to specific things.  Be generally respectful of
> people putting in any good work even if you don't like the style,
> and try to learn a new thing or two every now and then.
>
> BTW here are some nice generic suggestions from the Lisp world,
> written by two fenomenal programmers.  I love reading this from
> time to time:
>
>    https://www.cs.umd.edu/~nau/cmsc421/norvig-lisp-style.pdf

Thanks, that was well written, João. I agree.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Alan Mackenzie]

Hello, João.

On Thu, Nov 09, 2023 at 11:06:02 +0000, João Távora wrote:
> On Thu, Nov 9, 2023 at 10:05 AM Alan Mackenzie <acm@muc.de> wrote:

> > >    If it needed any confirmation, I too like cl-lib and I too help
> > >    maintain other people's code in the Emacs core tree as well as
> > >    maintaining a number of libraries I have authored.
> >
> > There's a difference between liking cl-lib and advocating its
> > indiscriminate use.  I don't think you've done the latter in this (and
> > related) threads.

> Yes, you're right.  Indeed I don't' advocate for its indiscriminate use,
> just as I don't advocate for indiscriminate use of anything, except
> perhaps drinking water and brushing teeth.

> > Nobody who likes cl-lib has yet addressed the point made by Richard and
> > (less eloquently) by me, namely that the incorporation and use of cl-lib
> > swells the size and complexity of Emacs Lisp to the point of making
> > maintenance difficult.  What is your view on this important point?

> That it doesn't make maintenance any more difficult than any other
> Elisp construct, be it very old and curiously named like 'rplacd' or
> much, much newer like `seq-do` or `pcase-lambda`.

OK, thanks for answering the question.  Please bare in mind that other
people's experience of maintaining code using cl-* is very different.

> My specific view on cl-lib.el is that it brings us a small part of
> the results of  non-trivial design work put in when important figures
> in the Lisp world met  regularly for many years to deliver what has
> proved to become excellent,  battle-tested, widely understood and
> impeccably documented programming abstractions.

> What I'm reading so far in this long discussion is that the argument
> of its detractors isn't really that cl-lib isn't good, but that
> it is superfluous and that learning it is a burden on maintainers.
> Well, it's just as superfluous as all of Elisp apart from two handfuls
> of primitives, I guess.

No, it's completely superfluous, even if it can be convenient.  Learning
it is a burden because its objects and abstractions are, well, very
abstract as well as being badly documented.  They have little
relationship to anything concrete in Emacs, and they are typically
complicated with many seemingly random details in a way that Emacs
primitives are not.

> Or any programming language for that matter, if you know enough
> machine code.  Or any other programming abstraction I happen not to be
> familiar with.

I am familiar with machine code.  It is perhaps pertinent to remark that
modern processors tend to the reduced instruction set model - that the
complicated "do it in one single complicated instruction" notion doesn't
help the writing of good compilers, or the running of programs
efficiently.  I suspect that analagous logic applies to programming
languages too.

[ .... ]

> João

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 10, 2023 at 1:16 PM Alan Mackenzie <acm@muc.de> wrote:

> > Or any programming language for that matter, if you know enough
> > machine code.  Or any other programming abstraction I happen not to be
> > familiar with.
>
> I am familiar with machine code.  It is perhaps pertinent to remark that
> modern processors tend to the reduced instruction set model - that the
> complicated "do it in one single complicated instruction" notion doesn't
> help the writing of good compilers, or the running of programs
> efficiently.  I suspect that analagous logic applies to programming
> languages too.

That's a clever and interesting perspective actually, but I would
tend to say no.  Programming languages are still written and read
by humans.  Human brains aren't designed the same way processors
are.  For one they aren't "designed" at all.  And we don't consume
information the same way a metal likes to, we don't unroll loops,
and at least I don't prefetch all of the supermarket shelf when
someone tells me to get olives, etc, etc.  Plus I'm horrible at
parallelism.  And we don't speak with reduced instruction sets,
we have many words with nuanced meanings.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Manuel Giraud via Emacs development discussions.]

João Távora <joaotavora@gmail.com> writes:

[...]

> and at least I don't prefetch all of the supermarket shelf when
> someone tells me to get olives, etc, etc.

😂 you should!  It sure takes some space at home but then you really
save time.
-- 
Manuel Giraud

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> That's a clever and interesting perspective actually, but I would
> tend to say no.  Programming languages are still written and read
> by humans.  Human brains aren't designed the same way processors
> are.  For one they aren't "designed" at all.  And we don't consume
> information the same way a metal likes to, we don't unroll loops,
> and at least I don't prefetch all of the supermarket shelf when
> someone tells me to get olives, etc, etc.  Plus I'm horrible at
> parallelism.  And we don't speak with reduced instruction sets,
> we have many words with nuanced meanings.

No, but humans are less still adapted for processing dozens of
obliquely-named constructs which wreck havoc on the control and data
flow of many a form.

Suppose that to retrieve a shopping cart, it were mandatory to connect
half a dozen unmarked pulleys and pneumatic tubes to one another in a
particular arrangement before depositing your coin in one end of the
contrivance, rather than to merely insert the coin in a small
receptacle.  Would you not be so deterred from availing yourself of the
gratis trolley rentals?

Re: What's missing in ELisp that makes people want to use cl-lib?

[Michael Heerdegen]

Dmitry Gutov <dmitry@gutov.dev> writes:

> I like cl-lib (it could be optimized better still, but it's good for
> certain uses), and this discussion doesn't look like it will reach any
> constructive conclusion, so it didn't seem worth it to spend time on
> it.

I also find it a bit exaggerating: cl-lib is large, but the majority of
its uses in Emacs are from a small subset.  In cases where it's
objectively beneficial, using its stuff can make the code actually
better readable.  But everybody should, as always, just select the
simplest tools when coding.  Please let's avoid a discussion about what
different opinions about readability and simplicity we have.

But of course: when we continue to migrate the most useful stuff in
cl-lib out to other libraries (like we did with generic functions,
generalized variables, sequence functions, ...), answering this question
might become harder in the future.

Michael.

Re: What's missing in ELisp that makes people want to use cl-lib?

[T.V Raman]

Michael Heerdegen <michael_heerdegen@web.de> writes:


One more fan of cl, and cl-libs is a good solution and a nice compromise
 between having the expressiveness of cl, without loading all of cl at
 runtime.

Creating and migrating to cl-libs was a large effort:

A. For the creators of cl-libs who carefully did the refactoring,
   and for
   B: Package authors who paid attention to the "cl used at runtime "
   warning to properly namespace-qualify cl calls,  and remove  (require
   'cl) 


So I dont quite understand the apparent desire to negate all of that work?
   
>
>> I like cl-lib (it could be optimized better still, but it's good for
>> certain uses), and this discussion doesn't look like it will reach any
>> constructive conclusion, so it didn't seem worth it to spend time on
>> it.
>
> I also find it a bit exaggerating: cl-lib is large, but the majority of
> its uses in Emacs are from a small subset.  In cases where it's
> objectively beneficial, using its stuff can make the code actually
> better readable.  But everybody should, as always, just select the
> simplest tools when coding.  Please let's avoid a discussion about what
> different opinions about readability and simplicity we have.
>
> But of course: when we continue to migrate the most useful stuff in
> cl-lib out to other libraries (like we did with generic functions,
> generalized variables, sequence functions, ...), answering this question
> might become harder in the future.
>
> Michael.
>
>

-- 

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Alan Mackenzie <acm@muc.de> writes:

> That is the current reality.

Maybe you could elaborate on what the plan then could look like?
Or is it not about a plan, but something else? 

Re: What's missing in ELisp that makes people want to use cl-lib?

[Alan Mackenzie]

On Thu, Nov 09, 2023 at 08:24:14 +0100, Gerd Möllmann wrote:
> Alan Mackenzie <acm@muc.de> writes:

>>Just take one look at this "reality" you're so supportive of: the
>>widespread use of cl-lib, not just in people's own projects, but
>>throughout the core of Emacs, has multiplied the size of Lisp language
>>part of Emacs by a factor of around 3.   This is a gross increase in
>>complexity for maintainers that is not justified by the slight increase
>>in facility that cl-lib (along with things like seq.el and oclosures)
>>gives.

>>Throughout this long discussion, this indiscriminate use of cl-lib has
>>been supported only by occasional contributers.  Those who actually
>>maintain other people's code, apart from (I think) Eli, Richard and
>>myself, have been conspicuously silent.  None of us three have favoured
>>such use of cl-lib.

>>Occasional contributors may be fascinated by cl-lib, and learn enough
>>of it to use random bits of it in their code.  The trouble is, each
>>such contributor uses a different piece of cl-lib, with the result that
>>those who end up maintaining it need to know a far greater part of it
>>just to cope.

>>This factor of 3 is, I believe, a significant barrier to new
>>programmers coming into Emacs; Elisp is just that much more difficult
>>than it was in the past.  And it isn't just for newcomers that it is
>>more difficult.  I spend a significant amount of debugging time having
>>to look up doc strings and manual pages for obscure cl-lib (etc.)
>>functions.

> > That is the current reality.

> Maybe you could elaborate on what the plan then could look like?
> Or is it not about a plan, but something else? 

You stripped all the context.  I've put it back again.

As a concrete plan, I would propose the following for discussion:

We should deprecate those functions/macros/variables in cl-lib that have
no doc string, or a substandard one.  This includes "internal" functions,
too.  Also to be deprecated are obscure functions/m/v (such as
cl-labels).

Having done this, we recode code currently using those deprecated f/m/v.

(Here a "substandard" doc string is contrasted with an adequate one,
which does all of the following:
(i) It says what the function/macro _does_, or what the variable _is_.
(ii) It describes the form and meaning of each parameter, and its
  relationship to (i).
(iii) If the return value is significant, it describes this.
(iv) It describes all effects on the global state, such as where it
  writes results to, and suchlike.)

This would reduce the size of cl-lib to something more manageable than
what we have at the moment.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 09/11/2023 12:34, Alan Mackenzie wrote:
> As a concrete plan, I would propose the following for discussion:
> 
> We should deprecate those functions/macros/variables in cl-lib that have
> no doc string, or a substandard one.  This includes "internal" functions,
> too.  Also to be deprecated are obscure functions/m/v (such as
> cl-labels).

I'm not sure we can remove cl-labels -- it's useful enough for the cases 
where it does help, and there are callers inside and outside of Emacs.

Whatever other removals, would probably need to be discussed case-by-case.

> Having done this, we recode code currently using those deprecated f/m/v.
> 
> (Here a "substandard" doc string is contrasted with an adequate one,
> which does all of the following:
> (i) It says what the function/macro_does_, or what the variable_is_.
> (ii) It describes the form and meaning of each parameter, and its
>    relationship to (i).
> (iii) If the return value is significant, it describes this.
> (iv) It describes all effects on the global state, such as where it
>    writes results to, and suchlike.)

Improving cl-lib's documentation would be a welcome effort.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Thu, Nov 9, 2023 at 11:49 AM Dmitry Gutov <dmitry@gutov.dev> wrote:

> Improving cl-lib's documentation would be a welcome effort.

For sure, and not a hard one as well, as all those functions and
macros are pretty good, often flawless emulations of CL functions that
are impeccably documented in

  http://www.lispworks.com/documentation/HyperSpec/Front/

Which is of free access (though not of a compatible license, I
think).  But if people can point to the 5 most confusing functions
they think are poorly documented, I volunteer to rewrite the
docstrings for them.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Alan Mackenzie]

Hello, João.

On Thu, Nov 09, 2023 at 12:40:24 +0000, João Távora wrote:
> On Thu, Nov 9, 2023 at 11:49 AM Dmitry Gutov <dmitry@gutov.dev> wrote:

> > Improving cl-lib's documentation would be a welcome effort.

> For sure, and not a hard one as well, as all those functions and
> macros are pretty good, often flawless emulations of CL functions that
> are impeccably documented in

>   http://www.lispworks.com/documentation/HyperSpec/Front/

> Which is of free access (though not of a compatible license, I
> think).  But if people can point to the 5 most confusing functions
> they think are poorly documented, I volunteer to rewrite the
> docstrings for them.

How much are you prepared to do?  I don't have a list of the _most_
confusing doc strings, there are too many to chose from.  But starting
at the start of cl-macs.el, we have:

(i) cl--compiler-macro-list*; completely undocumented.
(ii) cl--simple-expr-p: Talks about "side effects", but not what they
  are side effects of.  Doesn't describe it's parameters or return
  value.  It's unclear what it is that "executes quickly".
(iii) cl--expr-contains: It's unclear what X and Y are, and what "refers
  to" means.
(iv) cl--expr-contains-any; completely undocumented.
(v) cl--expr-depends-p: It's unclear what X and Y are, though Y appears
  to be some sort of container of symbols.  It's unclear what sort of
  "dependency" the function handles, or what "may" means in the context.

There are many more.

> João

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Thu, Nov 9, 2023 at 1:36 PM Alan Mackenzie <acm@muc.de> wrote:
>
> Hello, João.
>
> On Thu, Nov 09, 2023 at 12:40:24 +0000, João Távora wrote:
> > On Thu, Nov 9, 2023 at 11:49 AM Dmitry Gutov <dmitry@gutov.dev> wrote:
>
> > > Improving cl-lib's documentation would be a welcome effort.
>
> > For sure, and not a hard one as well, as all those functions and
> > macros are pretty good, often flawless emulations of CL functions that
> > are impeccably documented in
>
> >   http://www.lispworks.com/documentation/HyperSpec/Front/
>
> > Which is of free access (though not of a compatible license, I
> > think).  But if people can point to the 5 most confusing functions
> > they think are poorly documented, I volunteer to rewrite the
> > docstrings for them.
>
> How much are you prepared to do?  I don't have a list of the _most_
> confusing doc strings, there are too many to chose from.  But starting
> at the start of cl-macs.el, we have:
>
> (i) cl--compiler-macro-list*; completely undocumented.
> (ii) cl--simple-expr-p: Talks about "side effects", but not what they
>   are side effects of.  Doesn't describe it's parameters or return
>   value.  It's unclear what it is that "executes quickly".
> (iii) cl--expr-contains: It's unclear what X and Y are, and what "refers
>   to" means.
> (iv) cl--expr-contains-any; completely undocumented.
> (v) cl--expr-depends-p: It's unclear what X and Y are, though Y appears
>   to be some sort of container of symbols.  It's unclear what sort of
>   "dependency" the function handles, or what "may" means in the context.
>
> There are many more.

These are all internal functions and implementation details.  They're
not necessary at all for users of cl-lib.el, only for its developers.
What problem are you trying to solve by enhancing these docstrings?  I thought
the problem here was code that _used_ cl-lib.el, not hacking on
cl-lib.el itself.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

João Távora wrote:

> These are all internal functions and implementation details.
> They're not necessary at all for users of cl-lib.el, only
> for its developers. What problem are you trying to solve by
> enhancing these docstrings? I thought the problem here was
> code that _used_ cl-lib.el, not hacking on cl-lib.el itself.

It is still nuch better than removing or rewriting it, so
let's do that.

We can use checkdoc to find out what docstrings are not
submitted or not correctly formated. Those should be pretty
quick fixes, to get away with those warnings.

After we have done that, if someone says some specific
docstring is still lacking we can focus on such individual
cases in more detail.

Here, to get the list from, e.g., cl-lib.el

(defun check-package-style ()
  (interactive)
  (let ((msg "Style check..."))
    (message msg)
    (checkdoc-current-buffer t)
    (message "%sdone" msg) ))

And the list is

cl-lib.el:133: Argument ‘keys’ should appear (as KEYS) in the doc string
cl-lib.el:160: All variables and subroutines might as well have a documentation string
cl-lib.el:166: All variables and subroutines might as well have a documentation string
cl-lib.el:197: All variables and subroutines might as well have a documentation string
cl-lib.el:197: All variables and subroutines might as well have a documentation string
cl-lib.el:226: Arguments occur in the doc string out of order
cl-lib.el:258: Argument ‘specs’ should appear (as SPECS) in the doc string
cl-lib.el:357: Argument ‘cl-func’ should appear (as CL-FUNC) in the doc string
cl-lib.el:450: Argument ‘rest’ should appear (as REST) in the doc string
cl-lib.el:488: Argument ‘cl-item’ should appear (as CL-ITEM) in the doc string
cl-lib.el:501: Argument ‘cl-new’ should appear (as CL-NEW) in the doc string
cl-lib.el:510: All variables and subroutines might as well have a documentation string
cl-lib.el:544: All variables and subroutines might as well have a documentation string
cl-lib.el:582: Probably "returns" should be imperative "return"

Here are all the cl-lib files

cl-extra.el
cl-generic.el
cl-indent.el
cl-lib.el
cl-loaddefs.el
cl-macs.el
cl-preloaded.el
cl-print.el
cl-seq.el

GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, cairo version 1.16.0)
of 2023-11-01 [commit
8eb9b82ce58a8323af21f52625a401e19e279a9c]

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Alan Mackenzie]

Hello, João.

On Thu, Nov 09, 2023 at 13:41:48 +0000, João Távora wrote:
> On Thu, Nov 9, 2023 at 1:36 PM Alan Mackenzie <acm@muc.de> wrote:

> > Hello, João.

> > On Thu, Nov 09, 2023 at 12:40:24 +0000, João Távora wrote:
> > > On Thu, Nov 9, 2023 at 11:49 AM Dmitry Gutov <dmitry@gutov.dev> wrote:

> > > > Improving cl-lib's documentation would be a welcome effort.

> > > For sure, and not a hard one as well, as all those functions and
> > > macros are pretty good, often flawless emulations of CL functions that
> > > are impeccably documented in

> > >   http://www.lispworks.com/documentation/HyperSpec/Front/

> > > Which is of free access (though not of a compatible license, I
> > > think).  But if people can point to the 5 most confusing functions
> > > they think are poorly documented, I volunteer to rewrite the
> > > docstrings for them.

> > How much are you prepared to do?  I don't have a list of the _most_
> > confusing doc strings, there are too many to chose from.  But starting
> > at the start of cl-macs.el, we have:

> > (i) cl--compiler-macro-list*; completely undocumented.
> > (ii) cl--simple-expr-p: Talks about "side effects", but not what they
> >   are side effects of.  Doesn't describe it's parameters or return
> >   value.  It's unclear what it is that "executes quickly".
> > (iii) cl--expr-contains: It's unclear what X and Y are, and what "refers
> >   to" means.
> > (iv) cl--expr-contains-any; completely undocumented.
> > (v) cl--expr-depends-p: It's unclear what X and Y are, though Y appears
> >   to be some sort of container of symbols.  It's unclear what sort of
> >   "dependency" the function handles, or what "may" means in the context.

> > There are many more.

> These are all internal functions and implementation details.  They're
> not necessary at all for users of cl-lib.el, only for its developers.

So, you're not prepared to rewrite their doc strings as you promised
yesterday, any more.

> What problem are you trying to solve by enhancing these docstrings?

Being able to debug Emacs problems.

> I thought the problem here was code that _used_ cl-lib.el, not hacking
> on cl-lib.el itself.

cl-lib uses itself, and needs debugging too.  Debugging code that uses
abstractions often requires penetrating these abstractions and checking
their innards.

Still, if you're not prepared to fix doc strings in internal functions,
here are five external functions with inadequate doc strings:

(i) cl-fill: The doc string misnames some of the parameters.  It is
unclear what "fill" means, and the forms of CL-SEQ and CL-ITEM aren't
specified.  Is each element of CL-SEQ "filled" with the entirety of
CL-ITEM, or what?  Finally, there is no documentation of CL-KEYS, beyond
stating that two particular keywords are "supported", whatever that
means.  What do :start and :end actually do?  The return value is not
specified.

(ii) cl-replace: Much the same problems as for cl-fill.

(iii) cl-remove: Same again.  Additionally, it is not specified what an
"occurrence" is; this presumably involves some equality test, and which
one is used is not specified.  Again the effect of the keywords is
unspecified.  There are keywords used in its code which aren't listed in
the doc string, namely :if and :if-not.

(iv) cl-remove-if: Much the same again.

(v) cl-remove-if-not: ... and again.

I found these simply by starting at the beginning of cl-seq and listing
pretty much each external function I found in a simple search.  There are
a lot more like these.

Still, even having just those five things properly documented would be
worthwhile.  Thanks for volunteering to do this!

> João

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 10, 2023 at 12:13 PM Alan Mackenzie <acm@muc.de> wrote:
>
> Hello, João.
>
> On Thu, Nov 09, 2023 at 13:41:48 +0000, João Távora wrote:
> > On Thu, Nov 9, 2023 at 1:36 PM Alan Mackenzie <acm@muc.de> wrote:
>
> > > Hello, João.
>
> > > On Thu, Nov 09, 2023 at 12:40:24 +0000, João Távora wrote:
> > > > On Thu, Nov 9, 2023 at 11:49 AM Dmitry Gutov <dmitry@gutov.dev> wrote:
>
> > > > > Improving cl-lib's documentation would be a welcome effort.
>
> > > > For sure, and not a hard one as well, as all those functions and
> > > > macros are pretty good, often flawless emulations of CL functions that
> > > > are impeccably documented in
>
> > > >   http://www.lispworks.com/documentation/HyperSpec/Front/
>
> > > > Which is of free access (though not of a compatible license, I
> > > > think).  But if people can point to the 5 most confusing functions
> > > > they think are poorly documented, I volunteer to rewrite the
> > > > docstrings for them.
>
> > > How much are you prepared to do?  I don't have a list of the _most_
> > > confusing doc strings, there are too many to chose from.  But starting
> > > at the start of cl-macs.el, we have:
>
> > > (i) cl--compiler-macro-list*; completely undocumented.
> > > (ii) cl--simple-expr-p: Talks about "side effects", but not what they
> > >   are side effects of.  Doesn't describe it's parameters or return
> > >   value.  It's unclear what it is that "executes quickly".
> > > (iii) cl--expr-contains: It's unclear what X and Y are, and what "refers
> > >   to" means.
> > > (iv) cl--expr-contains-any; completely undocumented.
> > > (v) cl--expr-depends-p: It's unclear what X and Y are, though Y appears
> > >   to be some sort of container of symbols.  It's unclear what sort of
> > >   "dependency" the function handles, or what "may" means in the context.
>
> > > There are many more.
>
> > These are all internal functions and implementation details.  They're
> > not necessary at all for users of cl-lib.el, only for its developers.
>
> So, you're not prepared to rewrite their doc strings as you promised
> yesterday, any more.

I thought the whole problem is that code using cl-lib.el is hard to use.
You said cl-labels was "obscure", so I was prepared to enhance its
docstring, yes.  That's a public macro.  Along with 4 other "obscure"
public ones.  Documentation cl-lib's implementation details is something
else entirely.  It doesn't hurt if there's suspicion that cl-lib.el is
misbehaving in function or efficiency.   But that's not the matter in
discussion here and it doesn't solve the presumed problem of not
understanding code that makes uses cl-lib.

> > What problem are you trying to solve by enhancing these docstrings?
> Being able to debug Emacs problems.

What problem specifically are you having trouble with?  What do you
ultimately want to achieve that you can't without docstrings for
cl-lib.el's implementation details.  This might be an XY problem
(https://en.wikipedia.org/wiki/XY_problem)

> cl-lib uses itself, and needs debugging too.  Debugging code that uses
> abstractions often requires penetrating these abstractions and checking
> their innards.

Well, no.  I don't go read OS source code whenever I have doubts
about how to use a system call or how to read a given piece of
code that uses that system call, I read the manual page for the system
call itself.  Just as I don't pull up a magnifying glass to check if
my CPU's transistors are doing what they should, I trust the data sheet
describing the instructions.  So, no, at least not desirably.  That's
the whole argument for good docstrings: that it relieves you from
having to penetrate innards  to understand a function's interface.
Feels a bit odd to be defending age-old ideas of encapsulation and
information-hiding here.

To summarize, I thought it would be absolutely clear that I volunteered
to write docstrings for public functions, public interfaces.  I'm
amazed it wasn't, but I apologize for the misunderstanding nonetheless.
Docstrings for things that the user sees, is confused by, finds
obscure, etc. Give me 5 of  those, and I'll rework their docstrings for
clarity.  I suggest cl-labels since you singled it out and some
sequence manipulating functions which just say: "Keywords supported:
:test :test-not :key :start :end :from-end" but don't explain what
each of those keyword arguments do.

Another useful thing would be to extend eldoc support in Elisp mode
to be able to highlight the currently active keyword argument.  I'll
also look into that, but no promises.

Thanks,
João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Fri, 10 Nov 2023 12:53:10 +0000
> Cc: Dmitry Gutov <dmitry@gutov.dev>,
>  Gerd Möllmann <gerd.moellmann@gmail.com>, 
>  Björn Bidar <bjorn.bidar@thaodan.de>, emacs-devel@gnu.org
> 
> > cl-lib uses itself, and needs debugging too.  Debugging code that uses
> > abstractions often requires penetrating these abstractions and checking
> > their innards.
> 
> Well, no.  I don't go read OS source code whenever I have doubts
> about how to use a system call or how to read a given piece of
> code that uses that system call, I read the manual page for the system
> call itself.

This misses the point which I think Alan wanted to make.  The point is
that when some cl-lib function seems to return an unexpected result,
one needs to debug that function.  And since Edebug has only
rudimentary and frequently completely inadequate support for debugging
cl-lib abstractions, one must then attempt at analyzing and
understanding those abstractions as part of debugging.

And if you never had such moments when some OS syscall produced an
unexpected result and the man pages failed to explain why, then I
guess you have yet a few TIL situations in your future ;-)

> Just as I don't pull up a magnifying glass to check if
> my CPU's transistors are doing what they should, I trust the data sheet
> describing the instructions.

Strawmen aside, this flies in the face of one of the great advantages
of working with Free Software: if you want and need to, you can find
out what happens on any level, including the lowest level of OS system
calls.  Trying to laugh it away is a non-starter around here.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Alan Mackenzie]

Hello, João.

On Fri, Nov 10, 2023 at 12:53:10 +0000, João Távora wrote:
> On Fri, Nov 10, 2023 at 12:13 PM Alan Mackenzie <acm@muc.de> wrote:

> > On Thu, Nov 09, 2023 at 13:41:48 +0000, João Távora wrote:
> > > On Thu, Nov 9, 2023 at 1:36 PM Alan Mackenzie <acm@muc.de> wrote:

> > > > On Thu, Nov 09, 2023 at 12:40:24 +0000, João Távora wrote:
> > > > > On Thu, Nov 9, 2023 at 11:49 AM Dmitry Gutov <dmitry@gutov.dev> wrote:

> > > > > > Improving cl-lib's documentation would be a welcome effort.

> > > > > For sure, and not a hard one as well, as all those functions and
> > > > > macros are pretty good, often flawless emulations of CL functions that
> > > > > are impeccably documented in

> > > > >   http://www.lispworks.com/documentation/HyperSpec/Front/

> > > > > Which is of free access (though not of a compatible license, I
> > > > > think).  But if people can point to the 5 most confusing functions
> > > > > they think are poorly documented, I volunteer to rewrite the
> > > > > docstrings for them.

> > > > How much are you prepared to do?  I don't have a list of the _most_
> > > > confusing doc strings, there are too many to chose from.  But starting
> > > > at the start of cl-macs.el, we have:

> > > > (i) cl--compiler-macro-list*; completely undocumented.
> > > > (ii) cl--simple-expr-p: Talks about "side effects", but not what they
> > > >   are side effects of.  Doesn't describe it's parameters or return
> > > >   value.  It's unclear what it is that "executes quickly".
> > > > (iii) cl--expr-contains: It's unclear what X and Y are, and what "refers
> > > >   to" means.
> > > > (iv) cl--expr-contains-any; completely undocumented.
> > > > (v) cl--expr-depends-p: It's unclear what X and Y are, though Y appears
> > > >   to be some sort of container of symbols.  It's unclear what sort of
> > > >   "dependency" the function handles, or what "may" means in the context.

> > > > There are many more.

> > > These are all internal functions and implementation details.  They're
> > > not necessary at all for users of cl-lib.el, only for its developers.

> > So, you're not prepared to rewrite their doc strings as you promised
> > yesterday, any more.

> I thought the whole problem is that code using cl-lib.el is hard to use.

More precisely, it's hard to understand and to debug, precisely because
it uses cl-* functions.

> You said cl-labels was "obscure", so I was prepared to enhance its
> docstring, yes.  That's a public macro.  Along with 4 other "obscure"
> public ones.

Ah, so your offer of fixing 5 doc strings is limited to obscure objects.
That wasn't clear from the outset, and wasn't what you said.

Documentation won't help cl-labels become any less obscure.  I've
debugged code using it, but I havn't any clue what it does.  There is
nothing about its functionality which sticks in memory.  Its name is
poor - "labels" isn't a verb, and what it does has nothing to do with
labels.

> Documentation cl-lib's implementation details is something else
> entirely.  It doesn't hurt if there's suspicion that cl-lib.el is
> misbehaving in function or efficiency.   But that's not the matter in
> discussion here and it doesn't solve the presumed problem of not
> understanding code that makes uses cl-lib.

It would solve part of the problem.

> > > What problem are you trying to solve by enhancing these docstrings?
> > Being able to debug Emacs problems.

> What problem specifically are you having trouble with?  What do you
> ultimately want to achieve that you can't without docstrings for
> cl-lib.el's implementation details.  This might be an XY problem
> (https://en.wikipedia.org/wiki/XY_problem)

I'm not having any specific problem at the moment.  The need for accurate
readable doc strings, particularly for code as difficult and obscure as
cl-*, should be self-evident.

I don't know what an "XY problem" is and can't be bothered to look up
your reference.  Thanks for not explaining clearly what you mean.

> > cl-lib uses itself, and needs debugging too.  Debugging code that uses
> > abstractions often requires penetrating these abstractions and checking
> > their innards.

> Well, no.  I don't go read OS source code whenever I have doubts
> about how to use a system call or how to read a given piece of
> code that uses that system call, I read the manual page for the system
> call itself.

One Emacs bug I solved involved hacking into the Linux kernel.  The
solution to the bug was to upgrade to a specific kernel version or later.

> Just as I don't pull up a magnifying glass to check if my CPU's
> transistors are doing what they should, I trust the data sheet
> describing the instructions.  So, no, at least not desirably.  That's
> the whole argument for good docstrings: that it relieves you from
> having to penetrate innards  to understand a function's interface.
> Feels a bit odd to be defending age-old ideas of encapsulation and
> information-hiding here.

Total strawman argument.

> To summarize, I thought it would be absolutely clear that I volunteered
> to write docstrings for public functions, public interfaces.

No, it wasn't clear.

> I'm amazed it wasn't, but I apologize for the misunderstanding
> nonetheless.  Docstrings for things that the user sees, is confused by,
> finds obscure, etc. Give me 5 of  those, and I'll rework their
> docstrings for clarity.

I did so in my last post, and you've just snipped them without comment.
That's twice I've given you five bad doc strings, and you aren't acting
on either set.  That took a significant amount of my time.

Thanks.

It seems to me you are going back on your undertaking to amend these five
doc strings.

> I suggest cl-labels since you singled it out and some sequence
> manipulating functions which just say: "Keywords supported: :test
> :test-not :key :start :end :from-end" but don't explain what each of
> those keyword arguments do.

That would certainly be useful, yes.

> Another useful thing would be to extend eldoc support in Elisp mode
> to be able to highlight the currently active keyword argument.  I'll
> also look into that, but no promises.

That is something else entirely, unrelated to the poor quality of the doc
strings that we're talking about.

> Thanks,
> João

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 10, 2023 at 2:25 PM Alan Mackenzie <acm@muc.de> wrote:

> I did so in my last post, and you've just snipped them without comment.
> That's twice I've given you five bad doc strings, and you aren't acting
> on either set.  That took a significant amount of my time.

Those are not public functions, Alan.  Even Dmitry pointed that out
immediately.

> It seems to me you are going back on your undertaking to amend these five
> doc strings.

Stop insinuating that.  I'm sorry you lost time looking at those
functions (by why???).  But it's not my fault you have no distinction of
public/private interface.  Or at least not one I share.  But I do, and
the Emacs documentation does.  The Emacs and Elisp manual doesn't
describe '--' functions, period. Even checkdoc.el has this concept.
Docstrings for internal functions don't hurt, but they're not the
essential parts to describe a library.  Surely not the first things
you should focus on, because internals are by definition prone to
one could re-implement all of cl-lib.el keeping the same interfaces
and their docstrings with wholly new internals.  I don't know why
am I'm having to clarify this.

Anyway, take care, Alan.
João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Alan Mackenzie]

Hello again, João.

On Fri, Nov 10, 2023 at 14:35:38 +0000, João Távora wrote:
> On Fri, Nov 10, 2023 at 2:25 PM Alan Mackenzie <acm@muc.de> wrote:

> > I did so in my last post, and you've just snipped them without comment.
> > That's twice I've given you five bad doc strings, and you aren't acting
> > on either set.  That took a significant amount of my time.

> Those are not public functions, Alan.  Even Dmitry pointed that out
> immediately.

The second set were public functions.  And as Eli pointed out,
non-public functions need documenting too.

> > It seems to me you are going back on your undertaking to amend these
> > five doc strings.

> Stop insinuating that.

You made an undertaking, the text of which was still in this thread
until you snipped it from your last post.

I now accept you won't be amending the doc strings I suggested, or
probably any others.  I think that's a shame, and I'm disappointed.

> I'm sorry you lost time looking at those functions (by why???).  But
> it's not my fault you have no distinction of public/private interface.

What sort of a smear is that meant to be?  It's just that I'm aware of
the need to document internal functions, you seem not to be.  As a
cl-lib enthusiast, you would have been best placed to fix this doc.

> Or at least not one I share.  But I do, and the Emacs documentation
> does.  The Emacs and Elisp manual doesn't describe '--' functions,
> period. Even checkdoc.el has this concept.  Docstrings for internal
> functions don't hurt, but they're not the essential parts to describe
> a library.  Surely not the first things you should focus on, because
> internals are by definition prone to one could re-implement all of
> cl-lib.el keeping the same interfaces and their docstrings with wholly
> new internals.  I don't know why am I'm having to clarify this.

I don't think there's much point continuing this bit of the conversation
any further.

> Anyway, take care, Alan.
> João

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 10, 2023 at 3:11 PM Alan Mackenzie <acm@muc.de> wrote:

> > Those are not public functions, Alan.  Even Dmitry pointed that out
> > immediately.
>
> The second set were public functions.  And as Eli pointed out,
> non-public functions need documenting too.

Oh then I missed that second set in all this jumble.  Can you please
post it again?

Thanks,
João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Alan Mackenzie]

On Fri, Nov 10, 2023 at 15:14:05 +0000, João Távora wrote:
> On Fri, Nov 10, 2023 at 3:11 PM Alan Mackenzie <acm@muc.de> wrote:

> > > Those are not public functions, Alan.  Even Dmitry pointed that out
> > > immediately.
> >
> > The second set were public functions.  And as Eli pointed out,
> > non-public functions need documenting too.

> Oh then I missed that second set in all this jumble.  Can you please
> post it again?

It's in my post in this thread posted at Date: Fri, 10 Nov 2023 12:13:25
+0000.

Or to be found at
https://lists.gnu.org/archive/html/emacs-devel/2023-11/msg00439.html.

> Thanks,
> João

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 10, 2023 at 3:47 PM Alan Mackenzie <acm@muc.de> wrote:
>
> On Fri, Nov 10, 2023 at 15:14:05 +0000, João Távora wrote:
> > On Fri, Nov 10, 2023 at 3:11 PM Alan Mackenzie <acm@muc.de> wrote:
>
> > > > Those are not public functions, Alan.  Even Dmitry pointed that out
> > > > immediately.
> > >
> > > The second set were public functions.  And as Eli pointed out,
> > > non-public functions need documenting too.
>
> > Oh then I missed that second set in all this jumble.  Can you please
> > post it again?
>
> It's in my post in this thread posted at Date: Fri, 10 Nov 2023 12:13:25
> +0000.
>
> Or to be found at
> https://lists.gnu.org/archive/html/emacs-devel/2023-11/msg00439.html.

Alright, here are two of the five functions documented in a patch.
Will look at the other 3 later.  By the way, if you have a web browser,
the so-called Common Lisp hyperspec is an excellent resource
to learn to work with CL functions.

Here is the doc for FILL and REPLACE:

  http://www.lispworks.com/documentation/HyperSpec/Body/f_fill.htm
  http://www.lispworks.com/documentation/HyperSpec/Body/f_replac.htm

Disclamer:  I'm open to tweaking these docstrings I wrote to
some reasonable degree, like grammar fixes, reordering sentences,
 etc.  Constructive criticism only. I will not indulge endless
hair-splitting, bikeshedding or deep personal considerations of
what proper docstrings should contain and look like, especially
if I suspect it is fueled by any deep disdain (or "generic aversion"
as someone put it) of the cl-lib.el library, such as has been patent
so far in some posts of this discussion so far.

IOW I agree with you that the current docstrings are subpar, so
this is why I volunteered to enhance them, but this is my honest
writing and my own considerations of what a good docstring should
look like, based on many years experience with Lisp and Emacs, and
how it may help a honest user of these functions put them to good
use or read code containing them.

And now the patch for 2 of the 5 functions you mentioned,
João

diff --git a/lisp/emacs-lisp/cl-seq.el b/lisp/emacs-lisp/cl-seq.el
index ec481121ae6..dd9f37b0a82 100644
--- a/lisp/emacs-lisp/cl-seq.el
+++ b/lisp/emacs-lisp/cl-seq.el
@@ -162,9 +162,19 @@ cl-reduce

 ;;;###autoload
 (defun cl-fill (cl-seq cl-item &rest cl-keys)
-  "Fill the elements of SEQ with ITEM.
-\nKeywords supported:  :start :end
-\n(fn SEQ ITEM [KEYWORD VALUE]...)"
+  "Replace elements of SEQ between START and END with ITEM.
+SEQ is a Lisp sequence.  It is destructively modified and
+returned.
+
+START and END are indexes like in `aref' or `elt'.  They
+designate the subsequence of SEQ to operate on.  If END is nil,
+process the sequence to the end. They default to 0 and nil,
+respectively, meaning process the whole SEQ.
+
+START and END are keyword arguments.  See info node `(cl) Program
+Structure > Argument Lists' for details.
+
+\n(fn SEQ ITEM &key START END...)"
   (cl--parsing-keywords ((:start 0) :end) ()
     (if (listp cl-seq)
  (let ((p (nthcdr cl-start cl-seq))
@@ -182,10 +192,30 @@ cl-fill

 ;;;###autoload
 (defun cl-replace (cl-seq1 cl-seq2 &rest cl-keys)
-  "Replace the elements of SEQ1 with the elements of SEQ2.
-SEQ1 is destructively modified, then returned.
-\nKeywords supported:  :start1 :end1 :start2 :end2
-\n(fn SEQ1 SEQ2 [KEYWORD VALUE]...)"
+  "Replace the elements of SEQ1 with elements of SEQ2.
+SEQ1 and SEQ2 are both Lisp sequences.  SEQ1 is destructively
+modified and returned.
+
+START1, END1, START2 and END2 can be indexes like in `aref' or
+`elt'.  Each pair designates two subsequences of SEQ1 and SEQ2,
+respectively, to operate on.  If END1 or END2 is nil, consider
+the respective sequences to the end.
+
+Consecutive elements of the subsequence of SEQ1 are replaced by
+consecutive elements of the subsequence of SEQ2.
+
+If the subsequences vary in length, the shorter one determines
+how many elements are replaced.  Extra elements in either
+subsequence are ignored.
+
+START1 and START2 default to 0, END1 and END2 default to nil,
+meaning replace as much of SEQ1 as possible with elements from
+SEQ2.
+
+START1, END1, START2 and END2 are keyword arguments.  See info
+node `(cl) Program Structure > Argument Lists' for details.
+
+\n(fn SEQ1 SEQ2 &key START1 END1 START2 END2...)"
   (cl--parsing-keywords ((:start1 0) :end1 (:start2 0) :end2) ()
     (if (and (eq cl-seq1 cl-seq2) (<= cl-start2 cl-start1))
  (or (= cl-start1 cl-start2)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Alan Mackenzie]

Hello, João.

On Fri, Nov 10, 2023 at 16:10:36 +0000, João Távora wrote:

[ .... ]

> Alright, here are two of the five functions documented in a patch.
> Will look at the other 3 later.

Thanks!  These two look like just the job!

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 10, 2023 at 8:24 PM Alan Mackenzie <acm@muc.de> wrote:
>
> Hello, João.
>
> On Fri, Nov 10, 2023 at 16:10:36 +0000, João Távora wrote:
>
> [ .... ]
>
> > Alright, here are two of the five functions documented in a patch.
> > Will look at the other 3 later.
>
> Thanks!  These two look like just the job!

Very happy you like them, will do the others ASAP, and
maybe even some more.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Michael Heerdegen]

Alan Mackenzie <acm@muc.de> writes:

> I don't think there's much point continuing this bit of the
> conversation any further.

There is much point.  Please, again, let's calm down a bit.  João is not
personally responsible for the things you criticize.  He will maybe not
be able to fix all of these points in a couple of days.  And he does not
have to prove that he can.  This doesn't mean it is not possible in a
longer term.

Alan, I totally understand that you are annoyed.  But impatience is not
helpful in this case, it will rather prevent that we get things done in
a good way.

Michael.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Alan Mackenzie]

Hello, Michael.

Thanks for the kind message!

On Sat, Nov 11, 2023 at 07:21:25 +0100, Michael Heerdegen wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > I don't think there's much point continuing this bit of the
> > conversation any further.

> There is much point.  Please, again, let's calm down a bit.  João is not
> personally responsible for the things you criticize.  He will maybe not
> be able to fix all of these points in a couple of days.  And he does not
> have to prove that he can.  This doesn't mean it is not possible in a
> longer term.

> Alan, I totally understand that you are annoyed.  But impatience is not
> helpful in this case, it will rather prevent that we get things done in
> a good way.

There was actually a specific misunderstanding between João and me,
which has now been resolved.  He has, so far, amended two doc strings
beautifully.

> Michael.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: What's missing in ELisp that makes people want to use cl-lib?

[Michael Heerdegen]

João Távora <joaotavora@gmail.com> writes:

> I suggest cl-labels since you singled it out and some
> sequence manipulating functions which just say: "Keywords supported:
> :test :test-not :key :start :end :from-end" but don't explain what
> each of those keyword arguments do.

I dunno if this would fit in our concept, but it would be nice if those
functions supporting keywords had a link (or an initially folded
subsection) where the keywords are explained in a quick way.  One needs
them too seldom to remember all the details until the next time.

Michael. 

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Sat, Nov 11, 2023 at 6:08 AM Michael Heerdegen
<michael_heerdegen@web.de> wrote:
>
> João Távora <joaotavora@gmail.com> writes:
>
> > I suggest cl-labels since you singled it out and some
> > sequence manipulating functions which just say: "Keywords supported:
> > :test :test-not :key :start :end :from-end" but don't explain what
> > each of those keyword arguments do.
>
> I dunno if this would fit in our concept, but it would be nice if those
> functions supporting keywords had a link (or an initially folded
> subsection) where the keywords are explained in a quick way.  One needs
> them too seldom to remember all the details until the next time.

I think these keyword arguments are just like normal elisp
optional arguments, they are just passed by name.  So the
same problems of writing/explaining them apply.

As to whether keyword arguments are intrinsically needed more
or less frequently than optional arguments is IMO, very
debatable, and entirely dependent on the function.  Again, just
like optional args. I don't think I've ever used read-from-minibuffer's
optional KEYMAP arg, for example.  Or INHERIT-INPUT-METHOD to
completing-read.  I use :KEY and :TEST quite often for cl-lib sequence
less frequently :FROM-END and even less frequently  :TEST-NOT.

Where I can agree with you is that is seems a bit suboptimal to
re-describe -- specifically for CL keyword arguments, _not_ keyword
arguments in general --  the semantics of the elements of the
standard set :KEY, :TEST, :FROM-END, etc.  I.e. there could
be some shortcuts to help facilitate this writing in cl-lib.el,
maybe some special docstring construct.  But copy-pasting it's
not that horrible either, and they can be summarily described
in single line with consistent style between the functions that
use them.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I think these keyword arguments are just like normal elisp
  > optional arguments, they are just passed by name.  So the
  > same problems of writing/explaining them apply.

  > As to whether keyword arguments are intrinsically needed more
  > or less frequently than optional arguments is IMO, very
  > debatable, and entirely dependent on the function.

I think both of these questions are minor issues that threaten to act
as red herrings.  Keyword arguments are better than optional
positional arguments in some circumstances, and the optional positional
arguments are better in some circumstances.

Keyword arguments are better -- sometimes much better -- when one
function accepts many different optional arguments.  There are
functions in Emacs that accept many positional optional arguments, and
some of them would be cleaner if they accepted keyword arguments.

Why don't they?  Because Emacs Lisp doesn't make it easy to accept
keyword arguments.  It is especially a pain in a primitive.  And many
of those functions were defined before Emacs Lisp had such a thing as
a keyword, let alone a keyword argument.

On the other hand, positional optional arguments are better when a
function accepts only one or two of them and they are used often.
They are easier to pass, and easier to remember,

Keyword arguments are easier to use when a function accepts many
different arguments pass.  But this can backfire when it encourages
people to give a function many keyword arguments, on the assumption
that "It doesn't cost us anything to give it more."

It may not cost much more at run time to decode those arguments,
but it costs a lot more to remember what they do, and to use them.

What Common Lisp did to sequence functions was the worst of
everything.  It gave a set of many keyword arguments to functions
which didn't have any, then changed the defaults for those functions
so that one usually needed to pass those keyword arguments,

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 09/11/2023 15:36, Alan Mackenzie wrote:
> (i) cl--compiler-macro-list*; completely undocumented.
> (ii) cl--simple-expr-p: Talks about "side effects", but not what they
>    are side effects of.  Doesn't describe it's parameters or return
>    value.  It's unclear what it is that "executes quickly".
> (iii) cl--expr-contains: It's unclear what X and Y are, and what "refers
>    to" means.
> (iv) cl--expr-contains-any; completely undocumented.
> (v) cl--expr-depends-p: It's unclear what X and Y are, though Y appears
>    to be some sort of container of symbols.  It's unclear what sort of
>    "dependency" the function handles, or what "may" means in the context.

These aren't public-facing functions, though.

Adding docs wouldn't hurt, but it's not like these could be removed due 
to the absence of them (that should depend on which public callers they 
have).

Re: What's missing in ELisp that makes people want to use cl-lib?

[Alan Mackenzie]

Hello, Dmitry.

On Thu, Nov 09, 2023 at 13:48:34 +0200, Dmitry Gutov wrote:

> On 09/11/2023 12:34, Alan Mackenzie wrote:
> > As a concrete plan, I would propose the following for discussion:

> > We should deprecate those functions/macros/variables in cl-lib that have
> > no doc string, or a substandard one.  This includes "internal" functions,
> > too.  Also to be deprecated are obscure functions/m/v (such as
> > cl-labels).

> I'm not sure we can remove cl-labels -- it's useful enough for the cases 
> where it does help, and there are callers inside and outside of Emacs.

> Whatever other removals, would probably need to be discussed case-by-case.

There are probably too many such functions/macros/variables to make that
practicable.  If we did do this, the discussion for each f/m/v would
likely balloon out into a thread as long as this one, and nothing much
would get done.

> > Having done this, we recode code currently using those deprecated f/m/v.

> > (Here a "substandard" doc string is contrasted with an adequate one,
> > which does all of the following:
> > (i) It says what the function/macro_does_, or what the variable_is_.
> > (ii) It describes the form and meaning of each parameter, and its
> >    relationship to (i).
> > (iii) If the return value is significant, it describes this.
> > (iv) It describes all effects on the global state, such as where it
> >    writes results to, and suchlike.)

> Improving cl-lib's documentation would be a welcome effort.

Yes, people have been saying this for years.  I wrote a chapter on
cl-print.el for cl.info a couple of months back.  Other than that nobody
seems prepared to do anything much on this front.

What is needed is a sustained effort from a cl-lib enthusiast to fix the
50 - 100 doc strings which are missing or substandard.  Well, João has
offered to start off with 5 doc strings, which is good.  Maybe something
will come of that.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Alan Mackenzie <acm@muc.de> writes:

> On Thu, Nov 09, 2023 at 08:24:14 +0100, Gerd Möllmann wrote:
>> Alan Mackenzie <acm@muc.de> writes:
>
>>>Just take one look at this "reality" you're so supportive of: the
>>>widespread use of cl-lib, not just in people's own projects, but
>>>throughout the core of Emacs, has multiplied the size of Lisp language
>>>part of Emacs by a factor of around 3.   This is a gross increase in
>>>complexity for maintainers that is not justified by the slight increase
>>>in facility that cl-lib (along with things like seq.el and oclosures)
>>>gives.
>
>>>Throughout this long discussion, this indiscriminate use of cl-lib has
>>>been supported only by occasional contributers.  Those who actually
>>>maintain other people's code, apart from (I think) Eli, Richard and
>>>myself, have been conspicuously silent.  None of us three have favoured
>>>such use of cl-lib.
>
>>>Occasional contributors may be fascinated by cl-lib, and learn enough
>>>of it to use random bits of it in their code.  The trouble is, each
>>>such contributor uses a different piece of cl-lib, with the result that
>>>those who end up maintaining it need to know a far greater part of it
>>>just to cope.
>
>>>This factor of 3 is, I believe, a significant barrier to new
>>>programmers coming into Emacs; Elisp is just that much more difficult
>>>than it was in the past.  And it isn't just for newcomers that it is
>>>more difficult.  I spend a significant amount of debugging time having
>>>to look up doc strings and manual pages for obscure cl-lib (etc.)
>>>functions.
>
>> > That is the current reality.
>
>> Maybe you could elaborate on what the plan then could look like?
>> Or is it not about a plan, but something else? 
>
> You stripped all the context.  I've put it back again.
>
> As a concrete plan, I would propose the following for discussion:

Thanks for that. I find it much easier to digest than this clean/unclean
thing.

And wow, that will not be popular :-).

>
> We should deprecate those functions/macros/variables in cl-lib that have
> no doc string, or a substandard one.  This includes "internal" functions,
> too.  Also to be deprecated are obscure functions/m/v (such as
> cl-labels).

Am I right in assuming that this is not about the documentation itself,
but just some selection criterium for reducing the size of cl-lib?

> Having done this, we recode code currently using those deprecated
> f/m/v.

What would recode mean? Using seq/map?

>
> (Here a "substandard" doc string is contrasted with an adequate one,
> which does all of the following:
> (i) It says what the function/macro _does_, or what the variable _is_.
> (ii) It describes the form and meaning of each parameter, and its
>   relationship to (i).
> (iii) If the return value is significant, it describes this.
> (iv) It describes all effects on the global state, such as where it
>   writes results to, and suchlike.)
>
> This would reduce the size of cl-lib to something more manageable than
> what we have at the moment.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Alan Mackenzie]

Hello, Gerd.

On Thu, Nov 09, 2023 at 13:19:28 +0100, Gerd Möllmann wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > On Thu, Nov 09, 2023 at 08:24:14 +0100, Gerd Möllmann wrote:
> >> Alan Mackenzie <acm@muc.de> writes:

> >>>Just take one look at this "reality" you're so supportive of: the
> >>>widespread use of cl-lib, not just in people's own projects, but
> >>>throughout the core of Emacs, has multiplied the size of Lisp language
> >>>part of Emacs by a factor of around 3.   This is a gross increase in
> >>>complexity for maintainers that is not justified by the slight increase
> >>>in facility that cl-lib (along with things like seq.el and oclosures)
> >>>gives.

> >>>Throughout this long discussion, this indiscriminate use of cl-lib has
> >>>been supported only by occasional contributers.  Those who actually
> >>>maintain other people's code, apart from (I think) Eli, Richard and
> >>>myself, have been conspicuously silent.  None of us three have favoured
> >>>such use of cl-lib.

> >>>Occasional contributors may be fascinated by cl-lib, and learn enough
> >>>of it to use random bits of it in their code.  The trouble is, each
> >>>such contributor uses a different piece of cl-lib, with the result that
> >>>those who end up maintaining it need to know a far greater part of it
> >>>just to cope.

> >>>This factor of 3 is, I believe, a significant barrier to new
> >>>programmers coming into Emacs; Elisp is just that much more difficult
> >>>than it was in the past.  And it isn't just for newcomers that it is
> >>>more difficult.  I spend a significant amount of debugging time having
> >>>to look up doc strings and manual pages for obscure cl-lib (etc.)
> >>>functions.

> >> > That is the current reality.

> >> Maybe you could elaborate on what the plan then could look like?
> >> Or is it not about a plan, but something else? 

> > You stripped all the context.  I've put it back again.

> > As a concrete plan, I would propose the following for discussion:

> Thanks for that. I find it much easier to digest than this clean/unclean
> thing.

> And wow, that will not be popular :-).

It will be a mixed popular/unpopular, with probably few people in the
middle.  Back when you were the Maintainer, I think there was a ban on
the use of cl.el and friends, except for at compilation time.  I don't
think that caused any problems.

> > We should deprecate those functions/macros/variables in cl-lib that have
> > no doc string, or a substandard one.  This includes "internal" functions,
> > too.  Also to be deprecated are obscure functions/m/v (such as
> > cl-labels).

> Am I right in assuming that this is not about the documentation itself,
> but just some selection criterium for reducing the size of cl-lib?

The most troublesome cl-lib functions are those without adequate
documentation.  They're the ones that waste unbounded amounts of time
when trying to debug something which uses them.  That's why I'd like to
do away with them.  There are few people willing to fix the doc strings
in cl-lib, though João has volunteered to make a start.

> > Having done this, we recode code currently using those deprecated
> > f/m/v.

> What would recode mean? Using seq/map?

I hadn't really thought of seq.el or map.el.  What do they do?  But it's
clear that working code can be written without these or cl-lib.

> > (Here a "substandard" doc string is contrasted with an adequate one,
> > which does all of the following:
> > (i) It says what the function/macro _does_, or what the variable _is_.
> > (ii) It describes the form and meaning of each parameter, and its
> >   relationship to (i).
> > (iii) If the return value is significant, it describes this.
> > (iv) It describes all effects on the global state, such as where it
> >   writes results to, and suchlike.)

> > This would reduce the size of cl-lib to something more manageable than
> > what we have at the moment.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Alan Mackenzie wrote:

> Just take one look at this "reality" you're so supportive
> of: the widespread use of cl-lib, not just in people's own
> projects, but throughout the core of Emacs, has multiplied
> the size of Lisp language part of Emacs by a factor of
> around 3.

So say I write a function like this, which has nothing to do
with cl-lib BTW.

(defun nand (&rest conditions)
  (when (member nil conditions)
    t) )

Some would say it is consistent, since we have `not' and
`and', so why not NAND as well?

Others would say it is cleaner in source than (not (and ...))
- especially if you find use of that recurring, since it is
a single function, not a combination of two.

Let's then say I put it in a package that makes its way into
Emacs, and some people start to use it since they also find it
consistent, clean, and useful.

What has then happened?

Have I introduced something that is clean, consistent,
and useful?

Or have I increased the size of the Lisp language by 1, making
it harder for the maintainers to do their work?

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Alan Mackenzie wrote:

> The most troublesome cl-lib functions are those without
> adequate documentation. They're the ones that waste
> unbounded amounts of time when trying to debug something
> which uses them. That's why I'd like to do away with them.

Why not instead provide them with docstrings, I think everyone
can agree they should have them?

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Bob Rogers]

   From: Emanuel Berg <incal@dataswamp.org>
   Date: Thu, 09 Nov 2023 16:13:29 +0100

   Alan Mackenzie wrote:

   > The most troublesome cl-lib functions are those without
   > adequate documentation. They're the ones that waste
   > unbounded amounts of time when trying to debug something
   > which uses them. That's why I'd like to do away with them.

   Why not instead provide them with docstrings, I think everyone
   can agree they should have them?

As a long-time CL/EL hacker, I will also undertake to document cl-lib
functions.  I think that is a much easier, and safer, alternative to
trying to recode cl-* usage.

   FWIW, I am also of the philosophy that simple functions should be
used where possible and cl-* functions where necessary, all in the name
of keeping my own code concise and readable.  But, alas, readability is
in the eye of the beholder.

					-- Bob Rogers
					   http://www.rgrjr.com/

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >    Why not instead provide them with docstrings, I think everyone
  >    can agree they should have them?

  > As a long-time CL/EL hacker, I will also undertake to document cl-lib
  > functions.  I think that is a much easier, and safer, alternative to
  > trying to recode cl-* usage.

Those functions certainly should have doc strings, and adding some is
a good thing to do.  But that will not cure the larger problem
caused by too much usage of those functions.  They were meant to be used
in a few obsure corners, not used so often that maintainers need to know
about them.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Bob Rogers]

   From: Richard Stallman <rms@gnu.org>
   Date: Fri, 10 Nov 2023 22:13:51 -0500

   Those functions certainly should have doc strings, and adding some is
   a good thing to do.

Naturally.

   But that will not cure the larger problem caused by too much usage of
   those functions.  They were meant to be used in a few obsure corners,
   not used so often that maintainers need to know about them.

   -- 
   Dr Richard Stallman (https://stallman.org)
   Chief GNUisance of the GNU Project (https://gnu.org)
   Founder, Free Software Foundation (https://fsf.org)
   Internet Hall-of-Famer (https://internethalloffame.org)

I know Emacs has many obscure corners.  But I didn't think that any were
beyond the reach of the maintainers.  ;-}

					-- Bob

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >    But that will not cure the larger problem caused by too much usage of
  >    those functions.  They were meant to be used in a few obsure corners,
  >    not used so often that maintainers need to know about them.

  > I know Emacs has many obscure corners.  But I didn't think that any were
  > beyond the reach of the maintainers.  ;-}

I have the impression that that is meant as a clever riposte to the
words you qupted, but I don't understand the point.  The words you qupted
are not clever wit.  They are meant to be serious and straightforward.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Bob Rogers]

   From: Richard Stallman <rms@gnu.org>
   Date: Sun, 12 Nov 2023 22:07:47 -0500

   [[[ To any NSA and FBI agents reading my email: please consider    ]]]
   [[[ whether defending the US Constitution against all enemies,     ]]]
   [[[ foreign or domestic, requires you to follow Snowden's example. ]]]

     >    But that will not cure the larger problem caused by too much usage of
     >    those functions.  They were meant to be used in a few obsure corners,
     >    not used so often that maintainers need to know about them.

     > I know Emacs has many obscure corners.  But I didn't think that any were
     > beyond the reach of the maintainers.  ;-}

   I have the impression that that is meant as a clever riposte to the
   words you qupted, but I don't understand the point.  The words you qupted
   are not clever wit.  They are meant to be serious and straightforward.

It seemed odd to me that anything could be beyond what the maintainers
would need to know and deal with.  But I guess that ignores the matter
of threshold.  So I withdraw my off-the-mark jibe.

					-- Bob

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > It seemed odd to me that anything could be beyond what the maintainers
  > would need to know and deal with.  But I guess that ignores the matter
  > of threshold.

You understand it now.

                   So I withdraw my off-the-mark jibe.

Thank you.  I appreciate that.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Alan Mackenzie <acm@muc.de> writes:

>> Thanks for that. I find it much easier to digest than this clean/unclean
>> thing.
>
>> And wow, that will not be popular :-).
>
> It will be a mixed popular/unpopular, with probably few people in the
> middle.  Back when you were the Maintainer, I think there was a ban on
> the use of cl.el and friends, except for at compilation time.  I don't
> think that caused any problems.

Not that I remember, but it's of course a long time ago.

But somehow, CL discussions and the clean/unclean thing from Richard
were always present. At least I seem to remember that.

>> > We should deprecate those functions/macros/variables in cl-lib that have
>> > no doc string, or a substandard one.  This includes "internal" functions,
>> > too.  Also to be deprecated are obscure functions/m/v (such as
>> > cl-labels).
>
>> Am I right in assuming that this is not about the documentation itself,
>> but just some selection criterium for reducing the size of cl-lib?
>
> The most troublesome cl-lib functions are those without adequate
> documentation.  They're the ones that waste unbounded amounts of time
> when trying to debug something which uses them.  That's why I'd like to
> do away with them.  There are few people willing to fix the doc strings
> in cl-lib, though João has volunteered to make a start.

Ok, thanks.

>> > Having done this, we recode code currently using those deprecated
>> > f/m/v.
>
>> What would recode mean? Using seq/map?
>
> I hadn't really thought of seq.el or map.el.  What do they do?

I would describe both as sets of generic functions for sequence handling
and mapping over sequences. I'm personally not using them, as I wrote here:

  https://lists.gnu.org/archive/html/emacs-devel/2023-11/msg00122.html

That spawned answers from Eli and a sub-thread from Richard, which I
personally found unconvincing.

But I'd recommend in any case that you build your own opinion.

> But it's
> clear that working code can be written without these or cl-lib.
>
>> > (Here a "substandard" doc string is contrasted with an adequate one,
>> > which does all of the following:
>> > (i) It says what the function/macro _does_, or what the variable _is_.
>> > (ii) It describes the form and meaning of each parameter, and its
>> >   relationship to (i).
>> > (iii) If the return value is significant, it describes this.
>> > (iv) It describes all effects on the global state, such as where it
>> >   writes results to, and suchlike.)
>
>> > This would reduce the size of cl-lib to something more manageable than
>> > what we have at the moment.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Philip Kaludercic wrote:

> Could you explain what you mean by "traditional" Emacs Lisp?

Traditional Elisp does not use concepts from ML and CL - it
merely implements them.

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: rms@gnu.org,  joaotavora@gmail.com,  adam@alphapapa.net,
>   emacs-devel@gnu.org,  stefankangas@gmail.com
> Date: Thu, 02 Nov 2023 21:26:00 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > AFAIU, it means that Emacs Lisp traditionally doesn't use keyword
> > arguments, except as relatively rare exceptions.
> 
> It is used rather prominently in `define-minor-mode' or
> `define-derived-mode', or do you specifically mean keyword arguments to
> functions?

I did mention exceptions, didn't I?

And define-derived-mode supports keyword arguments only since Emacs
22; the original implementation didn't.

> >> what constitutes "Emacs Lisp"?  It would
> >> seem peculiar if it were to be defined by the arbitrary decisions of the
> >> past, constrained by the contingent circumstances of the time.
> >
> > Those "arbitrary decisions" are what got us to where we are now, 40
> > years later.  So some respect for those "arbitrary decisions" is due,
> > I think.
> 
> No disrespect meant, but I am not sure we are thinking of the same
> things.  An "arbitrary decision" usually doesn't matter much, like
> calling a function rplacd or setcdr.  If a decision got us to where we
> are now, I would say it wasn't that arbitrary, but a good one?

Exactly my point.  So what did you mean by "It would seem peculiar if
[what constitutes Emacs Lisp] were to be defined by the arbitrary
decisions of the past"?

> > IMNSHO, extending Emacs Lisp as the language is not the main goal of
> > Emacs development.  Emacs Lisp is not a programming language on its
> > own, it is a language for implementing and extending features and
> > extensions in Emacs.  Thus, the main goal of Emacs development is to
> > develop applications and application-level features, and provide more
> > opportunities for extending the core where that is considered useful.
> > What we have in Emacs Lisp is IMO ample for that purpose.  Moreover,
> > most participants in Emacs development are not experts in programming
> > languages, their expertise is elsewhere (which is definitely a Good
> > Thing).
> 
> Of course not extending it for its own sake, but I would assume that
> making it easier for users to write practical and useful code should be
> something that is valued.

We should consider such additions carefully, weighing their advantages
against the disadvantages: introducing "alien" syntax, making the
language larger, etc.

> > Objectively, adding new syntax and semantics to Emacs Lisp does make
> > the source code harder to read and maintain, because it makes the
> > language larger and requires familiarization with those new language
> > features, which more often than not look and feel completely different
> > from the "traditional" Emacs Lisp.  So even if we conclude that these
> > additions are worthwhile, we should not pretend they come without a
> > price, and IMO we should think carefully whether their use is
> > justified before we use them in each and every case.
> 
> Could you explain what you mean by "traditional" Emacs Lisp?

Basically, the language as it is, without macros whose syntax is
different from Emacs Lisp.  For example, cl-loop has syntax that to my
eyes is starkly not Emacs Lisp, because it uses many keyword-like
parts that look like they were lifted from Fortran.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: rms@gnu.org,  joaotavora@gmail.com,  adam@alphapapa.net,
>>   emacs-devel@gnu.org,  stefankangas@gmail.com
>> Date: Thu, 02 Nov 2023 21:26:00 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > AFAIU, it means that Emacs Lisp traditionally doesn't use keyword
>> > arguments, except as relatively rare exceptions.
>> 
>> It is used rather prominently in `define-minor-mode' or
>> `define-derived-mode', or do you specifically mean keyword arguments to
>> functions?
>
> I did mention exceptions, didn't I?
>
> And define-derived-mode supports keyword arguments only since Emacs
> 22; the original implementation didn't.

Ok, I interpreted this as in "few usages", not in "few definitions".

>> >> what constitutes "Emacs Lisp"?  It would
>> >> seem peculiar if it were to be defined by the arbitrary decisions of the
>> >> past, constrained by the contingent circumstances of the time.
>> >
>> > Those "arbitrary decisions" are what got us to where we are now, 40
>> > years later.  So some respect for those "arbitrary decisions" is due,
>> > I think.
>> 
>> No disrespect meant, but I am not sure we are thinking of the same
>> things.  An "arbitrary decision" usually doesn't matter much, like
>> calling a function rplacd or setcdr.  If a decision got us to where we
>> are now, I would say it wasn't that arbitrary, but a good one?
>
> Exactly my point.  So what did you mean by "It would seem peculiar if
> [what constitutes Emacs Lisp] were to be defined by the arbitrary
> decisions of the past"?

That "setcar" is supposed to be more Elisp-ish than "rplacd", or things
like that. 

>> > IMNSHO, extending Emacs Lisp as the language is not the main goal of
>> > Emacs development.  Emacs Lisp is not a programming language on its
>> > own, it is a language for implementing and extending features and
>> > extensions in Emacs.  Thus, the main goal of Emacs development is to
>> > develop applications and application-level features, and provide more
>> > opportunities for extending the core where that is considered useful.
>> > What we have in Emacs Lisp is IMO ample for that purpose.  Moreover,
>> > most participants in Emacs development are not experts in programming
>> > languages, their expertise is elsewhere (which is definitely a Good
>> > Thing).
>> 
>> Of course not extending it for its own sake, but I would assume that
>> making it easier for users to write practical and useful code should be
>> something that is valued.
>
> We should consider such additions carefully, weighing their advantages
> against the disadvantages: introducing "alien" syntax, making the
> language larger, etc.

What I had in mind were extensions like the recent `with-restriction'.
Defining something like this came to my mine on more than one occasion
when combining `save-restriction' and `narrow-to-region' in exactly the
same pattern as I had done many times before, because it seems natural
to have it combined in a single form.  I don't think this is necessarily
alien, perhaps even natural in the long term.

>> > Objectively, adding new syntax and semantics to Emacs Lisp does make
>> > the source code harder to read and maintain, because it makes the
>> > language larger and requires familiarization with those new language
>> > features, which more often than not look and feel completely different
>> > from the "traditional" Emacs Lisp.  So even if we conclude that these
>> > additions are worthwhile, we should not pretend they come without a
>> > price, and IMO we should think carefully whether their use is
>> > justified before we use them in each and every case.
>> 
>> Could you explain what you mean by "traditional" Emacs Lisp?
>
> Basically, the language as it is, without macros whose syntax is
> different from Emacs Lisp.  For example, cl-loop has syntax that to my
> eyes is starkly not Emacs Lisp, because it uses many keyword-like
> parts that look like they were lifted from Fortran.

Then again (cl-)loop is a peculiar example; even among CL programmers
there are many that dislike using it on the same grounds.  As a macro,
it doesn't really even attempt to blend into the surrounding language,
but uses special keywords and syntax parsing, that has both its
advantages and disadvantages, but it is still used if only because CL
doesn't have a simple `while' macro.

This is a much more stark example than pushnew.  I could imagine that
pushnew could be implemented without a keyword argument, if we accepted
three arguments (newelt place compare-fn).

But there are a plenty of definitions in cl-lib that in some form or
another I think would be nice to have in Elisp proper as well: list*,
flet, defmacro/defgeneric, defstruct would be some examples that come to
mind.  This is not only my personal perspective, but the feeling I get
from reviewing Elisp packages that include cl-lib for only two or three
definitions.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: rms@gnu.org,  joaotavora@gmail.com,  adam@alphapapa.net,
>   emacs-devel@gnu.org,  stefankangas@gmail.com
> Date: Fri, 03 Nov 2023 07:48:21 +0000
> 
> >> >> what constitutes "Emacs Lisp"?  It would
> >> >> seem peculiar if it were to be defined by the arbitrary decisions of the
> >> >> past, constrained by the contingent circumstances of the time.
> >> >
> >> > Those "arbitrary decisions" are what got us to where we are now, 40
> >> > years later.  So some respect for those "arbitrary decisions" is due,
> >> > I think.
> >> 
> >> No disrespect meant, but I am not sure we are thinking of the same
> >> things.  An "arbitrary decision" usually doesn't matter much, like
> >> calling a function rplacd or setcdr.  If a decision got us to where we
> >> are now, I would say it wasn't that arbitrary, but a good one?
> >
> > Exactly my point.  So what did you mean by "It would seem peculiar if
> > [what constitutes Emacs Lisp] were to be defined by the arbitrary
> > decisions of the past"?
> 
> That "setcar" is supposed to be more Elisp-ish than "rplacd", or things
> like that. 

I didn't mean traditional names, I meant traditional syntax and
semantics of constructs.

> >> Of course not extending it for its own sake, but I would assume that
> >> making it easier for users to write practical and useful code should be
> >> something that is valued.
> >
> > We should consider such additions carefully, weighing their advantages
> > against the disadvantages: introducing "alien" syntax, making the
> > language larger, etc.
> 
> What I had in mind were extensions like the recent `with-restriction'.

I doubt that this is what bothers Richard (and me, to some extent).

> >> Could you explain what you mean by "traditional" Emacs Lisp?
> >
> > Basically, the language as it is, without macros whose syntax is
> > different from Emacs Lisp.  For example, cl-loop has syntax that to my
> > eyes is starkly not Emacs Lisp, because it uses many keyword-like
> > parts that look like they were lifted from Fortran.
> 
> Then again (cl-)loop is a peculiar example; even among CL programmers
> there are many that dislike using it on the same grounds.

Grepping our own sources for "(cl-loop" comes back with more than 760
hits.  That's more than one use for every 2 Lisp files in our tree.

And cl-loop is not the only example of additions whose syntax deviates
from Emacs Lisp traditions.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> >> Could you explain what you mean by "traditional" Emacs Lisp?
>> >
>> > Basically, the language as it is, without macros whose syntax is
>> > different from Emacs Lisp.  For example, cl-loop has syntax that to my
>> > eyes is starkly not Emacs Lisp, because it uses many keyword-like
>> > parts that look like they were lifted from Fortran.
>> 
>> Then again (cl-)loop is a peculiar example; even among CL programmers
>> there are many that dislike using it on the same grounds.
>
> Grepping our own sources for "(cl-loop" comes back with more than 760
> hits.  That's more than one use for every 2 Lisp files in our tree.
>
> And cl-loop is not the only example of additions whose syntax deviates
> from Emacs Lisp traditions.

It certainly has fans, though I believe this is more due to the utility
and flexibility than the syntax.  If there were a Elisp'ish alternative
(in the sense in which pcase is also an alternative to cl-case --
assuming that you see pcase as Elisp'ish?), then I honestly believe it
could be more popular than cl-loop.  I would be disappointed if being
Elisp'ish would mean that these kinds of language features per se
wouldn't be permissible.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > what constitutes "Emacs Lisp"?  It would
  > > seem peculiar if it were to be defined by the arbitrary decisions of the
  > > past, constrained by the contingent circumstances of the time.

  > Those "arbitrary decisions" are what got us to where we are now, 40
  > years later.  So some respect for those "arbitrary decisions" is due,
  > I think.

Not only that, but -- these decusions were not arbitrary in the first
place.  They were based on thought and embodied an idea of design.
That's why they add up to a coherent whole.

And yes, they add up to Emacs Lisp as it is -- "where we are now" is
the sum of them.

  > > 1. Not standardised; it is possible to extend the language without
  > >    having to worry about how many implementations will follow along

  > IMNSHO, extending Emacs Lisp as the language is not the main goal of
  > Emacs development.  Emacs Lisp is not a programming language on its
  > own, it is a language for implementing and extending features and
  > extensions in Emacs.

That is absolutely right!  But there is a second error in the point
(1) that you are responding to: the idea that extending a license is
good, that more complexity in the form of language constructs

Complexity of a language imposes a cost on all users of that language.
Sometimes the right choice is to refuse ti extend the language.

  > > Emacs Lisp can learn from interesting ideas that other
  > > languages provide, adapt and add them, making them available to
  > > everyone.

  > It certainly can.  The question is: should it?  Since we are not
  > driven by any standard, it is completely up to us to make those
  > decisions, and we should IMO make them judiciously and carefully,
  > taking the downsides into consideration.  In particular, I hope people
  > agree that making a language too large and complex is not a good
  > thing in the long run.

I think the same.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Petteri Hintsanen]

Eli Zaretskii <eliz@gnu.org> writes:

>> Emacs Lisp can learn from interesting ideas that other
>> languages provide, adapt and add them, making them available to
>> everyone.
>
> It certainly can.  The question is: should it?  Since we are not
> driven by any standard, it is completely up to us to make those
> decisions, and we should IMO make them judiciously and carefully,
> taking the downsides into consideration.  In particular, I hope people
> agree that making a language too large and complex is not a good
> thing in the long run.

I don't know anything about Common Lisp, and only fundamentals of Emacs
Lisp, but I think the above is really the crux of this discussion.

I believe that the complexity of our tools (including programming
languages) affects the complexity of our products.  Therefore, a kind of
methodological minimalism is valid, as it tends to lead to simpler
solutions, which are almost always preferable.

Additionally, since debugging is at least twice as hard as writing new
code--and I'd extrapolate that debugging code written by others is at
least four times as hard--any efforts to make debugging easier are
commendable.  Often this boils down to general principles: write
clearly, don't be too clever, do the same thing the same way, accumulate
idioms [1, 2].  Limiting the number of acceptable language features,
either by preference or deprecation, is another way towards this aim.

Would it be useful to document recommended and/or avoidable practices,
for example by extending Appendix D (Tips and Conventions) in Emacs Lisp
reference manual?  Or maybe a separate set of guidelines for code that
is, or is likely to be, included in Emacs itself?

For a recent example of a growing hodgepodge complexity, consider C++.
C++98 was already a Byzantine language.  C++11 made it significantly
more complex, and the recent editions are borderline crazy.  Even the
creator of the language seems to agree [3].  Nowadays I cannot recommend
learning C++ for any novice programmer in good faith.  It would be sad
to see Emacs Lisp take the same path.  (Even though I agree this is
comparing apples to oranges; C++ is truly a general programming language
in the widest sense, while Emacs Lisp is not.)

Thanks,
Petteri

[1] Kernighan and Plauger: The Elements of Programming Style.  (1978)
[2] A J Perlis: Epigrams in Programming.  (1982)
[3] B Stroustrup: Remember the Vasa!  (2018)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Petteri Hintsanen <petterih@iki.fi>
> Cc: Philip Kaludercic <philipk@posteo.net>,  rms@gnu.org,
>   joaotavora@gmail.com,  adam@alphapapa.net,  emacs-devel@gnu.org,
>   stefankangas@gmail.com
> Date: Sun, 12 Nov 2023 13:08:20 +0200
> 
> Would it be useful to document recommended and/or avoidable practices,
> for example by extending Appendix D (Tips and Conventions) in Emacs Lisp
> reference manual?

It would, but first we need to have a kind of consensus on what those
practices are.  Currently, as this discussion demonstrates, we don't
have such a consensus.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Petteri Hintsanen wrote:

> I believe that the complexity of our tools (including
> programming languages) affects the complexity of our
> products. Therefore, a kind of methodological minimalism is
> valid, as it tends to lead to simpler solutions, which are
> almost always preferable.

Minimalism is not a clear concept. If we imagine an ideal
hospital, the best in the world at everything, is this
"maximalist" because it does everything and has all the best
equipment, or is it "minimalist" because it ONLY has that and
nothing unnecessary?

Instead of thinking if one should have a lot or a little,
I think one should have the best stuff available, and have
that as well organized as possible. Because if it isn't good,
and not organized, having just a little of it won't
help, really.

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > IMNSHO, extending Emacs Lisp as the language is not the main goal of
  > Emacs development.  Emacs Lisp is not a programming language on its
  > own, it is a language for implementing and extending features and
  > extensions in Emacs.  Thus, the main goal of Emacs development is to
  > develop applications and application-level features, and provide more
  > opportunities for extending the core where that is considered useful.

I agree fully.

  > Objectively, adding new syntax and semantics to Emacs Lisp does make
  > the source code harder to read and maintain, because it makes the
  > language larger and requires familiarization with those new language
  > features, which more often than not look and feel completely different
  > from the "traditional" Emacs Lisp.  So even if we conclude that these
  > additions are worthwhile, we should not pretend they come without a
  > price, and IMO we should think carefully whether their use is
  > justified before we use them in each and every case.

I agree with that point too.

Extensions to Emacs Lisp can have very different effects
on the use of Emacs Lisp.  We should distinguish between

1. Narrow-purpose extensions, such as open-network-stream

2. General-purposes extensions, such as pcase.

The narrow-purpose exensions tend to be useful without increasing the
effective complexity of programming.  If you want to do what
open-network-stream does, you will find it very helpful.  Otherwise,
it probably won't be called from the code you work on, so its
existence won't affect you at all -- you will never need to know about
it.

General-purposes estensions are just the opposite.  They could show up
in any code whatsoever, so everyone will need to learn about all of
them.  Thus, they burden everyone.  But they are never _needed_ in any
code/


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

[ Richard, with this message, I am also replying to your message in the
other thread ]

On Wed, Nov 1, 2023 at 1:48 AM Richard Stallman <rms@gnu.org> wrote:

>   >   Anyway, the point is that to hack on long-lived files
>   > such as lisp/minibuffer.el, one can't really "ignore" the new
>   > dictionary of seq.el anymore.
>
> That's exactly the problem.

Then why this laser-focus on cl-lib.el?  Why not criticize the use of
seq.el and map.el and pcase.el all of which "add many functions, which
have many details"?  Why are these libraries seemingly exempt from this
discussion?

The above question is mostly rhetorical, I am _not_ criticizing these
libraries or advocating restricting seq.el and map.el and pcase.el.

Reading certain parts of Emacs core has become impossible without
getting well acquainted with Stefan's Monnier pcase.el which I had to do
some 7-8 years ago.  It was an entirely alien creature when I first
saw it.  It is directly inspired by pattern matching of the ML language.

When reading a pcase,  I occasionally think to myself that it's overkill
complexity, that "in the old days" it was all much simpler.  In many
other cases I've learned to appreciate, and now it's part of my Lisp
toolbox too.

I think it's important to allow the programmers and maintainers working
on specific sections of the code a certain freedom to recommend
or select certain abstractions.  But Emacs's Lisp code base is now
very extensive and any kind of core-wide ban or rewrites of certain
patterns is counter-productive and highly contentious.  Let's be
very careful when going down that path.

On Thu, Nov 2, 2023 at 2:28 AM Richard Stallman <rms@gnu.org> wrote:
>
> It might be ok to add some keyword arguments to `sort', which are more
> unusual and complex to use, but not to simple constructs like
> `pushnew'.  This is Emacs Lisp, not Common Lisp.

Adding keyword arguments to a given function doesn't transform
Emacs Lisp into Common Lisp in any meaningful way.  Just as
translating a work by Shakespeare into Portuguese doesn't transform
Portuguese into English.  It just allows Portuguese speakers
to enjoy some very decent, maybe even essential, poetry.

Has Emacs Lisp become ML through the use of pcase.el?  Of course not.
It's become easier to write many things in Emacs Lisp, if anything.

Frequently, the argument advanced for not wanting to import feature
A or B from this other language, is "Emacs Lisp is to be kept
simple and not have this complexity".  Fair enough, programmers' craft
is foremost about managing complexity.  So very often a decision to
simplify here and now breeds enormous complexity elsewhere later.
Simplicity that proved effective and beneficial 30 years ago, in
a different context, might not really be those things today, or
30 years from now.

For example, the early decision not to have namespaces in Emacs Lisp,
taken in the name of simplicity, may have facilitated development of
Emacs features in a time where there were reasonably few packages.
But as the ecosystem grew, it created -- and still creates  -- many
challenges.

For example, one of the challenges it creates is this discussion
right here:  to be able to easily delimit what that "core" or
"standard" Emacs Lisp is -- which is what Philip Addressed.  There
is in fact no such thing.

This is _not_ necessarily advocating for namespaces/packages in Emacs
Lisp, it's just my way of pointing out that the argument of "simplicity"
is much more nuanced.  Is a keyword-argument-less 'pushnew' simpler
than 'cl-pushnew'?  Yes it is, undoubtedly.  Will the former breed
less complexity than the latter?  Very unlikely IMO.

Instead we should look at how much more complex than the hypothetical
'eq'-comparing version of 'pushnew' is `cl-pushnew` in the end.  In
the argumentless version it's exactly the same.  When a keyword
argument _is_ needed (say we need to test with #'equal or
#'my-special-comp)  doesn't that increase in complexity pay off
three-fold almost immediately?  IOW versatility of a given utility
must obviously be taken in consideration.

By the way, if it hasn't yet become apparent to you or other readers,
the keywords usually accepted to cl-sort, cl-pushnew, etc all belong
to a fairly small dictionary with consistent meaning.  For example,
the ':key' accepted by 'cl-sort' tells the thing to consider when
comparing, and the ':key' accepted by  'cl-pushnew' also tells
the thing to consider when comparing.  It is not more useful to
'cl-sort' than it is to 'cl-pushnew'.

Keyword arguments are IMHO one of the most elegant mechanisms to
promote maintainability, and generally typing less code.
Of course, there might be other possible, unique and original ways of
achieving the same of course, just as pcase.el showed me there were
other ways of doing destructuring.   But choosing any one option
just for the sake of "not being like any other language" is not the
right path to take.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

João Távora wrote:

>>> Anyway, the point is that to hack on long-lived files such
>>> as lisp/minibuffer.el, one can't really "ignore" the new
>>> dictionary of seq.el anymore.
>>
>> That's exactly the problem.
>
> Then why this laser-focus on cl-lib.el? Why not criticize
> the use of seq.el and map.el and pcase.el all of which "add
> many functions, which have many details"? Why are these
> libraries seemingly exempt from this discussion?

Indeed, this has been mentioned several times now and the
answer we get is "There is no special treatment of cl-lib.el".
Hard to believe.

> The above question is mostly rhetorical, I am _not_
> criticizing these libraries or advocating restricting seq.el
> and map.el and pcase.el.
>
> Reading certain parts of Emacs core has become impossible
> without getting well acquainted with Stefan's Monnier
> pcase.el which I had to do some 7-8 years ago. It was an
> entirely alien creature when I first saw it. It is directly
> inspired by pattern matching of the ML language.
>
> When reading a pcase, I occasionally think to myself that
> it's overkill complexity, that "in the old days" it was all
> much simpler. In many other cases I've learned to
> appreciate, and now it's part of my Lisp toolbox too.
>
> I think it's important to allow the programmers and
> maintainers working on specific sections of the code
> a certain freedom to recommend or select certain
> abstractions. But Emacs's Lisp code base is now very
> extensive and any kind of core-wide ban or rewrites of
> certain patterns is counter-productive and highly
> contentious. Let's be very careful when going down
> that path.

Very much so.

> Has Emacs Lisp become ML through the use of pcase.el?
> Of course not. It's become easier to write many things in
> Emacs Lisp, if anything.

It has become the SME(cl), the Standard Meta Emacs
Common Lisp. Deal with it.

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > It might be more aesthetically or ideologically pleasing, but it would 
  > be much less useful.

If we add a function `pushnew' to Emacs Lisp, it will not have a :text
argument.  We are not going to start adding those arguments to
ordinary Emavs Lisp functions.

We could define `pushnew-equal' as well as `pushnew'.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > It might be more aesthetically or ideologically pleasing, but it would 
  > be much less useful.

If we add a function `pushnew' to Emacs Lisp, it will have a calling convention
like other Emacs Lisp functions -- not using CL-style keyword arguments.

We could define two functions, `pushnew' and `pushnq', if they are
both worth adding.  Or maybe the version that uses `equal' would DTRT
for most calls and we would only need that one.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)