Re: [External] : Re: Shrinking the C core

Re: [External] : Re: Shrinking the C core

[Arthur Miller]

>[[[ To any NSA and FBI agents reading my email: please consider    ]]]
>[[[ whether defending the US Constitution against all enemies,     ]]]
>[[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>  > Of course, but nobody suggests it is all-in. `length' is not a keyworded
>  > in CL either.
>
>Oops, I thought it was -- but the time when I implemented and used
>Common Lisp was 40 years ago.
>
>I am pretty sure `member' used keyword arguments, and I think that
>getting behavior equivalent to traditional Lisp `member' required
>specifying a keyword argument.  I never forgot that, during that
>period, because I got reminded of it almost every day.
>
>  >  But it is convenient to have keywords in
>  > some places, like for example in define-minor-mode or make-process, or
>  > even "new" define-keymap.
>
>I would not object to using keyword arguments for functions like that
>-- complex and cumbersome to use, and not used often.

Well, than we pretty much agree 100% about keyword args. 40 years ago I
was playing in the woods, but as I have learned Lisp some barely 2 or 3
years ago, thanks to Emacs Lisp, that is pretty much how I understand
keyword arguments and why they are used, so that is why it felt a bit
strange when you said you don't want them, but I understand now that you
might have felt aversion if they were used everywhere; it certainly is a
PITA in functions with small number of arguments.

Question is also, if Maclisp was developed further, to current day, if
and how it would evolve itself. As time goes buy we discover new idioms,
patterns, have different needs etc. These things changes and new ideas
emerges; it is very much probable that Maclisp itself would look
differently than what it was when CL was conceived. I am sure they
wouldn't have those paires of complementatry functions
(remove-if/remove-if-not) which make for a big part of CL verbosity too.

RE: [External] : Re: Shrinking the C core

[Drew Adams]

[Moving reply to e-tangents.]

> > I would not object to using keyword arguments for functions
> > like that -- complex and cumbersome to use, and not used often.
> 
> Well, than we pretty much agree 100% about keyword args.

+1.

Except maybe for the "not used often".  Depends on the
"often" use.  If used often without any keyword args (or
optional args, for that matter), what difference does it
make that the function _has_ keyword/optional args?  By
definition and convention, such args are generally not
for the most common (the default) use case.

But the general point, I think, is that keyword (and
optional) args turn one function into a family of several,
and that can be useful.  But the "base" function, called without such args, can be just as convenient as if there
were no such args possible.

Re: [External] : Re: Shrinking the C core

[Arthur Miller]

Drew Adams <drew.adams@oracle.com> writes:

> [Moving reply to e-tangents.]
>
>> > I would not object to using keyword arguments for functions
>> > like that -- complex and cumbersome to use, and not used often.
>> 
>> Well, than we pretty much agree 100% about keyword args.
>
> +1.
>
> Except maybe for the "not used often".  Depends on the
> "often" use.  If used often without any keyword args (or
> optional args, for that matter), what difference does it
> make that the function _has_ keyword/optional args?  By
> definition and convention, such args are generally not
> for the most common (the default) use case.

Well yes, of course, it is corect. I think it depends on what "not often
used" referred to. I interepretted it as not often used argument, not
the function itself. Those more often used but possible to omit are of
course usually optional and in front of the optional list, but those
that are used in some more specialized circumstance or by very few are
perhaps good candidates for keyword arguments? I don't know, that is how
I perecieve it.

> But the general point, I think, is that keyword (and
> optional) args turn one function into a family of several,
> and that can be useful.  But the "base" function, called without such args, can be just as convenient as if there
> were no such args possible.

Yes indeed; and I agree with what you wrote in your longer second mail
about that. They do lessen the need for more functions which means less
API surface to document and potentially lookup and learn. Isn't a good
example that window functions that I wanted to make into same once in
the context of help/info patch? I don't remember which it were tbh
:). Something with selecting window on just current frame or all frames
or something like that.

Re: [External] : Re: Shrinking the C core

[Emanuel Berg]

Arthur Miller wrote:

> Yes indeed; and I agree with what you wrote in your longer
> second mail about that. They do lessen the need for more
> functions which means less API surface to document and
> potentially lookup and learn. Isn't a good example that
> window functions that I wanted to make into same once in the
> context of help/info patch? I don't remember which it were
> tbh :). Something with selecting window on just current
> frame or all frames or something like that.

I think, in general and ranked from best to worse,

1. Many functions, all with few arguments
2. One function with not-that-many optional arguments
3. One function with keyword arguments
4. One function with too many optional arguments

Documentation is not a problem for case 1 since small
functions with clear names and few arguments typically
requires just 1~3 lines of documentation.

This documentation is also easy to browse, you won't have to
navigate the help buffer or search it for what an argument or
keyword does, written far below somewhere in the text.

-- 
underground experts united
https://dataswamp.org/~incal

RE: [External] : Re: Shrinking the C core

[Drew Adams]

> I think, in general and ranked from best to worse,
> 
> 1. Many functions, all with few arguments
> 2. One function with not-that-many optional arguments
> 3. One function with keyword arguments
> 4. One function with too many optional arguments

I don't think any such generalization is
very helpful - regardless of how one might
choose to order those.

> Documentation is not a problem for case 1 since small
> functions with clear names and few arguments typically
> requires just 1~3 lines of documentation.
> 
> This documentation is also easy to browse, you won't have to
> navigate the help buffer or search it for what an argument or
> keyword does, written far below somewhere in the text.

I think you're missing the point I made about
a "family" of functions.

Just having separate "many functions, all with
few arguments" can make it more difficult to
understand which is which, what's what, and
when to use this one or that one.

More generally, it's harder to see what the
familial relations are.  You can't as easily
see the forest because of all the trees.

Re: [External] : Re: Shrinking the C core

[Emanuel Berg]

Drew Adams wrote:

> I think you're missing the point I made about a "family"
> of functions.
>
> Just having separate "many functions, all with few
> arguments" can make it more difficult to understand which is
> which, what's what, and when to use this one or that one.
>
> More generally, it's harder to see what the familial
> relations are. You can't as easily see the forest because of
> all the trees.

Indeed, you have to be familiar with the functions in order to
use them. Just as you have to be familiar with the optional
arguments and keywords to use them. So there is no difference
in that regard.

But I think the functions are easier to see than arguments and
keywords which are inside functions, because when I use
functions, I'm myself outside, not inside of them.

Or put it like this: If I have a computer that can do 100
things, I prefer 100 functions to 2 functions, one with
50 optional arguments and the other with 50 keywords.

-- 
underground experts united
https://dataswamp.org/~incal

RE: [External] : Re: Shrinking the C core

[Drew Adams]

> > I think you're missing the point I made about a "family"
> > of functions.
> >
> > Just having separate "many functions, all with few
> > arguments" can make it more difficult to understand which is
> > which, what's what, and when to use this one or that one.
> >
> > More generally, it's harder to see what the familial
> > relations are. You can't as easily see the forest because of
> > all the trees.
> 
> Indeed, you have to be familiar with the functions in order to
> use them. Just as you have to be familiar with the optional
> arguments and keywords to use them. So there is no difference
> in that regard.

There is a difference.  The doc of for the family,
i.e., all the variants of the function, is in one
place.  And that doc can, and typically does,
describe them together, i.e., it describes their
relationships.

And another, obvious, difference, for things like
apropos search and completion of commands.

> But I think the functions are easier to see than arguments and
> keywords which are inside functions, because when I use
> functions, I'm myself outside, not inside of them.

"The functions" can't be "easier to see" than the
single function, which encompasses the family.

Of course, you haven't made clear what you mean
by "to see" or your being "outside"/"inside"
functions.  So maybe you have some unsaid meanings
that fit your "easier to see" characterization.
 
> Or put it like this: If I have a computer that can do 100
> things, I prefer 100 functions to 2 functions, one with
> 50 optional arguments and the other with 50 keywords.

There's no need to exaggerate or deal with such
a level of abstraction.  The ability to have
&optional, &rest, and keyword arguments doesn't
prevent anyone from not making use of any of those
and instead defining 8000 separate functions (to
further the exaggeration).

Do you use &optional?  &rest?  Why, since you
apparently have a blanket rule that using
multiple separate functions is always better.

Re: [External] : Re: Shrinking the C core

[Emanuel Berg]

Drew Adams wrote:

> "The functions" can't be "easier to see" than the single
> function

Many functions that each do little are easier to see than few
functions that each do a lot, because with many functions that
each do little, the function names are often enough to
understand what they do and how they work - and they require
none or very little documentation.

But with few functions that each do a lot, one has to check
the documentation for each function, and the documentation
will be huge, because few functions that each do a lot either
have to have an almost endless list of arguments or rely on an
equally long list of possible keywords to control
their behavior.

For example, (+ ...), (- ...) etc are preferable to
(arithmetic :operation 'addition ...),
(arithmetic :operation 'subtraction ...) etc.

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: Shrinking the C core

[Yuri Khan]

On Sun, 17 Sept 2023 at 17:46, Emanuel Berg <incal@dataswamp.org> wrote:

> For example, (+ ...), (- ...) etc are preferable to
> (arithmetic :operation 'addition ...),
> (arithmetic :operation 'subtraction ...) etc.

Let’s reframe that in more real-life examples. What is preferable and why:

1. a couple of functions
   a. ‘(my-sort-by SEQ LESSP)’ where LESSP is a function accepting two
elements X and Y and returning ‘t’ when X should be sorted before Y,
and
   b. ‘(my-sort-on SEQ KEY)’ where KEY is a function accepting an
element X and returning a number such that sorting by that number
yields the desired order; or

2. a single function that could be called as ‘(my-sort SEQ :by LESSP)’
or ‘(my-sort SEQ :key KEY)’, with the same semantics as above?

Does your answer change if we want to extend the API to also support
sorting in descending order? If it does, how?

Re: [External] : Re: Shrinking the C core

[Emanuel Berg]

Yuri Khan wrote:

> 1. a couple of functions
>    a. ‘(my-sort-by SEQ LESSP)’ where LESSP is a function accepting two
>        elements X and Y and returning ‘t’ when X should be
>        sorted before Y, and
>    b. ‘(my-sort-on SEQ KEY)’ where KEY is a function accepting an
>        element X and returning a number such that sorting by
>        that number yields the desired order; or
>
> 2. a single function that could be called as ‘(my-sort SEQ
>    :by LESSP)’ or ‘(my-sort SEQ :key KEY)’, with the same
>    semantics as above?
>
> Does your answer change if we want to extend the API to also
> support sorting in descending order? If it does, how?

If there are several tendencies and in particular if those can
be combined one can absolutely use keywords (and extend by
adding more), but then we are at a level of complexity which
would translate to a huge amount of envisioned simple, one or
two argument functions, so instead of doing that, if one were
to stay at the simple function end of the spectrum, one would
use combinations of such simple functions, e.g.
(reverse (sort ... ))

At the opposite end where functions are insanely complex,
keywords are a way to make the interface more clear and
facilitate the submission of arguments from code.

So it is good keywords exist when it is complicated, in
general I prefer it when it is simple but sometimes the
complexity is such it is actually easier to not try to keep it
simple - since that is impossible - but to stay at the complex
side and take it from there.

Optional arguments don't really belong on that simple/complex
scale, rather they should be used when there is an intuitive
default, for example (forward-line &optional N) where
N defaults to 1.

-- 
underground experts united
https://dataswamp.org/~incal

RE: [External] : Re: Shrinking the C core

[Drew Adams]

> Many functions that each do little are easier to see than few
> functions that each do a lot, because with many functions that
> each do little, the function names are often enough to
> understand what they do and how they work - and they require
> none or very little documentation.
> 
> But with few functions that each do a lot, one has to check
> the documentation for each function, and the documentation
> will be huge, because few functions that each do a lot either
> have to have an almost endless list of arguments or rely on an
> equally long list of possible keywords to control
> their behavior.
> 
> For example, (+ ...), (- ...) etc are preferable to
> (arithmetic :operation 'addition ...),
> (arithmetic :operation 'subtraction ...) etc.

You continue to ignore this in your posts:

  The ability to have &optional, &rest, and keyword
  arguments doesn't prevent anyone from _not_ making
  use of any of those and instead defining 8000
  separate functions (to further the exaggeration).

  Do _you_ ever use &optional or &rest when defining
  functions?  If so, why, since you apparently have
  a blanket rule that using multiple separate
  functions is always better?
 
Everything you've said against keyword args (even
just allowing them?) applies equally to &optional
and &rest.  Should they be outlawed or deprecated?

Please answer the question: do you define functions
that use those?  If so, how do you reconcile that
practice with your claim that it's always better to
define separate functions instead?  Why do you use

 (defun foo (arg1 &optional arg2 arg3 arg4)...)

instead of these, since you argue that these must
be better?

 (defun foo1 (arg1) ...)
 (defun foo2 (arg1 arg2) ...)
 (defun foo3 (arg1 arg2 arg3) ...)
 (defun foo4 (arg1 arg2 arg3 arg4) ...)

It's a no-brainer that:

1. Providing the _possibility_ of using &optional,
   &rest - and, yes, keyword args - doesn't _oblige_
   anyone to define functions that use those.

   Such arguments just _allow_ you to define, and
   users to use, a function that takes the place
   of multiple related functions.  Or that serves
   as a helper/workhorse, to _define_ multiple
   related functions.

2. That provides a place and a name for the family
   as a whole, and a place for their _relations_
   to be set forth explicitly in doc.  You get not
   only individual views of particular trees but
   views of tree communities: particular forests.

3. Keyword args let users use fewer actual args
   than when &optional args are used - nils that
   are only positional placeholders disappear.

4. Keyword args make clear what the intention of
   each arg is (it's named!).  Contrast that with
   having to consult the doc each time to figure
   out what each positional arg means.

When calling a function, both (1) the need to stick
in positional-placeholder nils and (2) the need to
identify a keyword arg by its name make function
calls more verbose (an argument you could have made,
but didn't, in favor of not using &optional, &rest,
and keyword args).

It's a tradeoff, for not needing to define and use
more functions.  When defining functions _you_ get
to make that tradeoff decision.  That's the point -
the language doesn't decide for you, by offering
only one possibility.  Lisp says, "You're welcome!"

If every function had no relatives, you might have
an argument.  But if that were the case then no one
would ever define or use &optional or &rest or
keyword args, and those never would have been added
to Lisp in the first place.

(BTW, &rest is essentially just a shortcut for an
explicit list argument.  Some languages make you
pass a vector/sequence/list each time, even when
that's empty: [], "", ().)

Finally, perhaps the most common uses of &optional,
&rest, and keyword args are for defining a family
workhorse function, which is then used to define
other functions with fewer or no such args, which
serve particularly common use cases.

And often the most common use case, even for end
users, is just to call the workhorse function with
no such actual args.  IOW, the "family" function
is often, maybe even typically, defined so that it
can be used as is, with no such args, to provide a
useful default behavior.

Thus it has been, for decades.  And thus it will
continue to be...

Re: [External] : Re: Shrinking the C core

[Emanuel Berg]

Drew Adams wrote:

> your claim that it's always better to define separate
> functions instead?

You are claiming that - but I disagree.

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: Shrinking the C core

[Emanuel Berg]

Arthur Miller wrote:

> I am sure they wouldn't have those paires of complementatry
> functions (remove-if/remove-if-not) which make for a big
> part of CL verbosity too.

What is wrong with them?

We have them in Elisp as well, `cl-remove-if' and
`cl-remove-if-not', both in cl-seq.el.

Those functions are pretty useful IMO including use which
includes the keywords, all tho that is optional.

The only thing negative is the documentation, see for example
the docstring of `cl-remove-if-not':

  Remove all items not satisfying PREDICATE in SEQ.
  This is a non-destructive function; it makes a copy of SEQ
  if necessary to avoid corrupting the original SEQ.

  Keywords supported:  :key :count :start :end :from-end

As you see, it only says what keywords are "supported", not
what they express or how they effect the execution of
the function.

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: Shrinking the C core

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > We have them in Elisp as well, `cl-remove-if' and
  > `cl-remove-if-not', both in cl-seq.el.

A partial emulation of some Common Lisp functions is present
in the cl-lib library, for emulation purposes.  It is not supposed to
be used a lot.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [External] : Re: Shrinking the C core

[Alfred M. Szmidt]

   [[[ To any NSA and FBI agents reading my email: please consider    ]]]
   [[[ whether defending the US Constitution against all enemies,     ]]]
   [[[ foreign or domestic, requires you to follow Snowden's example. ]]]

     > We have them in Elisp as well, `cl-remove-if' and
     > `cl-remove-if-not', both in cl-seq.el.

   A partial emulation of some Common Lisp functions is present
   in the cl-lib library, for emulation purposes.  It is not supposed to
   be used a lot.

And if there are functions or features that make sense from Common
Lisp, they can always be added piecemeal.  There is no need to make
Emacs Lisp complicated for the sake of compatibility with Common Lisp.

Someone mentioned &optional, and with default values -- that would be
a nice addition to Emacs Lisp.

Re: [External] : Re: Shrinking the C core

[Emanuel Berg]

Alfred M. Szmidt wrote:

>>> We have them in Elisp as well, `cl-remove-if' and
>>> `cl-remove-if-not', both in cl-seq.el.
>>
>> A partial emulation of some Common Lisp functions is
>> present in the cl-lib library, for emulation purposes.
>> It is not supposed to be used a lot.
>
> And if there are functions or features that make sense from
> Common Lisp, they can always be added piecemeal. There is no
> need to make Emacs Lisp complicated for the sake of
> compatibility with Common Lisp.

I don't know why cl-lib was added to GNU Emacs and Emacs Lisp,
maybe it was, as you say, for emulation purposes and for the
sake of compatibility with Common Lisp. But the way it is used
today, as we just heard in 28% of vanilla Emacs files, isn't
because of CL emulation or compatibility purposes, but because
it adds useful features and covers aspects that non-cl-lib
Elisp leaves blank.

For functions like `cl-incf', where there is no corresponding
"incf" [I don't know whatever happened to it or why it was
dropped, or maybe it wasn't ever there before cl-lib?] one
could setup aliases or simply drop the "cl-" prefix (and set
an "cl-incf" alias to the new "incf" as not to break existing
code).

Then, for functions like `cl-defun' and `defun', one would
examine if those could be merged into a new "defun", with the
CL features brought over, e.g. the default &optional argument
value syntax. Here one would again use aliases to cover
the back.

For functions that cannot be merged because they are
inherently different, if such cases exist, one would keep the
"cl-" prefix and the incompatible non-cl-lib Elisp function.

But one could also just leave it the way it is! Since those
prefixes are hardly a big problem. And especially not compared
to all the good features cl-lib brings to our game.

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: Shrinking the C core

[Alfred M. Szmidt]

One should think carefully before thinking of what one should do, or
would do.  What one can do is anything... and that is not always the
best thing, these things should be done very carefully and on a case
by case basis.  So lets not go crazy.

RE: [External] : Re: Shrinking the C core

[Drew Adams]

> For functions like `cl-incf', where there is no corresponding
> "incf" [I don't know whatever happened to it or why it was
> dropped, or maybe it wasn't ever there before cl-lib?] one
> could setup aliases or simply drop the "cl-" prefix (and set
> an "cl-incf" alias to the new "incf" as not to break existing
> code).

The CL emulation was originally in file `cl.el' and
its helper files, such as `cl-macs.el'.  At that time
functions/macros such as `cl-incf' and `cl-case' had
no `cl-' prefix.  E.g., Emacs 20:
___

 incf is a Lisp macro in `cl'.
 (incf PLACE &optional X)

 (incf PLACE [X]): increment PLACE by X (1 by default).
 PLACE may be a symbol, or any generalized variable allowed by `setf'.
 The return value is the incremented value of PLACE.
___

 case is a Lisp macro in `cl-macs'.
 (case EXPR &rest CLAUSES)

 (case EXPR CLAUSES...): evals EXPR, chooses from CLAUSES on that value.
 Each clause looks like (KEYLIST BODY...).  EXPR is evaluated and compared
 against each key in each KEYLIST; the corresponding BODY is evaluated.
 If no clause succeeds, case returns nil.  A single atom may be used in
 place of a KEYLIST of one atom.  A KEYLIST of `t' or `otherwise' is
 allowed only in the final clause, and matches if no other keys match.
 Key values are compared by `eql'.
___

There's no `incf' or `case' in Elisp that would
conflict with the CL-emulation definitions, but
when Someone decided to prefix CL thingies with
`cl-' the decision was to do it wholesale, not
just where there was a collision/conflict with
existing Elisp thingies.

Re: [External] : Re: Shrinking the C core

[Emanuel Berg]

Drew Adams wrote:

>> For functions like `cl-incf', where there is no
>> corresponding "incf" [I don't know whatever happened to it
>> or why it was dropped, or maybe it wasn't ever there before
>> cl-lib?] [...]
>
> The CL emulation was originally in file `cl.el' and its
> helper files, such as `cl-macs.el'. At that time
> functions/macros such as `cl-incf' and `cl-case' had no
> `cl-' prefix. [...]
>
> There's no `incf' or `case' in Elisp that would conflict
> with the CL-emulation definitions, but when Someone decided
> to prefix CL thingies with `cl-' the decision was to do it
> wholesale, not just where there was a collision/conflict
> with existing Elisp thingies.

Ah, I see!

Thanks for explaining.

So I remembered correctly then, that I did use "incf" before
`cl-incf'!

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: Shrinking the C core

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I think the reason we added `cl-' to all the CL emulations is to
inform people that they are not parts of what Emacs Lisp supports.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [External] : Re: Shrinking the C core

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > But the way it is used
  > today, as we just heard in 28% of vanilla Emacs files, isn't
  > because of CL emulation or compatibility purposes, but because
  > it adds useful features and covers aspects that non-cl-lib
  > Elisp leaves blank.

I consider this a problem.  Such frequent use of the cl facilities --
even though it is just the macros -- adds those cl macros to what
people need to know to understand those files.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [External] : Re: Shrinking the C core

[Emanuel Berg]

Richard Stallman wrote:

>> But the way it is used today, as we just heard in 28% of
>> vanilla Emacs files, isn't because of CL emulation or
>> compatibility purposes, but because it adds useful features
>> and covers aspects that non-cl-lib Elisp leaves blank.
>
> I consider this a problem. Such frequent use of the cl
> facilities -- even though it is just the macros -- adds
> those cl macros to what people need to know to understand
> those files.

Okay, I understand. I must admit I don't really care about
other people at such a fine grained level, but let's think
then ...

I think that non-programmers are a lost cause in this case
anyway and for programmers the complexity increase from
non-cl-lib Elisp to cl-lib is pretty much negligible?

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: Shrinking the C core

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Tue, 19 Sep 2023 13:21:59 +0200
> 
> for programmers the complexity increase from non-cl-lib Elisp to
> cl-lib is pretty much negligible?

No, it isn't.  There's quite a lot of different syntax and semantics
that need to be learned and mastered.

Re: [External] : Re: Shrinking the C core

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I think that non-programmers are a lost cause in this case
  > anyway and for programmers the complexity increase from
  > non-cl-lib Elisp to cl-lib is pretty much negligible?

This dichotomy oversimplifies the various levels of skill and
knowledge that people have for Emacs Lisp programming.  People's skill
at programming varies across a wide range.  Programmers' knowledge of
Emacs Lisp likewise -- there is so much you could learn if you want to
but few have time to learn it all.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [External] : Re: Shrinking the C core

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Sat, 16 Sep 2023 20:46:24 -0400
> 
> A partial emulation of some Common Lisp functions is present
> in the cl-lib library, for emulation purposes.  It is not supposed to
> be used a lot.

466 out of 1637 Lisp files in Emacs require cl-lib (some of them only
during compilation, i.e. they use only the macros).

Re: [External] : Re: Shrinking the C core

[Emanuel Berg]

Eli Zaretskii wrote:

>> A partial emulation of some Common Lisp functions is
>> present in the cl-lib library, for emulation purposes.
>> It is not supposed to be used a lot.
>
> 466 out of 1637 Lisp files in Emacs require cl-lib (some of
> them only during compilation, i.e. they use only the
> macros).

466 out of 1637 files is 28%. It seems that, regardless of any
anti cl-lib sentiments present, the issue has been decided on
the field already.

(format "%d%%" (round (* 100 (/ 466 1637.0)))) ; 28%

The only thing that would have made it more striking is if
Emacs had 1337 files instead.

-- 
underground experts united
https://dataswamp.org/~incal

RE: [External] : Re: Shrinking the C core

[Drew Adams]

> (format "%d%%" (round (* 100 (/ 466 1637.0)))) ; 28%


OT, but wrt `format':

Recently I've seen some questions on SE.Emacs that
resulted from users trying to use Common Lisp's
`format' in Elisp, that is, trying to use `format'
with DESTINATION first arg and with Common Lisp's
kind of CONTROL-STRING arg.

Re: [External] : Re: Shrinking the C core

[Emanuel Berg]

Drew Adams wrote:

>> (format "%d%%" (round (* 100 (/ 466 1637.0)))) ; 28%
>
> OT, but wrt `format':
>
> Recently I've seen some questions on SE.Emacs that resulted
> from users trying to use Common Lisp's `format' in Elisp,
> that is, trying to use `format' with DESTINATION first arg
> and with Common Lisp's kind of CONTROL-STRING arg.

Maybe we should have a "cl-format" then for the CL format of
`format'?

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: Shrinking the C core

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > 466 out of 1637 Lisp files in Emacs require cl-lib (some of them only
  > during compilation, i.e. they use only the macros).

If a file uses cl only during compilation (for macros) it is
not much of a problem.  How many use it at run time?

I reported one file a few weeks ago that is always (or nearly always)
loaded and uses cl-lib, forcin it to be nearly always loaded too.
That should be fixed.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [External] : Re: Shrinking the C core

[Emanuel Berg]

Richard Stallman wrote:

>> 466 out of 1637 Lisp files in Emacs require cl-lib (some of
>> them only during compilation, i.e. they use only the
>> macros).
>
> If a file uses cl only during compilation (for macros) it is
> not much of a problem. How many use it at run time?
>
> I reported one file a few weeks ago that is always (or
> nearly always) loaded and uses cl-lib, forcin it to be
> nearly always loaded too. That should be fixed.

But if it is shipped and included with vanilla Emacs, which it
is, how can it be either any more or any less (dis)encouraged
for use than anything else included on the same premises?

If something isn't part of vanilla Emacs one can maybe say
"see if you can do without it, because if you use it, it has
to be brought in externally adding complexity" - perhaps.

But it _is_ included so in terms of technology it is on the
same level as everything else included. Are we gonna have one
big toolbox containing a bunch of tools, but on certain tools
put on little stickers saying "don't use this"?

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: Shrinking the C core

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Sun, 15 Oct 2023 04:46:45 +0200
> 
> Richard Stallman wrote:
> 
> >> 466 out of 1637 Lisp files in Emacs require cl-lib (some of
> >> them only during compilation, i.e. they use only the
> >> macros).
> >
> > If a file uses cl only during compilation (for macros) it is
> > not much of a problem. How many use it at run time?
> >
> > I reported one file a few weeks ago that is always (or
> > nearly always) loaded and uses cl-lib, forcin it to be
> > nearly always loaded too. That should be fixed.
> 
> But if it is shipped and included with vanilla Emacs, which it
> is, how can it be either any more or any less (dis)encouraged
> for use than anything else included on the same premises?
> 
> If something isn't part of vanilla Emacs one can maybe say
> "see if you can do without it, because if you use it, it has
> to be brought in externally adding complexity" - perhaps.
> 
> But it _is_ included so in terms of technology it is on the
> same level as everything else included. Are we gonna have one
> big toolbox containing a bunch of tools, but on certain tools
> put on little stickers saying "don't use this"?

This thread is not for casual users of Emacs, it is for active Emacs
developers and maintainers.  So please just read it and don't try to
chime in, as that doesn't help.

Re: [External] : Re: Shrinking the C core

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: incal@dataswamp.org, emacs-devel@gnu.org
> Date: Sat, 14 Oct 2023 21:53:56 -0400
> 
>   > 466 out of 1637 Lisp files in Emacs require cl-lib (some of them only
>   > during compilation, i.e. they use only the macros).
> 
> If a file uses cl only during compilation (for macros) it is
> not much of a problem.  How many use it at run time?

226.

> I reported one file a few weeks ago that is always (or nearly always)
> loaded and uses cl-lib, forcin it to be nearly always loaded too.
> That should be fixed.

Which file was that?  I cannot reason about its causes to load cl-lib
without knowing the details, which include looking into the file and
considering what it does.

Re: [External] : Re: Shrinking the C core

[Emanuel Berg]

Eli Zaretskii wrote:

>>> 466 out of 1637 Lisp files in Emacs require cl-lib (some
>>> of them only during compilation, i.e. they use only the
>>> macros).
>> 
>> If a file uses cl only during compilation (for macros) it
>> is not much of a problem. How many use it at run time?
>
> 226.

That means that, out of a total number of 1637 Lisp files in
vanilla Emacs,

466 files, or 28%, require cl-lib overall;

240 files, or 15%, use cl-lib at compile time; and

226 files, or 14%, use cl-lib at run time.

-- 
underground experts united
https://dataswamp.org/~incal

Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I've changed the subject line to match what we are talking about.

  > 226 files, or 14%, use cl-lib at run time.

These cases may be more or less problematical, but not necessarily
equally so.

Which of those files are usually loaded when you start Emacs with
no arguments?  Those we should certainly fix.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Richard Stallman wrote:

> I've changed the subject line to match what we are
> talking about.
>
>> 226 files, or 14%, use cl-lib at run time.
>
> These cases may be more or less problematical, but not
> necessarily equally so.
>
> Which of those files are usually loaded when you start Emacs
> with no arguments? Those we should certainly fix.

I don't know, but at least one since you get the cl-lib stuff
even at 'emacs -Q'.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Thu, 19 Oct 2023 06:14:52 +0200
> 
> Richard Stallman wrote:
> 
> >> 226 files, or 14%, use cl-lib at run time.
> >
> > These cases may be more or less problematical, but not
> > necessarily equally so.
> >
> > Which of those files are usually loaded when you start Emacs
> > with no arguments? Those we should certainly fix.
> 
> I don't know, but at least one since you get the cl-lib stuff
> even at 'emacs -Q'.

That is not true, since in "emacs -Q" I get this:

  (featurep 'cl-lib) => nil

both in GUI and TTY (i.e. -nw) sessions.

So if cl-lib is loaded at startup, Emacs does that because of some
user customizations, not by default.

Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Eli Zaretskii wrote:

>>>> 226 files, or 14%, use cl-lib at run time.
>>>
>>> These cases may be more or less problematical, but not
>>> necessarily equally so.
>>>
>>> Which of those files are usually loaded when you start
>>> Emacs with no arguments? Those we should certainly fix.
>> 
>> I don't know, but at least one since you get the cl-lib
>> stuff even at 'emacs -Q'.
>
> That is not true, since in "emacs -Q" I get this:
>
>   (featurep 'cl-lib) => nil
>
> both in GUI and TTY (i.e. -nw) sessions.
>
> So if cl-lib is loaded at startup, Emacs does that because
> of some user customizations, not by default.

It is the debugger. Try this two times:

  (cl-loop for i from 0 to 10 do (+ 1 1))

The first time `cl-loop' isn't loaded, the second time it is.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Thu, 19 Oct 2023 07:18:47 +0200
> 
> Eli Zaretskii wrote:
> 
> >   (featurep 'cl-lib) => nil
> >
> > both in GUI and TTY (i.e. -nw) sessions.
> >
> > So if cl-lib is loaded at startup, Emacs does that because
> > of some user customizations, not by default.
> 
> It is the debugger.

Then that's fine, as the debugger is not normally loaded in a
production session.

Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

> It is the debugger. Try this two times:
>
>   (cl-loop for i from 0 to 10 do (+ 1 1))
>
> The first time `cl-loop' isn't loaded, the second time
> it is.

debug.el `require' cl-lib as its first line of non-comment
source, and then goes on to use it 6 times: `cl-prin1' (2),
`cl-defstruct' (1), and `cl-getf' (3).

cl-prin1 is found in cl-print.el, cl-defstruct in cl-macs.el
and cl-getf in cl-extra.el.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > debug.el `require' cl-lib as its first line of non-comment
  > source, and then goes on to use it 6 times: `cl-prin1' (2),
  > `cl-defstruct' (1), and `cl-getf' (3).

  > cl-prin1 is found in cl-print.el, cl-defstruct in cl-macs.el
  > and cl-getf in cl-extra.el.

It sounds like this should be easy to fix.

I would expect that there is an important reason to call cl-prin1.
But I'd expect it is easy to avoid cl-getf -- that using
it struct some developer as "more elegant".

As for cl-defstruct, that is a macro, so it should not be an issue really.

Does this mean it would be simple to fix debug.el by writing out cl-getf,
loading cl-print.el, and not requirng cl-lib?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Lisp files that load cl-lib in problematical ways

[Stephen Berman]

On Thu, 19 Oct 2023 07:54:12 +0300 Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Emanuel Berg <incal@dataswamp.org>
>> Date: Thu, 19 Oct 2023 06:14:52 +0200
>>
>> Richard Stallman wrote:
>>
>> >> 226 files, or 14%, use cl-lib at run time.
>> >
>> > These cases may be more or less problematical, but not
>> > necessarily equally so.
>> >
>> > Which of those files are usually loaded when you start Emacs
>> > with no arguments? Those we should certainly fix.
>>
>> I don't know, but at least one since you get the cl-lib stuff
>> even at 'emacs -Q'.
>
> That is not true, since in "emacs -Q" I get this:
>
>   (featurep 'cl-lib) => nil
>
> both in GUI and TTY (i.e. -nw) sessions.

I also get this in a freshly updated build from master both with -Q and
with -Q -nw.  In a freshly updated build from emacs-29 I also get it
with -Q -nw, but with just -Q (i.e. the GUI) it returns `t', and
evaluating `features' returns (time-date subr-x cl-loaddefs cl-lib
rmc...).  And in a GUI build from master from October 8, (featurep
'cl-lib) returns `t' and features returns (time-date subr-x cl-loaddefs
cl-lib rmc...).  In the GUI build from current master, features returns
(rmc iso-transl tooltip...), so it looks like a recent change in master
stopped loading subr-x at startup.

Steve Berman

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Stephen Berman <stephen.berman@gmx.net>
> Cc: Emanuel Berg <incal@dataswamp.org>,  emacs-devel@gnu.org
> Date: Thu, 19 Oct 2023 09:34:39 +0200
> 
> On Thu, 19 Oct 2023 07:54:12 +0300 Eli Zaretskii <eliz@gnu.org> wrote:
> 
> >   (featurep 'cl-lib) => nil
> >
> > both in GUI and TTY (i.e. -nw) sessions.
> 
> I also get this in a freshly updated build from master both with -Q and
> with -Q -nw.  In a freshly updated build from emacs-29 I also get it
> with -Q -nw, but with just -Q (i.e. the GUI) it returns `t', and
> evaluating `features' returns (time-date subr-x cl-loaddefs cl-lib
> rmc...).  And in a GUI build from master from October 8, (featurep
> 'cl-lib) returns `t' and features returns (time-date subr-x cl-loaddefs
> cl-lib rmc...).

Please try figuring out what loads cl-lib in your case, as I cannot
reproduce this with the latest emacs-29 branch.

Re: Lisp files that load cl-lib in problematical ways

[Stephen Berman]

On Thu, 19 Oct 2023 10:55:24 +0300 Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Stephen Berman <stephen.berman@gmx.net>
>> Cc: Emanuel Berg <incal@dataswamp.org>,  emacs-devel@gnu.org
>> Date: Thu, 19 Oct 2023 09:34:39 +0200
>>
>> On Thu, 19 Oct 2023 07:54:12 +0300 Eli Zaretskii <eliz@gnu.org> wrote:
>>
>> >   (featurep 'cl-lib) => nil
>> >
>> > both in GUI and TTY (i.e. -nw) sessions.
>>
>> I also get this in a freshly updated build from master both with -Q and
>> with -Q -nw.  In a freshly updated build from emacs-29 I also get it
>> with -Q -nw, but with just -Q (i.e. the GUI) it returns `t', and
>> evaluating `features' returns (time-date subr-x cl-loaddefs cl-lib
>> rmc...).  And in a GUI build from master from October 8, (featurep
>> 'cl-lib) returns `t' and features returns (time-date subr-x cl-loaddefs
>> cl-lib rmc...).
>
> Please try figuring out what loads cl-lib in your case, as I cannot
> reproduce this with the latest emacs-29 branch.

Neither can I now; I must have mistakenly used the previous build of
emacs-29 (also from October 8), which was still open.  Sorry for the
false alarm.

Steve Berman

Re: Lisp files that load cl-lib in problematical ways

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stephen Berman <stephen.berman@gmx.net>
>> Cc: Emanuel Berg <incal@dataswamp.org>,  emacs-devel@gnu.org
>> Date: Thu, 19 Oct 2023 09:34:39 +0200
>> 
>> On Thu, 19 Oct 2023 07:54:12 +0300 Eli Zaretskii <eliz@gnu.org> wrote:
>> 
>> >   (featurep 'cl-lib) => nil
>> >
>> > both in GUI and TTY (i.e. -nw) sessions.
>> 
>> I also get this in a freshly updated build from master both with -Q and
>> with -Q -nw.  In a freshly updated build from emacs-29 I also get it
>> with -Q -nw, but with just -Q (i.e. the GUI) it returns `t', and
>> evaluating `features' returns (time-date subr-x cl-loaddefs cl-lib
>> rmc...).  And in a GUI build from master from October 8, (featurep
>> 'cl-lib) returns `t' and features returns (time-date subr-x cl-loaddefs
>> cl-lib rmc...).
>
> Please try figuring out what loads cl-lib in your case, as I cannot
> reproduce this with the latest emacs-29 branch.

Maybe the native compiler as it was triggered for some native
compilation?

  Andrea

Re: Lisp files that load cl-lib in problematical ways

[Andrea Corallo]

Andrea Corallo <acorallo@gnu.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Stephen Berman <stephen.berman@gmx.net>
>>> Cc: Emanuel Berg <incal@dataswamp.org>,  emacs-devel@gnu.org
>>> Date: Thu, 19 Oct 2023 09:34:39 +0200
>>> 
>>> On Thu, 19 Oct 2023 07:54:12 +0300 Eli Zaretskii <eliz@gnu.org> wrote:
>>> 
>>> >   (featurep 'cl-lib) => nil
>>> >
>>> > both in GUI and TTY (i.e. -nw) sessions.
>>> 
>>> I also get this in a freshly updated build from master both with -Q and
>>> with -Q -nw.  In a freshly updated build from emacs-29 I also get it
>>> with -Q -nw, but with just -Q (i.e. the GUI) it returns `t', and
>>> evaluating `features' returns (time-date subr-x cl-loaddefs cl-lib
>>> rmc...).  And in a GUI build from master from October 8, (featurep
>>> 'cl-lib) returns `t' and features returns (time-date subr-x cl-loaddefs
>>> cl-lib rmc...).
>>
>> Please try figuring out what loads cl-lib in your case, as I cannot
>> reproduce this with the latest emacs-29 branch.
>
> Maybe the native compiler as it was triggered for some native
> compilation?

Okay, I confirm that comp.el loads cl-lib, so any jit compilation
triggered loads that.

emacs -Q on my system at the first run (not in the followings) loads
cl-lib because of that.

The compiler really needs cl-lib while running as needs to understand
user defined types (cl structs).

OTOH one thing we could do, if that's important, is to split the code
that only drives the async compilation (that actually happens in a
subprocess) so we don't load cl-lib in the Emacs the user is actually
using.  This should not be that hard (optimism mode on).

  Andrea

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Andrea Corallo <acorallo@gnu.org>
> Cc: Stephen Berman <stephen.berman@gmx.net>,  incal@dataswamp.org,
>   emacs-devel@gnu.org
> Date: Thu, 19 Oct 2023 09:44:51 -0400
> 
> Okay, I confirm that comp.el loads cl-lib, so any jit compilation
> triggered loads that.

This is OK, not a problem.

> OTOH one thing we could do, if that's important, is to split the code
> that only drives the async compilation (that actually happens in a
> subprocess) so we don't load cl-lib in the Emacs the user is actually
> using.  This should not be that hard (optimism mode on).

I don't see the need, but I will not object if you want to make such
changes.

Re: Lisp files that load cl-lib in problematical ways

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <acorallo@gnu.org>
>> Cc: Stephen Berman <stephen.berman@gmx.net>,  incal@dataswamp.org,
>>   emacs-devel@gnu.org
>> Date: Thu, 19 Oct 2023 09:44:51 -0400
>> 
>> Okay, I confirm that comp.el loads cl-lib, so any jit compilation
>> triggered loads that.
>
> This is OK, not a problem.
>
>> OTOH one thing we could do, if that's important, is to split the code
>> that only drives the async compilation (that actually happens in a
>> subprocess) so we don't load cl-lib in the Emacs the user is actually
>> using.  This should not be that hard (optimism mode on).
>
> I don't see the need, but I will not object if you want to make such
> changes.

Okay I'll think about it, might be a positive change even leaving aside
the cl-lib discussion.

Thanks

  Andrea

Re: Lisp files that load cl-lib in problematical ways

[Andrea Corallo]

Andrea Corallo <acorallo@gnu.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Andrea Corallo <acorallo@gnu.org>
>>> Cc: Stephen Berman <stephen.berman@gmx.net>,  incal@dataswamp.org,
>>>   emacs-devel@gnu.org
>>> Date: Thu, 19 Oct 2023 09:44:51 -0400
>>> 
>>> Okay, I confirm that comp.el loads cl-lib, so any jit compilation
>>> triggered loads that.
>>
>> This is OK, not a problem.
>>
>>> OTOH one thing we could do, if that's important, is to split the code
>>> that only drives the async compilation (that actually happens in a
>>> subprocess) so we don't load cl-lib in the Emacs the user is actually
>>> using.  This should not be that hard (optimism mode on).
>>
>> I don't see the need, but I will not object if you want to make such
>> changes.
>
> Okay I'll think about it, might be a positive change even leaving aside
> the cl-lib discussion.

So I did some experimentation (scratch/comp-run branch) where I splitted
the code needed to run the (async) native compiler at runtime into a
separete file (comp-run.el).

I think it's a good change as:

1- instead of loading almost 6000 lines of compiler only code
   (comp.el+comp-cstr.el not including dependencies) a normal user now
   has to load only ~500 LOC.  The rest will be loaded only by the async
   workers.

2- it is conceptually correct to divide the code needed at runtime from
   the one necessary to actually compile.

3- at the first startup on my configuration only gv.el gets native
   compiled!

Limitations so far are that:

1- if any of the async compilation will have to report a warning,
   warnings.el will require icons.el that will still require cl-lib.el.

2- if cl-lib.el and other dependencies of the native-compiler are not
   native compiled because of some other package gets loaded, they might
   stay bytecode.  In this case the native compilers running in
   background _might_ have some performance degradation.  I guess is
   more theoretical than practical but we might want to have a look
   before committing.

Bests!

  Andrea

Re: Lisp files that load cl-lib in problematical ways

[Andrea Corallo]

Andrea Corallo <acorallo@gnu.org> writes:

> Andrea Corallo <acorallo@gnu.org> writes:
>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>>>> From: Andrea Corallo <acorallo@gnu.org>
>>>> Cc: Stephen Berman <stephen.berman@gmx.net>,  incal@dataswamp.org,
>>>>   emacs-devel@gnu.org
>>>> Date: Thu, 19 Oct 2023 09:44:51 -0400
>>>> 
>>>> Okay, I confirm that comp.el loads cl-lib, so any jit compilation
>>>> triggered loads that.
>>>
>>> This is OK, not a problem.
>>>
>>>> OTOH one thing we could do, if that's important, is to split the code
>>>> that only drives the async compilation (that actually happens in a
>>>> subprocess) so we don't load cl-lib in the Emacs the user is actually
>>>> using.  This should not be that hard (optimism mode on).
>>>
>>> I don't see the need, but I will not object if you want to make such
>>> changes.
>>
>> Okay I'll think about it, might be a positive change even leaving aside
>> the cl-lib discussion.
>
> So I did some experimentation (scratch/comp-run branch) where I splitted
> the code needed to run the (async) native compiler at runtime into a
> separete file (comp-run.el).
>
> I think it's a good change as:
>
> 1- instead of loading almost 6000 lines of compiler only code
>    (comp.el+comp-cstr.el not including dependencies) a normal user now
>    has to load only ~500 LOC.  The rest will be loaded only by the async
>    workers.
>
> 2- it is conceptually correct to divide the code needed at runtime from
>    the one necessary to actually compile.
>
> 3- at the first startup on my configuration only gv.el gets native
>    compiled!

That's a little more sorry for the mistake, in this conf the list of
native compiled files at startup is: gv.el compile.el ansi-color.el
comint.el text-property-search.el ring.el ansi-osc.el
display-line-numbers.el

  Andrea

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Andrea Corallo <acorallo@gnu.org>
> Cc: stephen.berman@gmx.net,  incal@dataswamp.org,  emacs-devel@gnu.org
> Date: Wed, 25 Oct 2023 18:31:52 -0400
> 
> That's a little more sorry for the mistake, in this conf the list of
> native compiled files at startup is: gv.el compile.el ansi-color.el
> comint.el text-property-search.el ring.el ansi-osc.el
> display-line-numbers.el

Can you figure out why is display-line-numbers compiled in that case?

Re: Lisp files that load cl-lib in problematical ways

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <acorallo@gnu.org>
>> Cc: stephen.berman@gmx.net,  incal@dataswamp.org,  emacs-devel@gnu.org
>> Date: Wed, 25 Oct 2023 18:31:52 -0400
>>
>> That's a little more sorry for the mistake, in this conf the list of
>> native compiled files at startup is: gv.el compile.el ansi-color.el
>> comint.el text-property-search.el ring.el ansi-osc.el
>> display-line-numbers.el
>
> Can you figure out why is display-line-numbers compiled in that case?

Sorry I had to improve my methodology to see what's really compiled just
starting up emacs -Q.  Depending on the methodology used to perform the
measure something else can be compiled.  This is the correct list I
believe:

gv.el comint.el compile.el ansi-osc.el ansi-color.el ring.el
text-property-search.el comp-run.el

Thanks

  Andrea

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Andrea Corallo <acorallo@gnu.org>
> Cc: stephen.berman@gmx.net,  incal@dataswamp.org,  emacs-devel@gnu.org
> Date: Wed, 25 Oct 2023 18:02:52 -0400
> 
> So I did some experimentation (scratch/comp-run branch) where I splitted
> the code needed to run the (async) native compiler at runtime into a
> separete file (comp-run.el).
> 
> I think it's a good change as:
> 
> 1- instead of loading almost 6000 lines of compiler only code
>    (comp.el+comp-cstr.el not including dependencies) a normal user now
>    has to load only ~500 LOC.  The rest will be loaded only by the async
>    workers.
> 
> 2- it is conceptually correct to divide the code needed at runtime from
>    the one necessary to actually compile.
> 
> 3- at the first startup on my configuration only gv.el gets native
>    compiled!

Sounds good to me, thanks.

> Limitations so far are that:
> 
> 1- if any of the async compilation will have to report a warning,
>    warnings.el will require icons.el that will still require cl-lib.el.

I don't see this as a significant problem.  Here's my data point: I
use desktop.el to save/restore my production sessions, which tend to
be large, so when a new version of Emacs starts, I have more than 200
packages natively-compiled in the background during the first 2
minutes, and not a single warning emitted.  Granted, those are all
bundled packages, but why shouldn't we expect third-party packages to
live up to the same standards?

And in addition, users can always turn off the warnings if loading
cl-lib bothers them.

> 2- if cl-lib.el and other dependencies of the native-compiler are not
>    native compiled because of some other package gets loaded, they might
>    stay bytecode.  In this case the native compilers running in
>    background _might_ have some performance degradation.  I guess is
>    more theoretical than practical but we might want to have a look
>    before committing.

Doesn't happen here: I see that cl-lib is natively-compiled in the
very first group of files compiled after Emacs starts.

Re: Lisp files that load cl-lib in problematical ways

[Andrea Corallo]

Andrea Corallo <acorallo@gnu.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Andrea Corallo <acorallo@gnu.org>
>>> Cc: Stephen Berman <stephen.berman@gmx.net>,  incal@dataswamp.org,
>>>   emacs-devel@gnu.org
>>> Date: Thu, 19 Oct 2023 09:44:51 -0400
>>> 
>>> Okay, I confirm that comp.el loads cl-lib, so any jit compilation
>>> triggered loads that.
>>
>> This is OK, not a problem.
>>
>>> OTOH one thing we could do, if that's important, is to split the code
>>> that only drives the async compilation (that actually happens in a
>>> subprocess) so we don't load cl-lib in the Emacs the user is actually
>>> using.  This should not be that hard (optimism mode on).
>>
>> I don't see the need, but I will not object if you want to make such
>> changes.
>
> Okay I'll think about it, might be a positive change even leaving aside
> the cl-lib discussion.

Okay in scratch/comp-run we have a branch that does not require to load
comp.el for jit compilations and for installing already existing
trampolines (compiling a new trampolines indeed require to load the
compiler).

It achieves that adding comp-run.el for runtime dependencies.

Still the compiler is loaded for C-h f, to solve that I think we'd need
to add a third file (ex comp-common.el), not sure it's worth ATM.

BR

  Andrea

Re: Lisp files that load cl-lib in problematical ways

[Andrea Corallo]

Andrea Corallo <acorallo@gnu.org> writes:

> Andrea Corallo <acorallo@gnu.org> writes:
>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>>>> From: Andrea Corallo <acorallo@gnu.org>
>>>> Cc: Stephen Berman <stephen.berman@gmx.net>,  incal@dataswamp.org,
>>>>   emacs-devel@gnu.org
>>>> Date: Thu, 19 Oct 2023 09:44:51 -0400
>>>> 
>>>> Okay, I confirm that comp.el loads cl-lib, so any jit compilation
>>>> triggered loads that.
>>>
>>> This is OK, not a problem.
>>>
>>>> OTOH one thing we could do, if that's important, is to split the code
>>>> that only drives the async compilation (that actually happens in a
>>>> subprocess) so we don't load cl-lib in the Emacs the user is actually
>>>> using.  This should not be that hard (optimism mode on).
>>>
>>> I don't see the need, but I will not object if you want to make such
>>> changes.
>>
>> Okay I'll think about it, might be a positive change even leaving aside
>> the cl-lib discussion.
>
> Okay in scratch/comp-run we have a branch that does not require to load
> comp.el for jit compilations and for installing already existing
> trampolines (compiling a new trampolines indeed require to load the
> compiler).
>
> It achieves that adding comp-run.el for runtime dependencies.
>
> Still the compiler is loaded for C-h f, to solve that I think we'd need
> to add a third file (ex comp-common.el), not sure it's worth ATM.

I think what is now in scratch/comp-run does what we want.  If there are
no objections I'll push it to master after some polishing in the
following days.

Bests

  Andrea

Re: Lisp files that load cl-lib in problematical ways

[Andrea Corallo]

Andrea Corallo <acorallo@gnu.org> writes:

> Andrea Corallo <acorallo@gnu.org> writes:
>
>> Andrea Corallo <acorallo@gnu.org> writes:
>>
>>> Eli Zaretskii <eliz@gnu.org> writes:
>>>
>>>>> From: Andrea Corallo <acorallo@gnu.org>
>>>>> Cc: Stephen Berman <stephen.berman@gmx.net>,  incal@dataswamp.org,
>>>>>   emacs-devel@gnu.org
>>>>> Date: Thu, 19 Oct 2023 09:44:51 -0400
>>>>> 
>>>>> Okay, I confirm that comp.el loads cl-lib, so any jit compilation
>>>>> triggered loads that.
>>>>
>>>> This is OK, not a problem.
>>>>
>>>>> OTOH one thing we could do, if that's important, is to split the code
>>>>> that only drives the async compilation (that actually happens in a
>>>>> subprocess) so we don't load cl-lib in the Emacs the user is actually
>>>>> using.  This should not be that hard (optimism mode on).
>>>>
>>>> I don't see the need, but I will not object if you want to make such
>>>> changes.
>>>
>>> Okay I'll think about it, might be a positive change even leaving aside
>>> the cl-lib discussion.
>>
>> Okay in scratch/comp-run we have a branch that does not require to load
>> comp.el for jit compilations and for installing already existing
>> trampolines (compiling a new trampolines indeed require to load the
>> compiler).
>>
>> It achieves that adding comp-run.el for runtime dependencies.
>>
>> Still the compiler is loaded for C-h f, to solve that I think we'd need
>> to add a third file (ex comp-common.el), not sure it's worth ATM.
>
> I think what is now in scratch/comp-run does what we want.  If there are
> no objections I'll push it to master after some polishing in the
> following days.

Okay I've installed the changes.

To summarize now comp.el is loaded only when a trampoline need to be
compiled (there's no other way around for sync compilation to happen
indeed).  In all other cases (jit compilation, installation of exinting
trampoline, C-h f) comp.el is *not* loaded in the main Emacs process.

As a consequence of this cl-lib will not be loaded as well, the only
caveat is when a warning has to be reported (form the jit compilation
for instance) cl-lib will be loaded as it's an indirect dependecy of
warnings.el.

Best Regards

  Andrea

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Andrea Corallo <acorallo@gnu.org>
> Cc: stephen.berman@gmx.net,  incal@dataswamp.org,  emacs-devel@gnu.org
> Date: Thu, 09 Nov 2023 06:41:03 -0500
> 
> > I think what is now in scratch/comp-run does what we want.  If there are
> > no objections I'll push it to master after some polishing in the
> > following days.
> 
> Okay I've installed the changes.

Thanks, but please in the future allow a bit longer than just 17 hours
for people to review the changes and post comments.

I see some warnings when compiling the new files in a build without
native-compilation:

  In comp-eln-load-path-eff:
  emacs-lisp/comp-common.el:519:31: Warning: reference to free variable `comp-native-version-dir'
  emacs-lisp/comp-common.el:522:11: Warning: reference to free variable `native-comp-eln-load-path'

  In native-compile-async-skip-p:
  emacs-lisp/comp-run.el:124:54: Warning: reference to free variable `comp--no-native-compile'
  In comp-run-async-workers:
  emacs-lisp/comp-run.el:309:14: Warning: reference to free variable `comp-deferred-pending-h'
  In comp-subr-trampoline-install:
  emacs-lisp/comp-run.el:336:21: Warning: reference to free variable `native-comp-enable-subr-trampolines'
  emacs-lisp/comp-run.el:338:34: Warning: reference to free variable `comp-installed-trampolines-h'

  In end of data:
  emacs-lisp/comp-run.el:342:8: Warning: the function `comp--install-trampoline' is not known to be defined.
  emacs-lisp/comp-run.el:291:37: Warning: the function `native-elisp-load' is not known to be defined.
  emacs-lisp/comp-run.el:211:35: Warning: the function `comp-el-to-eln-filename' is not known to be defined.

Re: Lisp files that load cl-lib in problematical ways

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <acorallo@gnu.org>
>> Cc: stephen.berman@gmx.net,  incal@dataswamp.org,  emacs-devel@gnu.org
>> Date: Thu, 09 Nov 2023 06:41:03 -0500
>> 
>> > I think what is now in scratch/comp-run does what we want.  If there are
>> > no objections I'll push it to master after some polishing in the
>> > following days.
>> 
>> Okay I've installed the changes.
>
> Thanks, but please in the future allow a bit longer than just 17 hours
> for people to review the changes and post comments.

You're right sorry, I had the impression this branch of the thread was
becoming a monologue and not many people were interested so I proceeded,
also the weekend I'll not be responsive so I wanted to have it in with
some fixing for fixing in case.  Noted for the next time.

> I see some warnings when compiling the new files in a build without
> native-compilation:
>
>   In comp-eln-load-path-eff:
>   emacs-lisp/comp-common.el:519:31: Warning: reference to free variable `comp-native-version-dir'
>   emacs-lisp/comp-common.el:522:11: Warning: reference to free variable `native-comp-eln-load-path'
>
>   In native-compile-async-skip-p:
>   emacs-lisp/comp-run.el:124:54: Warning: reference to free variable `comp--no-native-compile'
>   In comp-run-async-workers:
>   emacs-lisp/comp-run.el:309:14: Warning: reference to free variable `comp-deferred-pending-h'
>   In comp-subr-trampoline-install:
>   emacs-lisp/comp-run.el:336:21: Warning: reference to free variable `native-comp-enable-subr-trampolines'
>   emacs-lisp/comp-run.el:338:34: Warning: reference to free variable `comp-installed-trampolines-h'
>
>   In end of data:
>   emacs-lisp/comp-run.el:342:8: Warning: the function `comp--install-trampoline' is not known to be defined.
>   emacs-lisp/comp-run.el:291:37: Warning: the function `native-elisp-load' is not known to be defined.
>   emacs-lisp/comp-run.el:211:35: Warning: the function `comp-el-to-eln-filename' is not known to be defined.

Thanks 13a1797d5b6 fixes all of these here.

  Andrea

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Andrea Corallo <acorallo@gnu.org>
> Cc: stephen.berman@gmx.net,  incal@dataswamp.org,  emacs-devel@gnu.org
> Date: Thu, 09 Nov 2023 11:10:13 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Thanks, but please in the future allow a bit longer than just 17 hours
> > for people to review the changes and post comments.
> 
> You're right sorry, I had the impression this branch of the thread was
> becoming a monologue and not many people were interested so I proceeded,
> also the weekend I'll not be responsive so I wanted to have it in with
> some fixing for fixing in case.  Noted for the next time.

Silence is not necessarily a sign that no one is paying attention...

> Thanks 13a1797d5b6 fixes all of these here.

Hmm... I still see warnings:

  In native-compile-async-skip-p:
  emacs-lisp/comp-run.el:124:54: Warning: reference to free variable `comp--no-native-compile'

  In comp-run-async-workers:
  emacs-lisp/comp-run.el:309:14: Warning: reference to free variable `comp-deferred-pending-h'

  In comp-subr-trampoline-install:
  emacs-lisp/comp-run.el:336:21: Warning: reference to free variable `native-comp-enable-subr-trampolines'
  emacs-lisp/comp-run.el:338:34: Warning: reference to free variable `comp-installed-trampolines-h'

  In end of data:
  emacs-lisp/comp-run.el:342:8: Warning: the function `comp--install-trampoline' is not known to be defined.
  emacs-lisp/comp-run.el:291:37: Warning: the function `native-elisp-load' is not known to be defined.
  emacs-lisp/comp-run.el:211:35: Warning: the function `comp-el-to-eln-filename' is not known to be defined.

  In comp--func-arity:
  emacs-lisp/comp.el:1179:26: Warning: reference to free variable `comp-subr-arities-h'

  In comp-final:
  emacs-lisp/comp.el:3146:56: Warning: reference to free variable `native-comp-eln-load-path'

  In comp--trampoline-abs-filename:
  emacs-lisp/comp.el:3217:29: Warning: reference to free variable `native-comp-enable-subr-trampolines'
  emacs-lisp/comp.el:3221:46: Warning: reference to free variable `comp-native-version-dir'

  In comp-compile-all-trampolines:
  emacs-lisp/comp.el:3409:21: Warning: reference to free variable `native-comp-eln-load-path'

  In comp-lookup-eln:
  emacs-lisp/comp.el:3421:15: Warning: reference to free variable `native-comp-eln-load-path'
  emacs-lisp/comp.el:3423:48: Warning: reference to free variable `comp-native-version-dir'

  In batch-native-compile:
  emacs-lisp/comp.el:3458:25: Warning: reference to free variable `native-comp-eln-load-path'

  In batch-byte+native-compile:
  emacs-lisp/comp.el:3502:24: Warning: reference to free variable `native-comp-eln-load-path'

  In native-compile-prune-cache:
  emacs-lisp/comp.el:3515:25: Warning: reference to free variable `native-comp-eln-load-path'
  emacs-lisp/comp.el:3527:32: Warning: reference to free variable `comp-native-version-dir'

  In end of data:
  emacs-lisp/comp.el:3420:25: Warning: the function `comp-el-to-eln-rel-filename' is not known to be defined.
  emacs-lisp/comp.el:3389:18: Warning: the function `native-elisp-load' is not known to be defined.
  emacs-lisp/comp.el:3115:6: Warning: the function `comp--release-ctxt' is not known to be defined.
  emacs-lisp/comp.el:3112:4: Warning: the function `comp--init-ctxt' is not known to be defined.
  emacs-lisp/comp.el:3109:6: Warning: the function `comp--compile-ctxt-to-file' is not known to be defined.
  emacs-lisp/comp.el:804:12: Warning: the function `comp-el-to-eln-filename' is not known to be defined.

Re: Lisp files that load cl-lib in problematical ways

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <acorallo@gnu.org>
>> Cc: stephen.berman@gmx.net,  incal@dataswamp.org,  emacs-devel@gnu.org
>> Date: Thu, 09 Nov 2023 11:10:13 -0500
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > Thanks, but please in the future allow a bit longer than just 17 hours
>> > for people to review the changes and post comments.
>> 
>> You're right sorry, I had the impression this branch of the thread was
>> becoming a monologue and not many people were interested so I proceeded,
>> also the weekend I'll not be responsive so I wanted to have it in with
>> some fixing for fixing in case.  Noted for the next time.
>
> Silence is not necessarily a sign that no one is paying attention...
>
>> Thanks 13a1797d5b6 fixes all of these here.
>
> Hmm... I still see warnings:

Apologies again, my test was just wrong, 9c9b87639f9 should do a better
job.

Thanks

  Andrea

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Andrea Corallo <acorallo@gnu.org>
> Cc: stephen.berman@gmx.net,  incal@dataswamp.org,  emacs-devel@gnu.org
> Date: Thu, 09 Nov 2023 13:34:06 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Thanks 13a1797d5b6 fixes all of these here.
> >
> > Hmm... I still see warnings:
> 
> Apologies again, my test was just wrong, 9c9b87639f9 should do a better
> job.

Compiles cleanly now, thanks.

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > That is not true, since in "emacs -Q" I get this:

  >   (featurep 'cl-lib) => nil

  > both in GUI and TTY (i.e. -nw) sessions.

Interesting.  I am glad I was mistaken about that file.
But I wonder how I made the mistake.

I recall sending a first email some weeks ago in which I said which
file caused loading of cl-lib.  But I can't find that outgoing message now.
Did anyone here receive that message?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Which of those files are usually loaded when you start Emacs
  > > with no arguments? Those we should certainly fix.

  > I don't know, but at least one since you get the cl-lib stuff
  > even at 'emacs -Q'.

That reply gives no useful information.  It only makes
a specious argument.

I'm starting to think that your presence on the list
msy be motivated by the goak of being a troll.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Lisp files that load cl-lib in problematical ways

[Alan Mackenzie]

Hello, Richard.

On Wed, Oct 18, 2023 at 21:28:41 -0400, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]

> I've changed the subject line to match what we are talking about.

>   > 226 files, or 14%, use cl-lib at run time.

> These cases may be more or less problematical, but not necessarily
> equally so.

> Which of those files are usually loaded when you start Emacs with
> no arguments?  Those we should certainly fix.

With the following command, started from the lisp directory:

$ sed 's/^[^;]*(load "\(.*\)".*$/\1.el/; s/\.el\.el/.el/; t; d' loadup.el | \
sort | xargs grep -c "^[^;]*(cl-" | sed '/:0/d'

, I get the following files, with counts of '(cl-' usages:

abbrev.el:2
dnd.el:4
emacs-lisp/byte-run.el:1
emacs-lisp/cconv.el:15
emacs-lisp/cl-generic.el:178
emacs-lisp/cl-preloaded.el:45
emacs-lisp/eldoc.el:12
emacs-lisp/lisp-mode.el:14
emacs-lisp/nadvice.el:7
emacs-lisp/oclosure.el:31
emacs-lisp/seq.el:51
emacs-lisp/shorthands.el:1
emacs-lisp/syntax.el:6
emacs-lisp/tabulated-list.el:1
emacs-lisp/timer.el:1
font-lock.el:1
frame.el:6
international/mule-cmds.el:4
international/ucs-normalize.el:3
isearch.el:3
ldefs-boot.el:3
loaddefs.el:3
minibuffer.el:12
progmodes/elisp-mode.el:23
progmodes/prog-mode.el:1
register.el:23
replace.el:1
select.el:6
simple.el:9
startup.el:3
subr.el:1
tab-bar.el:4
term/android-win.el:7
term/haiku-win.el:11
term/ns-win.el:9
term/pc-win.el:7
term/pgtk-win.el:9
term/w32-win.el:12
term/x-win.el:8
uniquify.el:2
vc/vc-hooks.el:1

..  That's a total of 541 occurrences, just in the files that get dumped
with the Emacs binary.

I fear that that's the way things will stay, because it would be too
much work to fix.  But more because there are enough contributors who
think that cl-lib is just an ordinary part of Emacs to be used freely
without restraint.  I am not among this group of contributors.

> -- 
> Dr Richard Stallman (https://stallman.org)
> Chief GNUisance of the GNU Project (https://gnu.org)
> Founder, Free Software Foundation (https://fsf.org)
> Internet Hall-of-Famer (https://internethalloffame.org)

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> Date: Thu, 19 Oct 2023 05:27:03 +0000
> Cc: Emanuel Berg <incal@dataswamp.org>, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> With the following command, started from the lisp directory:
> 
> $ sed 's/^[^;]*(load "\(.*\)".*$/\1.el/; s/\.el\.el/.el/; t; d' loadup.el | \
> sort | xargs grep -c "^[^;]*(cl-" | sed '/:0/d'
> 
> , I get the following files, with counts of '(cl-' usages:
> 
> abbrev.el:2
> dnd.el:4
> emacs-lisp/byte-run.el:1
> emacs-lisp/cconv.el:15
> emacs-lisp/cl-generic.el:178
> emacs-lisp/cl-preloaded.el:45
> emacs-lisp/eldoc.el:12
> emacs-lisp/lisp-mode.el:14
> emacs-lisp/nadvice.el:7
> emacs-lisp/oclosure.el:31
> emacs-lisp/seq.el:51
> emacs-lisp/shorthands.el:1
> emacs-lisp/syntax.el:6
> emacs-lisp/tabulated-list.el:1
> emacs-lisp/timer.el:1
> font-lock.el:1
> frame.el:6
> international/mule-cmds.el:4
> international/ucs-normalize.el:3
> isearch.el:3
> ldefs-boot.el:3
> loaddefs.el:3
> minibuffer.el:12
> progmodes/elisp-mode.el:23
> progmodes/prog-mode.el:1
> register.el:23
> replace.el:1
> select.el:6
> simple.el:9
> startup.el:3
> subr.el:1
> tab-bar.el:4
> term/android-win.el:7
> term/haiku-win.el:11
> term/ns-win.el:9
> term/pc-win.el:7
> term/pgtk-win.el:9
> term/w32-win.el:12
> term/x-win.el:8
> uniquify.el:2
> vc/vc-hooks.el:1
> 
> ..  That's a total of 541 occurrences, just in the files that get dumped
> with the Emacs binary.

Those use cl-lib during byte-compilation, they don't load cl-lib at
run time.  So this is not what Richard asked about, AFAIU.

> I fear that that's the way things will stay, because it would be too
> much work to fix.  But more because there are enough contributors who
> think that cl-lib is just an ordinary part of Emacs to be used freely
> without restraint.  I am not among this group of contributors.

We cannot possibly expect people to contribute code if we force them
not to use the macros they are used to.  If cl-lib is not loaded as
result, that is good enough for us, I think.

Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Eli Zaretskii wrote:

>> That's a total of 541 occurrences, just in the files that
>> get dumped with the Emacs binary.
>
> Those use cl-lib during byte-compilation, they don't load
> cl-lib at run time. So this is not what Richard asked
> about, AFAIU.

The "levels" introduced in this discussion are, it would seem

  - files that are used by vanilla Emacs that use cl-lib
  
  - such files that use it at compile time

  - such files that use it a run time

  - such files that are loaded when Emacs is executed with no
    arguments

but - what is the difference? I think it is pretty clear that
cl-lib is used quite broadly, probably because people find
it useful.

So what does it matter if it is "purged" from one or several
of these levels? What gain is there if that is achieved?

>> I fear that that's the way things will stay, because it
>> would be too much work to fix. But more because there are
>> enough contributors who think that cl-lib is just an
>> ordinary part of Emacs to be used freely without restraint.
>> I am not among this group of contributors.
>
> We cannot possibly expect people to contribute code if we
> force them not to use the macros they are used to. If cl-lib
> is not loaded as result, that is good enough for us,
> I think.

Again, what is the harm loading cl-lib?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Thu, 19 Oct 2023 13:19:06 +0200
> 
> The "levels" introduced in this discussion are, it would seem
> 
>   - files that are used by vanilla Emacs that use cl-lib
>   
>   - such files that use it at compile time
> 
>   - such files that use it a run time
> 
>   - such files that are loaded when Emacs is executed with no
>     arguments
> 
> but - what is the difference? I think it is pretty clear that
> cl-lib is used quite broadly, probably because people find
> it useful.
> 
> So what does it matter if it is "purged" from one or several
> of these levels? What gain is there if that is achieved?

It matters to us because we decided long ago to avoid loading cl-lib
at runtime in vanilla Emacs.  There are several good reasons: bloat,
unnecessary namespace pollution, etc.  But all this is not really
relevant for the code that you (or any one of us) write for their own
personal use, it is only relevant to code contributed to Emacs.

Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Eli Zaretskii wrote:

>> The "levels" introduced in this discussion are, it would
>> seem
>> 
>>   - files that are used by vanilla Emacs that use cl-lib
>>   
>>   - such files that use it at compile time
>> 
>>   - such files that use it a run time
>> 
>>   - such files that are loaded when Emacs is executed with
>>     no arguments
>> 
>> but - what is the difference? I think it is pretty clear
>> that cl-lib is used quite broadly, probably because people
>> find it useful.
>> 
>> So what does it matter if it is "purged" from one or
>> several of these levels? What gain is there if that
>> is achieved?
>
> It matters to us because we decided long ago to avoid
> loading cl-lib at runtime in vanilla Emacs.

But as we have seen in practice it _is_ loaded, be it by the
debugger, by native compilation, or any other package or piece
of Elisp that uses it, or uses something else that is using
it. It is all interconnected, and because it is used so much
already, using Emacs in practice implies having cl-lib loaded
at run time.

> There are several good reasons: bloat, unnecessary namespace
> pollution, etc.

People use it because it is useful: there is `cl-loop',
`cl-incf', `cl-decf', `cl-pushnew' and many others examples of
that. Those are not complicated to use, they are all prefixed
with cl- (as we see) and have good docstrings.

> But all this is not really relevant for the code that you
> (or any one of us) write for their own personal use, it is
> only relevant to code contributed to Emacs.

But we don't know what people who are starting to write Elisp
today for their personal use go on to contribute to Emacs
packages that make it to the ELPAs, or core Emacs for that
matter. That whole distinction does not make any sense to me.
If it is harmful, which it isn't, no one should use it and it
doesn't matter in what context, be it personal use, ELPA
packages, core Emacs, compile time, run time - what does
it matter?

On the other hand if it _is_ useful - and the number of people
using it and the number of files where it is used, and the
number of occurrences indicate that it is - there is no reason
to disencourage people from using it, anywhere.

You, and everyone else writing Elisp for that matter, are of
coures equally permitted not to use it, if they consider it
harmful, it just shouldn't be labeled detrimental as a policy
with really no good arguments supporting that claim that we
have seen thus far at least.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 1074 bytes --]


Emanuel Berg <incal@dataswamp.org> writes:

>> But all this is not really relevant for the code that you
>> (or any one of us) write for their own personal use, it is
>> only relevant to code contributed to Emacs.
>
> But we don't know what people who are starting to write Elisp
> today for their personal use go on to contribute to Emacs
> packages that make it to the ELPAs, or core Emacs for that
> matter. That whole distinction does not make any sense to me.

I have quite a few things in my .emmacs.d/init.el that are right for me
but would not be suitable for core.

That starts with (setq gc-cons-threshold (* 100 1024 1024))
;; Make gc pauses faster by decreasing the threshold again (from the increased initial).
(setq gc-cons-threshold (* 20 1024 1024))
;; speed up reading from external processes
(setq read-process-output-max (* 1024 1024)) ;; 1mb

and continues through my exwm setup, reaching key-chords and interactive
ido preview.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Dr. Arne Babenhauserheide wrote:

>> But we don't know what people who are starting to write
>> Elisp today for their personal use go on to contribute to
>> Emacs packages that make it to the ELPAs, or core Emacs for
>> that matter. That whole distinction does not make any sense
>> to me.
>
> I have quite a few things in my .emmacs.d/init.el that are
> right for me but would not be suitable for core.
>
> That starts with (setq gc-cons-threshold (* 100 1024 1024))
> ;; Make gc pauses faster by decreasing the threshold again
> (from the increased initial).
> (setq gc-cons-threshold (* 20 1024 1024))
> ;; speed up reading from external processes
> (setq read-process-output-max (* 1024 1024)) ;; 1mb
>
> and continues through my exwm setup, reaching key-chords and
> interactive ido preview.

Case closed :)

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Fri, 20 Oct 2023 05:02:14 +0200
> 
> Eli Zaretskii wrote:
> 
> >> So what does it matter if it is "purged" from one or
> >> several of these levels? What gain is there if that
> >> is achieved?
> >
> > It matters to us because we decided long ago to avoid
> > loading cl-lib at runtime in vanilla Emacs.
> 
> But as we have seen in practice it _is_ loaded, be it by the
> debugger, by native compilation, or any other package or piece
> of Elisp that uses it, or uses something else that is using
> it. It is all interconnected, and because it is used so much
> already, using Emacs in practice implies having cl-lib loaded
> at run time.

It is okay for cl-lib to be loaded when packages that use it are
loaded.  The same happens with gazillion other packages in Emacs; for
example, try loading Gnus or Org or Eshell, and see how much stuff
unrelated to these packages per se gets loaded as result, because
those packages need some non-default infrastructure.

What matters to us is that this unnecessary stuff is not loaded in
"emacs -Q", i.e. is not dumped and preloaded into the bare
uncustomized Emacs session.  This lets users start from the leanest
possible Emacs, then load what they need, without being "punished" by
loads of stuff they never need or want.

For Emacs to preload unnecessary stuff is unclean, and we try to avoid
that uncleanliness.

> > There are several good reasons: bloat, unnecessary namespace
> > pollution, etc.
> 
> People use it because it is useful: there is `cl-loop',
> `cl-incf', `cl-decf', `cl-pushnew' and many others examples of
> that. Those are not complicated to use, they are all prefixed
> with cl- (as we see) and have good docstrings.

This is not relevant to the issue at hand.

> > But all this is not really relevant for the code that you
> > (or any one of us) write for their own personal use, it is
> > only relevant to code contributed to Emacs.
> 
> But we don't know what people who are starting to write Elisp
> today for their personal use go on to contribute to Emacs
> packages that make it to the ELPAs, or core Emacs for that
> matter.

That's okay, they will learn it on-the-fly, from review comments to
their contributions, like with the rest of our coding conventions,
which are usually not clear to new contributors when they first
contribute.

> You, and everyone else writing Elisp for that matter, are of
> coures equally permitted not to use it, if they consider it
> harmful, it just shouldn't be labeled detrimental as a policy
> with really no good arguments supporting that claim that we
> have seen thus far at least.

See above: this is not relevant to the issue at hand.

Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Eli Zaretskii wrote:

> What matters to us is that this unnecessary stuff is not
> loaded in "emacs -Q", i.e. is not dumped and preloaded into
> the bare uncustomized Emacs session. This lets users start
> from the leanest possible Emacs, then load what they need,
> without being "punished" by loads of stuff they never need
> or want.

In practice cl-lib will be loaded pretty much instantly even
from minor use, including from the starting point that is the
minimal 'emacs -Q'.

But if we focus on this level in particular, defined by
'emacs -Q' and no use whatsoever on top of that, then cl-lib
isn't loaded as you say.

Because 'emacs -Q' should load as little as possible, it makes
sense not to load cl-lib and, as has been showed, that doesn't
happen either.

So I suppose everyone is happy about that then. It will also
be easy for you to keep it that way by not introducing it to
anything that is loaded by 'emacs -Q'.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Jens Schmidt]

On 2023-10-20  08:49, Eli Zaretskii wrote:

> For Emacs to preload unnecessary stuff is unclean, and we try to avoid
> that uncleanliness.

I have to admit that I don't fully understand the dump/bootstrap logic
of Emacs (yet), but doesn't inhibit the following test in Frequire already
any/most uncleanliness:

      /* This is to make sure that loadup.el gives a clear picture
         of what files are preloaded and when.  */
      if (will_dump_p () && !will_bootstrap_p ())
        {
          /* Avoid landing here recursively while outputting the
             backtrace from the error.  */
          gflags.will_dump_ = false;
          error ("(require %s) while preparing to dump",
                 SDATA (SYMBOL_NAME (feature)));
        }

At least that error fires during dump ("Dump mode: pdump") when I try to
require, for example, cl-lib from a file loaded from loadup.el.

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> Date: Sat, 21 Oct 2023 17:17:24 +0200
> Cc: emacs-devel@gnu.org
> From: Jens Schmidt <jschmidt4gnu@vodafonemail.de>
> 
> On 2023-10-20  08:49, Eli Zaretskii wrote:
> 
> > For Emacs to preload unnecessary stuff is unclean, and we try to avoid
> > that uncleanliness.
> 
> I have to admit that I don't fully understand the dump/bootstrap logic
> of Emacs (yet), but doesn't inhibit the following test in Frequire already
> any/most uncleanliness:
> 
>       /* This is to make sure that loadup.el gives a clear picture
>          of what files are preloaded and when.  */
>       if (will_dump_p () && !will_bootstrap_p ())
>         {
>           /* Avoid landing here recursively while outputting the
>              backtrace from the error.  */
>           gflags.will_dump_ = false;
>           error ("(require %s) while preparing to dump",
>                  SDATA (SYMBOL_NAME (feature)));
>         }

No, because there's more than one way of loading a Lisp package.
Moreover, we could have a situation where cl-lib is not preloaded, but
just starting "emacs -Q" loads it (e.g., via some autoloaded feature),
and that also should be avoided.

Re: Lisp files that load cl-lib in problematical ways

[Björn Bidar]

Emanuel Berg <incal@dataswamp.org> writes:

>> There are several good reasons: bloat, unnecessary namespace
>> pollution, etc.
>
> People use it because it is useful: there is `cl-loop',
> `cl-incf', `cl-decf', `cl-pushnew' and many others examples of
> that. Those are not complicated to use, they are all prefixed
> with cl- (as we see) and have good docstrings.
>
>> But all this is not really relevant for the code that you
>> (or any one of us) write for their own personal use, it is
>> only relevant to code contributed to Emacs.
>
> But we don't know what people who are starting to write Elisp
> today for their personal use go on to contribute to Emacs
> packages that make it to the ELPAs, or core Emacs for that
> matter. That whole distinction does not make any sense to me.
> If it is harmful, which it isn't, no one should use it and it
> doesn't matter in what context, be it personal use, ELPA
> packages, core Emacs, compile time, run time - what does
> it matter?

From my point of view in some way it does sound like that some are
against it for ideological reasons rather than technical.
E.g. those that makes code unreadable or lets rewrite something to drop
the dependency to not load it.
It does sound like a generational conflict, newer developers write code
sometimes different than older ones.
Why not use cl-lib when it makes it easier to achieve what I want easier
and easier to understand than "vanilla" elisp.
But as other said, this wasn't the point of the thread even when some
messages did sound very ideological.

>
> On the other hand if it _is_ useful - and the number of people
> using it and the number of files where it is used, and the
> number of occurrences indicate that it is - there is no reason
> to disencourage people from using it, anywhere.

I think these things are also often a long the lines we don't want you
to use thing XX for a reason, so discourage them from even mentioning
it.
Ideological vs practicism. 
Similar as when people mention other repositories for package.el that
are not from Gnu..

Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Björn Bidar wrote:

>>> But all this is not really relevant for the code that you
>>> (or any one of us) write for their own personal use, it is
>>> only relevant to code contributed to Emacs.
>>
>> But we don't know what people who are starting to write
>> Elisp today for their personal use go on to contribute to
>> Emacs packages that make it to the ELPAs, or core Emacs for
>> that matter. That whole distinction does not make any sense
>> to me. If it is harmful, which it isn't, no one should use
>> it and it doesn't matter in what context, be it personal
>> use, ELPA packages, core Emacs, compile time, run time -
>> what does it matter?
>
> From my point of view in some way it does sound like that
> some are against it for ideological reasons rather than
> technical. E.g. those that makes code unreadable or lets
> rewrite something to drop the dependency to not load it.
> It does sound like a generational conflict, newer developers
> write code sometimes different than older ones. Why not use
> cl-lib when it makes it easier to achieve what I want easier
> and easier to understand than "vanilla" elisp. But as other
> said, this wasn't the point of the thread even when some
> messages did sound very ideological.

Exactly, that's the way it sounds.

>> On the other hand if it _is_ useful - and the number of
>> people using it and the number of files where it is used,
>> and the number of occurrences indicate that it is - there
>> is no reason to disencourage people from using
>> it, anywhere.
>
> I think these things are also often a long the lines we
> don't want you to use thing XX for a reason, so discourage
> them from even mentioning it. Ideological vs practicism.

Again, exactly. Only it is difficult to pinpoint what is the
ideology or set of thoughts that drives the anti-cl-lib
tendency.

As for practicality, I think that has been showed many times
over by now, not the least in the amount of usage already
present in Emacs. There is no going around that, contributors
to Emacs have already voted with their fingers in favor of
cl-lib.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > But as we have seen in practice it _is_ loaded, be it by the
  > debugger, by native compilation, or any other package or piece
  > of Elisp that uses it,

Let's fix the debugger, and native compilation, not to load cl-lib.

It's not so bad if you load a specialized package that serves your code
and it uses cl-lib.  But the debugger can be loaded to debug anything,
even programs you have nothing to do with.  It's rude for the debugger
to load cl-lib.

Likewise for native compilation.  You might recompile anything.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Sun, 22 Oct 2023 22:08:55 -0400
> 
>   > But as we have seen in practice it _is_ loaded, be it by the
>   > debugger, by native compilation, or any other package or piece
>   > of Elisp that uses it,
> 
> Let's fix the debugger, and native compilation, not to load cl-lib.

That's not practical.  I don't see any reason to waste our resources
on doing that.

> It's not so bad if you load a specialized package that serves your code
> and it uses cl-lib.  But the debugger can be loaded to debug anything,
> even programs you have nothing to do with.  It's rude for the debugger
> to load cl-lib.

The debugger uses cl-lib for good reasons.

> Likewise for native compilation.  You might recompile anything.

Likewise here.

I don't see a problem in these using cl-lib, and don't think we should
be bothered about that.

Re: Lisp files that load cl-lib in problematical ways

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Richard Stallman <rms@gnu.org>
>> Cc: emacs-devel@gnu.org
>> Date: Sun, 22 Oct 2023 22:08:55 -0400
>>
>> Let's fix the debugger, and native compilation, not to load cl-lib.
>
> That's not practical.  I don't see any reason to waste our resources
> on doing that.

+1

> I don't see a problem in these using cl-lib, and don't think we should
> be bothered about that.

Yes, let's focus our efforts on fixing real problems.

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Björn Bidar <bjorn.bidar@thaodan.de>
> Date: Sun, 22 Oct 2023 03:52:29 +0300
> 
> >From my point of view in some way it does sound like that some are
> against it for ideological reasons rather than technical.

I gave specific technical reasons for why we are trying to avoid
preloading cl-lib.

> Ideological vs practicism. 

It isn't.  Again, I explained the policy.  That Emanuel disregards
what I said and keeps claiming it's ideology or that people prefer
cl-lib etc. is not relevant to the actual issue at hand, nor to
specific questions Richard asked (and I answered).  This is purely a
technical, practical issue of interest to Emacs maintainers, it has no
bearing on how people write their own code or Emacs extensions they
submit to Emacs.

Of course, if cl-lib is used without any justification, patch review
comments will usually point that out, but in that case cl-lib is not
different from any other package, because Lisp code should not
gratuitously load stuff it can do without.  IOW, this is a matter of
writing clean code, not of preferring or not preferring cl-lib.

Re: Lisp files that load cl-lib in problematical ways

[Alan Mackenzie]

Hello, Björn.

On Sun, Oct 22, 2023 at 03:52:29 +0300, Björn Bidar wrote:

[ .... ]

> >From my point of view in some way it does sound like that some are
> against it [the use of cl-lib] for ideological reasons rather than
> technical.  E.g. those that makes code unreadable or lets rewrite
> something to drop the dependency to not load it.

You seem to be saying that avoiding making code unreadable (i.e.
difficult to maintain) is an "ideological" thing.  Here "ideological"
means "can be dismissed without consideration", I think.

> It does sound like a generational conflict, newer developers write code
> sometimes different than older ones.

I think it's more likely to be a "conflict" between those who have to
maintain other people's code and those who merely write it.  But seeing
as how I'm the only member of the first group to express a view on this,
it's difficult to be sure.

> Why not use cl-lib when it makes it easier to achieve what I want easier
> and easier to understand than "vanilla" elisp.

That's a big "when" you use.  If the use of cl-lib actually did make
things "easier to understand", I would agree with you.  But it doesn't.
I'm speaking from bitter experience here, like trying to track down a bug
at one o'clock in the morning, and encountering some obscure cl-lib
function or macro.

A great number of them are obscure.  They have inadequate doc-strings, if
any at all.  That is to say, doc-strings which don't say what the thing
does, and don't say what the parameters mean or give their form.

As an example, take the last macro in cl-macs.el, `cl-deftype'.  Its doc
string is just

    "Define NAME as a new data type.
    The type name can then be used in `cl-typecase', `cl-check-type', etc."

..  At 01:00, I need to start asking what is NAME?  Is it a symbol?  Is it
a string?  Is it, perhaps, some type of list structure?  What does it
mean to "define" it?  And what exactly is meant by a "data type"?  And
there are the other two parameters ARGLIST and BODY which are not
mentioned at all, and would appear to have nothing to do with defining
data types.  In practice, this means having to study the source code of
several functions/macros in detail, something which would be not be
needed had it not been used in the first place.  This is unrewarding,
tedious, and time consuming.

This poor standard of doc strings is typical throughout cl-lib, even if
not universal.

> But as other said, this wasn't the point of the thread even when some
> messages did sound very ideological.

Some, perhaps, but not all.

> > On the other hand if it _is_ useful - and the number of people
> > using it and the number of files where it is used, and the
> > number of occurrences indicate that it is - there is no reason
> > to disencourage people from using it, anywhere.

I have outlined a reason above.  Fifty years ago, you could have been
saying the same about the goto statement.

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> Date: Mon, 23 Oct 2023 11:07:27 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> This poor standard of doc strings is typical throughout cl-lib, even if
> not universal.

Patches to improve the documentation of CL macros and functions will
be most welcome.

Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Eli Zaretskii wrote:

>> This poor standard of doc strings is typical throughout
>> cl-lib, even if not universal.
>
> Patches to improve the documentation of CL macros and
> functions will be most welcome.

`elint-current-buffer' does not find anything to correct in
cl-lib.el.

(checkdoc-current-buffer t) says the following:

*** cl-lib.el: checkdoc-current-buffer
cl-lib.el:133: Argument ‘keys’ should appear (as KEYS) in the doc string
cl-lib.el:160: All variables and subroutines might as well have a documentation string
cl-lib.el:166: All variables and subroutines might as well have a documentation string
cl-lib.el:197: All variables and subroutines might as well have a documentation string
cl-lib.el:197: All variables and subroutines might as well have a documentation string
cl-lib.el:226: Arguments occur in the doc string out of order
cl-lib.el:258: Argument ‘specs’ should appear (as SPECS) in the doc string
cl-lib.el:357: Argument ‘cl-func’ should appear (as CL-FUNC) in the doc string
cl-lib.el:450: Argument ‘rest’ should appear (as REST) in the doc string
cl-lib.el:488: Argument ‘cl-item’ should appear (as CL-ITEM) in the doc string
cl-lib.el:501: Argument ‘cl-new’ should appear (as CL-NEW) in the doc string
cl-lib.el:510: All variables and subroutines might as well have a documentation string
cl-lib.el:539: All variables and subroutines might as well have a documentation string
cl-lib.el:577: Probably "returns" should be imperative "return"

I CC this to the maintainer, but like to add that I personally
have not experienced the documentation of cl-lib bad in
any way. That said, for a person who knows the code well,
those warnings should be a pretty quick fix.

There are other files to cl-lib, of course.

Emacs version:
  GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, cairo
  version 1.16.0) of 2023-10-10 [commit
  239db5d5162f6f9a4a6735e176c8d306f18e50e7]

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Alan Mackenzie wrote:

>> From my point of view in some way it does sound like that
>> some are against it [the use of cl-lib] for ideological
>> reasons rather than technical. E.g. those that makes code
>> unreadable or lets rewrite something to drop the dependency
>> to not load it.
>
> You seem to be saying that avoiding making code unreadable
> (i.e. difficult to maintain) is an "ideological" thing.

From Eli we heard cl-lib adds bloat and fills up the global
namespace. From RMS and you we hear it makes code more
difficult to maintain.

At the same time Eli says it is just another Elisp package and
library which should be treated as any other, i.e. it should
only be used and brought in when needed. This, of course,
I think everyone will agree on.

But the other arguments are more difficult to see from here.
Rather than bloat and filling up the namespace unnecessarily,
I think it makes Elisp more powerful. So to me, cl-lib is
a good example och packages we do want, and the more
the merrier.

When there is no clear way to do something using non-cl-lib
Elisp, I see no harm in using cl-lib (on the contrary) and
I still don't understand why it appears to be considered OK in
some places and some stages of Emacs execution and not others.
Wherever it is useful and provides expressive and
computational power otherwise unavailable, it should be OK to
use. With the possible exception of 'emacs -Q', since there
the minimalist policy makes sense and one can make
that argument.

One can also think that cl-lib's presence can bring guys from
the Common Lisp world here, which is of course a positive
thing. And it doesn't mean now all Emacs will be Common Lisp.

> That's a big "when" you use. If the use of cl-lib actually
> did make things "easier to understand", I would agree with
> you. But it doesn't.

E.g. (cl-incf some-var) isn't better and more clean than (setq
some-var (1+ some-var)) ? I think it is! Other examples are,
as has been mentioned, `cl-decf' and `cl-pushnew'. I can see
that `cl-loop' has some higher threshold to understand and to
get used to but (1) it adds so much expressive power increase
in complexity is unavoidable and expected; and, (2) it still
should be no match for programmers maintaining Emacs?

> As an example, take the last macro in cl-macs.el,
> `cl-deftype'. Its doc string is just [...]

Well, we should improve the docstrings whenever they are
insufficient, this is the same as any other library.

>> On the other hand if it _is_ useful - and the number of
>> people using it and the number of files where it is used,
>> and the number of occurrences indicate that it is - there
>> is no reason to disencourage people from using
>> it, anywhere.
>
> I have outlined a reason above. Fifty years ago, you could
> have been saying the same about the goto statement.

50 years ago, some programming languages were so limited the
goto was needed. Today, some are so powerful, one can add
cl-lib to make them even better. And indeed, we now have
`cl-return', even ;)

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Mon, 23 Oct 2023 14:18:51 +0200
> 
> >From Eli we heard cl-lib adds bloat and fills up the global
> namespace. From RMS and you we hear it makes code more
> difficult to maintain.
> 
> At the same time Eli says it is just another Elisp package and
> library which should be treated as any other, i.e. it should
> only be used and brought in when needed. This, of course,
> I think everyone will agree on.

So you get to decide who do you listen to.  Choose wisely.

Re: Lisp files that load cl-lib in problematical ways

[Alan Mackenzie]

Hello, Eli.

On Thu, Oct 19, 2023 at 10:47:37 +0300, Eli Zaretskii wrote:
> > Date: Thu, 19 Oct 2023 05:27:03 +0000
> > Cc: Emanuel Berg <incal@dataswamp.org>, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > With the following command, started from the lisp directory:

> > $ sed 's/^[^;]*(load "\(.*\)".*$/\1.el/; s/\.el\.el/.el/; t; d' loadup.el | \
> > sort | xargs grep -c "^[^;]*(cl-" | sed '/:0/d'

> > , I get the following files, with counts of '(cl-' usages:

> > abbrev.el:2
[ .... ]
> > vc/vc-hooks.el:1

> > ..  That's a total of 541 occurrences, just in the files that get dumped
> > with the Emacs binary.

> Those use cl-lib during byte-compilation, they don't load cl-lib at
> run time.

They do, I think (see below).

> So this is not what Richard asked about, AFAIU.

Yes, you're right.

So, I counted all the occurrences of "cl-" not inside an
eval-when-compile, by piping all the dumped .el files through the
following script:

find-run-time-cl.el:
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defun run-time-cl ()
  (save-excursion
    (goto-char (point-min))
    (let ((count 0))
      (while (and (< (point) (point-max))
		  (re-search-forward "^[^;]*(\\(eval-when-compile\\|cl-\\)"
				     nil 'stop))
	(if (string= (match-string-no-properties 1) "cl-")
	    (progn
	      (setq count (1+ count)))
	      ;; (message "%s:%s %s" (buffer-file-name) (line-number-at-pos)
	      ;; 	       (buffer-substring-no-properties (line-beginning-position)
	      ;; 					       (line-end-position)))
	  (goto-char (1- (match-beginning 1)))
	  (forward-list)))
      (prin1 (format "%s:%s\n" (buffer-file-name) count) t))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

$ for f in `sed 's/^[^;]*(load "\(.*\)".*$/\1.el/; s/\.el\.el/.el/; t; d' \
  loadup.el | sort`; \
  do ../src/emacs -Q -batch -l ~/find-run-time-cl.el \
  --eval "(progn (find-file \"$f\") (run-time-cl))"; \
  done | sed '/:0$/d' | sed 's%^.*/lisp/%%'

and got the following results:

abbrev.el:2
dnd.el:4
emacs-lisp/byte-run.el:1
emacs-lisp/cconv.el:13
emacs-lisp/cl-generic.el:56
emacs-lisp/cl-preloaded.el:16
emacs-lisp/eldoc.el:6
emacs-lisp/lisp-mode.el:6
emacs-lisp/nadvice.el:2
emacs-lisp/oclosure.el:8
emacs-lisp/seq.el:25
emacs-lisp/shorthands.el:1
emacs-lisp/syntax.el:5
emacs-lisp/tabulated-list.el:1
emacs-lisp/timer.el:1
font-lock.el:1
frame.el:5
international/mule-cmds.el:3
international/ucs-normalize.el:1
isearch.el:3
ldefs-boot.el:3
loaddefs.el:3
minibuffer.el:12
progmodes/elisp-mode.el:11
progmodes/prog-mode.el:1
register.el:3
replace.el:1
select.el:3
simple.el:9
startup.el:1
subr.el:1
tab-bar.el:4
term/android-win.el:4
term/haiku-win.el:3
term/ns-win.el:2
term/pc-win.el:6
term/pgtk-win.el:3
term/w32-win.el:4
term/x-win.el:2
uniquify.el:2
vc/vc-hooks.el:1

That may not be 541 occurrences, but it's still 239.  Incidentally, when
I start emacs -Q, cl-lib isn't yet loaded for me, either.

> > I fear that that's the way things will stay, because it would be too
> > much work to fix.  But more because there are enough contributors who
> > think that cl-lib is just an ordinary part of Emacs to be used freely
> > without restraint.  I am not among this group of contributors.

> We cannot possibly expect people to contribute code if we force them
> not to use the macros they are used to.  If cl-lib is not loaded as
> result, that is good enough for us, I think.

We "force" them to use Emacs Lisp, but they still contribute.  The
problem is that use of cl- makes maintenance of other people's code much
harder, at least for me.  I'm sure I'm not alone.

cl- probably multiplies the size of the Lisp language parts of Elisp
(i.e. things like defun, defmacro, if, while, cond, ...) by around three.
I'm not able to learn all these fancy macros which are used rarely enough
not to be learnable, but frequently enough to cause maintenance headaches
- every time I encounter `if-let' and the like, it just stops me in my
tracks.  I either have to guess what it means, which makes me
uncomfortable, or laboriously consult a doc string, if any.

Yes, I know if-let isn't a cl- function, and its doc string is of
reasonable quality.  Many cl- functions don't have doc strings, or have
inadequate ones, however.

There was a time in the medium past when the use of run time cl- was not
allowed.  Emacs was then, from my point of view, much easier to maintain.
At some stage this rule was done away with, but I don't recall there
being any discussion on emacs-devel about this change.  There should have
been such discussion.

But we are where we are, as you said.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> Date: Thu, 19 Oct 2023 12:34:42 +0000
> Cc: rms@gnu.org, incal@dataswamp.org, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> So, I counted all the occurrences of "cl-" not inside an
> eval-when-compile, by piping all the dumped .el files through the
> following script:
> 
> find-run-time-cl.el:
> ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
> (defun run-time-cl ()
>   (save-excursion
>     (goto-char (point-min))
>     (let ((count 0))
>       (while (and (< (point) (point-max))
> 		  (re-search-forward "^[^;]*(\\(eval-when-compile\\|cl-\\)"
> 				     nil 'stop))
> 	(if (string= (match-string-no-properties 1) "cl-")
> 	    (progn
> 	      (setq count (1+ count)))
> 	      ;; (message "%s:%s %s" (buffer-file-name) (line-number-at-pos)
> 	      ;; 	       (buffer-substring-no-properties (line-beginning-position)
> 	      ;; 					       (line-end-position)))
> 	  (goto-char (1- (match-beginning 1)))
> 	  (forward-list)))
>       (prin1 (format "%s:%s\n" (buffer-file-name) count) t))))
> ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
> 
> $ for f in `sed 's/^[^;]*(load "\(.*\)".*$/\1.el/; s/\.el\.el/.el/; t; d' \
>   loadup.el | sort`; \
>   do ../src/emacs -Q -batch -l ~/find-run-time-cl.el \
>   --eval "(progn (find-file \"$f\") (run-time-cl))"; \
>   done | sed '/:0$/d' | sed 's%^.*/lisp/%%'
> 
> and got the following results:
> 
> abbrev.el:2
> dnd.el:4
> emacs-lisp/byte-run.el:1
> emacs-lisp/cconv.el:13
> emacs-lisp/cl-generic.el:56
> emacs-lisp/cl-preloaded.el:16
> emacs-lisp/eldoc.el:6
> emacs-lisp/lisp-mode.el:6
> emacs-lisp/nadvice.el:2
> emacs-lisp/oclosure.el:8
> emacs-lisp/seq.el:25
> emacs-lisp/shorthands.el:1
> emacs-lisp/syntax.el:5
> emacs-lisp/tabulated-list.el:1
> emacs-lisp/timer.el:1
> font-lock.el:1
> frame.el:5
> international/mule-cmds.el:3
> international/ucs-normalize.el:1
> isearch.el:3
> ldefs-boot.el:3
> loaddefs.el:3
> minibuffer.el:12
> progmodes/elisp-mode.el:11
> progmodes/prog-mode.el:1
> register.el:3
> replace.el:1
> select.el:3
> simple.el:9
> startup.el:1
> subr.el:1
> tab-bar.el:4
> term/android-win.el:4
> term/haiku-win.el:3
> term/ns-win.el:2
> term/pc-win.el:6
> term/pgtk-win.el:3
> term/w32-win.el:4
> term/x-win.el:2
> uniquify.el:2
> vc/vc-hooks.el:1

If the above is correct, then how do you explain that in "emacs -Q" I
don't see the cl-lib feature present?

My conclusion is that your program has a bug.

> That may not be 541 occurrences, but it's still 239.  Incidentally, when
> I start emacs -Q, cl-lib isn't yet loaded for me, either.

Exactly.  How come, if all of the above-mentioned files load it?

> > We cannot possibly expect people to contribute code if we force them
> > not to use the macros they are used to.  If cl-lib is not loaded as
> > result, that is good enough for us, I think.
> 
> We "force" them to use Emacs Lisp, but they still contribute.  The
> problem is that use of cl- makes maintenance of other people's code much
> harder, at least for me.  I'm sure I'm not alone.

I'm sorry, but the above is how I feel we should treat our
contributors, if we want to keep the existing ones and attract new
ones.

Re: Lisp files that load cl-lib in problematical ways

[Alan Mackenzie]

Hello, Eli.

On Thu, Oct 19, 2023 at 15:55:34 +0300, Eli Zaretskii wrote:
> > Date: Thu, 19 Oct 2023 12:34:42 +0000
> > Cc: rms@gnu.org, incal@dataswamp.org, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > So, I counted all the occurrences of "cl-" not inside an
> > eval-when-compile, by piping all the dumped .el files through the
> > following script:

[ .... ]

> > and got the following results:

> > abbrev.el:2
[ .... ]
> > vc/vc-hooks.el:1

> If the above is correct, then how do you explain that in "emacs -Q" I
> don't see the cl-lib feature present?

Because the files that use cl- don't have a (require 'cl-lib) in them.
Quite a lot of the function calls are things like cl-assert, cl-incf, or
cl-defgeneric, which won't cause the loading of cl- until they are
actually run.

> My conclusion is that your program has a bug.

Maybe, but there are definitely run time uses of cl- in our dumped .el
files, such as a cl-incf in font-lock.el which creates the gradually
increasing row of dots which are displayed for large files (or, at least,
used to be).

> > That may not be 541 occurrences, but it's still 239.  Incidentally, when
> > I start emacs -Q, cl-lib isn't yet loaded for me, either.

> Exactly.  How come, if all of the above-mentioned files load it?

See above.  

> > > We cannot possibly expect people to contribute code if we force them
> > > not to use the macros they are used to.  If cl-lib is not loaded as
> > > result, that is good enough for us, I think.

> > We "force" them to use Emacs Lisp, but they still contribute.  The
> > problem is that use of cl- makes maintenance of other people's code much
> > harder, at least for me.  I'm sure I'm not alone.

> I'm sorry, but the above is how I feel we should treat our
> contributors, if we want to keep the existing ones and attract new
> ones.

I'm not urging you to change anything; merely expressing regret at the
existing state of affairs, and possibly inviting contributors in general
to use cl-lib a little less.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> Date: Thu, 19 Oct 2023 13:28:28 +0000
> Cc: rms@gnu.org, incal@dataswamp.org, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > If the above is correct, then how do you explain that in "emacs -Q" I
> > don't see the cl-lib feature present?
> 
> Because the files that use cl- don't have a (require 'cl-lib) in them.
> Quite a lot of the function calls are things like cl-assert, cl-incf, or
> cl-defgeneric, which won't cause the loading of cl- until they are
> actually run.

Then there's no problem, from where I stand.

> Maybe, but there are definitely run time uses of cl- in our dumped .el
> files, such as a cl-incf in font-lock.el which creates the gradually
> increasing row of dots which are displayed for large files (or, at least,
> used to be).

Again, not a problem.

Re: Lisp files that load cl-lib in problematical ways

[Michael Heerdegen]

Alan Mackenzie <acm@muc.de> writes:

> > If the above is correct, then how do you explain that in "emacs -Q" I
> > don't see the cl-lib feature present?
>
> Because the files that use cl- don't have a (require 'cl-lib) in them.
> Quite a lot of the function calls are things like cl-assert, cl-incf, or
> cl-defgeneric, which won't cause the loading of cl- until they are
> actually run.

These are all macros.  Unless you explicitly load the .el source files,
nothing of these will cause loading cl-lib when the according code is
run.

Michael.

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Quite a lot of the function calls are things like cl-assert, cl-incf, or
  > cl-defgeneric, which won't cause the loading of cl- until they are
  > actually run.

Those are macros, right?  These macros are not supposed to need cl-lib
at run time; they are supposed to be handled (when the code is
compiled) at compile time.

Maybe people changed these macros and caused them to load cl-lib.

But maybe not all of them.

I looked into cl-assert and found that it generates a call to
cl--assertion-failed, which is defined in xo cl-preloaded.el.
So I think that macro is ok -- the problem in question doesn't
seem to happen in cl-assert.

Which of the cl- macros really generate calls that require cl-lib
to be loaded?  We should fix those.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Sat, 21 Oct 2023 01:21:16 -0400
> 
> Which of the cl- macros really generate calls that require cl-lib
> to be loaded?

I don't think there are any.

> We should fix those.

If they exist, yes.

Re: Lisp files that load cl-lib in problematical ways

[Alan Mackenzie]

Hello, Eli.

On Sat, Oct 21, 2023 at 10:42:05 +0300, Eli Zaretskii wrote:
> > From: Richard Stallman <rms@gnu.org>
> > Cc: emacs-devel@gnu.org
> > Date: Sat, 21 Oct 2023 01:21:16 -0400

> > Which of the cl- macros really generate calls that require cl-lib
> > to be loaded?

> I don't think there are any.

> > We should fix those.

> If they exist, yes.

Just as a matter of interest, what loads cl-lib in my .emacs is
desktop.el.  It does this for a single call to cl-list* which could be
seen as rather contrived.

So any Emacs session loading desktop will also be loading cl-lib; that's
virtually any Emacs session in which real work will be getting done, I
think.

It would be fairly easy to remove cl-lib from desktop.el.  Or, on the
other hand, we could just as well preload cl-lib, and save an
unmeasurable amount of loading time at each Emacs startup.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Lisp files that load cl-lib in problematical ways

[Gerd Möllmann]

Alan Mackenzie <acm@muc.de> writes:

> It would be fairly easy to remove cl-lib from desktop.el.  Or, on the
> other hand, we could just as well preload cl-lib, and save an
> unmeasurable amount of loading time at each Emacs startup.

+1

RE: [External] : Re: Lisp files that load cl-lib in problematical ways

[Drew Adams]

> > It would be fairly easy to remove cl-lib from desktop.el.

+1

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Loading cl-lib when the user doesn't really need it is wasteful.
To load it unconditionally would only be more so.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> Date: Sat, 21 Oct 2023 17:53:31 +0000
> Cc: rms@gnu.org, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> Just as a matter of interest, what loads cl-lib in my .emacs is
> desktop.el.  It does this for a single call to cl-list* which could be
> seen as rather contrived.
> 
> So any Emacs session loading desktop will also be loading cl-lib; that's
> virtually any Emacs session in which real work will be getting done, I
> think.
> 
> It would be fairly easy to remove cl-lib from desktop.el.  Or, on the
> other hand, we could just as well preload cl-lib, and save an
> unmeasurable amount of loading time at each Emacs startup.

No, we will not preload cl-lib.  But if some package loads cl-lib for
a small number of functions that can be easily replaced with
non-cl-lib functions, patches to make such packages leaner and meaner
will be welcome.

Re: Lisp files that load cl-lib in problematical ways

[Alan Mackenzie]

Hello, Eli.

On Sat, Oct 21, 2023 at 22:00:16 +0300, Eli Zaretskii wrote:
> > Date: Sat, 21 Oct 2023 17:53:31 +0000
> > Cc: rms@gnu.org, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > Just as a matter of interest, what loads cl-lib in my .emacs is
> > desktop.el.  It does this for a single call to cl-list* which could be
> > seen as rather contrived.

> > So any Emacs session loading desktop will also be loading cl-lib; that's
> > virtually any Emacs session in which real work will be getting done, I
> > think.

> > It would be fairly easy to remove cl-lib from desktop.el.  Or, on the
> > other hand, we could just as well preload cl-lib, and save an
> > unmeasurable amount of loading time at each Emacs startup.

> No, we will not preload cl-lib.  But if some package loads cl-lib for
> a small number of functions that can be easily replaced with
> non-cl-lib functions, patches to make such packages leaner and meaner
> will be welcome.

Apologies, I was mistaken, it's not quite so simple.  desktop.el itself
could have its cl-lib dependency easily removed, but it loads
frameset.el, the bit of desktop that saves frame layouts.

frameset.el makes moderately heavy use of cl-lib, including at least two
externally visible cl-defun's where keyword parameters are used.  It
would be a lot of work to replace cl-lib in this file, and doing this
might be viewed as ungenerous towards its author, Juanma Barranquero.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Lisp files that load cl-lib in problematical ways

[Gerd Möllmann]

Alan Mackenzie <acm@muc.de> writes:

> frameset.el makes moderately heavy use of cl-lib, including at least two
> externally visible cl-defun's where keyword parameters are used.  It
> would be a lot of work to replace cl-lib in this file, and doing this
> might be viewed as ungenerous towards its author, Juanma Barranquero.

Yes, that, and it would make an unmeasurable difference.

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > frameset.el makes moderately heavy use of cl-lib, including at least two
  > externally visible cl-defun's where keyword parameters are used.  It
  > would be a lot of work to replace cl-lib in this file

cl-defun is a macro; using it ought to work without causing cl-lib to
be loaded.  So if frameset.el does load cl-lib, the next question is
what it is in the file that actually does so.

We might discover an underlying bug by which some construction loads
cl-lib but shouldn't do so.  Or we might find a few things frameset.el
which load cl-lib for clear reasons, but which would be a small
change.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: eliz@gnu.org, emacs-devel@gnu.org
> Date: Tue, 24 Oct 2023 22:44:49 -0400
> 
>   > frameset.el makes moderately heavy use of cl-lib, including at least two
>   > externally visible cl-defun's where keyword parameters are used.  It
>   > would be a lot of work to replace cl-lib in this file
> 
> cl-defun is a macro; using it ought to work without causing cl-lib to
> be loaded.  So if frameset.el does load cl-lib, the next question is
> what it is in the file that actually does so.

It uses several cl-lib functions: cl-every, cl-find-if,
cl-delete-if-not, and some others.

This code was written before we had seq.el, which is nowadays
preloaded.  So I guess we could rewrite frameset.el to use seq.el
instead (and a few cl-macs macros it uses).  If someone wants to work
on this, please do.

In any case, I see no reason to continue asking this kind of questions
on the list.  Each such question can be easily answered by using
M-. and "C-h f", so anyone who is interested in understanding why a
given Lisp package loads cl-lib already has the answers at their
fingertips.  There's no reason to ask questions on the list that can
be easily answered in a few seconds.

I also very much doubt that what people will find by using "C-h f" and
M-. will be "bugs", since such bugs would have been found long ago.
It bothers the heck out of me to hear assessments like that which
clearly hint that the quality of our code is deemed to be so low; it
is not.

And again, let's not keep this discussion, as it just wastes bandwidth
and energy on something that can be found within seconds by anyone who
wants to know.

Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Eli Zaretskii wrote:

> I also very much doubt that what people will find by using
> "C-h f" and M-. will be "bugs", since such bugs would have
> been found long ago. It bothers the heck out of me to hear
> assessments like that which clearly hint that the quality of
> our code is deemed to be so low; it is not.

?

Who is saying that?

And what code is that?

> And again, let's not keep this discussion, as it just wastes
> bandwidth and energy on something that can be found within
> seconds by anyone who wants to know.

Indeed, let's drop the cl-lib discussion. Our collective
investigation into the matter has shown that we should all let
it be. Muck around with it further, possible problems ahead
for what in practice will be very close to zero gain.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Which of the cl- macros really generate calls that require cl-lib
  > > to be loaded?

  > I don't think there are any.

If there aren't any, that's good -- but I think we have found a few.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Lisp files that load cl-lib in problematical ways

[Alan Mackenzie]

Hello, Richard.

On Sat, Oct 21, 2023 at 01:21:16 -0400, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]

>   > Quite a lot of the function calls are things like cl-assert, cl-incf, or
>   > cl-defgeneric, which won't cause the loading of cl- until they are
>   > actually run.

> Those are macros, right?  These macros are not supposed to need cl-lib
> at run time; they are supposed to be handled (when the code is
> compiled) at compile time.

A lot of them are macros, yes.  When I filter out the macros from my
list, and also functions of the form cl-generic-* and cl--generic-*
(cl-generic is loaded at dump time), I'm left with these functions:

* cl-call-next-method
* cl--class-parents
cl-member
* cl-next-method-p
cl-old-struct-compat-mode
cl-plusp
cl-print-object
* cl--slot-descriptor-name
* cl--slot-descriptor-props
* cl--struct-class-children-sym
* cl--struct-class-parents
* cl--struct-register-child

..  I've marked with an asterisk those that are in cl-preloaded.el or
cl-generic.el.

Of the remaining functions:
(i) cl-member is used only in a compiler macro for add-to-list in
  subr.el.  This is OK.
(ii) cl-old-struct-compat-mode is called from cl-struct-define in
  cl-preloaded.el.  It is an auto-load function which only gets loaded
  when cl-struct-define is called with an obsolete calling convention.
(iii) cl-plusp is a defsubst, effectively a macro.
(iv) cl-print-object is a defgeneric function in cl-print.el.  There is
  a cl-defmethod of cl-print-object in nadvice.el.  This contains calls
  to cl-print-object methods defined in cl-print.el  Since
  cl-print-object is an auto-load, this should only cause cl-lib to get
  loaded at run time, not dump time.

So it would appear there are no cl- functions used in the dumped .el
files which would cause cl-lib to be loaded.

There was only one cl- special variable which got bound to a non-nil
value in the dumped .el files.  That was cl--generic-edebug-name,
defined and bound in cl-generic.el.

It would take a more rigorous analysis to find any cl- variables bound
to nil in these files.

> Maybe people changed these macros and caused them to load cl-lib.

> But maybe not all of them.

I'm convinced this is not the case, unless it is by binding cl- special
variables to nil.

> I looked into cl-assert and found that it generates a call to
> cl--assertion-failed, which is defined in xo cl-preloaded.el.
> So I think that macro is ok -- the problem in question doesn't
> seem to happen in cl-assert.

> Which of the cl- macros really generate calls that require cl-lib
> to be loaded?  We should fix those.

After scanning the source code, I'm convinced there aren't any, with the
above proviso.

> -- 
> Dr Richard Stallman (https://stallman.org)
> Chief GNUisance of the GNU Project (https://gnu.org)
> Founder, Free Software Foundation (https://fsf.org)
> Internet Hall-of-Famer (https://internethalloffame.org)

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > So it would appear there are no cl- functions used in the dumped .el
  > files which would cause cl-lib to be loaded.

That is good news.  Thanks for studying this.

Nonetheless, we do have some problems.  For instance, the debugger
should not load cl-lib.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Lisp files that load cl-lib in problematical ways

[Alan Mackenzie]

Hello, Richard.

On Sun, Oct 22, 2023 at 22:11:31 -0400, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]

>   > So it would appear there are no cl- functions used in the dumped .el
>   > files which would cause cl-lib to be loaded.

> That is good news.  Thanks for studying this.

> Nonetheless, we do have some problems.  For instance, the debugger
> should not load cl-lib.

I would agree, but edebug _does_ load cl-lib.  So does desktop, as I've
already pointed out.  Even calling M-: (backtrace) causes cl-lib to get
loaded.  All of these, and likely many others, would take a lot of work
to fix.

It seems that while emacs -Q doesn't itself load cl-lib, there is now
very little, if anything, one can do in Emacs which doesn't load this
library.  I find this regrettable, but can't see there's much to be done
about it.

> -- 
> Dr Richard Stallman (https://stallman.org)
> Chief GNUisance of the GNU Project (https://gnu.org)
> Founder, Free Software Foundation (https://fsf.org)
> Internet Hall-of-Famer (https://internethalloffame.org)

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> Date: Mon, 23 Oct 2023 12:54:33 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> Even calling M-: (backtrace) causes cl-lib to get loaded.

That's because we use cl-print to display the backtrace.  That change
was done intentionally and for good reasons.  So that is not a
"problem" that needs to be "fixed", from where I stand.

> It seems that while emacs -Q doesn't itself load cl-lib, there is now
> very little, if anything, one can do in Emacs which doesn't load this
> library.  I find this regrettable, but can't see there's much to be done
> about it.

Neither should we do anything about it (except fixing places where
this is done unintentionally, if there are such places).

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Even calling M-: (backtrace) causes cl-lib to get loaded.

  > That's because we use cl-print to display the backtrace.  That change
  > was done intentionally and for good reasons.  So that is not a
  > "problem" that needs to be "fixed", from where I stand.

Can this load cl-print.el instead of cl-lib.el?  Would that do the job?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: acm@muc.de, emacs-devel@gnu.org
> Date: Tue, 24 Oct 2023 22:46:53 -0400
> 
>   > > Even calling M-: (backtrace) causes cl-lib to get loaded.
> 
>   > That's because we use cl-print to display the backtrace.  That change
>   > was done intentionally and for good reasons.  So that is not a
>   > "problem" that needs to be "fixed", from where I stand.
> 
> Can this load cl-print.el instead of cl-lib.el?  Would that do the job?

No.  If you look at cl-print.el, you will immediately see why: it uses
cl-incf and cl-find-class.

Once again: we do pay attention to this stuff, so the chance of
finding some random example where cl-lib is loaded due to a bug are
slim.  And in any case, it is not necessary to discuss this here, one
can just look in the code using "C-h f".

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Can this load cl-print.el instead of cl-lib.el?  Would that do the job?

  > No.  If you look at cl-print.el, you will immediately see why: it uses
  > cl-incf and cl-find-class.

Thanks for telling me about the problem at that next stage.
If I had tried that change I would have see why it did not work,
but I had only got as far as asking you.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: acm@muc.de, emacs-devel@gnu.org
> Date: Wed, 25 Oct 2023 22:27:34 -0400
> 
>   > > Can this load cl-print.el instead of cl-lib.el?  Would that do the job?
> 
>   > No.  If you look at cl-print.el, you will immediately see why: it uses
>   > cl-incf and cl-find-class.
> 
> Thanks for telling me about the problem at that next stage.
> If I had tried that change I would have see why it did not work,
> but I had only got as far as asking you.

There's no need to try.  I didn't try, I just typed "C-x cl- C-s C-s..."
and looked at the hits, invoking "C-h f" where I didn't recognize the
symbol from memory.

Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Eli Zaretskii wrote:

>> Thanks for telling me about the problem at that next stage.
>> If I had tried that change I would have see why it did not
>> work, but I had only got as far as asking you.
>
> There's no need to try. I didn't try, I just typed "C-x cl-
> C-s C-s..." and looked at the hits, invoking "C-h f" where
> I didn't recognize the symbol from memory.

Oh, the help loads cl-lib. Do 'emacs -Q' and then C-h
f cl-loop ... it auto-completes and you can use it after that,
because the help itself loaded it.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Gregory Heytings]

>
> For instance, the debugger should not load cl-lib.
>

Pretty much everything loads cl-lib nowadays, for example:

- C-s / C-r / M-% / C-M-%
- C-h C-h
- C-h t
- C-h w
- C-h k C-g
- M-x apropos
- clicking in the echo area
- clicking on the modeline
- hovering over the modeline or over the toolbar
- browsing the menus
- clicking on a completion in *Completions*
- CC Mode
- Shell-script mode
- VC (and opening any file in a version-controlled repository)
- Dired
- M-g g (or c or i)
- M-x shell
- M-x eshell
- M-x M-n
- M-. and M-? and M-, (Xref)
- C-x (
- M-&
- C-x m and Gnus and Rmail
- EWW
- Org

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> Date: Mon, 23 Oct 2023 14:06:33 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: emacs-devel@gnu.org
> 
> Pretty much everything loads cl-lib nowadays, for example:

That's not the issue, from my POV.  the issue is whether all these
places load cl-lib for good reasons.  If not, they shouldn't.

Are any of these commands loading cl-lib without a good reason?

Re: Lisp files that load cl-lib in problematical ways

[Gregory Heytings]

>> Pretty much everything loads cl-lib nowadays, for example:
>
> That's not the issue, from my POV.  the issue is whether all these 
> places load cl-lib for good reasons.  If not, they shouldn't.
>
> Are any of these commands loading cl-lib without a good reason?
>

I don't know.  I was only pointing out that the debugger is just one of 
the many cases, and not the most likely one, in which cl-lib is loaded. 
I was also hinting that it would perhaps make sense to preload cl-lib, 
given that it's practically impossible to use Emacs without loading 
cl-lib.

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> Date: Mon, 23 Oct 2023 14:48:29 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: rms@gnu.org, emacs-devel@gnu.org
> 
> 
> >> Pretty much everything loads cl-lib nowadays, for example:
> >
> > That's not the issue, from my POV.  the issue is whether all these 
> > places load cl-lib for good reasons.  If not, they shouldn't.
> >
> > Are any of these commands loading cl-lib without a good reason?
> >
> 
> I don't know.  I was only pointing out that the debugger is just one of 
> the many cases, and not the most likely one, in which cl-lib is loaded. 
> I was also hinting that it would perhaps make sense to preload cl-lib, 
> given that it's practically impossible to use Emacs without loading 
> cl-lib.

It is also "practically impossible" to use Emacs without loading some
other packages, like help-fns, for example.  But we still don't (and
shouldn't) preload them.

Preloading unnecessary stuff is a slippery slope that once we start
there's no way of knowing where we end.  So let's not.

Re: Lisp files that load cl-lib in problematical ways

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

> Preloading unnecessary stuff is a slippery slope that once we start
> there's no way of knowing where we end.  So let's not.

AFAIK, the main drawback is increased memory consumption.  The main
benefit is that Emacs starts faster.

Are there any other important considerations?

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Mon, 23 Oct 2023 15:11:06 -0700
> Cc: rms@gnu.org, emacs-devel@gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Preloading unnecessary stuff is a slippery slope that once we start
> > there's no way of knowing where we end.  So let's not.
> 
> AFAIK, the main drawback is increased memory consumption.  The main
> benefit is that Emacs starts faster.
> 
> Are there any other important considerations?

Introduction of symbols that are not needed.  Also, some packages
modify the run-time environment when they load.

Re: Lisp files that load cl-lib in problematical ways

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stefan Kangas <stefankangas@gmail.com>
>> Date: Mon, 23 Oct 2023 15:11:06 -0700
>> Cc: rms@gnu.org, emacs-devel@gnu.org
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > Preloading unnecessary stuff is a slippery slope that once we start
>> > there's no way of knowing where we end.  So let's not.
>> 
>> AFAIK, the main drawback is increased memory consumption.  The main
>> benefit is that Emacs starts faster.
>> 
>> Are there any other important considerations?
>
> Introduction of symbols that are not needed.  Also, some packages
> modify the run-time environment when they load.

And the pure space or W32 dumped_heap occupied increases as more files
are preloaded.  Until such time as they are removed, this remains an
important factor in deciding what to load.

Re: Lisp files that load cl-lib in problematical ways

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

> Introduction of symbols that are not needed.

Sure, but that's rather subjective.  One could also argue that it is a
good thing to have certain symbols available immediately, for example
because they are typically needed.

> Also, some packages modify the run-time environment when they load.

Yeah, we do have some bugs in that department.  :-(

Re: Lisp files that load cl-lib in problematical ways

[Ihor Radchenko]

Stefan Kangas <stefankangas@gmail.com> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> Introduction of symbols that are not needed.
>
> Sure, but that's rather subjective.  One could also argue that it is a
> good thing to have certain symbols available immediately, for example
> because they are typically needed.

Could it be customizeable somehow?

I have seen articles using portable dumper to pre-load more libraries,
but it was rather cumbersome and fragile exercise. It might be of
interest for people interested in reducing the startup time to have
a command like M-x dump-emacs-using-already-loaded-built-in-libraries.
That way, users can pre-load the libraries that will be loaded anyway.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: Eli Zaretskii <eliz@gnu.org>, gregory@heytings.org, rms@gnu.org,
>  emacs-devel@gnu.org
> Date: Tue, 24 Oct 2023 09:04:54 +0000
> 
> Stefan Kangas <stefankangas@gmail.com> writes:
> 
> > Eli Zaretskii <eliz@gnu.org> writes:
> >
> >> Introduction of symbols that are not needed.
> >
> > Sure, but that's rather subjective.  One could also argue that it is a
> > good thing to have certain symbols available immediately, for example
> > because they are typically needed.
> 
> Could it be customizeable somehow?

What do you want to be customizable, and how?

> I have seen articles using portable dumper to pre-load more libraries,
> but it was rather cumbersome and fragile exercise. It might be of
> interest for people interested in reducing the startup time to have
> a command like M-x dump-emacs-using-already-loaded-built-in-libraries.
> That way, users can pre-load the libraries that will be loaded anyway.

Isn't that already available using dump-emacs-portable?  Or am I
missing something?

Re: Lisp files that load cl-lib in problematical ways

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> > Sure, but that's rather subjective.  One could also argue that it is a
>> > good thing to have certain symbols available immediately, for example
>> > because they are typically needed.
>> 
>> Could it be customizeable somehow?
>
> What do you want to be customizable, and how?

What I have in mind is a custom list of packages to be dumped.

>> I have seen articles using portable dumper to pre-load more libraries,
>> but it was rather cumbersome and fragile exercise. It might be of
>> interest for people interested in reducing the startup time to have
>> a command like M-x dump-emacs-using-already-loaded-built-in-libraries.
>> That way, users can pre-load the libraries that will be loaded anyway.
>
> Isn't that already available using dump-emacs-portable?  Or am I
> missing something?

(1) It does not work interactively.
(2) Even making it work non-interactively is not necessarily trivial.
    See https://archive.casouri.cat/note/2020/painless-transition-to-portable-dumper/index.html

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: stefankangas@gmail.com, gregory@heytings.org, rms@gnu.org,
>  emacs-devel@gnu.org
> Date: Tue, 24 Oct 2023 12:41:13 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> > Sure, but that's rather subjective.  One could also argue that it is a
> >> > good thing to have certain symbols available immediately, for example
> >> > because they are typically needed.
> >> 
> >> Could it be customizeable somehow?
> >
> > What do you want to be customizable, and how?
> 
> What I have in mind is a custom list of packages to be dumped.

You can easily edit lisp/loadup.el and add whatever you want.  So this
possibility already exists.

> >> I have seen articles using portable dumper to pre-load more libraries,
> >> but it was rather cumbersome and fragile exercise. It might be of
> >> interest for people interested in reducing the startup time to have
> >> a command like M-x dump-emacs-using-already-loaded-built-in-libraries.
> >> That way, users can pre-load the libraries that will be loaded anyway.
> >
> > Isn't that already available using dump-emacs-portable?  Or am I
> > missing something?
> 
> (1) It does not work interactively.

Doesn't M-: work?

> (2) Even making it work non-interactively is not necessarily trivial.
>     See https://archive.casouri.cat/note/2020/painless-transition-to-portable-dumper/index.html

These are bugs we still need to find and fix.  But adding arbitrary
Lisp packages to loadup.el is also likely to cause bugs, since
dumping Emacs is a tricky business in any case.

Re: Lisp files that load cl-lib in problematical ways

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> > What do you want to be customizable, and how?
>> 
>> What I have in mind is a custom list of packages to be dumped.
>
> You can easily edit lisp/loadup.el and add whatever you want.  So this
> possibility already exists.

Indeed, custom dump is already possible, as described in the article I
linked to.

I meant something more interactive that takes care about arranging all
the internal details like running batch Emacs, setting up load-path, and
loading the packages.

>> > Isn't that already available using dump-emacs-portable?  Or am I
>> > missing something?
>> 
>> (1) It does not work interactively.
>
> Doesn't M-: work?

Nope.

Debugger entered--Lisp error: (error "Dumping Emacs currently works only in batch mode.  If you’d like it to work interactively, please consider contributing a patch to Emacs.")
  (dump-emacs-portable "~/tmp/emacs-pdumped")
  (eval (dump-emacs-portable "~/tmp/emacs-pdumped") t)

>> (2) Even making it work non-interactively is not necessarily trivial.
>>     See https://archive.casouri.cat/note/2020/painless-transition-to-portable-dumper/index.html
>
> These are bugs we still need to find and fix.  But adding arbitrary
> Lisp packages to loadup.el is also likely to cause bugs, since
> dumping Emacs is a tricky business in any case.

That's my point. If we can carefully select built-in packages that can
be safely dumped, we can generate Emacs executable tailored for user
configs.

For example, imaginary M-x dump-emacs-portable-interactively might check
the loaded features, filter them according to built-in libraries known
to be safe for dumping, and generate Emacs image that will pre-load
these libraries. This way, Emacs startup time will be reduced while not
adding unnecessary bloat, with "unnecessary" defined by the user config
rather than general considerations that must apply to every possible
Emacs user.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: stefankangas@gmail.com, gregory@heytings.org, rms@gnu.org,
>  emacs-devel@gnu.org
> Date: Tue, 24 Oct 2023 13:01:45 +0000
> 
> > These are bugs we still need to find and fix.  But adding arbitrary
> > Lisp packages to loadup.el is also likely to cause bugs, since
> > dumping Emacs is a tricky business in any case.
> 
> That's my point. If we can carefully select built-in packages that can
> be safely dumped, we can generate Emacs executable tailored for user
> configs.
> 
> For example, imaginary M-x dump-emacs-portable-interactively might check
> the loaded features, filter them according to built-in libraries known
> to be safe for dumping, and generate Emacs image that will pre-load
> these libraries. This way, Emacs startup time will be reduced while not
> adding unnecessary bloat, with "unnecessary" defined by the user config
> rather than general considerations that must apply to every possible
> Emacs user.

Feel free to work on such a feature, I'm sure some will like it,
especially people who for some reason start Emacs frequently.  It's a
lot of non-trivial work, though.

Re: Lisp files that load cl-lib in problematical ways

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> Feel free to work on such a feature, I'm sure some will like it,
> especially people who for some reason start Emacs frequently.  It's a
> lot of non-trivial work, though.

Not sure if I can allocate enough time soon enough, but as a quick
experimentation I tried

----- /tmp/dump.el -----
(require 'cl-lib)
(dump-emacs-portable "/tmp/emacs-dumped")
-------- end -----------

and got surprised by

> ./src/emacs -Q --batch -l /tmp/dump.el 
Dumping fingerprint: 83db7cb9b32874a4c4a0b923b2ca94f0d08a9a22ff41e18a2f92290276ef7d07

Error: error ("Trying to dump non fixed-up eln file")
  mapbacktrace(#f(compiled-function (evald func args flags) #<bytecode -0x154b0bb3d851781b>))
  debug-early-backtrace()
  debug-early(error (error "Trying to dump non fixed-up eln file"))
  dump-emacs-portable("/tmp/emacs-dumped")
  eval-buffer(#<buffer  *load*> nil "/tmp/dump.el" nil t)
  load-with-code-conversion("/tmp/dump.el" "/tmp/dump.el" nil t)
  load("/tmp/dump.el" nil t)
  command-line-1(("-l" "/tmp/dump.el"))
  command-line()
  normal-top-level()
Trying to dump non fixed-up eln file

Is it a bug? Or do I miss something important?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: stefankangas@gmail.com, gregory@heytings.org, rms@gnu.org,
>  emacs-devel@gnu.org
> Date: Tue, 24 Oct 2023 13:33:01 +0000
> 
> ----- /tmp/dump.el -----
> (require 'cl-lib)
> (dump-emacs-portable "/tmp/emacs-dumped")
> -------- end -----------
> 
> and got surprised by
> 
> > ./src/emacs -Q --batch -l /tmp/dump.el 
> Dumping fingerprint: 83db7cb9b32874a4c4a0b923b2ca94f0d08a9a22ff41e18a2f92290276ef7d07
> 
> Error: error ("Trying to dump non fixed-up eln file")
>   mapbacktrace(#f(compiled-function (evald func args flags) #<bytecode -0x154b0bb3d851781b>))
>   debug-early-backtrace()
>   debug-early(error (error "Trying to dump non fixed-up eln file"))
>   dump-emacs-portable("/tmp/emacs-dumped")
>   eval-buffer(#<buffer  *load*> nil "/tmp/dump.el" nil t)
>   load-with-code-conversion("/tmp/dump.el" "/tmp/dump.el" nil t)
>   load("/tmp/dump.el" nil t)
>   command-line-1(("-l" "/tmp/dump.el"))
>   command-line()
>   normal-top-level()
> Trying to dump non fixed-up eln file
> 
> Is it a bug? Or do I miss something important?

I think you are trying to do something that isn't supported.  Andrea,
am I right?

Your original idea was to add stuff to loadup, not to dump an already
dumped Emacs, and with AOT *.eln stuff on top of that.  Preloaded
*.eln files are handled specially by loadup.el during the dumping
process, and you tried doing that without observing the protocol of
*.eln dumping.

Re: Lisp files that load cl-lib in problematical ways

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Ihor Radchenko <yantar92@posteo.net>
>> Cc: stefankangas@gmail.com, gregory@heytings.org, rms@gnu.org,
>>  emacs-devel@gnu.org
>> Date: Tue, 24 Oct 2023 13:33:01 +0000
>> 
>> ----- /tmp/dump.el -----
>> (require 'cl-lib)
>> (dump-emacs-portable "/tmp/emacs-dumped")
>> -------- end -----------
>> 
>> and got surprised by
>> 
>> > ./src/emacs -Q --batch -l /tmp/dump.el 
>> Dumping fingerprint: 83db7cb9b32874a4c4a0b923b2ca94f0d08a9a22ff41e18a2f92290276ef7d07
>> 
>> Error: error ("Trying to dump non fixed-up eln file")
>>   mapbacktrace(#f(compiled-function (evald func args flags) #<bytecode -0x154b0bb3d851781b>))
>>   debug-early-backtrace()
>>   debug-early(error (error "Trying to dump non fixed-up eln file"))
>>   dump-emacs-portable("/tmp/emacs-dumped")
>>   eval-buffer(#<buffer  *load*> nil "/tmp/dump.el" nil t)
>>   load-with-code-conversion("/tmp/dump.el" "/tmp/dump.el" nil t)
>>   load("/tmp/dump.el" nil t)
>>   command-line-1(("-l" "/tmp/dump.el"))
>>   command-line()
>>   normal-top-level()
>> Trying to dump non fixed-up eln file
>> 
>> Is it a bug? Or do I miss something important?
>
> I think you are trying to do something that isn't supported.  Andrea,
> am I right?
>
> Your original idea was to add stuff to loadup, not to dump an already
> dumped Emacs, and with AOT *.eln stuff on top of that.  Preloaded
> *.eln files are handled specially by loadup.el during the dumping
> process, and you tried doing that without observing the protocol of
> *.eln dumping.


Yes that's correct, so far we never supported re-dumping Emacs with
elns.

I think it should be rather easy tho, I'll try to investigate a bit.

Bests

  Andrea

Re: Lisp files that load cl-lib in problematical ways

[Corwin Brust]

[-- Attachment #1: Type: text/plain, Size: 2062 bytes --]

On Tue, Oct 24, 2023 at 9:31 AM Andrea Corallo <acorallo@gnu.org> wrote:

> Eli Zaretskii <eliz@gnu.org> writes:
>
> >> From: Ihor Radchenko <yantar92@posteo.net>
> >> Cc: stefankangas@gmail.com, gregory@heytings.org, rms@gnu.org,
> >>  emacs-devel@gnu.org
> >> Date: Tue, 24 Oct 2023 13:33:01 +0000
> >>
> >> ----- /tmp/dump.el -----
> >> (require 'cl-lib)
> >> (dump-emacs-portable "/tmp/emacs-dumped")
> >> -------- end -----------
> >>
> >> and got surprised by
> >>
> >> Error: error ("Trying to dump non fixed-up eln file")
>

FTR, I tried the same experiment and got the same result.  Glad to know I'm
not the only person who thought to try this approach :)

I then also tried creating a lisp/site-load.el  with just (load
"emacs-lisp/cl-lib") and got this seemingly circular error, suggesting it
might not be so easy to include cl-lib in the dumper.  (Or maybe just that
I have no idea what I'm doing, which is true.)

Loading emacs-lisp/cl-lib (native compiled elisp)...

Error: error ("(require cl-macs) while preparing to dump")
  mapbacktrace(#[1028
"\1\4\203\24\0\301\302!\210\300\4!\210\301\303!\210\202\35\0\301\304!\210\3\3B\262\1\211\2035\0\300\1@!\210\211A\211\262\2\2035\0\301\305!\210\202!\0\301\306!\207"
[prin1 princ "  " "(" "  (" " " ")\n"] 7 "\n\n(fn EVALD FUNC ARGS FLAGS)"])
  debug-early-backtrace()
  debug-early(error (error "(require cl-macs) while preparing to dump"))
  require(cl-macs)

byte-code("\300\301!\210\302\303\304\305#\204\24\0\306\307!\210\306\310!\210\300\207"
[provide cl-lib load "cl-loaddefs" noerror quiet require cl-macs cl-seq] 4)
  load("emacs-lisp/cl-lib")
  load("loadup.el")
(require cl-macs) while preparing to dump
make[1]: *** [Makefile:682: emacs.pdmp] Error 127
make[1]: Leaving directory '/h/emacs-dev/emacs-30/src'
make: *** [Makefile:554: src] Error 2


>
> Yes that's correct, so far we never supported re-dumping Emacs with
> elns.
>
> I think it should be rather easy tho, I'll try to investigate a bit.
>
>
Thanks for looking into this!

[-- Attachment #2: Type: text/html, Size: 3231 bytes --]

Re: Lisp files that load cl-lib in problematical ways

[Andrea Corallo]

Andrea Corallo <acorallo@gnu.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Ihor Radchenko <yantar92@posteo.net>
>>> Cc: stefankangas@gmail.com, gregory@heytings.org, rms@gnu.org,
>>>  emacs-devel@gnu.org
>>> Date: Tue, 24 Oct 2023 13:33:01 +0000
>>> 
>>> ----- /tmp/dump.el -----
>>> (require 'cl-lib)
>>> (dump-emacs-portable "/tmp/emacs-dumped")
>>> -------- end -----------
>>> 
>>> and got surprised by
>>> 
>>> > ./src/emacs -Q --batch -l /tmp/dump.el 
>>> Dumping fingerprint: 83db7cb9b32874a4c4a0b923b2ca94f0d08a9a22ff41e18a2f92290276ef7d07
>>> 
>>> Error: error ("Trying to dump non fixed-up eln file")
>>>   mapbacktrace(#f(compiled-function (evald func args flags) #<bytecode -0x154b0bb3d851781b>))
>>>   debug-early-backtrace()
>>>   debug-early(error (error "Trying to dump non fixed-up eln file"))
>>>   dump-emacs-portable("/tmp/emacs-dumped")
>>>   eval-buffer(#<buffer  *load*> nil "/tmp/dump.el" nil t)
>>>   load-with-code-conversion("/tmp/dump.el" "/tmp/dump.el" nil t)
>>>   load("/tmp/dump.el" nil t)
>>>   command-line-1(("-l" "/tmp/dump.el"))
>>>   command-line()
>>>   normal-top-level()
>>> Trying to dump non fixed-up eln file
>>> 
>>> Is it a bug? Or do I miss something important?
>>
>> I think you are trying to do something that isn't supported.  Andrea,
>> am I right?
>>
>> Your original idea was to add stuff to loadup, not to dump an already
>> dumped Emacs, and with AOT *.eln stuff on top of that.  Preloaded
>> *.eln files are handled specially by loadup.el during the dumping
>> process, and you tried doing that without observing the protocol of
>> *.eln dumping.
>
>
> Yes that's correct, so far we never supported re-dumping Emacs with
> elns.
>
> I think it should be rather easy tho, I'll try to investigate a bit.

Okay with f7d88f4a047 on master I'm now able to re-dump a native
compiled Emacs!  I followed essentially what Ihor attemped so I loaded
cl-lib before re-dumping as well.

Not 100% sure loadup.el is the best place to host the definition of the
the function and the two new variables I've introduced.  Happy to move
things around if necessary.

Thanks

  Andrea

Re: Lisp files that load cl-lib in problematical ways

[Ihor Radchenko]

Andrea Corallo <acorallo@gnu.org> writes:

>> I think it should be rather easy tho, I'll try to investigate a bit.
>
> Okay with f7d88f4a047 on master I'm now able to re-dump a native
> compiled Emacs!  I followed essentially what Ihor attemped so I loaded
> cl-lib before re-dumping as well.

I confirm.
I was able to dump all the extra built-in libraries Emacs loads, except
Org mode without errors on the latest master.
Now, experiencing various funny glitches :)

But even pre-loading extra built-in libraries reduced startup time
noticeably:

;; With dumper: "7.715609 seconds"
;; Without dumper: "9.866784 seconds"

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Lisp files that load cl-lib in problematical ways

[Corwin Brust]

[-- Attachment #1: Type: text/plain, Size: 937 bytes --]

On Tue, Oct 24, 2023 at 2:43 PM Ihor Radchenko <yantar92@posteo.net> wrote:

> Andrea Corallo <acorallo@gnu.org> writes:
>
> >> I think it should be rather easy tho, I'll try to investigate a bit.
> >
> > Okay with f7d88f4a047 on master I'm now able to re-dump a native
> > compiled Emacs!  I followed essentially what Ihor attemped so I loaded
> > cl-lib before re-dumping as well.
>
> I confirm.
>

It worked for me also, trying from a MSYS2/MINGW64 shell, on Windows.

$ bin/emacs --batch -Q -eval '(let ((custom-theme-load-path (list
"<path>/.emacs.d/elpa/cyberpunk-2019-theme-20191008.1133/"))) (require
'"'"'erc) (load-theme '"'"'cyberpunk-2019 t) (enable-theme
'"'"'cyberpunk-2019) (dump-emacs-portable "erc.pdmp"))'

After which this works:

$ bin/runemacs.exe --dump-file erc.pdmp -fs -l <my-erc-setup>

I've been connected for around 13 hours, so far, so I think it's working :)

Thank you Andrea.

[-- Attachment #2: Type: text/html, Size: 1719 bytes --]

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Tue, 24 Oct 2023 01:35:21 -0700
> Cc: gregory@heytings.org, rms@gnu.org, emacs-devel@gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Introduction of symbols that are not needed.
> 
> Sure, but that's rather subjective.

No, it's completely objective: we add symbols to obarray that don't
need to be there.

> One could also argue that it is a good thing to have certain symbols
> available immediately, for example because they are typically
> needed.

We have autoloading to take care of that.

Re: Lisp files that load cl-lib in problematical ways

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

>>> Introduction of symbols that are not needed.
>>
>> Sure, but that's rather subjective.
>
> No, it's completely objective: we add symbols to obarray ...

This part is objective, indeed.

> ... that don't need to be there.

But this part seems more subjective (based on a very strict definition
of "need", a lot of now-preloaded things don't really need to be).

>> One could also argue that it is a good thing to have certain symbols
>> available immediately, for example because they are typically
>> needed.
>
> We have autoloading to take care of that.

Yes, of course.  But we also have preloading.  ;-)

Re: Lisp files that load cl-lib in problematical ways

[Björn Bidar]

Stefan Kangas <stefankangas@gmail.com> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> Preloading unnecessary stuff is a slippery slope that once we start
>> there's no way of knowing where we end.  So let's not.
>
> AFAIK, the main drawback is increased memory consumption.  The main
> benefit is that Emacs starts faster.

About how much memory are we talking here? All this sounds like talk
 about micro optimizations.
Emacs speed is more limited about it being single threaded most of the
 time than anything else.
 

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Björn Bidar <bjorn.bidar@thaodan.de>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Gregory Heytings <gregory@heytings.org>,
>   rms@gnu.org,  emacs-devel@gnu.org
> Date: Tue, 24 Oct 2023 11:10:06 +0300
> 
> Stefan Kangas <stefankangas@gmail.com> writes:
> 
> > Eli Zaretskii <eliz@gnu.org> writes:
> >
> >> Preloading unnecessary stuff is a slippery slope that once we start
> >> there's no way of knowing where we end.  So let's not.
> >
> > AFAIK, the main drawback is increased memory consumption.  The main
> > benefit is that Emacs starts faster.
> 
> About how much memory are we talking here? All this sounds like talk
>  about micro optimizations.

If we ignore such "micro optimizations", we will quickly bloat Emacs,
since those small amounts add up.  We have a lot of relatively small
potential additions that we make a point of not adding, because if we
decide adding one is okay, then why not add all the rest?  We don't
add such stuff to subr.el or to simple.el or to files.el, and keep
them separate, precisely because adding them all will not be
insignificant.

The only way of keeping this kind of bloat at bay is to resist adding
anything, no matter how small, because, like I said, it's a slippery
slope.

> Emacs speed is more limited about it being single threaded most of the
>  time than anything else.

I don't see the relevance, sorry.  We can work on making Emacs faster
without bloating it.

Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Eli Zaretskii wrote:

>> About how much memory are we talking here? All this sounds
>> like talk about micro optimizations.
>
> If we ignore such "micro optimizations", we will quickly
> bloat Emacs, since those small amounts add up. We have a lot
> of relatively small potential additions that we make a point
> of not adding, because if we decide adding one is okay, then
> why not add all the rest? We don't add such stuff to subr.el
> or to simple.el or to files.el, and keep them separate,
> precisely because adding them all will not be insignificant.
>
> The only way of keeping this kind of bloat at bay is to
> resist adding anything, no matter how small, because, like
> I said, it's a slippery slope.

While I agree one shouldn't ignore micro optimizations in
general, you keep refering to useful Elisp as bloat. That is
disrespectful and also not true, it is useful code, and where
it is or isn't added or loaded doesn't change it into
something it isn't.

Also, instead of the general "slippery slope" model of thought
and offered explanation, it is better to either have a clear
policy "we add stuff that [...]". Then one can say "we don't
intend to include some-package.el because it doesn't fall
under our policy what stuff to add, because it is [...] and
the policy is [...]".

If one cannot formulate a general policy it will instead have
to be dealt with on a case-by-case basis. But then one would
like to hear the arguments for each such case suggested, and
in terms of what that package brings (or lacks) and where it
is suggested to be added, not that you want to keep "this kind
of bloat at bay".

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Tue, 24 Oct 2023 14:43:48 +0200
> 
> Eli Zaretskii wrote:
> 
> > If we ignore such "micro optimizations", we will quickly
> > bloat Emacs, since those small amounts add up. We have a lot
> > of relatively small potential additions that we make a point
> > of not adding, because if we decide adding one is okay, then
> > why not add all the rest? We don't add such stuff to subr.el
> > or to simple.el or to files.el, and keep them separate,
> > precisely because adding them all will not be insignificant.
> >
> > The only way of keeping this kind of bloat at bay is to
> > resist adding anything, no matter how small, because, like
> > I said, it's a slippery slope.
> 
> While I agree one shouldn't ignore micro optimizations in
> general, you keep refering to useful Elisp as bloat.

No, I said that loading stuff that _is_not_necessary_ will _cause_
bloat.  Please don't misrepresent what I wrote.

> That is disrespectful and also not true, it is useful code, and
> where it is or isn't added or loaded doesn't change it into
> something it isn't.

See above.

> Also, instead of the general "slippery slope" model of thought
> and offered explanation, it is better to either have a clear
> policy "we add stuff that [...]".

The policy exists and is clear: we don' preload anything that isn't
needed by Emacs either during the dumping process or when it starts
after dumping.

Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Eli Zaretskii wrote:

> No, I said that loading stuff that _is_not_necessary_ will
> _cause_ bloat. Please don't misrepresent what I wrote.
>
>> That is disrespectful and also not true, it is useful code,
>> and where it is or isn't added or loaded doesn't change it
>> into something it isn't.

> See above.

Yeah, okay but I think it is better to avoid that word?

>> Also, instead of the general "slippery slope" model of
>> thought and offered explanation, it is better to either
>> have a clear policy "we add stuff that [...]".
>
> The policy exists and is clear: we don' preload anything
> that isn't needed by Emacs either during the dumping process
> or when it starts after dumping.

OK, you have a policy! Good :)

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Björn Bidar]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Björn Bidar <bjorn.bidar@thaodan.de>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  Gregory Heytings <gregory@heytings.org>,
>>   rms@gnu.org,  emacs-devel@gnu.org
>> Date: Tue, 24 Oct 2023 11:10:06 +0300
>> 
>> Stefan Kangas <stefankangas@gmail.com> writes:
>> 
>> > Eli Zaretskii <eliz@gnu.org> writes:
>> >
>> >> Preloading unnecessary stuff is a slippery slope that once we start
>> >> there's no way of knowing where we end.  So let's not.
>> >
>> > AFAIK, the main drawback is increased memory consumption.  The main
>> > benefit is that Emacs starts faster.
>> 
>> About how much memory are we talking here? All this sounds like talk
>>  about micro optimizations.
>
> If we ignore such "micro optimizations", we will quickly bloat Emacs,
> since those small amounts add up.  We have a lot of relatively small
> potential additions that we make a point of not adding, because if we
> decide adding one is okay, then why not add all the rest?  We don't
> add such stuff to subr.el or to simple.el or to files.el, and keep
> them separate, precisely because adding them all will not be
> insignificant.

I get that point but in this case we talk about doing something
slightly different to not load cl-lib while eventually cl-lib is loaded
anyway.

>
>> Emacs speed is more limited about it being single threaded most of the
>>  time than anything else.
>
> I don't see the relevance, sorry.  We can work on making Emacs faster
> without bloating it.

From my pov Emacs start up time or speed is more limted by Emacs being
single threaded rather than loading a few kbs of code that will be
loaded later anyway.

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Björn Bidar <bjorn.bidar@thaodan.de>
> Cc: stefankangas@gmail.com,  gregory@heytings.org,  rms@gnu.org,
>   emacs-devel@gnu.org
> Date: Tue, 24 Oct 2023 18:29:55 +0300
> 
> > If we ignore such "micro optimizations", we will quickly bloat Emacs,
> > since those small amounts add up.  We have a lot of relatively small
> > potential additions that we make a point of not adding, because if we
> > decide adding one is okay, then why not add all the rest?  We don't
> > add such stuff to subr.el or to simple.el or to files.el, and keep
> > them separate, precisely because adding them all will not be
> > insignificant.
> 
> I get that point but in this case we talk about doing something
> slightly different to not load cl-lib while eventually cl-lib is loaded
> anyway.

How is cl-lib different from any other package that is extremely
likely to be loaded close to a beginning of a production session?

The rationale for not loading anything we don't strictly need is that
different users have different use patterns, and Emacs can be used in
different modes: a GUI or TTY interactive session (with various user
customizations that load packages), a batch command, in a script using
--script, as a daemon, etc.  Each one of these needs different
optional modes needs to load different packages; preloading things
that are frequently loaded is therefore likely to load stuff needed by
some other mode/configuration, and that is simply not clean, besides
the fact that it "punishes" users by having them load stuff they don't
need.  E.g., my production session loads no less than 214 packages
right when it starts.  Why would we do that when everything works
without it?  What kind of problem are we trying to solve here?  I see
no problem.

> >> Emacs speed is more limited about it being single threaded most of the
> >>  time than anything else.
> >
> > I don't see the relevance, sorry.  We can work on making Emacs faster
> > without bloating it.
> 
> >From my pov Emacs start up time or speed is more limted by Emacs being
> single threaded rather than loading a few kbs of code that will be
> loaded later anyway.

So our POVs differ.  Lat's agree to disagree about that.

Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Eli Zaretskii wrote:

> E.g., my production session loads no less than 214 packages
> right when it starts. Why would we do that when everything
> works without it? What kind of problem are we trying to
> solve here? I see no problem.

We are trying to understand why guys grep the source and
traces this huge software component down to individual
instances of its use, and like, "hey, can we remove that?".

Gonna do that to those other 213 packages as well?

So let me ask, what is the special attention to cl-lib? To me
it looks like - not saying it is like that - ideology, if you
don't like CL maybe you see cl-lib like some infiltration from
CL? And for this reason don't like it? Well, if so, sorry to
say guys, but you lost the battle then! We are not getting rid
of this infiltration.

Might and Glory to Elisp(cl)!

:P

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Tue, 24 Oct 2023 19:54:11 +0200
> 
> Eli Zaretskii wrote:
> 
> So let me ask, what is the special attention to cl-lib?

For the gazillionth time: it isn't!  Not as far as preloading it is
considered.

> To me it looks like - not saying it is like that - ideology, if you
> don't like CL maybe you see cl-lib like some infiltration from CL?
> And for this reason don't like it? Well, if so, sorry to say guys,
> but you lost the battle then! We are not getting rid of this
> infiltration.

Not relevant to the issue of loading cl-lib.

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > How is cl-lib different from any other package that is extremely
  > likely to be loaded close to a beginning of a production session?

I think that is the crucial question -- it shows the deep difference.

Most packages do a specific kind of job and have just a few entry
points that relate to that job.  Consider sendmail.el, for instance.
It Has commands to initialize a message and to send it, and some
relating to editing one.  Most of the package is internals
that you wouldn't need to know about in order to _use_ sendmail.

CL makes a big contrast with that. It is mainly entry points.  And
many of these entry points try to be as multi-purpose as possible.

What are the implications of this?

If your package FROB wants to send an email, and it uses sendmail.el
to do that, that won't add much complexity to Emacs or FROB.  Using
sendmail.tl won't make FROB harder to maintain, because as a
maintainer of FROB there is only a little you need to know as a
consequnce of its use of sendmail.  Use of sendmail.el from the few
places that want to send an email just is not a problem.

This is why I'm concered with how many packages use CL, but not
concerned with how many use sendmail.el.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: bjorn.bidar@thaodan.de, stefankangas@gmail.com,
> 	gregory@heytings.org, emacs-devel@gnu.org
> Date: Thu, 26 Oct 2023 22:12:18 -0400
> 
>   > How is cl-lib different from any other package that is extremely
>   > likely to be loaded close to a beginning of a production session?
> 
> I think that is the crucial question -- it shows the deep difference.
> 
> Most packages do a specific kind of job and have just a few entry
> points that relate to that job.  Consider sendmail.el, for instance.
> It Has commands to initialize a message and to send it, and some
> relating to editing one.  Most of the package is internals
> that you wouldn't need to know about in order to _use_ sendmail.
> 
> CL makes a big contrast with that. It is mainly entry points.  And
> many of these entry points try to be as multi-purpose as possible.

We have a lot of the CL variety as well.  Look at lisp/emacs-lisp/
directory: it is full of such "entry-point-only" packages, and most of
them aren't preloaded, they load on demand.  That's typical of any
Lisp package that provides infrastructure rather than applications,
like sendmail.el does.  cl-lib is one of those infrastructure
packages, but it isn't the only one, far from that.  We take care not
to preload such infrastructure packages unless they are needed by the
code which runs at build time, or are needed at startup.

So I again say: cl-lib is not different from any other infrastructure
package that is likely to be loaded because some Lisp program needs to
use it.

> This is why I'm concered with how many packages use CL, but not
> concerned with how many use sendmail.el.

Are you concerned with any of the following?

  easy-mmode.el
  eieio.el
  generic.el
  icons.el
  let-alist.el
  map.el
  rx.el
  subr-x.el
  unsafep.el
  warnings.el

They are all of the same nature: they provide utility functions to be
used by Lisp programs.

Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Eli Zaretskii wrote:

>> Pretty much everything loads cl-lib nowadays [...]
>
> That's not the issue, from my POV. the issue is whether all
> these places load cl-lib for good reasons. If not,
> they shouldn't.

But as you say, this is nothing specific to cl-lib.
Because without good reason, nothing should load anything.

> Are any of these commands loading cl-lib without
> a good reason?

There is no way to tell without examining each case. And case
here isn't a file but individual usage of whatever
cl-lib provides.

And, because cl-lib is just another Elisp library at the same
level and status as any other, we might generalize the
question from both ends, "Is any file loading anything without
good reason?" If so - yes, we should fix that.

And voilà, everyone agrees.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Björn Bidar]

Alan Mackenzie <acm@muc.de> writes:

> Hello, Eli.
>> > I fear that that's the way things will stay, because it would be too
>> > much work to fix.  But more because there are enough contributors who
>> > think that cl-lib is just an ordinary part of Emacs to be used freely
>> > without restraint.  I am not among this group of contributors.
>
>> We cannot possibly expect people to contribute code if we force them
>> not to use the macros they are used to.  If cl-lib is not loaded as
>> result, that is good enough for us, I think.
>
> We "force" them to use Emacs Lisp, but they still contribute.  The
> problem is that use of cl- makes maintenance of other people's code much
> harder, at least for me.  I'm sure I'm not alone.
>
> cl- probably multiplies the size of the Lisp language parts of Elisp
> (i.e. things like defun, defmacro, if, while, cond, ...) by around three.
> I'm not able to learn all these fancy macros which are used rarely enough
> not to be learnable, but frequently enough to cause maintenance headaches
> - every time I encounter `if-let' and the like, it just stops me in my
> tracks.  I either have to guess what it means, which makes me
> uncomfortable, or laboriously consult a doc string, if any.
>
> Yes, I know if-let isn't a cl- function, and its doc string is of
> reasonable quality.  Many cl- functions don't have doc strings, or have
> inadequate ones, however.
>
> There was a time in the medium past when the use of run time cl- was not
> allowed.  Emacs was then, from my point of view, much easier to maintain.
> At some stage this rule was done away with, but I don't recall there
> being any discussion on emacs-devel about this change.  There should have
> been such discussion.
>

I think if people find cl- useful there are reasons for that.
That code changes, things get harder to read if one doesn't know the new
code is normal I think.

Resisting against change, doesn't seem like the best option to me.
But the only change that is necessary from my point of view is that the
documentation is on par.

As someone that is new in lisp programming I found that these shortcuts
that macros like these allow make things easier but in sometimes harder
to understand what's going on.

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > and got the following results:

  > abbrev.el:2

A generally useful minor mode as abbrev.el should not load cl-lib.
Why does it?  I investigated.

abbrev--check-chars uses the macro cl-pushnew.  Generally we
arrange for cl- macros not to load cl-lib.  But cl-pushnew
is _defined_ in cl-lib.  It ought to be defined with and like
other cl- macros that are meant for users to use.

That's a bug, and probably not a deep one.  Would someone please fix
it?

I investigated byte-run.el.  I don't think it uses any cl- facility.
Rather, it declares a let-variable whose name starts with cl-.

I think that if we investigate the other general-purpose packages
in the list you posted (I copied it below), we wlll find many such bugs that
got in because developers were not on guard against them.

Once we fix them all, how about if we add a regression test
to verify that various packages anyone might load at any time
do not use cl-lib.  With that, bugs like this would get caught
right away.

Here's the list, with some files crossed off that I've just handled.

;; abbrev.el:2
dnd.el:4
;; emacs-lisp/byte-run.el:1
emacs-lisp/cconv.el:13
emacs-lisp/cl-generic.el:56
emacs-lisp/cl-preloaded.el:16
emacs-lisp/eldoc.el:6
emacs-lisp/lisp-mode.el:6
emacs-lisp/nadvice.el:2
emacs-lisp/oclosure.el:8
emacs-lisp/seq.el:25
emacs-lisp/shorthands.el:1
emacs-lisp/syntax.el:5
emacs-lisp/tabulated-list.el:1
emacs-lisp/timer.el:1
font-lock.el:1
frame.el:5
international/mule-cmds.el:3
international/ucs-normalize.el:1
isearch.el:3
ldefs-boot.el:3
loaddefs.el:3
minibuffer.el:12
progmodes/elisp-mode.el:11
progmodes/prog-mode.el:1
register.el:3
replace.el:1
select.el:3
simple.el:9
startup.el:1
subr.el:1
tab-bar.el:4
term/android-win.el:4
term/haiku-win.el:3
term/ns-win.el:2
term/pc-win.el:6
term/pgtk-win.el:3
term/w32-win.el:4
term/x-win.el:2
uniquify.el:2
vc/vc-hooks.el:1


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Sun, 22 Oct 2023 22:08:55 -0400
> 
>   > abbrev.el:2
> 
> A generally useful minor mode as abbrev.el should not load cl-lib.
> Why does it?  I investigated.
> 
> abbrev--check-chars uses the macro cl-pushnew.  Generally we
> arrange for cl- macros not to load cl-lib.  But cl-pushnew
> is _defined_ in cl-lib.  It ought to be defined with and like
> other cl- macros that are meant for users to use.

It would not have helped to define cl-pushnew on cl-macs, because
cl-pushnew calls cl-adjoin, which is a function defined on cl-lib.  So
basically cl-pushnew is a (rather thin) wrapper around cl-adjoin, and
thus it is correctly defined on the same file where cl-adjoin is
defined.  I see no problem here.

> I investigated byte-run.el.  I don't think it uses any cl- facility.
> Rather, it declares a let-variable whose name starts with cl-.

byte-run.el indeed doesn't need cl-lib, and loading byte-run.elc
doesn't load cl-lib.  So I see no problem here, either.  IOW, this is
a false positive due to the naïve search pattern used to detect this.

> I think that if we investigate the other general-purpose packages
> in the list you posted (I copied it below), we wlll find many such bugs that
> got in because developers were not on guard against them.

I very much doubt that we will find many such bugs, because we
actually pay close attention to these aspects as part of our patch
review process.  But, of course, there could be places we missed, so
if someone finds them, please do report them and/or propose patches to
fix them.

> Once we fix them all, how about if we add a regression test
> to verify that various packages anyone might load at any time
> do not use cl-lib.  With that, bugs like this would get caught
> right away.

It isn't easy to write such a test, since our test suite
infrastructure, ert.el, itself uses cl-lib extensively.  But if
someone can come up with a test, please do submit it, and TIA.

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > It would not have helped to define cl-pushnew on cl-macs, because
  > cl-pushnew calls cl-adjoin, which is a function defined on cl-lib.  So
  > basically cl-pushnew is a (rather thin) wrapper around cl-adjoin, and
  > thus it is correctly defined on the same file where cl-adjoin is
  > defined.

That is a good point -- for that particular solution I proposed.

But if we let ourselves be a little more flexible, how about this
solution: install this

        (while (string-match "\\W" abbrev pos)
          (or (memq (aref abbrev (match-beginning 0)) badchars)
              (push (aref abbrev (match-beginning 0)) badchars))
          (setq pos (1+ pos)))

instead of the current code:

        (while (string-match "\\W" abbrev pos)
          (cl-pushnew (aref abbrev (match-beginning 0)) badchars)
          (setq pos (1+ pos)))

A CL fan might say the latter is simpler, but for me, who isn't sure
just how to use cl-pushnew, the former seems conceptually simpler.

The two ecxpressions have two different kinds of complexity.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Richard Stallman wrote:

> But if we let ourselves be a little more flexible, how about
> this solution: install this
>
>         (while (string-match "\\W" abbrev pos)
>           (or (memq (aref abbrev (match-beginning 0)) badchars)
>               (push (aref abbrev (match-beginning 0)) badchars))
>           (setq pos (1+ pos)))
>
> instead of the current code:
>
>         (while (string-match "\\W" abbrev pos)
>           (cl-pushnew (aref abbrev (match-beginning 0)) badchars)
>           (setq pos (1+ pos)))
>
> A CL fan might say the latter is simpler [...]

Yes, but none of those codes are super-simple, to be fair,
are they?

BTW the CL case can be improved like this - if it isn't that
much simpler, then at least it is much neater, I'd say.

(while (string-match "\\W" abbrev pos)
  (cl-pushnew (aref abbrev (match-beginning 0)) badchars)
  (cl-incf pos) )

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Stefan Kangas]

Richard Stallman <rms@gnu.org> writes:

> But if we let ourselves be a little more flexible, how about this
> solution: install this
>
>         (while (string-match "\\W" abbrev pos)
>           (or (memq (aref abbrev (match-beginning 0)) badchars)
>               (push (aref abbrev (match-beginning 0)) badchars))
>           (setq pos (1+ pos)))
>
> instead of the current code:
>
>         (while (string-match "\\W" abbrev pos)
>           (cl-pushnew (aref abbrev (match-beginning 0)) badchars)
>           (setq pos (1+ pos)))
>
> A CL fan might say the latter is simpler, but for me, who isn't sure
> just how to use cl-pushnew, the former seems conceptually simpler.

The `cl-pushnew' docstring explains:

    Add X to the list stored in PLACE unless X is already in the list.

> The two ecxpressions have two different kinds of complexity.

The only difference I see is that new version is harder to read, and
more prone to bugs (as you have to write the same code twice).

I'd suggest avoiding these types of changes.

Re: Lisp files that load cl-lib in problematical ways

[Gerd Möllmann]

Stefan Kangas <stefankangas@gmail.com> writes:

> Richard Stallman <rms@gnu.org> writes:
>
>> But if we let ourselves be a little more flexible, how about this
>> solution: install this
>>
>>         (while (string-match "\\W" abbrev pos)
>>           (or (memq (aref abbrev (match-beginning 0)) badchars)
>>               (push (aref abbrev (match-beginning 0)) badchars))
>>           (setq pos (1+ pos)))
>>
>> instead of the current code:
>>
>>         (while (string-match "\\W" abbrev pos)
>>           (cl-pushnew (aref abbrev (match-beginning 0)) badchars)
>>           (setq pos (1+ pos)))
>>
>> A CL fan might say the latter is simpler, but for me, who isn't sure
>> just how to use cl-pushnew, the former seems conceptually simpler.
>
> The `cl-pushnew' docstring explains:
>
>     Add X to the list stored in PLACE unless X is already in the list.
>
>> The two ecxpressions have two different kinds of complexity.
>
> The only difference I see is that new version is harder to read, and
> more prone to bugs (as you have to write the same code twice).
>
> I'd suggest avoiding these types of changes.

+1

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > A CL fan might say the latter is simpler, but for me, who isn't sure
  > > just how to use cl-pushnew, the former seems conceptually simpler.

  > The `cl-pushnew' docstring explains:

  >     Add X to the list stored in PLACE unless X is already in the list.

Yes, it does -- but what I said is valid nonetheless.
We're talking about the complexity of different ways of writing code.

A reference to a function that you don't know has a kind of complexity.
The fact that you can look up that function doesn't eliminate that complexity.

  > The only difference I see is that new version is harder to read, and
  > more prone to bugs (as you have to write the same code twice).

Do you frequently use the cl- facilities such as `cl-pushnew'?  If so,
that might explain why you didn't _see_ the extra complexity that
comes from calling `cl-pushnew'.  That extra complexity exists for
people who don't habitually use cl facilities, and not for those who
do habitually use them.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: eliz@gnu.org, acm@muc.de, emacs-devel@gnu.org
> Date: Wed, 25 Oct 2023 22:27:54 -0400
> 
>   > The only difference I see is that new version is harder to read, and
>   > more prone to bugs (as you have to write the same code twice).
> 
> Do you frequently use the cl- facilities such as `cl-pushnew'?  If so,
> that might explain why you didn't _see_ the extra complexity that
> comes from calling `cl-pushnew'.  That extra complexity exists for
> people who don't habitually use cl facilities, and not for those who
> do habitually use them.

I think the need to be familiar with the cl-lib and cl-macs
functionalities is nowadays a requirement for any Emacs maintainer.
It can take time to familiarize oneself with them, but it isn't rocket
science.

Btw, some of the cl-lib facilities should not be used in new code if
they have equivalent implementations ins seq.el (which is nowadays
preloaded) and in map.el.  cl-generic.el is also preloaded, so should
be used in preference to the cl-lib alternatives.  And some frequently
needed cl-lib functions were moved to cl-preloaded.el, and are also
always available.

Re: Lisp files that load cl-lib in problematical ways

[Adam Porter]

Hi Eli,

I'm curious about this:

> Btw, some of the cl-lib facilities should not be used in new code if
> they have equivalent implementations ins seq.el (which is nowadays
> preloaded) and in map.el.

Since seq.el and map.el use generic methods, don't they have some 
additional overhead compared to type-specific functions, like cl-lib 
ones that are specific to lists?  Or is it small enough now that it 
doesn't matter anymore?

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> Date: Thu, 26 Oct 2023 04:31:14 -0500
> Cc: acm@muc.de, emacs-devel@gnu.org, rms@gnu.org, stefankangas@gmail.com
> From: Adam Porter <adam@alphapapa.net>
> 
> > Btw, some of the cl-lib facilities should not be used in new code if
> > they have equivalent implementations ins seq.el (which is nowadays
> > preloaded) and in map.el.
> 
> Since seq.el and map.el use generic methods, don't they have some 
> additional overhead compared to type-specific functions, like cl-lib 
> ones that are specific to lists?  Or is it small enough now that it 
> doesn't matter anymore?

I don't know.  Feel free to investigate and post the results.  If
there's a significant run-time penalty, we should work on making these
packages faster, since we consider them to be part of our
infrastructure.

Re: Lisp files that load cl-lib in problematical ways

[Dmitry Gutov]

On 26/10/2023 12:31, Adam Porter wrote:
> Hi Eli,
> 
> I'm curious about this:
> 
>> Btw, some of the cl-lib facilities should not be used in new code if
>> they have equivalent implementations ins seq.el (which is nowadays
>> preloaded) and in map.el.
> 
> Since seq.el and map.el use generic methods, don't they have some 
> additional overhead compared to type-specific functions, like cl-lib 
> ones that are specific to lists?  Or is it small enough now that it 
> doesn't matter anymore?

For most functions, the dispatch overhead might only show up on the 
radar for small sequences (otherwise it's dwarfed by useful work). And I 
haven't yet seen a case where it was specifically a problem.

cl-lib has its own performance problems, so the real answer is for the 
programmer to measure all alternatives and sometimes contribute 
performance fixes upstream ;)

Re: Lisp files that load cl-lib in problematical ways

[Alan Mackenzie]

Hello, Eli.

On Thu, Oct 26, 2023 at 09:55:45 +0300, Eli Zaretskii wrote:
> > From: Richard Stallman <rms@gnu.org>
> > Cc: eliz@gnu.org, acm@muc.de, emacs-devel@gnu.org
> > Date: Wed, 25 Oct 2023 22:27:54 -0400

> >   > The only difference I see is that new version is harder to read, and
> >   > more prone to bugs (as you have to write the same code twice).

> > Do you frequently use the cl- facilities such as `cl-pushnew'?  If so,
> > that might explain why you didn't _see_ the extra complexity that
> > comes from calling `cl-pushnew'.  That extra complexity exists for
> > people who don't habitually use cl facilities, and not for those who
> > do habitually use them.

> I think the need to be familiar with the cl-lib and cl-macs
> functionalities is nowadays a requirement for any Emacs maintainer.

For me, that is not possible.  I simply don't have the brain power to
memorize such a morass of arbitrary, poorly named, poorly documented
stuff.  I am forever cursed by having to look up doc strings each time I
encounter these objects.  Maybe not things like cl-incf, but
monstrosities such as cl-labels just leave me groaning.  I frequently
spend, perhaps, half of my debugging time trying to understand what some
cl-* does rather than concentrating on the problem to be debugged.

> It can take time to familiarize oneself with them, but it isn't rocket
> science.

How many contributors are actually familiar with the bulk of cl-lib?
I suspect, very few indeed.  Maybe Stefan M.  The people who have written
strongly in favour of cl-lib over the last week or two have not been
those who spend lots of time maintaining others' code.

I further suspect that many contributions using cl-lib are from people
who know a small part of cl-lib, find the functionality intriguing, and
thus use it.  The trouble is, each such contributor knows a different
part of it, so in aggregate the part that an Emacs maintainer needs to
know (or continually to look up) is large.

Note that Common Lisp never took off as a popular language.  There must
be reasons for this.

> Btw, some of the cl-lib facilities should not be used in new code if
> they have equivalent implementations ins seq.el (which is nowadays
> preloaded) and in map.el.  cl-generic.el is also preloaded, so should
> be used in preference to the cl-lib alternatives.  And some frequently
> needed cl-lib functions were moved to cl-preloaded.el, and are also
> always available.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Alan Mackenzie wrote:

> How many contributors are actually familiar with the bulk of
> cl-lib? I suspect, very few indeed. Maybe Stefan M.
> The people who have written strongly in favour of cl-lib
> over the last week or two have not been those who spend lots
> of time maintaining others' code.
>
> I further suspect that many contributions using cl-lib are
> from people who know a small part of cl-lib, find the
> functionality intriguing, and thus use it. The trouble is,
> each such contributor knows a different part of it, so in
> aggregate the part that an Emacs maintainer needs to know
> (or continually to look up) is large.

It is the nature of innovation. People like to be at the
front. They like to try out exotic new gear. And the people at
the rear understand this - even if it means they have to build
new facilities to accomodate for the new stuff once in
a while.

If we tell people at the front to stick with the old ways, the
old gear, they will simply leave our section for something
more interesting to do. And we won't get anywhere.

> Note that Common Lisp never took off as a popular language.
> There must be reasons for this.

Maintaining it is too difficult?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Bob Rogers]

   From: Emanuel Berg <incal@dataswamp.org>
   Date: Thu, 26 Oct 2023 13:32:43 +0200

   Alan Mackenzie wrote:

   . . .

   > Note that Common Lisp never took off as a popular language.  There
   > must be reasons for this.

   Maintaining it is too difficult?

Even before CL, Lisp systems tended to be memory-hungry.  Common Lisp
was designed as a large language, for a future where memory would be
cheap.  That future eventually came, but before it came the PC boom,
when tiny systems with relatively tiny amounts of RAM ruled the roost.
And Lisp was neglected even on more powerful workstations; some of you
will rememeber the Sun Sparcstation Pmeg problem that caused page table
thrashing if you tried to allocate more than 16MB.  (This is an over-
simplification for why Lisp popularity declined, of course.)

   This complexity and the resulting complications, I believe, are what
Richard was trying to avoid when he designed Emacs Lisp to be a simpler
dialect of Lisp, and why he sees cl-lib.el as a trojan horse that is
changing Emacs Lisp.  (And why I hope this bit of history is not quite
off-topic.)

					-- Bob Rogers
					   http://www.rgrjr.com/

Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Bob Rogers wrote:

> Even before CL, Lisp systems tended to be memory-hungry.
> Common Lisp was designed as a large language, for a future
> where memory would be cheap. That future eventually came,
> but before it came the PC boom, when tiny systems with
> relatively tiny amounts of RAM ruled the roost. And Lisp was
> neglected even on more powerful workstations; some of you
> will rememeber the Sun Sparcstation Pmeg problem that caused
> page table thrashing if you tried to allocate more than
> 16MB. (This is an over- simplification for why Lisp
> popularity declined, of course.)

Before the PC: too memory-hungry
On the PCs: not enough RAM
On the workstations: neglected

> This complexity and the resulting complications, I believe,
> are what Richard was trying to avoid when he designed Emacs
> Lisp to be a simpler dialect of Lisp, and why he sees
> cl-lib.el as a trojan horse that is changing Emacs Lisp.

If so, it has certainly succeeded! There used to be 'labels',
'incf', and 'decf' in Elisp. Now, not only do they face
competition from `cl-labels', `cl-incf', and `cl-decf' - they
don't exist anymore.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >    This complexity and the resulting complications, I believe, are what
  > Richard was trying to avoid when he designed Emacs Lisp to be a simpler
  > dialect of Lisp, and why he sees cl-lib.el as a trojan horse that is
  > changing Emacs Lisp.

I think you've put your finger on it.

I see nothing wrong with having an optional extension which implements
(more or less) Common Lisp in Emacs Lisp.  That's what I thought we
had done.

However, I object to incorporating (more or less) Common Lisp an essential part
of Emacs Lisp.  And it looks like we've been drifting into that.

There is no sharp line between the one and the other, and no simple
test to determine which of those two our current situation actually
is.  Rather, there is a spectrum that runs from "CL support is
available but you can ignore it" to "the CL constructs are an
essential and unavoidable part of Emacs Lisp."

The specific practical questions I've asked are efforts to evaluate
where we are now along that spectrum.  Of course, the answer to that
isn't precise either.  But I was very surprised to learn how far
Emacs has gone towards the latter end.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

What's missing in ELisp that makes people want to use cl-lib?

[Stefan Kangas]

Richard Stallman <rms@gnu.org> writes:

> The specific practical questions I've asked are efforts to evaluate
> where we are now along that spectrum.  Of course, the answer to that
> isn't precise either.  But I was very surprised to learn how far
> Emacs has gone towards the latter end.

It seems to me that the extent to which Emacs has shifted "towards the
latter end" is still open to interpretation.

More fundamentally, I believe the question we should be asking isn't the
quantity of cl-lib used in Emacs, but rather why these abstractions from
Common Lisp appeal to ELisp developers.  Most ELisp developers, I
presume, do not have a background in Common Lisp, indicating that their
choice to use cl-lib.el isn't driven by a mere affinity for CL.

Is it possible that certain abstractions or functions are filling a gap
in Emacs Lisp itself?  In the case of `cl-pushnew', I happen to think
that the answer is yes.

Discussing `cl-pushnew' and other cases like it might help us identify
areas for improvement in Emacs Lisp proper.  Perhaps we can borrow the
CL name/behavior, or perhaps we can come up with something even better.
That would be more constructive, and more helpful, than spending our
time changing code to not use cl-lib.el just for the sake of it.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Stefan Kangas wrote:

> More fundamentally, I believe the question we should be
> asking isn't the quantity of cl-lib used in Emacs, but
> rather why these abstractions from Common Lisp appeal to
> ELisp developers. Most ELisp developers, I presume, do not
> have a background in Common Lisp, indicating that their
> choice to use cl-lib.el isn't driven by a mere affinity
> for CL.

Good idea, I tried this to find out. Maybe the command can be
improved ...

$ grep -E -h -o 'cl-[-a-z]*' **/*.el | sort | uniq -c | sort -r -b -n | head -n 10

   1416 cl-defmethod
    979 cl-loop
    720 cl-incf
    614 cl-lib
    486 cl-assert
    309 cl-letf
    303 cl-seq
    281 cl-defstruct
    257 cl-defgeneric
    227 cl-case

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Sat, 28 Oct 2023 18:18:53 +0200
> 
> Stefan Kangas wrote:
> 
> > More fundamentally, I believe the question we should be
> > asking isn't the quantity of cl-lib used in Emacs, but
> > rather why these abstractions from Common Lisp appeal to
> > ELisp developers. Most ELisp developers, I presume, do not
> > have a background in Common Lisp, indicating that their
> > choice to use cl-lib.el isn't driven by a mere affinity
> > for CL.
> 
> Good idea, I tried this to find out. Maybe the command can be
> improved ...
> 
> $ grep -E -h -o 'cl-[-a-z]*' **/*.el | sort | uniq -c | sort -r -b -n | head -n 10
> 
>    1416 cl-defmethod
>     979 cl-loop
>     720 cl-incf
>     614 cl-lib
>     486 cl-assert
>     309 cl-letf
>     303 cl-seq
>     281 cl-defstruct
>     257 cl-defgeneric
>     227 cl-case

Some of these are macros in cl-macs, so not relevant to the discussion
about cl-lib.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Eli Zaretskii wrote:

>> $ grep -E -h -o 'cl-[-a-z]*' **/*.el | sort | uniq -c | sort -r -b -n |
>>  head -n 10
>> 
>>    1416 cl-defmethod
>>     979 cl-loop
>>     720 cl-incf
>>     614 cl-lib
>>     486 cl-assert
>>     309 cl-letf
>>     303 cl-seq
>>     281 cl-defstruct
>>     257 cl-defgeneric
>>     227 cl-case
>
> Some of these are macros in cl-macs, so not relevant to the
> discussion about cl-lib.

It is everything, remove the last 'head -n 10' to get it.

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Sat, 28 Oct 2023 21:08:33 +0200
> 
> Eli Zaretskii wrote:
> 
> >> $ grep -E -h -o 'cl-[-a-z]*' **/*.el | sort | uniq -c | sort -r -b -n |
> >>  head -n 10
> >> 
> >>    1416 cl-defmethod
> >>     979 cl-loop
> >>     720 cl-incf
> >>     614 cl-lib
> >>     486 cl-assert
> >>     309 cl-letf
> >>     303 cl-seq
> >>     281 cl-defstruct
> >>     257 cl-defgeneric
> >>     227 cl-case
> >
> > Some of these are macros in cl-macs, so not relevant to the
> > discussion about cl-lib.
> 
> It is everything, remove the last 'head -n 10' to get it.

What I wrote is valid even without 'head -n 10'.  My point is that
this kind of list is not useful for this discussion.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Eli Zaretskii wrote:

>>>> $ grep -E -h -o 'cl-[-a-z]*' **/*.el | sort | uniq -c | sort -r -b -n |
>>>>  head -n 10
>>>> 
>>>>    1416 cl-defmethod
>>>>     979 cl-loop
>>>>     720 cl-incf
>>>>     614 cl-lib
>>>>     486 cl-assert
>>>>     309 cl-letf
>>>>     303 cl-seq
>>>>     281 cl-defstruct
>>>>     257 cl-defgeneric
>>>>     227 cl-case
>>>
>>> Some of these are macros in cl-macs, so not relevant to the
>>> discussion about cl-lib.
>> 
>> It is everything, remove the last 'head -n 10' to get it.
>
> What I wrote is valid even without 'head -n 10'. My point is
> that this kind of list is not useful for this discussion.

Of course what you say is valid, since some part of everything
is always something.

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Jim Porter]

On 10/28/2023 8:39 AM, Stefan Kangas wrote:
> Is it possible that certain abstractions or functions are filling a gap
> in Emacs Lisp itself?  In the case of `cl-pushnew', I happen to think
> that the answer is yes.

I agree with this. I can speak about Eshell in particular, since it's 
historically used a bit of cl-lib, and I've added more usage to it.

By far the number one cl-lib thing Eshell uses is 'cl-assert'. This 
would be reasonably easy to do without cl-lib, but it's a pretty 
convenient syntax for, well... asserting some precondition for a 
function. That's been useful for helping to identify subtle bugs in 
Eshell's logic.

After that, the next most important cl-lib thing (in my opinion) is 
generic functions. I introduced this in Eshell to break apart some 
monolithic I/O code that had large 'cond' forms for each I/O target. 
Now, with generic functions, they're separate, and it's much easier for 
third parties to add new I/O target types. In the future, I'll likely 
use generic functions for a few more bits in Eshell.

Related to the above is 'cl-defstruct'. That's useful for working with 
generic functions, but I think there's also some benefit to using them 
instead of lists (or alists/plists) when you want something with a 
very-specific set of members (similar to a C struct or C++ class). 
However, I could do without CL-style structs for this case, excluding 
the generic function side.

Finally, Eshell uses a few miscellaneous cl-lib functions, some of which 
are probably necessary, and some of which are just conveniences. The 
(probable) necessities are: cl-progv and cl-mapc. The conveniences are: 
cl-loop, cl-flet, cl-list*, cl-remove-if, cl-incf, and cl-decf.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> Date: Sat, 28 Oct 2023 12:10:23 -0700
> Cc: emacs-devel@gnu.org
> From: Jim Porter <jporterbugs@gmail.com>
> 
> On 10/28/2023 8:39 AM, Stefan Kangas wrote:
> > Is it possible that certain abstractions or functions are filling a gap
> > in Emacs Lisp itself?  In the case of `cl-pushnew', I happen to think
> > that the answer is yes.
> 
> I agree with this. I can speak about Eshell in particular, since it's 
> historically used a bit of cl-lib, and I've added more usage to it.
> 
> By far the number one cl-lib thing Eshell uses is 'cl-assert'.

cl-assert is a macro in cl-macs, so it isn't relevant to this
discussion.

> After that, the next most important cl-lib thing (in my opinion) is 
> generic functions.

If you mean cl-defgeneric and cl-defmethod, they again are not in
cl-lib, so not relevant to this discussion.  Moreover, they are
defined in cl-generic.el, which is preloaded.

> Related to the above is 'cl-defstruct'.

cl-defstruct is again a macro in cl-macs, not in cl-lib.

> Finally, Eshell uses a few miscellaneous cl-lib functions, some of which 
> are probably necessary, and some of which are just conveniences. The 
> (probable) necessities are: cl-progv and cl-mapc. The conveniences are: 
> cl-loop, cl-flet, cl-list*, cl-remove-if, cl-incf, and cl-decf.

These are the only ones relevant to this discussion.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Jim Porter]

On 10/28/2023 12:14 PM, Eli Zaretskii wrote:
>> Date: Sat, 28 Oct 2023 12:10:23 -0700
>> Cc: emacs-devel@gnu.org
>> From: Jim Porter <jporterbugs@gmail.com>
>>
>> I agree with this. I can speak about Eshell in particular, since it's
>> historically used a bit of cl-lib, and I've added more usage to it.
>>
>> By far the number one cl-lib thing Eshell uses is 'cl-assert'.
> 
> cl-assert is a macro in cl-macs, so it isn't relevant to this
> discussion.

Thanks. I wasn't aware of this distinction. (I see that it was discussed 
elsewhere in the parent thread, but that thread is very big and I hadn't 
been following it closely.)

Is there any benefit in aliasing macros like this to non 'cl-FOO' names 
anyway? As a mere consumer of the various Common Lisp code in Emacs, 
it's not always clear to me what's ok to use versus what should ideally 
be avoided.

>> Finally, Eshell uses a few miscellaneous cl-lib functions, some of which
>> are probably necessary, and some of which are just conveniences. The
>> (probable) necessities are: cl-progv and cl-mapc. The conveniences are:
>> cl-loop, cl-flet, cl-list*, cl-remove-if, cl-incf, and cl-decf.
> 
> These are the only ones relevant to this discussion.

I see that 'cl-progv' is in cl-macs.el, so I guess that's not relevant. 
'cl-mapc' is in cl-extra.el, which I suppose is relevant here? I find it 
very helpful, but I'd be happy with any function/macro that would make 
it easy to iterate over multiple sequences at once.

Of the conveniences, I think the ones I'd miss the most are 'cl-incf' 
and 'cl-decf', which I use in Eshell for recounting some handles. (I 
could possibly rewrite that code to let the garbage collector do all the 
work, but that code interacts with some of the trickier bits in Eshell, 
so maybe it's best not to change it too much.)

'cl-remove-if' should be easy to replace with 'seq-filter' or somesuch. 
Likewise 'cl-list*' with 'nconc'. I know there's some controversy around 
'cl-loop' (I'm not a big fan myself), and I've thought about adding some 
new hooks to Eshell that would probably obviate the need for 'cl-loop' 
(which is used in em-extpipe.el).

So in conclusion, I guess the relevant functions that I care about are 
'cl-mapc', and to a lesser degree 'cl-incf'/'cl-decf'. Everything else 
could go away if we wanted to avoid them.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> Date: Sat, 28 Oct 2023 14:03:29 -0700
> Cc: stefankangas@gmail.com, rms@gnu.org, rogers@rgrjr.com, emacs-devel@gnu.org
> From: Jim Porter <jporterbugs@gmail.com>
> 
> Is there any benefit in aliasing macros like this to non 'cl-FOO' names 
> anyway?

I don't have an opinion on this, so I'll let others chime in.  In
general, introducing aliases has a minor downside: people will need to
learn two names instead of just one.

> I see that 'cl-progv' is in cl-macs.el, so I guess that's not relevant. 
> 'cl-mapc' is in cl-extra.el, which I suppose is relevant here?

Yes, cl-extra is just another part of cl-lib.

> I find it very helpful, but I'd be happy with any function/macro
> that would make it easy to iterate over multiple sequences at once.

Try looking in seq.el or map.el.

> 'cl-remove-if' should be easy to replace with 'seq-filter' or somesuch. 
> Likewise 'cl-list*' with 'nconc'. I know there's some controversy around 
> 'cl-loop' (I'm not a big fan myself), and I've thought about adding some 
> new hooks to Eshell that would probably obviate the need for 'cl-loop' 
> (which is used in em-extpipe.el).
> 
> So in conclusion, I guess the relevant functions that I care about are 
> 'cl-mapc', and to a lesser degree 'cl-incf'/'cl-decf'. Everything else 
> could go away if we wanted to avoid them.

Since seq.el is preloaded, I think we should prefer it to cl-lib.
Likewise map.el, although that isn't preloaded.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Of the conveniences, I think the ones I'd miss the most are 'cl-incf' 
  > and 'cl-decf', which I use in Eshell for recounting some handles.

If this pattern extends to many files -- if getf, incf and decf are
particularly useful among CL constructs -- making them standard parts of Emacs Lisp would be better than inducing lots of files to load cl-lib.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Jim Porter]

On 10/29/2023 7:09 PM, Richard Stallman wrote:
>    > Of the conveniences, I think the ones I'd miss the most are 'cl-incf'
>    > and 'cl-decf', which I use in Eshell for recounting some handles.
> 
> If this pattern extends to many files -- if getf, incf and decf are
> particularly useful among CL constructs -- making them standard parts of Emacs Lisp would be better than inducing lots of files to load cl-lib.

I don't think I'd fight tooth and nail to get incf/decf, but I'd 
certainly be happy to see them in standard Elisp.

I'm not sure how much others want to see setf-like constructs in 
Elisp[1], but I like them quite a bit. They fit in neatly alongside the 
C/C++ parts of my brain. Specifically, I find this style:

   (cl-incf (cdr output-target))
   - or -
   output_target.tail++;

a lot nicer than this:

   (setcdr output-target (1+ (cdr output-target)))
   - or -
   output_target.tail = output_target.tail + 1;

[1] I recall that some generalized values for setf were deprecated 
recently, but I think that was mainly for things where the setter had 
some non-obvious behavior.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Is it possible that certain abstractions or functions are filling a gap
  > in Emacs Lisp itself?

Yes, it could well be true for some of them.

                           In the case of `cl-pushnew', I happen to think
  > that the answer is yes.

Do people really want `cl-pushnew', with its keyword arguments?
Or would a simple `pushnew' be even more pleasing?

To define `pushnew' in Emacs Lisp, and document it in the Emacs Lisp
Reference Manual, would be much cleaner than spreading the use of
cl-lib to more files.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Adam Porter]

> Do people really want `cl-pushnew', with its keyword arguments?
> Or would a simple `pushnew' be even more pleasing?

It might be more aesthetically or ideologically pleasing, but it would 
be much less useful.  :test is often necessary for using EQUAL (as EQ is 
often the wrong function to use); without it, a hypothetical non-CL 
PUSHNEW function would often be unusable.  And :key makes it easy to use 
with, e.g. structs' accessor functions (which are very helpful for 
developers as they allow compile-time checking, completion, etc).

When CL-PUSHNEW is called without keyword arguments, it's as simple as a 
hypothetical non-CL PUSHNEW that offered no keyword arguments.  The 
keyword arguments provide optional flexibility; without them, one would 
have to write a much larger, more complex form to do the same things.

Finally, if there were an additional, non-CL PUSHNEW function added, I 
would think that would be much more confusing to users, especially new 
ones, for there to be a difference in behavior between PUSHNEW and 
CL-PUSHNEW.

So I would ask that CL-PUSHNEW become PUSHNEW as-is.

> To define `pushnew' in Emacs Lisp, and document it in the Emacs Lisp
> Reference Manual, would be much cleaner than spreading the use of
> cl-lib to more files.

I would be happy to see more cl-lib functions "moved" back into "core" 
Elisp.  Using Elisp without cl-lib feels restrictive to me.

By the way, with regard to CL-PUSHNEW, it seems surprising that no one's 
mentioned the existing counterpart in "core" Elisp, ADD-TO-LIST. 
Notably, its docstring says:

   This is handy to add some elements to configuration variables,
   but please do not abuse it in Elisp code, where you are usually
   better off using ‘push’ or ‘cl-pushnew’.

So even "core", non-cl-lib Elisp recommends the use of some cl-lib 
functions, i.e. CL-PUSHNEW.  That certainly suggests a "gap" in core 
Elisp functionality.

--Adam

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> Date: Mon, 30 Oct 2023 21:31:41 -0500
> Cc: emacs-devel@gnu.org, stefankangas@gmail.com
> From: Adam Porter <adam@alphapapa.net>
> 
> By the way, with regard to CL-PUSHNEW, it seems surprising that no one's 
> mentioned the existing counterpart in "core" Elisp, ADD-TO-LIST. 
> Notably, its docstring says:
> 
>    This is handy to add some elements to configuration variables,
>    but please do not abuse it in Elisp code, where you are usually
>    better off using ‘push’ or ‘cl-pushnew’.
> 
> So even "core", non-cl-lib Elisp recommends the use of some cl-lib 
> functions, i.e. CL-PUSHNEW.  That certainly suggests a "gap" in core 
> Elisp functionality.

This kind of remarks in Emacs doc strings are never about "gaps" of
any kind, they are about cleaner coding practices.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Tue, Oct 31, 2023 at 2:32 AM Adam Porter <adam@alphapapa.net> wrote:
>
> > Do people really want `cl-pushnew', with its keyword arguments?
> > Or would a simple `pushnew' be even more pleasing?
>
> It might be more aesthetically or ideologically pleasing, but it would
> be much less useful.  :test is often necessary for using EQUAL (as EQ is
> often the wrong function to use); without it, a hypothetical non-CL
> PUSHNEW function would often be unusable.  And :key makes it easy to use
> with, e.g. structs' accessor functions (which are very helpful for
> developers as they allow compile-time checking, completion, etc).
>
> When CL-PUSHNEW is called without keyword arguments, it's as simple as a
> hypothetical non-CL PUSHNEW that offered no keyword arguments.  The
> keyword arguments provide optional flexibility; without them, one would
> have to write a much larger, more complex form to do the same things.

Of course.

The other day I noticed we have a C function called 'car-less-than-car'
whose only point is to optimize a super-specific use of 'sort'.  Maybe
in 1993 we didn't have compiler macros but now we do, so this is the
wrong way to go about it.

We should just write

  (cl-sort foo #'< :key #'car)

which is much more modular, readable, reusable, and doesn't require
the programmer to know of this specific vocabulary.

Sure, "but what about the optimization?", I hear you ask.  Well if
you must have it then just use a compiler macro to have your cake and
eat it too.

In this case, something like

  (defun cl-sort-compiler-macro (form &rest args)
    (if (equal (cdr args) '(#'< :key #'car))
        `(cl-sort ,(car args) #'car-less-than-car)
      form))

and add it to the cl-sort declaration with

  (declare (compiler-macro 'cl-sort-compiler-macro))

> Finally, if there were an additional, non-CL PUSHNEW function added, I
> would think that would be much more confusing to users, especially new
> ones, for there to be a difference in behavior between PUSHNEW and
> CL-PUSHNEW.

Exactly.  Please don't do this, we have enough such cases already.
Either migrate the fully functional version or don't.

Also, it's not too late to add keyword arguments to our 'sort'

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

It might be ok to add some keyword arguments to `sort', which are more
unusual and complex to use, but not to simple constructs like
`pushnew'.  This is Emacs Lisp, not Common Lisp.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Jim Porter]

On 11/1/2023 7:28 PM, Richard Stallman wrote:
> It might be ok to add some keyword arguments to `sort', which are more
> unusual and complex to use, but not to simple constructs like
> `pushnew'.  This is Emacs Lisp, not Common Lisp.

I'm not advocating for (or against) this, but if we wanted something 
like 'pushnew' and didn't want to use keyword arguments, then what about 
these two functions?

   (defun pushnew (value place &optional testfun) ...)
   (defun pushnew-key (value place keyfun) ...)

There's no reason I can see that we'd *need* to have both forms in a 
single function. (Unless we specifically wanted to match the Common Lisp 
API, and that's what cl-lib is for.)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Philip Kaludercic]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> It might be ok to add some keyword arguments to `sort', which are more
> unusual and complex to use, but not to simple constructs like
> `pushnew'.  This is Emacs Lisp, not Common Lisp.

What does that last sentence mean?  I hope this is not a too
philosophical question, but what constitutes "Emacs Lisp"?  It would
seem peculiar if it were to be defined by the arbitrary decisions of the
past, constrained by the contingent circumstances of the time.
Certainly part of it is that it is part of a tradition of Lisp
implementations, that gives it a distinctive flavour, but to me the main
aspects and attractions of Emacs Lisp (in contrast to CL and Scheme)
have been:

1. Not standardised; it is possible to extend the language without
   having to worry about how many implementations will follow along

2. Accepting Unix; not trying to abstract over various different
   operating systems and file systems, making it easier to work with GNU
   and other POSIX-based environments.

The first point would be relevant here.  Just like if-let or
thread-first, Emacs Lisp can learn from interesting ideas that other
languages provide, adapt and add them, making them available to
everyone.  If a form expresses something that people frequently want to
do, but either have to elaborate using (unless (memq ...) (push ...)),
then we are making Emacs more useful and expressive by providing the
functionality OOTB.  And isn't that the real point of Emacs Lisp?

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: João Távora <joaotavora@gmail.com>,
>  adam@alphapapa.net, emacs-devel@gnu.org,  stefankangas@gmail.com
> Date: Thu, 02 Nov 2023 08:55:54 +0000
> 
> Richard Stallman <rms@gnu.org> writes:
> 
> > It might be ok to add some keyword arguments to `sort', which are more
> > unusual and complex to use, but not to simple constructs like
> > `pushnew'.  This is Emacs Lisp, not Common Lisp.
> 
> What does that last sentence mean?

AFAIU, it means that Emacs Lisp traditionally doesn't use keyword
arguments, except as relatively rare exceptions.

> what constitutes "Emacs Lisp"?  It would
> seem peculiar if it were to be defined by the arbitrary decisions of the
> past, constrained by the contingent circumstances of the time.

Those "arbitrary decisions" are what got us to where we are now, 40
years later.  So some respect for those "arbitrary decisions" is due,
I think.

> to me the main aspects and attractions of Emacs Lisp (in contrast to
> CL and Scheme) have been:
> 
> 1. Not standardised; it is possible to extend the language without
>    having to worry about how many implementations will follow along

IMNSHO, extending Emacs Lisp as the language is not the main goal of
Emacs development.  Emacs Lisp is not a programming language on its
own, it is a language for implementing and extending features and
extensions in Emacs.  Thus, the main goal of Emacs development is to
develop applications and application-level features, and provide more
opportunities for extending the core where that is considered useful.
What we have in Emacs Lisp is IMO ample for that purpose.  Moreover,
most participants in Emacs development are not experts in programming
languages, their expertise is elsewhere (which is definitely a Good
Thing).

Objectively, adding new syntax and semantics to Emacs Lisp does make
the source code harder to read and maintain, because it makes the
language larger and requires familiarization with those new language
features, which more often than not look and feel completely different
from the "traditional" Emacs Lisp.  So even if we conclude that these
additions are worthwhile, we should not pretend they come without a
price, and IMO we should think carefully whether their use is
justified before we use them in each and every case.

> Emacs Lisp can learn from interesting ideas that other
> languages provide, adapt and add them, making them available to
> everyone.

It certainly can.  The question is: should it?  Since we are not
driven by any standard, it is completely up to us to make those
decisions, and we should IMO make them judiciously and carefully,
taking the downsides into consideration.  In particular, I hope people
agree that making a language too large and complex is not a good
thing in the long run.

> If a form expresses something that people frequently want to
> do, but either have to elaborate using (unless (memq ...) (push ...)),
> then we are making Emacs more useful and expressive by providing the
> functionality OOTB.  And isn't that the real point of Emacs Lisp?

AFAIU, what Richard says is that if some form is frequently used, we
should indeed consider adding a primitive or an API for it, but that
primitive or API doesn't have to look like CL.  Cf the pushnew
argument.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: João Távora <joaotavora@gmail.com>,
>>  adam@alphapapa.net, emacs-devel@gnu.org,  stefankangas@gmail.com
>> Date: Thu, 02 Nov 2023 08:55:54 +0000
>> 
>> Richard Stallman <rms@gnu.org> writes:
>> 
>> > It might be ok to add some keyword arguments to `sort', which are more
>> > unusual and complex to use, but not to simple constructs like
>> > `pushnew'.  This is Emacs Lisp, not Common Lisp.
>> 
>> What does that last sentence mean?
>
> AFAIU, it means that Emacs Lisp traditionally doesn't use keyword
> arguments, except as relatively rare exceptions.

It is used rather prominently in `define-minor-mode' or
`define-derived-mode', or do you specifically mean keyword arguments to
functions?

>> what constitutes "Emacs Lisp"?  It would
>> seem peculiar if it were to be defined by the arbitrary decisions of the
>> past, constrained by the contingent circumstances of the time.
>
> Those "arbitrary decisions" are what got us to where we are now, 40
> years later.  So some respect for those "arbitrary decisions" is due,
> I think.

No disrespect meant, but I am not sure we are thinking of the same
things.  An "arbitrary decision" usually doesn't matter much, like
calling a function rplacd or setcdr.  If a decision got us to where we
are now, I would say it wasn't that arbitrary, but a good one?

>> to me the main aspects and attractions of Emacs Lisp (in contrast to
>> CL and Scheme) have been:
>> 
>> 1. Not standardised; it is possible to extend the language without
>>    having to worry about how many implementations will follow along
>
> IMNSHO, extending Emacs Lisp as the language is not the main goal of
> Emacs development.  Emacs Lisp is not a programming language on its
> own, it is a language for implementing and extending features and
> extensions in Emacs.  Thus, the main goal of Emacs development is to
> develop applications and application-level features, and provide more
> opportunities for extending the core where that is considered useful.
> What we have in Emacs Lisp is IMO ample for that purpose.  Moreover,
> most participants in Emacs development are not experts in programming
> languages, their expertise is elsewhere (which is definitely a Good
> Thing).

Of course not extending it for its own sake, but I would assume that
making it easier for users to write practical and useful code should be
something that is valued.

> Objectively, adding new syntax and semantics to Emacs Lisp does make
> the source code harder to read and maintain, because it makes the
> language larger and requires familiarization with those new language
> features, which more often than not look and feel completely different
> from the "traditional" Emacs Lisp.  So even if we conclude that these
> additions are worthwhile, we should not pretend they come without a
> price, and IMO we should think carefully whether their use is
> justified before we use them in each and every case.

Could you explain what you mean by "traditional" Emacs Lisp?  But yes, I
am not advocating for senselessly adding whatever seems remotely
interesting, of course.

>> Emacs Lisp can learn from interesting ideas that other
>> languages provide, adapt and add them, making them available to
>> everyone.
>
> It certainly can.  The question is: should it?  Since we are not
> driven by any standard, it is completely up to us to make those
> decisions, and we should IMO make them judiciously and carefully,
> taking the downsides into consideration.  In particular, I hope people
> agree that making a language too large and complex is not a good
> thing in the long run.
>
>> If a form expresses something that people frequently want to
>> do, but either have to elaborate using (unless (memq ...) (push ...)),
>> then we are making Emacs more useful and expressive by providing the
>> functionality OOTB.  And isn't that the real point of Emacs Lisp?
>
> AFAIU, what Richard says is that if some form is frequently used, we
> should indeed consider adding a primitive or an API for it, but that
> primitive or API doesn't have to look like CL.  Cf the pushnew
> argument.

Right, I agree with everything here.  The functionality, not the style
is of interest.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Bob Rogers]

   From: Philip Kaludercic <philipk@posteo.net>
   Date: Thu, 02 Nov 2023 21:26:00 +0000

   Eli Zaretskii <eliz@gnu.org> writes:

   >> From: Philip Kaludercic <philipk@posteo.net>
   >> Cc: João Távora <joaotavora@gmail.com>,
   >>  adam@alphapapa.net, emacs-devel@gnu.org,  stefankangas@gmail.com
   >> Date: Thu, 02 Nov 2023 08:55:54 +0000
   >> 
   >> Richard Stallman <rms@gnu.org> writes:
   >> 
   >> > It might be ok to add some keyword arguments to `sort', which
   >> > are more unusual and complex to use, but not to simple
   >> > constructs like `pushnew'.  This is Emacs Lisp, not Common Lisp.
   >> 
   >> What does that last sentence mean?
   >
   > . . .

   . . .

   >> what constitutes "Emacs Lisp"?  It would seem peculiar if it were
   >> to be defined by the arbitrary decisions of the past, constrained
   >> by the contingent circumstances of the time.
   >
   > Those "arbitrary decisions" are what got us to where we are now, 40
   > years later.  So some respect for those "arbitrary decisions" is due,
   > I think.

   No disrespect meant, but I am not sure we are thinking of the same
   things.  An "arbitrary decision" usually doesn't matter much, like
   calling a function rplacd or setcdr.  If a decision got us to where we
   are now, I would say it wasn't that arbitrary, but a good one?

I think we are not really talking about arbitrary decisions here, but
about language style decisions, which may seem arbitrary (especially if
you disagree with the style of the resulting language!) but do matter to
the consistency and coherence of the resulting language.  And, although
my experience only goes back to Emacs 18, I think the original Emacs
Lisp had a definite style that set it apart from other Lisp dialects.
But that was a long time ago, and Emacs Lisp has grown enormously (as
someone else pointed out early in the original thread) to support the
enormous growth in Emacs, so that original style is now much harder to
see.  (Indeed, I notice it mostly when updating my old code.)

   So, Richard, I see that you are fighting to preserve something real
and important -- your vision of Emacs Lisp as a coherent language -- and
I believe that may be your prerogative as its creator.  But I also
believe you may be a decade or two too late.

					-- Bob Rogers
					   http://www.rgrjr.com/

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Bob Rogers wrote:

> I think we are not really talking about arbitrary decisions
> here, but about language style decisions, which may seem
> arbitrary (especially if you disagree with the style of the
> resulting language!) but do matter to the consistency and
> coherence of the resulting language. And, although my
> experience only goes back to Emacs 18, I think the original
> Emacs Lisp had a definite style that set it apart from other
> Lisp dialects. But that was a long time ago, and Emacs Lisp
> has grown enormously (as someone else pointed out early in
> the original thread) to support the enormous growth in
> Emacs, so that original style is now much harder to see.
> (Indeed, I notice it mostly when updating my old code.)

So let's hear it then, what is it that sets Emacs 18 Elisp
apart from Emacs 30 Elisp

> So, Richard, I see that you are fighting to preserve
> something real and important -- your vision of Emacs Lisp as
> a coherent language --

and how is the Emacs 30 Elisp less coherent than the
Emacs 18 Elisp?

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Bob Rogers]

   From: Emanuel Berg <incal@dataswamp.org>
   Date: Fri, 03 Nov 2023 03:21:30 +0100

   Bob Rogers wrote:

   > I think we are not really talking about arbitrary decisions here,
   > but about language style decisions . . .

   So let's hear it then, what is it that sets Emacs 18 Elisp
   apart from Emacs 30 Elisp . . .

Briefly, it's now a much larger language . . .

   and how is the Emacs 30 Elisp less coherent than the Emacs 18 Elisp?

   -- 
   underground experts united
   https://dataswamp.org/~incal

. . . and the additions have come from diverse sources.  It started as a
Maclisp subset with extensions to support editing.  The issue is that
the programming features have grown, and not only from Common Lisp,
moving it away from its Maclisp roots.  (This, I believe, is the gist of
Richard's complaint.)  And the language editing features have (not
surprisingly) grown with Emacs' growth as an editor, as they have had to
do, which has unavoidably added to the learning curve you have to climb
before you can contribute to Emacs.  It has also made the language more
about editing and less about programming, which IMHO makes the style
question less acute.

					-- Bob

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Bob Rogers wrote:

>> So let's hear it then, what is it that sets Emacs 18 Elisp
>> apart from Emacs 30 Elisp . . .
>
> Briefly, it's now a much larger language . . .
>
>> and how is the Emacs 30 Elisp less coherent than the Emacs
>> 18 Elisp?
>
> . . . and the additions have come from diverse sources.
> It started as a Maclisp subset with extensions to support
> editing. The issue is that the programming features have
> grown, and not only from Common Lisp, moving it away from
> its Maclisp roots. (This, I believe, is the gist of
> Richard's complaint.) And the language editing features have
> (not surprisingly) grown with Emacs' growth as an editor, as
> they have had to do, which has unavoidably added to the
> learning curve you have to climb before you can contribute
> to Emacs. It has also made the language more about editing
> and less about programming, which IMHO makes the style
> question less acute.

I understand that it has expanded in size, this is the nature
of technology systems in general and perhaps even more so with
Lisp programming, since it is so easy and fun to do.

But I am still uncertain how this has changed the style and
reduced the coherence of Elisp.

Case in point, I don't see any such difference when I look at
cl-lib.el?

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Bob Rogers]

   From: Emanuel Berg <incal@dataswamp.org>
   Date: Fri, 03 Nov 2023 21:32:15 +0100

   Bob Rogers wrote:

   >> So let's hear it then, what is it that sets Emacs 18 Elisp
   >> apart from Emacs 30 Elisp . . .
   >
   > Briefly, it's now a much larger language . . .
   >
   >> and how is the Emacs 30 Elisp less coherent than the Emacs
   >> 18 Elisp?
   >
   > . . . and the additions have come from diverse sources . . .

   I understand that it has expanded in size, this is the nature
   of technology systems in general and perhaps even more so with
   Lisp programming, since it is so easy and fun to do.

But, just to point out the obvious, Emacs Lisp is unusual in that the
language and the application are inseparable.  In other language systems
(e.g. Python, Perl) the language, the application, and packages used to
build the application (even when distributed as part of the language)
are all distinct; in Emacs the boundaries are much fuzzier, undoubtedly
another consequence of Lisp.

   But I am still uncertain how this has changed the style and
   reduced the coherence of Elisp.

I'm sorry I'm apparently unable to be more concrete about explaining my
impressions.  Part of this is due to the elapsed time.  ISTR that, when
I started reading Elisp in the mid-90's, it had a "retro" feel compared
to the other Lisps I was using.  As time went on, I've had to hit C-h f
more and more to understand what was going on, not only for language
features (pcase has been mentioned), but for editing features like
indirect buffers and overlays as they were added -- not strictly part of
the language, but again, where's do you draw the boundary?  (I've also
had to read more C code to follow what's going on, but that's a separate
issue.)

   Case in point, I don't see any such difference when I look at
   cl-lib.el?

   -- 
   underground experts united
   https://dataswamp.org/~incal

Between cl-lib.el and what?

					-- Bob

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Bob Rogers wrote:

> I'm sorry I'm apparently unable to be more concrete about
> explaining my impressions. Part of this is due to the
> elapsed time. ISTR that, when I started reading Elisp in the
> mid-90's, it had a "retro" feel compared to the other Lisps
> I was using. As time went on, I've had to hit C-h f more and
> more to understand what was going on, not only for language
> features (pcase has been mentioned), but for editing
> features like indirect buffers and overlays as they were
> added -- not strictly part of the language, but again,
> where's do you draw the boundary?

To be honest, I have not seen any.

>> Case in point, I don't see any such difference when I look
>> at cl-lib.el?
>
> Between cl-lib.el and what?

We have to get the Emacs 18 source so we can compare :)

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Stephen Berman]

On Fri, 03 Nov 2023 23:46:22 +0100 Emanuel Berg <incal@dataswamp.org> wrote:

> Bob Rogers wrote:
>
>> I'm sorry I'm apparently unable to be more concrete about
>> explaining my impressions. Part of this is due to the
>> elapsed time. ISTR that, when I started reading Elisp in the
>> mid-90's, it had a "retro" feel compared to the other Lisps
>> I was using. As time went on, I've had to hit C-h f more and
>> more to understand what was going on, not only for language
>> features (pcase has been mentioned), but for editing
>> features like indirect buffers and overlays as they were
>> added -- not strictly part of the language, but again,
>> where's do you draw the boundary?
>
> To be honest, I have not seen any.
>
>>> Case in point, I don't see any such difference when I look
>>> at cl-lib.el?
>>
>> Between cl-lib.el and what?
>
> We have to get the Emacs 18 source so we can compare :)

You can find several versions here:

https://github.com/larsbrinkhoff/emacs-history

Steve Berman

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Stephen Berman wrote:

>> We have to get the Emacs 18 source so we can compare :)
>
> You can find several versions here:
>
> https://github.com/larsbrinkhoff/emacs-history

Several, including Emacs 18!

I followed the link to this page:

  http://bitsavers.org/bits/MIT/gnu/emacs_18.41.tar.gz

Now we can see clearly how the style has changed :)

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >    So, Richard, I see that you are fighting to preserve something real
  > and important -- your vision of Emacs Lisp as a coherent language -- and
  > I believe that may be your prerogative as its creator.  But I also
  > believe you may be a decade or two too late.

It is not too late if we find a few more contributors who appreciate
what is at stake.

The consistent style of Emacs Lisp is one part of it.
But there is a more practical part, which I've explained recently:
the duplication, even triplication of features for the same job.
The result of this is to multiply what one needs to know
to understand the code in Emacs.

If Emacs Lisp had only one interface for doing a particular job, it
would be enough to learn that one.  But when it has several different
interfaces for doing the same job, you end up needing to know all
those interfaces.   Any one file probably uses one of the interfaces;
to understand that file, knowing that one interface is emnough.
But the next file you look at probably uses a different interface,
and the next one yet another interface -- so in practice you need to
know all the redundant interfaces.

This puts an avoidable extra a burden on everyone who wants to read
the Lisp code of Emacs.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Richard Stallman wrote:

>> So, Richard, I see that you are fighting to preserve
>> something real and important -- your vision of Emacs Lisp
>> as a coherent language -- and I believe that may be your
>> prerogative as its creator. But I also believe you may be
>> a decade or two too late.
>
> It is not too late if we find a few more contributors who
> appreciate what is at stake [...]

Maybe, but how?

We have `cl-incf' and can't remove it as that would break the
CL emulation interface.

To the people who use it, and don't care about the CL
emulation part necessarily, they just think it is useful, we
can't say instead use

  (setq var (1+ var))

because we can't make the case that that is in any way better
than

  (cl-incf var)

We can however add a new function, "incf", that does the same
thing, only does not rely upon cl-lib in any way.

But how would this help the maintainers? Instead of
familiarizing themselves with `cl-incf', and all other
functions provided by cl-lib that are in use and people will
continue to use, they will have to learn the new functions
replacing them? So what difference will that make?

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Björn Bidar]

Emanuel Berg <incal@dataswamp.org> writes:

> But how would this help the maintainers? Instead of
> familiarizing themselves with `cl-incf', and all other
> functions provided by cl-lib that are in use and people will
> continue to use, they will have to learn the new functions
> replacing them? So what difference will that make?

It does sound more and more the fight for some kind of language purity
to not use cl-lib. Removed from reality like Don Quixote.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Alan Mackenzie]

Hello, Björn.

On Wed, Nov 08, 2023 at 17:00:06 +0200, Björn Bidar wrote:
> Emanuel Berg <incal@dataswamp.org> writes:

> > But how would this help the maintainers? Instead of
> > familiarizing themselves with `cl-incf', and all other
> > functions provided by cl-lib that are in use and people will
> > continue to use, they will have to learn the new functions
> > replacing them? So what difference will that make?

> It does sound more and more the fight for some kind of language purity
> to not use cl-lib. Removed from reality like Don Quixote.

That is verging on the offensively patronising, and I don't think it is
appropriate language for this mailing list.

Just take one look at this "reality" you're so supportive of: the
widespread use of cl-lib, not just in people's own projects, but
throughout the core of Emacs, has multiplied the size of Lisp language
part of Emacs by a factor of around 3.   This is a gross increase in
complexity for maintainers that is not justified by the slight increase
in facility that cl-lib (along with things like seq.el and oclosures)
gives.

Throughout this long discussion, this indiscriminate use of cl-lib has
been supported only by occasional contributers.  Those who actually
maintain other people's code, apart from (I think) Eli, Richard and
myself, have been conspicuously silent.  None of us three have favoured
such use of cl-lib.

Occasional contributors may be fascinated by cl-lib, and learn enough of
it to use random bits of it in their code.  The trouble is, each such
contributor uses a different piece of cl-lib, with the result that those
who end up maintaining it need to know a far greater part of it just to
cope.

This factor of 3 is, I believe, a significant barrier to new programmers
coming into Emacs; Elisp is just that much more difficult than it was in
the past.  And it isn't just for newcomers that it is more difficult.  I
spend a significant amount of debugging time having to look up doc
strings and manual pages for obscure cl-lib (etc.) functions.  This is
lost time, representing a diminution of my capacity to maintain Emacs.
I suspect it is similar for other maintainers.

That is the current reality.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Alan Mackenzie wrote:

> Occasional contributors may be fascinated by cl-lib, and
> learn enough of it to use random bits of it in their code.
> The trouble is, each such contributor uses a different piece
> of cl-lib, with the result that those who end up maintaining
> it need to know a far greater part of it just to cope.

Yes, this sounds like a good explanation how the situation has
appeared. We see how two worlds collide.

But how do you propose to improve the situation, then?

Maybe maintaining can be distributed so that the "occasional
contributors" are maintaining their own code?

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 08/11/2023 19:18, Alan Mackenzie wrote:
> Throughout this long discussion, this indiscriminate use of cl-lib has
> been supported only by occasional contributers.  Those who actually
> maintain other people's code, apart from (I think) Eli, Richard and
> myself, have been conspicuously silent.

I like cl-lib (it could be optimized better still, but it's good for 
certain uses), and this discussion doesn't look like it will reach any 
constructive conclusion, so it didn't seem worth it to spend time on it.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 992 bytes --]

On Wed, Nov 8, 2023, 20:20 Dmitry Gutov <dmitry@gutov.dev> wrote:

> On 08/11/2023 19:18, Alan Mackenzie wrote:
> > Throughout this long discussion, this indiscriminate use of cl-lib has
> > been supported only by occasional contributers.  Those who actually
> > maintain other people's code, apart from (I think) Eli, Richard and
> > myself, have been conspicuously silent.
>
> I like cl-lib (it could be optimized better still, but it's good for
> certain uses), and this discussion doesn't look like it will reach any
> constructive conclusion, so it didn't seem worth it to spend time on it.
>

If it needed any confirmation, I too like cl-lib and I too help maintain
other people's code in the Emacs core tree as well as maintaining a number
of libraries I have authored. I read many styles of Elisp here and am not
really phased by any particular programming technique being discussed here
(cl-lib, seq, pcase, verbose while loops with catch/throw, etc etc).

João

[-- Attachment #2: Type: text/html, Size: 1645 bytes --]

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

João Távora wrote:

>>> Throughout this long discussion, this indiscriminate use
>>> of cl-lib has been supported only by occasional
>>> contributers. Those who actually maintain other people's
>>> code, apart from (I think) Eli, Richard and myself, have
>>> been conspicuously silent.
>>
>> I like cl-lib (it could be optimized better still, but it's
>> good for certain uses), and this discussion doesn't look
>> like it will reach any constructive conclusion, so it
>> didn't seem worth it to spend time on it.
>
> If it needed any confirmation, I too like cl-lib and I too
> help maintain other people's code in the Emacs core tree as
> well as maintaining a number of libraries I have authored.
> I read many styles of Elisp here and am not really phased by
> any particular programming technique being discussed here
> (cl-lib, seq, pcase, verbose while loops with catch/throw,
> etc etc).

I like cl-lib and I maintain an ELPA package [1] - ELPA, as we
know, is part of GNU Emacs. It doesn't use cl-lib. Okay,
so bad example then.

[1] https://elpa.gnu.org/packages/wrap-search.html

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Alan Mackenzie]

Hello, João

On Wed, Nov 08, 2023 at 21:10:00 +0000, João Távora wrote:
>    On Wed, Nov 8, 2023, 20:20 Dmitry Gutov <[1]dmitry@gutov.dev> wrote:

>      On 08/11/2023 19:18, Alan Mackenzie wrote:
>>      > Throughout this long discussion, this indiscriminate use of
>>      > cl-lib has been supported only by occasional contributers.Â
>>      > Those who actually maintain other people's code, apart from (I
>>      > think) Eli, Richard and myself, have been conspicuously
>>      > silent.

>>      I like cl-lib (it could be optimized better still, but it's good
>>      for certain uses), and this discussion doesn't look like it will
>>      reach any constructive conclusion, so it didn't seem worth it to
>>      spend time on it.

>    If it needed any confirmation, I too like cl-lib and I too help
>    maintain other people's code in the Emacs core tree as well as
>    maintaining a number of libraries I have authored.

There's a difference between liking cl-lib and advocating its
indiscriminate use.  I don't think you've done the latter in this (and
related) threads.

Nobody who likes cl-lib has yet addressed the point made by Richard and
(less eloquently) by me, namely that the incorporation and use of cl-lib
swells the size and complexity of Emacs Lisp to the point of making
maintenance difficult.  What is your view on this important point?

>    I read many styles of Elisp here and am not really phased by any
>    particular programming technique being discussed here (cl-lib, seq,
>    pcase, verbose while loops with catch/throw, etc etc).  João

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Thu, Nov 9, 2023 at 10:05 AM Alan Mackenzie <acm@muc.de> wrote:

> >    If it needed any confirmation, I too like cl-lib and I too help
> >    maintain other people's code in the Emacs core tree as well as
> >    maintaining a number of libraries I have authored.
>
> There's a difference between liking cl-lib and advocating its
> indiscriminate use.  I don't think you've done the latter in this (and
> related) threads.

Yes, you're right.  Indeed I don't' advocate for its indiscriminate use,
just as I don't advocate for indiscriminate use of anything, except
perhaps drinking water and brushing teeth.

> Nobody who likes cl-lib has yet addressed the point made by Richard and
> (less eloquently) by me, namely that the incorporation and use of cl-lib
> swells the size and complexity of Emacs Lisp to the point of making
> maintenance difficult.  What is your view on this important point?

That it doesn't make maintenance any more difficult than any other
Elisp construct, be it very old and curiously named like 'rplacd' or
much, much newer like `seq-do` or `pcase-lambda`.

My specific view on cl-lib.el is that it brings us a small part of
the results of  non-trivial design work put in when important figures
in the Lisp world met  regularly for many years to deliver what has
proved to become excellent,  battle-tested, widely understood and
impeccably documented programming abstractions.

What I'm reading so far in this long discussion is that the argument
of its detractors isn't really that cl-lib isn't good, but that
it is superfluous and that learning it is a burden on maintainers.
Well, it's just as superfluous as all of Elisp apart from two handfuls
of primitives, I guess.  Or any programming language for that matter, if
you know enough machine code.  Or any other programming abstraction I
happen not to be familiar with.

Also I seem to hear that Elisp has some kind of ideal hard-to-define
identity or fingerprint and that it shouldn't try to become anything
else.  But this argument is very strange given Elisp is, like any
decent language, easy to extend.  Not to mention I struggle to see
the technical advantage in uniqueness for uniqueness sake.  A good
part of Elisp is about a special purpose function: editing buffers,
and an the equally important part is about doing things with bits in
memory, there's no reason to advocate for singularity here.

I also hear Elisp shouldn't become Common Lisp, but not only is the
use of cl-lib.el nowhere a step in that direction, but also -- somewhat
ironically -- if Elisp _were_ Common Lisp, then that hard-to-define
identity would be much easier to define and language extension would be
much easier to organize into compartments to facilitate policy-making.

Again, the only thing that has brought Elisp any closer to Common
Lisp significantly, was the introduction of lexical binding some 10
years ago.  Elisp looks a lot different now than it did in the 90's.
Closures everywhere, higher-order functions!  Shudder!

There's even talk of a continuation-passing style library, to
be called future.el or promise.el or something.  Oh dear, what
will be of us that we will have to evolve like any other language
in the world!

So I propose we let programmers use their judgement.  Really
respect people who write code for no money and give the copyright
away to the FSF.  Maybe suggest guidelines such as not introduce
cl-loop where a dolist would do the job just as economically and
elegantly. Don't use higher-order functions just because it looks
cool. But maybe do suggest to use cl-position with a :key and a
:test instead of a complex while loop with an auxiliary variable.
Or cl-set-difference instead of nested loops.  Suggest to express
intent, use abstractions.   Suggest to be consistent, be scrutinous,
be "discriminate", be mindful of the style of the current area
you are working on.

But don't suggest anything too hard, especially if it's not
modifying code you have authored.  Don't use arguments of authority
when you can point to specific things.  Be generally respectful of
people putting in any good work even if you don't like the style,
and try to learn a new thing or two every now and then.

BTW here are some nice generic suggestions from the Lisp world,
written by two fenomenal programmers.  I love reading this from
time to time:

   https://www.cs.umd.edu/~nau/cmsc421/norvig-lisp-style.pdf

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Thu, 9 Nov 2023 11:06:02 +0000
> Cc: Dmitry Gutov <dmitry@gutov.dev>,
>  Björn Bidar <bjorn.bidar@thaodan.de>, 
>  emacs-devel <emacs-devel@gnu.org>
> 
> So I propose we let programmers use their judgement.

That's fair enough, provided that those same programmers step up to
become the Emacs maintainers, and thus share the burden to at least
some extent.  Especially those of them who consider witty sarcasm a
useful style in a discussion like this one.

The following is left as an exercise for the interested readers: take
a non-trivial piece of code that you never in your life have laid your
eyes upon, and decide what to do with a serious bug report about that
code, with or without a patch proposed by someone.  A recent case in
point: bug#66970.  Bonus points for doing this with code in a
programming language you don't command 110%, like C if you are mainly
a Lisp programmer or vice versa.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Thu, Nov 9, 2023 at 11:38 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Thu, 9 Nov 2023 11:06:02 +0000
> > Cc: Dmitry Gutov <dmitry@gutov.dev>,
> >  Björn Bidar <bjorn.bidar@thaodan.de>,
> >  emacs-devel <emacs-devel@gnu.org>
> >
> > So I propose we let programmers use their judgement.
>
> That's fair enough, provided that those same programmers step up to
> become the Emacs maintainers, and thus share the burden to at least
> some extent.  Especially those of them who consider witty sarcasm a
> useful style in a discussion like this one.

Did you read the rest of the paragraph from which you extracted that
one line?  I'm advocating for being discriminate and

Also, I would hope this is not a suggestion that I don't step to share the
burden of maintaining things?  I do, certainly for all the features that I
have I added to core in the past 10 years and frequently also to parts
of the code those features interact with.  Do you often have to fix bugs in
flymake.el, elec-pair.el, icomplete.el, eldoc.el, jsonrpc.el, etc...
that I didn't care to take a look at in a reasonable time frame?  More
importantly, to what extent are you and others significantly or often
hindered by the style I use there?

I can't address what I don't know about.  I can't track the bug tracker
as closely as you do.  If I'm missing or late about bug reports about
such things, just ping me like you usually do.

Or are you suggesting I should dumb down my style for when I'm dead and
my code survives me?  Just so you know, I think that's not entirely
unreasonable.  In fact if you read the pdf I linked to, it's a valid
criteria for making decisions.   But it's definitely not the only one.
Who knows if Emacs 40's maintainer wouldn't have appreciated it if I
had smartened it up even more and use _only_ cl-lib.

Anyway, if there's some code of mine that's bothering you and you
find inescrutable, point me to it and we can discuss it, of course.

> The following is left as an exercise for the interested readers: take
> a non-trivial piece of code that you never in your life have laid your
> eyes upon, and decide what to do with a serious bug report about that
> code, with or without a patch proposed by someone.  A recent case in
> point: bug#66970.  Bonus points for doing this with code in a
> programming language you don't command 110%, like C if you are mainly
> a Lisp programmer or vice versa.

Do you need help reading some Elisp construct or debugging?
Is this meaningfully related to style in Lisp code?  You want help
reading hangul.el?  (which doesn't seem to use cl-lib.el btw)

Would you like help rewriting hangul-insert-character to support
overwrite-mode, perhaps by leveraging post-self-insert-hook?  I can
help with that.  If there is some deep required specific domain
knowledge about input methods or how the Korean language works, you
or someone else may have to help out with that.  If C code is involved,
it's not a problem.  I'll have a better look later.

The problem of maintaining other people's code (which btw is 95%
of all my professional experience) is not about programming techniques
or even languages at all, it's about domain knowledge.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Thu, Nov 9, 2023 at 12:34 PM João Távora <joaotavora@gmail.com> wrote:

> > The following is left as an exercise for the interested readers: take
> > a non-trivial piece of code that you never in your life have laid your
> > eyes upon, and decide what to do with a serious bug report about that
> > code, with or without a patch proposed by someone.  A recent case in
> > point: bug#66970.  Bonus points for doing this with code in a
> > programming language you don't command 110%, like C if you are mainly
> > a Lisp programmer or vice versa.

>  I'll have a better look later.

At first sight, didn't seem terribly hard.  I gave it some basic testing
seems to work like the OP in that bug would like it to.  Could be a start
I guess.  Let me know if you don't like the style.

diff --git a/lisp/leim/quail/hangul.el b/lisp/leim/quail/hangul.el
index 46a2e5a6ba2..c97362b1250 100644
--- a/lisp/leim/quail/hangul.el
+++ b/lisp/leim/quail/hangul.el
@@ -146,21 +146,26 @@ hangul-insert-character
       (progn
         (delete-region (region-beginning) (region-end))
         (deactivate-mark)))
-  (quail-delete-region)
-  (let ((first (car queues)))
-    (insert
-     (hangul-character
-      (+ (aref first 0) (hangul-djamo 'cho (aref first 0) (aref first 1)))
-      (+ (aref first 2) (hangul-djamo 'jung (aref first 2) (aref first 3)))
-      (+ (aref first 4) (hangul-djamo 'jong (aref first 4) (aref first 5))))))
-  (move-overlay quail-overlay (overlay-start quail-overlay) (point))
-  (dolist (queue (cdr queues))
-    (insert
-     (hangul-character
-      (+ (aref queue 0) (hangul-djamo 'cho (aref queue 0) (aref queue 1)))
-      (+ (aref queue 2) (hangul-djamo 'jung (aref queue 2) (aref queue 3)))
-      (+ (aref queue 4) (hangul-djamo 'jong (aref queue 4) (aref queue 5)))))
-    (move-overlay quail-overlay (1+ (overlay-start quail-overlay)) (point))))
+  (let* ((things-to-insert
+          (with-temp-buffer
+            (dolist (queue queues (mapcar #'identity (buffer-string)))
+              (insert
+               (hangul-character
+                (+ (aref queue 0) (hangul-djamo 'cho (aref queue 0)
(aref queue 1)))
+                (+ (aref queue 2) (hangul-djamo 'jung (aref queue 2)
(aref queue 3)))
+                (+ (aref queue 4) (hangul-djamo 'jong (aref queue 4)
(aref queue 5))))))))
+         (may-have-to-overwrite-p
+          (or (= (overlay-start quail-overlay) (overlay-end quail-overlay))
+              (cdr things-to-insert))))
+    (quail-delete-region)
+    (dolist (c things-to-insert)
+      (let ((last-command-event c)
+            (overwrite-mode (and overwrite-mode
+                                 may-have-to-overwrite-p
+                                 overwrite-mode)))
+        (self-insert-command 1)
+        (setq may-have-to-overwrite-p nil)))
+    (move-overlay quail-overlay (1- (point)) (point))))

 (defun hangul-djamo (jamo char1 char2)
   "Return the double Jamo index calculated from the arguments.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Thu, 9 Nov 2023 12:34:09 +0000
> Cc: acm@muc.de, dmitry@gutov.dev, bjorn.bidar@thaodan.de, emacs-devel@gnu.org
> 
> On Thu, Nov 9, 2023 at 11:38 AM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > > From: João Távora <joaotavora@gmail.com>
> > > Date: Thu, 9 Nov 2023 11:06:02 +0000
> > > Cc: Dmitry Gutov <dmitry@gutov.dev>,
> > >  Björn Bidar <bjorn.bidar@thaodan.de>,
> > >  emacs-devel <emacs-devel@gnu.org>
> > >
> > > So I propose we let programmers use their judgement.
> >
> > That's fair enough, provided that those same programmers step up to
> > become the Emacs maintainers, and thus share the burden to at least
> > some extent.  Especially those of them who consider witty sarcasm a
> > useful style in a discussion like this one.
> 
> Did you read the rest of the paragraph from which you extracted that
> one line?

You should by now know me well enough to realize that I read
everything here.  However, I respond only to the parts where I have
something useful and non-trivial to say.

> Also, I would hope this is not a suggestion that I don't step to share the
> burden of maintaining things?  I do, certainly for all the features that I
> have I added to core in the past 10 years and frequently also to parts
> of the code those features interact with.  Do you often have to fix bugs in
> flymake.el, elec-pair.el, icomplete.el, eldoc.el, jsonrpc.el, etc...
> that I didn't care to take a look at in a reasonable time frame?  More
> importantly, to what extent are you and others significantly or often
> hindered by the style I use there?

I said nothing about your style.  Neither do I doubt that you know
your own code very well.  We all know our own code well.  I'm asking
you not to judge the burden of the maintainers in maintaining code
they didn't write nor even see, until you get to our place and do it
for a while.  Only then you will be able to realize the role of
obscure programming style and use of unfamiliar syntax in making the
job harder than it has to be.

> The problem of maintaining other people's code (which btw is 95%
> of all my professional experience) is not about programming techniques
> or even languages at all, it's about domain knowledge.

It is both.  In particular, when you need to understand why some code
fragment misbehaves, you need to understand the implementation and
what it intends to do, not only why.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Eli Zaretskii wrote:

> We all know our own code well. I'm asking you not to judge
> the burden of the maintainers in maintaining code they
> didn't write nor even see [...]

We should distribute the maintenance effort so that to
a higher degree, people who wrote the code are maintaining it,
and, whenever that cannot be achieved, at least people who
take on "I'll maintain this package", it will then be
understood, for example, if the file contains a lot of cl-lib,
the maintainer who takes on that file will have to deal
with it, because it comes with the task.

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Thu, 09 Nov 2023 16:29:45 +0100
> 
> Eli Zaretskii wrote:
> 
> > We all know our own code well. I'm asking you not to judge
> > the burden of the maintainers in maintaining code they
> > didn't write nor even see [...]
> 
> We should distribute the maintenance effort so that to
> a higher degree, people who wrote the code are maintaining it,

This is a volunteer-based project, so we have no way of "distributing
the maintenance effort", unless people volunteer to do it.  In
particular, for absolute majority of the Emacs code, the people who
wrote the code are no longer on board.  That's the reality.

If you mean that the authors who are still here should be asked to
take care of their code, then this is already being done, always, and
cannot be done to any higher degree.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Eli Zaretskii wrote:

> This is a volunteer-based project, so we have no way of
> "distributing the maintenance effort", unless people
> volunteer to do it. In particular, for absolute majority of
> the Emacs code, the people who wrote the code are no longer
> on board. That's the reality.
>
> If you mean that the authors who are still here should be
> asked to take care of their code, then this is already being
> done, always, and cannot be done to any higher degree.

Yes, I understand, but we can still just hand out files to
people, "now you are maintaining this file". That person will
have to agree, of course.

For each such file handed out there is one file less for the
"maintainers of everything else" to handle.

Because what we heard was that there are often just a few uses
of cl-lib in some person's code, and that is manageable to
learn for the maintainer. Only in some other person's code,
there are also just a few such cases, but not necessarily the
same, so the maintainers of a lot of files, for them the
complexity increases for each file even tho the individual
files are not complex in their us of, say, cl-lib, really.

So if we give away a bunch of files to a larger group of
people, those people will not have an issue with cl-lib,
really, and at the same time both scope and complexity will be
reduced for the "everything else" maintainers.

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Sat, 11 Nov 2023 14:10:20 +0100
> 
> Eli Zaretskii wrote:
> 
> > This is a volunteer-based project, so we have no way of
> > "distributing the maintenance effort", unless people
> > volunteer to do it. In particular, for absolute majority of
> > the Emacs code, the people who wrote the code are no longer
> > on board. That's the reality.
> >
> > If you mean that the authors who are still here should be
> > asked to take care of their code, then this is already being
> > done, always, and cannot be done to any higher degree.
> 
> Yes, I understand, but we can still just hand out files to
> people, "now you are maintaining this file". That person will
> have to agree, of course.

In the real world, where I live, this is also unbecoming, and thus
practically impossible.  Good luck leading a volunteer project with
these ideas!

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Eli Zaretskii wrote:

>>> This is a volunteer-based project, so we have no way of
>>> "distributing the maintenance effort", unless people
>>> volunteer to do it. In particular, for absolute majority
>>> of the Emacs code, the people who wrote the code are no
>>> longer on board. That's the reality.
>>>
>>> If you mean that the authors who are still here should be
>>> asked to take care of their code, then this is already
>>> being done, always, and cannot be done to any
>>> higher degree.
>> 
>> Yes, I understand, but we can still just hand out files to
>> people, "now you are maintaining this file". That person
>> will have to agree, of course.
>
> In the real world, where I live, this is also unbecoming,
> and thus practically impossible. Good luck leading
> a volunteer project with these ideas!

What do you mean?

Why can't more people be involved?

But if they can't, then yes: the remaining three guys
maintaining hundreds of files will have to deal with a lot
of complexity, for sure.

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Sat, 11 Nov 2023 15:07:44 +0100
> 
> Eli Zaretskii wrote:
> 
> >> Yes, I understand, but we can still just hand out files to
> >> people, "now you are maintaining this file". That person
> >> will have to agree, of course.
> >
> > In the real world, where I live, this is also unbecoming,
> > and thus practically impossible. Good luck leading
> > a volunteer project with these ideas!
> 
> What do you mean?

I mean that's not how things work in this and other volunteer-based
projects.

> Why can't more people be involved?

They can, just not as simply as you put it, and what can be done is
already being done, by and large.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Eli Zaretskii wrote:

>> What do you mean?
>
> I mean that's not how things work in this and other
> volunteer-based projects.

Okay, I think it would be an improvement if more people did,
but each on a smaller number of files.

>> Why can't more people be involved?
>
> They can, just not as simply as you put it, and what can be
> done is already being done, by and large.

Did you ask, for example, on this list?

I have noticed in the package manager there are 70 packages
that are "built-in".

Maybe you can keep up the modularization process of core Emacs
in this way and create more and also see what people are up to
maintaining one or a couple of such packages?

I say this in connection to this issue as maintaining
a "package" sounds more appealing than maintaining some .el
file, I think more people would be attracted to it if phrased
and organized that way.

  allout                         2.3            built-in              extensive outline mode for use alone and with other modes
  allout-widgets                 1.0            built-in              Visually highlight allout outline structure.
  ansi-color                     3.4.2          built-in              translate ANSI escape sequences into faces
  antlr-mode                     2.2.3          built-in              major mode for ANTLR grammar files
  auth-source-pass               5.0.0          built-in              Integrate auth-source with password-store
  backtrace                      1.0            built-in              generic major mode for Elisp backtraces
  bind-key                       2.4.1          built-in              A simple way to manage personal keybindings
  cc-mode                        5.33.1         built-in              major mode for editing C and similar languages
  cedet                          2.0            built-in              Setup CEDET environment
  cfengine                       1.4            built-in              mode for editing Cfengine files
  cl-generic                     1.0            built-in              CLOS-style generic functions for Elisp
  cl-lib                         1.0            built-in              Common Lisp extensions for Emacs
  cl-print                       1.0            built-in              CL-style generic printing
  ede                            2.0            built-in              Emacs Development Environment gloss
  ediff                          2.81.6         built-in              a comprehensive visual interface to diff & patch
  eglot                          1.15           built-in              The Emacs Client for LSP servers
  eieio                          1.4            built-in              Enhanced Implementation of Emacs Interpreted Objects
  eieio-core                     1.4            built-in              Core implementation for eieio
  eldoc                          1.14.0         built-in              Show function arglist or variable docstring in echo area
  epg                            1.0.0          built-in              the EasyPG Library
  erc                            5.6snapshot    built-in              An Emacs Internet Relay Chat client
  eshell                         2.4.2          built-in              the Emacs command shell
  external-completion            0.1            built-in              Let external tools control completion style
  faceup                         0.0.6          built-in              Markup language for faces and font-lock regression testing
  feedmail                       11             built-in              assist other email packages to massage outgoing messages
  find-cmd                       0.6            built-in              Build a valid find(1) command with sexps
  flymake                        1.3.7          built-in              A universal on-the-fly syntax checker
  flymake-proc                   1.0            built-in              Flymake backend for external tools
  foldout                        1.10           built-in              folding extensions for outline-mode and outline-minor-mode
  gnus                           5.13           built-in              a newsreader for GNU Emacs
  htmlfontify                    0.21           built-in              htmlize a buffer/source tree with optional hyperlinks
  idlwave                        6.1.22         built-in              IDL editing mode for GNU Emacs
  image-dired                    0.5            built-in              use dired to browse and manipulate your images
  info-xref                      3              built-in              check external references in an Info document
  isearchb                       1.5            built-in              a marriage between iswitchb and isearch
  js                             9              built-in              Major mode for editing JavaScript
  json                           1.5            built-in              JavaScript Object Notation parser / generator
  jsonrpc                        1.0.17         built-in              JSON-RPC library
  let-alist                      1.0.6          built-in              Easily let-bind values of an assoc-list by their names
  map                            3.3.1          built-in              Map manipulation functions
  meta-mode                      1.0            built-in              major mode for editing Metafont or MetaPost sources
  mh-e                           8.6snapshot    built-in              GNU Emacs interface to the MH mail system
  mixal-mode                     0.4            built-in              Major mode for the mix asm language.
  nadvice                        1.0            built-in              Light-weight advice primitives for Elisp functions
  ntlm                           2.1.0          built-in              NTLM (NT LanManager) authentication support
  org                            9.6.10         built-in              Outline-based notes management and organizer
  package                        1.1.0          built-in              Simple package system for Emacs
  project                        0.10.0         built-in              Operations on the current project
  ps-mode                        1.1.9          built-in              PostScript mode for GNU Emacs
  pulse                          1.0            built-in              Pulsing Overlays
  python                         0.28           built-in              Python's flying circus support for Emacs
  ruby-mode                      1.2            built-in              Major mode for editing Ruby files
  ruby-ts-mode                   0.2            built-in              Major mode for editing Ruby files using tree-sitter
  semantic                       2.2            built-in              Semantic buffer evaluator.
  seq                            2.24           built-in              Sequence manipulation functions
  so-long                        1.1.2          built-in              Say farewell to performance problems with minified code.
  soap-client                    3.2.3          built-in              Access SOAP web services
  sql                            3.6            built-in              specialized comint.el for SQL interpreters
  srecode                        1.2            built-in              Semantic buffer evaluator.
  svg                            1.1            built-in              SVG image creation functions
  tabulated-list                 1.0            built-in              generic major mode for tabulated lists
  thunk                          1.0            built-in              Lazy form evaluation
  tildify                        4.6.1          built-in              adding hard spaces into texts
  tramp                          2.7.0pre       built-in              Transparent Remote Access, Multiple Protocol
  use-package                    2.4.5          built-in              A configuration macro for simplifying your .emacs
  vera-mode                      2.28           built-in              major mode for editing Vera files
  verilog-mode                   2023.6.6.141322628 built-in              major mode for editing verilog source in Emacs
  viper                          3.14.2         built-in              A full-featured Vi emulator for Emacs
  whitespace                     13.2.2         built-in              minor mode to visualize TAB, (HARD) SPACE, NEWLINE
  xref                           1.6.3          built-in              Cross-referencing commands

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Sat, 11 Nov 2023 15:54:29 +0100
> 
> Eli Zaretskii wrote:
> 
> >> Why can't more people be involved?
> >
> > They can, just not as simply as you put it, and what can be
> > done is already being done, by and large.
> 
> Did you ask, for example, on this list?

Yes, when that is appropriate.

> I have noticed in the package manager there are 70 packages
> that are "built-in".

See admin/MAINTAINERS.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Thu, Nov 9, 2023 at 3:05 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > that I didn't care to take a look at in a reasonable time frame?  More
> > importantly, to what extent are you and others significantly or often
> > hindered by the style I use there?
>
> I said nothing about your style.  Neither do I doubt that you know
> your own code very well.  We all know our own code well.  I'm asking
> you not to judge the burden of the maintainers in maintaining code
> they didn't write nor even see, until you get to our place and do it
> for a while.

Okay, and what I tried to say is that I _do_ have a valid judgement
of exactly this burden, to some quite significant degree, because not only
do I maintain "my" code, I have to wade through a lot of code that
I didn't write.  I didn't write any of icomplete.el and yet managed
to significantly enhance it, for example.  Or elisp-mode.el, or
minibuffer.el.  Or flymake.el, which I rewrote, backward compatibly.
Or eldoc.el, where I did likewise.  Or electric.el or so many other
pieces of code I have to visit and read to make sense of Emacs.
These aren't trivial files they're much larger.  I go
wherever M-. takes me.

So I have to read and understand lots of "alien" code in styles that
wouldn't be my first preference if I had written it from scratch,
often styles that lead to many "blerghs" and "yucks" which I know
we all do.  And sometimes that alien code was written by myself a
longer time ago.

And what I'm trying to say is that all those difficulties are much
much less importnat than shortcomings in domain knowledge.

> Only then you will be able to realize the role of
> obscure programming style and use of unfamiliar syntax in making the
> job harder than it has to be.

I am prepared to admit that that impact is heavier to someone
like you who is the first responder for any and every report to
parts of Emacs you didn't even know existed.  But I don't think
a uniform programming style, if that even exists (let alone
being feasible to enforce) would make your job much easier.

Maybe _somewhat_ easier, but to what loss?  Aggravated contributors,
endless hair-splitting of what exactly good style is.   And style
preferences change anyway: what I "liked" 10 years ago is not what
I like now. It's really a bad idea.

But it's definitely a good idea to review any _new_ contributions to
basic Elisp constructs clearly.  I'm not proposing a free-for-all.

I've said this often without any kind of irony or sarcasm that I think
you're an excellent maintainer.  This is not to "butter you up": I want
you to be happy maintaining Emacs for a long time.  IMHO, if I am allowed,
you should keep doing what you do, which is try to ping the specialists
in each domain, and prompt non-specialists too like you did with
bug#66970, to which I hope to have provided at least the start of
a fix.

João Távora

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Thu, 9 Nov 2023 15:39:48 +0000
> Cc: acm@muc.de, dmitry@gutov.dev, bjorn.bidar@thaodan.de, emacs-devel@gnu.org
> 
> And what I'm trying to say is that all those difficulties are much
> much less importnat than shortcomings in domain knowledge.

Not IME, FWIW.

> Maybe _somewhat_ easier, but to what loss?  Aggravated contributors,
> endless hair-splitting of what exactly good style is.   And style
> preferences change anyway: what I "liked" 10 years ago is not what
> I like now. It's really a bad idea.

You will not find any message from me that says we should ban cl-lib.
Maybe you are confusing me with someone else.  All I'm saying is that
people should not dismiss this aspect so easily, when they consider
the subject.  It is not just a coding style issue.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Thu, Nov 9, 2023 at 3:51 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Thu, 9 Nov 2023 15:39:48 +0000
> > Cc: acm@muc.de, dmitry@gutov.dev, bjorn.bidar@thaodan.de, emacs-devel@gnu.org
> >
> > And what I'm trying to say is that all those difficulties are much
> > much less importnat than shortcomings in domain knowledge.
>
> Not IME, FWIW.

Fair enough.  If you ever find my code whose style you think
needs rewriting, or need help translating another piece of
code to more familiar Lisp, let me know.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> That it doesn't make maintenance any more difficult than any other
> Elisp construct, be it very old and curiously named like 'rplacd' or
> much, much newer like `seq-do` or `pcase-lambda`.
>
> My specific view on cl-lib.el is that it brings us a small part of
> the results of  non-trivial design work put in when important figures
> in the Lisp world met  regularly for many years to deliver what has
> proved to become excellent,  battle-tested, widely understood and
> impeccably documented programming abstractions.
>
> What I'm reading so far in this long discussion is that the argument
> of its detractors isn't really that cl-lib isn't good, but that
> it is superfluous and that learning it is a burden on maintainers.
> Well, it's just as superfluous as all of Elisp apart from two handfuls
> of primitives, I guess.  Or any programming language for that matter, if
> you know enough machine code.  Or any other programming abstraction I
> happen not to be familiar with.
>
> Also I seem to hear that Elisp has some kind of ideal hard-to-define
> identity or fingerprint and that it shouldn't try to become anything
> else.  But this argument is very strange given Elisp is, like any
> decent language, easy to extend.  Not to mention I struggle to see
> the technical advantage in uniqueness for uniqueness sake.  A good
> part of Elisp is about a special purpose function: editing buffers,
> and an the equally important part is about doing things with bits in
> memory, there's no reason to advocate for singularity here.
>
> I also hear Elisp shouldn't become Common Lisp, but not only is the
> use of cl-lib.el nowhere a step in that direction, but also -- somewhat
> ironically -- if Elisp _were_ Common Lisp, then that hard-to-define
> identity would be much easier to define and language extension would be
> much easier to organize into compartments to facilitate policy-making.
>
> Again, the only thing that has brought Elisp any closer to Common
> Lisp significantly, was the introduction of lexical binding some 10
> years ago.  Elisp looks a lot different now than it did in the 90's.
> Closures everywhere, higher-order functions!  Shudder!
>
> There's even talk of a continuation-passing style library, to
> be called future.el or promise.el or something.  Oh dear, what
> will be of us that we will have to evolve like any other language
> in the world!
>
> So I propose we let programmers use their judgement.  Really
> respect people who write code for no money and give the copyright
> away to the FSF.  Maybe suggest guidelines such as not introduce
> cl-loop where a dolist would do the job just as economically and
> elegantly. Don't use higher-order functions just because it looks
> cool. But maybe do suggest to use cl-position with a :key and a
> :test instead of a complex while loop with an auxiliary variable.
> Or cl-set-difference instead of nested loops.  Suggest to express
> intent, use abstractions.   Suggest to be consistent, be scrutinous,
> be "discriminate", be mindful of the style of the current area
> you are working on.

Such casuistry and non-sequiturs are the reasons I'm glad I have thus
far given this conversation a generously wide berth.  But one's
restraint has its limits, and I suppose mine have been overstepped,
since I can't keep myself from delving head-first into this madhouse.
Here goes...

> Well, it's just as superfluous as all of Elisp apart from two handfuls
> of primitives, I guess.  Or any programming language for that matter, if
> you know enough machine code.  Or any other programming abstraction I
> happen not to be familiar with.

cl-lib does certainly provide some value for money, by way of constructs
that lend themselves to many general situations, but it does not provide
this value in the lion's share of the circumstances these constructs are
used in.  Consider the blithe misapplication of CL generics throughout
the window system code:

(cl-defmethod gui-backend-set-selection (type value
                                              &context (window-system android))
  ;; First, try to turn value into a string.
  ;; Don't set anything if that did not work.
  (when-let ((string (android-encode-select-string value)))
    (cond ((eq type 'CLIPBOARD)
           (android-set-clipboard string))
          ((eq type 'PRIMARY)
           (setq android-primary-selection string))
          ((eq type 'SECONDARY)
           (setq android-secondary-selection string)))))

do we really expect window-system to be anything _but_ android, so long
as a window system has been initialized, when android-win.el is loaded?
Why was it necessary to convolute the code when g-b-s-s (and its ilk)
could as easily have been defined as an ordinary function in each of the
window system files?

For a second example, how about cl-list*?  Is there really a purpose
for this function, in light of how it can always be replaced by:

  (nconc (list a b) c)

and demands callers load cl-lib.el?  The call to this function in
desktop.el is particularly dubitable: as far as the eye can see, it is
the singular reason that file requires cl-lib, its remainder being
implemented with the cut and dry Emacs Lisp builtins that are well known
to us all.  This abject misuse of cl-lib extensions, not an unqualified
opposition to extending Emacs Lisp, is the catalyst of most misgivings
towards it.

Like begets like, even imitations of like: as the corpus of files which
load cl-lib for legitimate purposes burgeons, so does that of files
which load it for positively frivolous reasons.  Thus usage of cl-lib
should be discouraged in general, and users firmly exhorted to seek
replacements.  Even if there are situations where it is not superfluous,
their existence doesn't outweigh the proliferation of that where it is,
and ultimately it stands us in better stead to treat it as such in
general.

> That it doesn't make maintenance any more difficult than any other
> Elisp construct, be it very old and curiously named like 'rplacd' or
> much, much newer like `seq-do` or `pcase-lambda`.

pcase-lambda and seq-do (plus the boatload of seq and pcase functions)
are not "other Elisp constructs," not in my book.  seq functions are all
generic functions, and their documentation is nothing short of
inscrutable; take the example of seq-do's:

Apply FUNCTION to each element of SEQUENCE.
Presumably, FUNCTION has useful side effects.
Return SEQUENCE.

What is a useful side effect?  Does this imply FUNCTION can modify
SEQUENCE as it is being iterated through?

and each time I read the documentation for pcase, I wind up bewildered
by the needlessly laconic style it is written in.  Most of its users
don't display a persuasive need for it either, a demographic of which
frameset.el is representative: much of its work is also replicated in
the *fns.c files and frame.c, which seem to fare equally well, despite
being authored in a language that largely enjoys an absence of byzantine
constructs.

If doc strings are meant to be incomprehensible without recourse to the
Lisp reference manual, then I suggest they not be written at all.  A doc
string that alludes to and accentuates concepts such as "useful side
effects" without providing the reader with a definition or a reference
as to where he can obtain such a definition is not edifying; it leaves
an unpleasant vacuum in the reader's mind that rankles until he bows to
fate and reads the manual anyway.

rplacd and rplaca are aliases to setcar and setcdr; Emacs Lisp code
ought to use the latter, and aliases and the wholesale introduction of
functions from other languages are plainly incommensurable.

> My specific view on cl-lib.el is that it brings us a small part of
> the results of  non-trivial design work put in when important figures
> in the Lisp world met  regularly for many years to deliver what has
> proved to become excellent,  battle-tested, widely understood and
> impeccably documented programming abstractions.

It's a fair bet that Common Lisp users today number fewer than Emacs
users; of course this does not immediately reflect on the character of
features present in Common Lisp, but it is a consideration to bear in
mind.

The quality of the documentation, which for cl-lib is far from
irreproachable, cannot allay the challenge posed by learning to read and
write a language that is in plenty of respects at variance with the
language Emacs users and developers have learned.  When the latter is in
fact documented beyond reproach (save for rough spots such as the said
pcase), more than adequate for programming in Emacs, and understood by
all Emacs users, the profundity of the deliberations underlying the
design of Common Lisp and the eminence of its designers are not
justifications for its pervading Emacs code, let alone to the degree
that it has by now.

> Also I seem to hear that Elisp has some kind of ideal hard-to-define
> identity or fingerprint and that it shouldn't try to become anything
> else.  But this argument is very strange given Elisp is, like any
> decent language, easy to extend.  Not to mention I struggle to see
> the technical advantage in uniqueness for uniqueness sake.  A good
> part of Elisp is about a special purpose function: editing buffers,
> and an the equally important part is about doing things with bits in
> memory, there's no reason to advocate for singularity here.

The relative ease of extension implies that we should be more
circumspect in judging which extensions to allow--that is the basis of
much of the argumentation here, not the unsupportable notion that there
is an identity to Emacs Lisp which must be preserved at all costs.

> I also hear Elisp shouldn't become Common Lisp, but not only is the
> use of cl-lib.el nowhere a step in that direction, but also -- somewhat
> ironically -- if Elisp _were_ Common Lisp, then that hard-to-define
> identity would be much easier to define and language extension would be
> much easier to organize into compartments to facilitate policy-making.

This is self-evidently wooden language.  I expect the likes of
"compartmentalizing a program for policy-making" from suits, not from
free software developers.

> Again, the only thing that has brought Elisp any closer to Common
> Lisp significantly, was the introduction of lexical binding some 10
> years ago.  Elisp looks a lot different now than it did in the 90's.

[...]

> Closures everywhere, higher-order functions!  Shudder!

This is not ipso facto a net positive and is also beside the point.

> There's even talk of a continuation-passing style library, to
> be called future.el or promise.el or something.  Oh dear, what
> will be of us that we will have to evolve like any other language
> in the world!

I certainly hope never to see the day that becomes widespread in Emacs
code.

> So I propose we let programmers use their judgement.  Really
> respect people who write code for no money and give the copyright
> away to the FSF.  Maybe suggest guidelines such as not introduce
> cl-loop where a dolist would do the job just as economically and
> elegantly. Don't use higher-order functions just because it looks
> cool. But maybe do suggest to use cl-position with a :key and a
> :test instead of a complex while loop with an auxiliary variable.
> Or cl-set-difference instead of nested loops.  Suggest to express
> intent, use abstractions.   Suggest to be consistent, be scrutinous,
> be "discriminate", be mindful of the style of the current area
> you are working on.

Respect for volunteers is no doubt laudable, but it is very much an
position to be considered alone which everything you've presented above
does not impact and nobody takes exception to.  From it, and even
supposing for the sake of argument that everything else you've said is
incontrovertible fact, it doesn't really follow that we should not seek
to reduce the incidence of cl-lib usage in core code.  Now I can't be
certain that is the object of your militating, if you will, though
promoting cl-set-difference and cl-position as alternatives to hallowed
Emacs Lisp constructs does suggest so, but the principle argument
against these constructs is that they are alien to the small group of
individuals charged with trudging through code where people want to make
use of them.  So no contentions resting on their virtues and
distinctions rather than their prevalence among us can hold water...

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Thu, Nov 9, 2023 at 1:46 PM Po Lu <luangruo@yahoo.com> wrote:
>
> João Távora <joaotavora@gmail.com> writes:

> This is self-evidently wooden language.

And all the time I thought I was being cheerful.

> I expect the likes of
> "compartmentalizing a program for policy-making" from suits, not from
> free software developers.

Are _you_ the suit?  I missed the point of most of your very
elegant text, but isn't policy-making exactly what you (and I) are
trying to do, i.e. discourage some things and encourage others?

What's the gripe with compartments?  We don't have all of Elisp in
a single library?  I was talking about namespaces by the way,
if you didn't catch the reference (admittedly obscure)

Also, is really cl-set-difference to take the difference
of two sets worse than writing a "hallowed" multi-line
construct??  Maybe I'm not seeing what the construct is, so
what is your preferred Elisp hallowed way to take the
difference of two sets, say make a list  that contains
all items that appear in LIST1 but not LIST2?  Or to find
the index of a certain element of a sequence...

These comparisons, taken one by one, are the hard evidence of that
allows us to determine what is "alien" or not, not theological
consecrations of personal tastes of a given group, _any_ group.
For example, I don't think your 'nconc'  form is more readable
than 'cl-list*'.  It's slightly less readable IMO.  We could alias
`list*` to nconc though, even better.  But I can agree it's not
worth requiring cl-lib.el just for cl-list* alone.

Much like you prefer setcar to rplacd. Which I also do, btw.  But
I prefer (setf (car ...))  even more.  But you'll frequently see
me using setcar and setcdr (or even rplacwahtevercannevertypeit)
if that's the style of the file I'm working on.  Because I too
have my personal tastes, but contrary to others I can let go of
them without too much fuss.

Stefan M's pcase in particular is great.  Its docstring is tough,
I agree, and could use some examples (I always end up grepping
for examples).  But that macro is a lifesaver.

The way you talk about seq.el and pcase.el makes me think you
want to ban _all the things_, not just cl-lib.  I'm confused about
what you want Elisp code in core to look like.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Thu, Nov 9, 2023 at 2:28 PM João Távora <joaotavora@gmail.com> wrote:

> than 'cl-list*'.  It's slightly less readable IMO.  We could alias
> `list*` to nconc though, even better.

oops, no we couldn't.  Same job, different calling convention.
I just don't use either that much, so here again my suggestion
is to just use what fits better with the style of surrounding code.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> On Thu, Nov 9, 2023 at 2:28 PM João Távora <joaotavora@gmail.com>
> wrote:
>
>> than 'cl-list*'.  It's slightly less readable IMO.  We could alias
>> `list*` to nconc though, even better.
>
> oops, no we couldn't.  Same job, different calling convention.
> I just don't use either that much, so here again my suggestion
> is to just use what fits better with the style of surrounding code.

There is no reason to require a 21 KB file for a six line function.
None.  Yet some of our files do just that; surely this is a
consideration on equal footing with one's own perception of style?

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 10, 2023 at 12:03 AM Po Lu <luangruo@yahoo.com> wrote:
>

> There is no reason to require a 21 KB file for a six line function.
> None.  Yet some of our files do just that; surely this is a
> consideration on equal footing with one's own perception of style?

I said I agree with you.  So go there and replace cl-list* with nconc.
Or put it in a much smaller preloaded file, it doesn't have
dependencies.  Or just define it in terms of nconc if it's really equivalent.

But cl-lib usage and usefulness is really much more than that tiny aspect
you're focusing on.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> I said I agree with you.  So go there and replace cl-list* with nconc.
> Or put it in a much smaller preloaded file, it doesn't have
> dependencies.  Or just define it in terms of nconc if it's really
> equivalent.
>
> But cl-lib usage and usefulness is really much more than that tiny
> aspect you're focusing on.

cl-list* is just one example of a theme that constantly repeats itself
throughout our code.  Removing this one instance is futile, since
there's plenty more where it came from.  The only remedy is for everyone
to develop a general aversion to cl-lib.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 10, 2023 at 2:43 AM Po Lu <luangruo@yahoo.com> wrote:

> cl-list* is just one example of a theme that constantly repeats itself
> throughout our code.  Removing this one instance is futile,

You're just contradicting yourself.  First, you argue that it's easy
to replace and has these immediate benefits and then when someone agrees
with you, you say you don't want to.

Also btw, it might not be easy to replace at all cl-list* can be used
with higher order #'apply in contexts where nconc can't.  Perhaps
you despise the programmer who wrote

  (apply #'cl-list* input-data)

But perhaps they did it to avoid writing a multi-line raw loop choo choo
train again.

So it's not true what you said originally that everytime you find a
cl-list* it can be cleanly replaced by nconc.  Not if you understand
or even appreciate function composition.

> since there's plenty more where it came from.   The only remedy is
> for everyone to develop a general aversion to cl-lib.

Be consistent: find the places using simple forms of cl-list* that
presumably (as you said) are the only motivations for the whole
cl-lib.el 27kb loading.  I didn't check but this was your opening
argument after all.  So do that and you'll have effectively and
consensually helped Emacs's code base.  You'll be closer to your
dream goal, using Emacs without loading cl-lib.el.   Be consistent
with your words, else you're just hand-waving and advocating
for irrationality.  "develop a general aversion"??? I like to think
of programmers as scientists, not acolytes to some sect.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> You're just contradicting yourself.  First, you argue that it's easy
> to replace and has these immediate benefits and then when someone agrees
> with you, you say you don't want to.

I didn't say I don't want to, I said that removing this one instance
will not solve the underlying readiness to resort to _non-preloaded_
files, of which cl-lib is one, and an egregious offender.

> Also btw, it might not be easy to replace at all cl-list* can be used
> with higher order #'apply in contexts where nconc can't.  Perhaps
> you despise the programmer who wrote
>
>   (apply #'cl-list* input-data)
>
> But perhaps they did it to avoid writing a multi-line raw loop choo choo
> train again.
>
> So it's not true what you said originally that everytime you find a
> cl-list* it can be cleanly replaced by nconc.  Not if you understand
> or even appreciate function composition.

I don't appreciate "function combination," whatever that is.

In a language such as C, the function of a single block is clear.  Each
block commences with a list of one declarator for each variable.

Variables are supplied to and operated on by a list of arithmetic or
data operators and procedures, subject to control constructs all
programmers learn, and a value derived from one or more expressions
holding these variables is returned to complete the block's procedure.

Thus as data changes hands between operators, functions and variables,
its movement can be discerned with only a handful of operators and
control constructs in memory.  This is how a computer operates, or what
_you_ do when working out long division on a piece of paper, or when you
set yourself to any other task entailing the observation of a series of
steps while responding to changing circumstances in the process.

Emacs Lisp is well capable of aligning to such requirements as well;
today many would implement the old float-to-string with cl-loop and
floating point numbers with structs, but in the past they were
implemented with cons cells and while loops, without us being any the
worse for wear.

(defun float-to-string (fnum &optional sci)
  "Convert the floating point number to a decimal string.
Optional second argument non-nil means use scientific notation."
  (let* ((value (fabs fnum)) (sign (< (car fnum) 0))
	 (power 0) (result 0) (str "") 
	 (temp 0) (pow10 _f1))

    (if (f= fnum _f0)
	"0"
      (if (f>= value _f1)			; find largest power of 10 <= value
	  (progn				; value >= 1, power is positive
	    (while (f<= (setq temp (f* pow10 highest-power-of-10)) value)
	      (setq pow10 temp
		    power (+ power decimal-digits)))
	    (while (f<= (setq temp (f* pow10 _f10)) value)
	      (setq pow10 temp
		    power (1+ power))))
	(progn				; value < 1, power is negative
	  (while (f> (setq temp (f/ pow10 highest-power-of-10)) value)
	    (setq pow10 temp
		  power (- power decimal-digits)))
	  (while (f> pow10 value)
	    (setq pow10 (f/ pow10 _f10)
		  power (1- power)))))
					  ; get value in range 100000 to 999999
      (setq value (f* (f/ value pow10) all-decimal-digs-minval)
	    result (ftrunc value))
      (let (int)
	(if (f> (f- value result) _f1/2)	; round up if remainder > 0.5
	    (setq int (1+ (fint result)))
	  (setq int (fint result)))
	(setq str (int-to-string int))
	(if (>= int 1000000)
	    (setq power (1+ power))))

      (if sci				; scientific notation
	  (setq str (concat (substring str 0 1) "." (substring str 1)
			    "E" (int-to-string power)))

					  ; regular decimal string
	(cond ((>= power (1- decimal-digits))
					  ; large power, append zeroes
	       (let ((zeroes (- power decimal-digits)))
		 (while (natnump zeroes)
		   (setq str (concat str "0")
			 zeroes (1- zeroes)))))

					  ; negative power, prepend decimal
	      ((< power 0)		; point and zeroes
	       (let ((zeroes (- (- power) 2)))
		 (while (natnump zeroes)
		   (setq str (concat "0" str)
			 zeroes (1- zeroes)))
		 (setq str (concat "0." str))))

	      (t				; in range, insert decimal point
	       (setq str (concat
			  (substring str 0 (1+ power))
			  "."
			  (substring str (1+ power)))))))

      (if sign				; if negative, prepend minus sign
	  (concat "-" str)
	str))))

If it is to these that you apply the moniker "multi-line raw loop choo
choo trains," then they are precisely what should be written more, not
less.

The "Lisp way" (more felicitously the cl-lib way) is to provide
functions to truckloads of other functions which interface them with one
another.

Each such function is a unique control construct.  To glean what a
function does, one must not only commit each of these control constructs
to memory, but as is all too often the case when the functions provided
to the control constructs are beyond its control, also examine each of
its callers, with labyrinths that put C++ to shame into the bargain,
like pcase-lambda and macros which rewrite the control flow of a form
beyond recognition.

> Be consistent: find the places using simple forms of cl-list* that
> presumably (as you said) are the only motivations for the whole
> cl-lib.el 27kb loading.  I didn't check but this was your opening
> argument after all.  So do that and you'll have effectively and
> consensually helped Emacs's code base.  You'll be closer to your
> dream goal, using Emacs without loading cl-lib.el.   Be consistent
> with your words, else you're just hand-waving and advocating
> for irrationality.  "develop a general aversion"??? I like to think
> of programmers as scientists, not acolytes to some sect.

If such invective and an unwillingness to reason from the specific to
the general is the best response you have, I won't condescend to answer.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 10, 2023 at 12:52 PM Po Lu <luangruo@yahoo.com> wrote:

> > So it's not true what you said originally that everytime you find a
> > cl-list* it can be cleanly replaced by nconc.  Not if you understand
> > or even appreciate function composition.
>
> I don't appreciate "function combination," whatever that is.

Composition, not "combination" You find it in mathematics.
Pretty useful many contexts.  IMHO of course.

Anyway, I don't think this argument has any end.  You don't seem to
have any concept of information hiding, abstraction reuse, etc.
Nor do you seem to have any inclination to start actually acting
on that purported plan of yours, even where there is clear
agreement on specific parts of it.  So we don't have a common basis
for discussion in this matter.   That's fine, what can I say?
Go assembly, go machine code!

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> Composition, not "combination" You find it in mathematics.
> Pretty useful many contexts.  IMHO of course.
>
> Anyway, I don't think this argument has any end.  You don't seem to
> have any concept of information hiding, abstraction reuse, etc.

Yes, why should I chase chimeras, when I could as easily get down to
work?

To mention an example that is perhaps a tadge closer to home,
c-forward-decl-or-cast-1 is a function you would certainly advocate the
balkanization of.  But if it is ultimately divvied up into dozens of
private functions, of which it remains the only caller, what use will
the CC Mode user have gotten out of the process?  CC Mode will only have
become that much slower, and, where CC Mode is concerned, speed is like
gold dust.

> Nor do you seem to have any inclination to start actually acting
> on that purported plan of yours, even where there is clear
> agreement on specific parts of it.

Where's the agreement?

Re: What's missing in ELisp that makes people want to use cl-lib?

[Manuel Giraud via Emacs development discussions.]

Po Lu <luangruo@yahoo.com> writes:

[...]

> If it is to these that you apply the moniker "multi-line raw loop choo
> choo trains," then they are precisely what should be written more, not
> less.

Just my 2 cents, but here you have chosen the code for parsing a
floating representation which is, of course, full of peculiarities and
corner cases.  It would not be that much different should it be written
in « pure » CL.

But at the other end of the spectrum, there is lots of code which is
just: walk through this collection and do "this" if "that".
-- 
Manuel Giraud

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 10/11/2023 14:52, Po Lu wrote:
> (defun float-to-string (fnum &optional sci)
>    "Convert the floating point number to a decimal string.
> Optional second argument non-nil means use scientific notation."
>    (let* ((value (fabs fnum)) (sign (< (car fnum) 0))
> 	 (power 0) (result 0) (str "")
> 	 (temp 0) (pow10 _f1))
> 
>      (if (f= fnum _f0)
> 	"0"
>        (if (f>= value _f1)			; find largest power of 10 <= value
> 	  (progn				; value >= 1, power is positive
> 	    (while (f<= (setq temp (f* pow10 highest-power-of-10)) value)
 >      .............
>        (if sign				; if negative, prepend minus sign
> 	  (concat "-" str)
> 	str))))
> 
> If it is to these that you apply the moniker "multi-line raw loop choo
> choo trains," then they are precisely what should be written more, not
> less.

I don't have the corresponding high-level implementation to compare to, 
but the main thing about the code you posted, is that one has to, 
basically, read every line (of which there are ~70) to understand what 
the function is doing and/or to find any irregularities in its 
implementation (e.g. when looking for a bug).

> The "Lisp way" (more felicitously the cl-lib way) is to provide
> functions to truckloads of other functions which interface them with one
> another.

A high-level approach is to use a set of agreed-upon abstractions 
(functional programming in general, across languages, has a basic set, 
but there can be additions) to write programs shorter.

> Each such function is a unique control construct.  To glean what a
> function does, one must not only commit each of these control constructs
> to memory, but as is all too often the case when the functions provided
> to the control constructs are beyond its control, also examine each of
> its callers, with labyrinths that put C++ to shame into the bargain,
> like pcase-lambda and macros which rewrite the control flow of a form
> beyond recognition.

There is, indeed, a mental cost to understanding the code which uses 
them -- one has to know the abstractions and be able to connect the 
names with the concepts. There aren't too many of them in use, though. 
We can argue whether cl-lib is an optimal set, or could be reduced 
(there are indeed many functions and macros in there that I never saw 
the point of using). But your proposed goal (full eventual removal of 
cl-lib, or anything like it) is a non-starter from my POV.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

Dmitry Gutov <dmitry@gutov.dev> writes:

> I don't have the corresponding high-level implementation to compare
> to, but the main thing about the code you posted, is that one has to,
> basically, read every line (of which there are ~70) to understand what
> the function is doing and/or to find any irregularities in its
> implementation (e.g. when looking for a bug).

By contrast, in the archetypal cl-lib user, one must read every line,
the doc strings of every cl-lib function it calls, and the several lines
adjacent to each place where it is called.  To speak nothing of cl-lib
macros which recast a form's control flow into unrecognizability.

> A high-level approach is to use a set of agreed-upon abstractions
> (functional programming in general, across languages, has a basic set,
> but there can be additions) to write programs shorter.

In this respect functional programming can be likened to poetry--after
writing a witty, caustic epigram inveighing against some perceived
iniquity, a poet is always very pleased with himself.  But should he
apply his manner of writing to communication with real people, as by
speaking in epigrams before a clerk at an unemployment office, it is
likely he will starve.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 11/11/2023 03:11, Po Lu wrote:
>> A high-level approach is to use a set of agreed-upon abstractions
>> (functional programming in general, across languages, has a basic set,
>> but there can be additions) to write programs shorter.
> In this respect functional programming can be likened to poetry--after
> writing a witty, caustic epigram inveighing against some perceived
> iniquity, a poet is always very pleased with himself.  But should he
> apply his manner of writing to communication with real people, as by
> speaking in epigrams before a clerk at an unemployment office, it is
> likely he will starve.

Poets don't use a common intermediate language which can be defined, 
studied and use by anybody who has the inclination and time to 
understand it. And makes sentences shorter.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

Dmitry Gutov <dmitry@gutov.dev> writes:

> Poets don't use a common intermediate language which can be defined,
> studied and use by anybody who has the inclination and time to
> understand it. And makes sentences shorter.

Poetic license can be defined, studied and employed by anyone with a
sufficient impetus to do so, which is why it is analogous to functional
programming, or to any other vogue for a particular programming practice
that enjoys a spot in the limelight once in a while.  But such time and
inclination are always in short supply.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Po Lu wrote:

> In this respect functional programming can be likened to
> poetry [...]

Isn't conventional wisdom on the contrary that functional
programming principles make programs shorter and more modular,
so more re-usable and with less overhead reading and
understanding individual parts at the detail-level?

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

Emanuel Berg <incal@dataswamp.org> writes:

> Isn't conventional wisdom on the contrary that functional
> programming principles make programs shorter and more modular,
> so more re-usable and with less overhead reading and
> understanding individual parts at the detail-level?

Of course, much as conventional wisdom generally extols the virtues of
practices that are "in" at any given moment.  I prefer to set store by
my eyes.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Michael Heerdegen]

Po Lu <luangruo@yahoo.com> writes:

> I don't appreciate "function combination," whatever that is.
>
> In a language such as C, the function of a single block is clear.  Each
> block commences with a list of one declarator for each variable.
> [...]
> If it is to these that you apply the moniker "multi-line raw loop choo
> choo trains," then they are precisely what should be written more, not
> less.

Side note: I'm not a "fan" of cl-lib, and I hate cl-loop.  I hate it
when I need to understand code that uses it.

But this discussion has some aspects that I don't like - forcing other
programmers to share the own likes and ways of programming, "without
discussing each decision" although these decision have a very broad
reach, are backward incompatible and throw away the long-year work of
very committed and not unintelligent people.  Be aware that some things
in cl-lib are very well written and you'll need a very long time to to
develop a replacement of even approximate quality.  This discussion has
too much from "Personally I don't like this, let's just throw it all
away and start from scratch" that I only know from maybe a bit naive
people in emacs-help.

I understand most of your points of criticism.  A lot of parts of cl-lib
are not very useful for core Emacs, are harmful when used, should be
avoided as much as possible.  OTOH, there also are the useful things,
`cl-labels' (it had been mentioned as a main example) is definitely
among them for me.  Simple, well-defined and used to implement other
things...  I can't believe that it is an expectation that somebody's
personal preferences and likes should be the only guide of how to
proceed.

cl-lib started as emulation package for Common Lisp.  It was a huge work
to get it where it is now.  We should definitely _not_ simply throw away
this achievement.

I see that some parts of cl-lib are not well documented, and there are
problems when debugging it's stuff.  But honestly, other parts of Emacs
have the same problems.  And the Edebug support of cl stuff has also
improved in the last time - this is possible (this question only applies
to macros anyway).  FWIW, just as a random opinion, I found the original
sparse documentation of `pcase' very fine (and yes, I understood it),
so, please note, we are different people having a different background.
Please, let's discuss this with a bit tolerance and a bit less anger and
sarcasm.  It is an important decision.

My goal would be like this:

 - keep cl-lib available to it's current extent for end users

 - try to get unnecessary cl-lib uses out of the code base

 - make useful cl ideas available in the core language

And: one main problem we have is that cl-lib is more or less a huge
monolithic block (as a `require'able library).  Why not split it up?  I
don't see why it is necessary that when I want to use a small useful cl
thing that cl-loop must be made available.  Such things could go to its
own library, name it cl-obscure when you want.

But I don't see a necessity to, for example, kick keyword support out of
reach for the Emacs code base because, there are still, few, use cases
where it fits.  We don't want to end in a situation where we have 15
places that all reinvent the same wheel in subtly different ways.  This
would not be an improvement.  There is no need to reinvent at all.  This
stuff is well tested and integrated.  As long as the fundamental
guideline is clear that using such features is limited to spare
situations where it really makes things simpler, I don't see a problem,
apart maybe from the fact that cl-lib is big.  I would like us to try to
factor it when the size is a problem (is it a practical, objective, or
more or less mainly only a psychological problem btw? - that has
actually been one of my main questions when following this thread).


Michael.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Date: Sat, 11 Nov 2023 07:01:39 +0100
> 
> My goal would be like this:
> 
>  - keep cl-lib available to it's current extent for end users
> 
>  - try to get unnecessary cl-lib uses out of the code base
> 
>  - make useful cl ideas available in the core language

These are exactly our goals.  We've been doing this for years: that's
how stuff like 'when', 'unless', 'push', 'pop', and others got into
the core.

IOW, the way to make some cl-lib functionality freely available in
core is to have a similar or identical implementation in the likes of
subr.el or seq.el, using our naming convention instead of cl-FOO.
We've been doing that for many years, and we will continue doing it.
People just have to understand that this is the way.

The problem (at least my problem) is that some of the people in this
discussion don't agree with this cautious and slow approach.  They
want the unconstrained liberty of using all or most of cl-lib and its
extra libraries everywhere, and they consider any request to avoid
using these facilities an unacceptable pressure.  They basically want
cl-lib, cl-extra, etc. preloaded and always available for use,
regardless of the aspects you mention above and without any
restrictions.  So your argument is not with Po Lu, your argument is
with others.

> But I don't see a necessity to, for example, kick keyword support out of
> reach for the Emacs code base because, there are still, few, use cases
> where it fits.

We use keyword arguments when there are many of them, or when a few of
them are obscure and rarely used.  From where I stand, the dispute is
not about banishing then -- that will not happen -- the dispute is
about how _much_ to use them.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Sat, Nov 11, 2023 at 7:56 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Michael Heerdegen <michael_heerdegen@web.de>
> > Date: Sat, 11 Nov 2023 07:01:39 +0100
> >
> > My goal would be like this:
> >
> >  - keep cl-lib available to it's current extent for end users
> >
> >  - try to get unnecessary cl-lib uses out of the code base
> >
> >  - make useful cl ideas available in the core language
>
> These are exactly our goals.  We've been doing this for years: that's
> how stuff like 'when', 'unless', 'push', 'pop', and others got into
> the core.
>
> IOW, the way to make some cl-lib functionality freely available in
> core is to have a similar or identical implementation in the likes of
> subr.el or seq.el, using our naming convention instead of cl-FOO.

I generally agree with the goal of bringing in cl-lib.el ideas into
core, but it must be said here that seq.el, despite being a sequence
processing library, should NOT be viewed as a drop-in replacement
for cl-lib.el counterparts

seq.el strength over cl-lib.el is that it is a generic sequence
processing library.  IMO it _should_ be used if there is the suspicion
that the sequences being in that data being processed by a given
function can be custom seq.el sequences.  Or when there is good
reason to suspect that one they might be such custom sequences.

I think that seq.el _should_ be used in those situations because
it expresses exactly this fact, so it becomes self-documenting.

I think the argument that Gerd is making in his economic
contributions is that very often seq.el is overkill.

For example, in its current form should _not_ be used when
there is certainty that the sequences are Lisp listsm because as I
showed elsewhere here, seq-some is significantly slower than
cl-some and cl-loop when processing lots of small lists.

Also, I know you don't like cl-loop but in terms of efficiency
it's sometimes a very good choice.  For example, processing
maintain list in order without it often requires a nreverse
at the end or even more code.  And the fact that cl-loop doesn't
need to do dispatching on different sequence types means it's
faster.  And less polymorphic of course, but sometimes you
simply don't _need_ polymorphism.

I do agree fully that writing

  (cl-loop for i in l do (foo i))

is kind of uncalled _in any situation_ if you can just write

  (dolist (i l) (foo j))

But when you need to process plists, for example, it becomes
very handy.  This construct, which does require knowing a bit
of the mini-language that is cl-loop,

  (cl-loop for (k v) on plist by #'cddr collect (cons k v))

is still the most concise way IMHO of processing a plist
for example.  Of course that is debatable.  Maybe you
prefer a raw while loop with some auxiliary variables, but
while that is more universal, it doesn't necessarily mean
it's more readable.  These are two different things that
people are mixing in here.  I would be on the fence if this example
would justify introducing cl-lib.el into a file or not.
Depends on how often this plist-processing business is needed.
Or whether there is a pair-sequence-processing util in seq.el
AND performance is not at stake.

And certainly we should not immediately shoot down a contribution
to the core without "batting an eyelid", as someone suggested
here, just because it doesn't fit a stylistic preference.

By the way, I also think Michael's idea of splitting up
cl-lib.el further into smaller bits is very good.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> But when you need to process plists, for example, it becomes
> very handy.  This construct, which does require knowing a bit
> of the mini-language that is cl-loop,
>
>   (cl-loop for (k v) on plist by #'cddr collect (cons k v))
>
> is still the most concise way IMHO of processing a plist
> for example.  Of course that is debatable.  Maybe you
> prefer a raw while loop with some auxiliary variables, but
> while that is more universal, it doesn't necessarily mean
> it's more readable.  These are two different things that
> people are mixing in here.  I would be on the fence if this example
> would justify introducing cl-lib.el into a file or not.
> Depends on how often this plist-processing business is needed.
> Or whether there is a pair-sequence-processing util in seq.el
> AND performance is not at stake.

It might be pithy, but I don't know what "processing" is being done.  So
I cannot really judge it in terms of readability to the cl-loop
initiated.

Would everyone who posts passages of code using cl-loop please also
affix their macroexpansions, as, for want of a better word, a
reconciliatory gesture acknowledging the existence of those who are not
completely au fait with such matters?  HHOS...

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Sat, Nov 11, 2023 at 1:17 PM Po Lu <luangruo@yahoo.com> wrote:

> It might be pithy, but I don't know what "processing" is being done.  So
> I cannot really judge it in terms of readability to the cl-loop
> initiated.

It's converting a plist into an alist.  And I used it precisely
an example of non-universal code, that is  nevertheless concise.

No one asked you to "judge it".  You may if you so desire, write
your own while loop version (check your results this time!) and
then we can compare if that universal, primitive-exclusive code,
that results is more readable than the cl-loop version.  If you
weren't generally averse to "balkanization", we could even
collaborate in designing a good flexible, well-named utility for
this common operation (not just plist-alist conversion, but
more types of plist processing) to add to the Emacs core.

But alas, my dear dictionary-loving friend, this would imply
we have to leave this spirited yet vacuous exchange of
pompous elucubrations behind us.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> It's converting a plist into an alist.  And I used it precisely
> an example of non-universal code, that is  nevertheless concise.
>
> No one asked you to "judge it".

You're arguing in favor of it, and to repudiate such arguments the
measure that is being used to compare it with other postulated
alternatives must also be applied to these alternatives, correct?

In view of the incidence of the word "conciseness" within both your
follow-ups, that is the measure in question.  I think the existence of a
minimum of one person who does not quite follow the code at hand is
enough to outweigh any advantages it might hold, when this is the
alternative in question:

(let ((list '(:a b :c d)) (other nil))
  (while list
    (push (cons (car list) (cadr list)) other)
    (setq list (cddr list)))
  (nreverse other))

Using cl-lib for this kind of job, and expecting others to understand
it, is like a philosopher upholding the Tractatus Logico-Philosophicus
as a universal cynosure for reasoning and exhorting laymen to hold their
thought to it.  Bizarrely enough, I have been subject to both these
kinds of exhortations.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Sat, Nov 11, 2023 at 2:07 PM Po Lu <luangruo@yahoo.com> wrote:

> > we could even collaborate in designing [...]
> > But alas, my dear dictionary-loving friend, this would imply
> > we have to leave this spirited yet vacuous exchange of
> > pompous elucubrations behind us.

> Using cl-lib for this kind of job, and expecting others to understand
> it, is like a philosopher upholding the Tractatus Logico-Philosophicus
> as a universal cynosure for reasoning and exhorting laymen to hold their
> thought to it.  Bizarrely enough, I have been subject to both these
> kinds of exhortations.

I guess that answers my question!

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Sat, 11 Nov 2023 13:09:11 +0000
> Cc: Michael Heerdegen <michael_heerdegen@web.de>, emacs-devel@gnu.org
> 
> On Sat, Nov 11, 2023 at 7:56 AM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > >  - keep cl-lib available to it's current extent for end users
> > >
> > >  - try to get unnecessary cl-lib uses out of the code base
> > >
> > >  - make useful cl ideas available in the core language
> >
> > These are exactly our goals.  We've been doing this for years: that's
> > how stuff like 'when', 'unless', 'push', 'pop', and others got into
> > the core.
> >
> > IOW, the way to make some cl-lib functionality freely available in
> > core is to have a similar or identical implementation in the likes of
> > subr.el or seq.el, using our naming convention instead of cl-FOO.
> 
> I generally agree with the goal of bringing in cl-lib.el ideas into
> core, but it must be said here that seq.el, despite being a sequence
> processing library, should NOT be viewed as a drop-in replacement
> for cl-lib.el counterparts

That (and the rest of your message) completely misses my point, and
doesn't even belong to this discussion, to tell the truth.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Sat, Nov 11, 2023 at 1:54 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > I generally agree with the goal of bringing in cl-lib.el ideas into
> > core, but it must be said here that seq.el, despite being a sequence
> > processing library, should NOT be viewed as a drop-in replacement
> > for cl-lib.el counterparts
>
> That (and the rest of your message) completely misses my point, and
> doesn't even belong to this discussion, to tell the truth.

I think addressed your suggestion to have "similar or identical
implementation [to cl-lib.el] in the likes of [..] seq.el" quite
directly.  So I don't understand your criticism of my contribution,
but fair enough, I think I don't have much more to say on the matter
anyway.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Sat, 11 Nov 2023 14:01:24 +0000
> Cc: michael_heerdegen@web.de, emacs-devel@gnu.org
> 
> On Sat, Nov 11, 2023 at 1:54 PM Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > > I generally agree with the goal of bringing in cl-lib.el ideas into
> > > core, but it must be said here that seq.el, despite being a sequence
> > > processing library, should NOT be viewed as a drop-in replacement
> > > for cl-lib.el counterparts
> >
> > That (and the rest of your message) completely misses my point, and
> > doesn't even belong to this discussion, to tell the truth.
> 
> I think addressed your suggestion to have "similar or identical
> implementation [to cl-lib.el] in the likes of [..] seq.el" quite
> directly.

In my suggestion seq.el was just one of the examples not the only, nor
the main, package for such additions.  So making it the only one, and
then objecting only to that, is a kind of a strawman, and completely
misses my point.

I included seq.el because I see nothing wrong with adding
cl-defmethods to seq.el with more efficient implementations for
specific types of sequences.  Whether it does or doesn't make sense
depends on the specifics, which are not on the table at this time.  I
certainly don't agree with rejecting the idea of adding specific
implementations to seq.el, let alone its wholesale rejection, and I
likewise agree that in some cases adding a specific implementation to
subr.el or somesuch could be a better idea.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1289 bytes --]

On Sat, Nov 11, 2023, 14:09 Eli Zaretskii <eliz@gnu.org> wrote:

> In my suggestion seq.el was just one of the examples not the only, nor
> the main, package for such additions.  So making it the only one, and
> then objecting only to that, is a kind of a strawman, and completely
> misses my point.
>

I objected to that part because I generally agree with the rest!!! I even
said so early! Certainly pointing out a detail I think needs clarification
can't be seen as a strawman. Only if I was trying to magnify it
unreasonably to bring down your whole plan, which I am obviously not.

I included seq.el because I see nothing wrong with adding
> cl-defmethods to seq.el with more efficient implementations for
> specific types of sequences.  Whether it does or doesn't make sense
> depends on the specifics, which are not on the table at this time.  I
> certainly don't agree with rejecting the idea of adding specific
> implementations to seq.el, let alone its wholesale rejection,


I don't reject it either. I just wrote that seq.el can not be "viewed as a
drop-in replacement", which in common programming talk means we have to
look at each "similar implemention" case by case and be attentive to where
we use each, or to how we recommend its usage.

João

[-- Attachment #2: Type: text/html, Size: 1951 bytes --]

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Sat, 11 Nov 2023 14:22:19 +0000
> Cc: Michael Heerdegen <michael_heerdegen@web.de>, emacs-devel <emacs-devel@gnu.org>
> 
>  I included seq.el because I see nothing wrong with adding
>  cl-defmethods to seq.el with more efficient implementations for
>  specific types of sequences.  Whether it does or doesn't make sense
>  depends on the specifics, which are not on the table at this time.  I
>  certainly don't agree with rejecting the idea of adding specific
>  implementations to seq.el, let alone its wholesale rejection,
> 
> I don't reject it either. I just wrote that seq.el can not be "viewed as a drop-in replacement", which in
> common programming talk means we have to look at each "similar implemention" case by case and
> be attentive to where we use each, or to how we recommend its usage.

No one said that seq.el is a "drop-in replacement" for anything.  We
prefer it to be used where that makes sense, because seq.el is
preloaded.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Sat, Nov 11, 2023 at 2:25 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Sat, 11 Nov 2023 14:22:19 +0000
> > Cc: Michael Heerdegen <michael_heerdegen@web.de>, emacs-devel <emacs-devel@gnu.org>
> >
> >  I included seq.el because I see nothing wrong with adding
> >  cl-defmethods to seq.el with more efficient implementations for
> >  specific types of sequences.  Whether it does or doesn't make sense
> >  depends on the specifics, which are not on the table at this time.  I
> >  certainly don't agree with rejecting the idea of adding specific
> >  implementations to seq.el, let alone its wholesale rejection,
> >
> > I don't reject it either. I just wrote that seq.el can not be "viewed as a drop-in replacement", which in
> > common programming talk means we have to look at each "similar implemention" case by case and
> > be attentive to where we use each, or to how we recommend its usage.
>
> No one said that seq.el is a "drop-in replacement" for anything.

I didn't say you did!!  I just contributed that perspective of mine.

> We prefer it to be used where that makes sense, because seq.el is
> preloaded.

Very well, and I contributed the opinion that the reason it is
preloaded is not sufficient in itself.  It must, as you now more
clearly underline, "make sense".  And IMHO it makes _more_
sense the greater the likelihood that custom sequence types
-- a core seq.el design tenet -- may be be involved.  I pointed
out performance considerations and the self-documenting
nature of that usage to be two _additional_ factors to take
in consideration.

I think this is logical and doesn't really contradict anything
you or Michael said.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

We have heard from some programmers and maintainers that they
see the complexity increase as a problem. We have to believe
them, they think like that. And not everyone else has
to agree.

But I have not heard a lot what should be done about it.
Not generally, how to not have that tendency in the future.

And also not what to do in the particular case being
discussed, namely cl-lib, which is already in use to a huge
extent - indeed, that is exactly what is, according to them,
what has created the huge increase in complexity.

I have only heard improve the docstrings, and I think everyone
agree that is a good idea.

But what more? In general, and with respect to cl-lib in
particular? What is the plan, from the people who think it is
a problem?

-- 
underground experts united
https://dataswamp.org/~incal

RE: [External] : Re: What's missing in ELisp that makes people want to use cl-lib?

[Drew Adams]

> We have heard from some programmers and maintainers that they
> see the complexity increase as a problem. We have to believe
> them, they think like that. And not everyone else has
> to agree.
> 
> But I have not heard a lot what should be done about it.
> Not generally, how to not have that tendency in the future.

Michael suggested splitting `cl-lib.el'.
I seconded that, as did some others.
That's a first step to tackling and
taming the beast.

> And also not what to do in the particular case being
> discussed, namely cl-lib, which is already in use to a huge
> extent - indeed, that is exactly what is, according to them,
> what has created the huge increase in complexity.
> 
> I have only heard improve the docstrings, and I think everyone
> agree that is a good idea.
> 
> But what more? In general, and with respect to cl-lib in
> particular? What is the plan, from the people who think it is
> a problem?

See above.

Re: [External] : Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Drew Adams wrote:

>> We have heard from some programmers and maintainers that
>> they see the complexity increase as a problem. We have to
>> believe them, they think like that. And not everyone else
>> has to agree.
>> 
>> But I have not heard a lot what should be done about it.
>> Not generally, how to not have that tendency in the future.
>
> Michael suggested splitting `cl-lib.el'. I seconded that, as
> did some others. That's a first step to tackling and taming
> the beast.

But it is also very easy for a programmer to use it the way it
is, as soon as you type "cl-" you know you want to `require'
cl-lib in that file.

To me, I can't say I care if libraries are big or small.
If they are big, probably it means they contain a lot of
useful stuff so all the better.

As you know, cl-lib is already split up in different files, it
is just the require part that is always just cl-lib and to me
that is practical for all the cl- stuff, what is the harm
with it?

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

João Távora <joaotavora@gmail.com> writes:

> I think the argument that Gerd is making in his economic
> contributions is that very often seq.el is overkill.

Sorry for being economic. I've never been very talkative.

About seq (and map): (a) to each his own, (b) I find it personally
useless, for me, and (c) I find it a mistake to make it something like a
preferred interface.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Michael Heerdegen
>  <michael_heerdegen@web.de>,  emacs-devel@gnu.org
> Date: Sat, 11 Nov 2023 15:53:31 +0100
> 
> I find it a mistake to make it something like a preferred interface.

You, and everyone else, are entitled to their opinions, but we made
this decision intentionally, and it wasn't made lightly.  Before we
declare it a mistake, I'd expect us to wait a bit, until some
unequivocal data to support the opinions is available.

It isn't like we as a project never made any mistakes, but there's no
reason to assume this decision is one of them until we have ample
evidence.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

> It isn't like we as a project never made any mistakes, but there's no
> reason to assume this decision is one of them until we have ample
> evidence.

Seq is 10 years in Emacs, its polymorphism is unused in the tree. Joao
showed that it's slow. Dmitry mentioned that it is not a full
replacement for what is in cl-lib. Po Lu mentioned the poor doc strings.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: joaotavora@gmail.com,  michael_heerdegen@web.de,  emacs-devel@gnu.org
> Date: Sat, 11 Nov 2023 19:07:59 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > It isn't like we as a project never made any mistakes, but there's no
> > reason to assume this decision is one of them until we have ample
> > evidence.
> 
> Seq is 10 years in Emacs

It is preloaded only since a little more than a year ago.

> its polymorphism is unused in the tree.

Searching for seq-* in the tree brings more than 590 hits in more than
170 Lisp files.

> Joao showed that it's slow.

No, he didn't.

> Dmitry mentioned that it is not a full replacement for what is in
> cl-lib.

Indeed, it is not a full replacement, and was never meant to be.  I
believe its name says that much.

> Po Lu mentioned the poor doc strings.

That can be said about a lot of functions in Emacs, unfortunately.
Including cl-lib, btw.  Which doesn't preclude us from using any of
those poorly-documented functions.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Eli Zaretskii wrote:

>>> It isn't like we as a project never made any mistakes, but
>>> there's no reason to assume this decision is one of them
>>> until we have ample evidence.
>> 
>> Seq is 10 years in Emacs
>
> It is preloaded only since a little more than a year ago.

Why is it important again if something is preloaded or not?

What does it mean in terms of technology?

And apart from technology, is it also some kind of
political signal?

I must admit, both points are vague to me. We understand that
the increase in complexity of Elisp because of the use of
cl-lib is unrelated to when cl-lib is loaded.

Because one would have to assume it is loaded when used.
And if it isn't used, it cannot really be a cause of
increased complexity.

Are you keeping it out of some preloaded subset of Elisp?
Is that "the last stand" vs Common Lisp that Elisp is making?

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Sat, 11 Nov 2023 21:53:32 +0100
> 
> Eli Zaretskii wrote:
> 
> >>> It isn't like we as a project never made any mistakes, but
> >>> there's no reason to assume this decision is one of them
> >>> until we have ample evidence.
> >> 
> >> Seq is 10 years in Emacs
> >
> > It is preloaded only since a little more than a year ago.
> 
> Why is it important again if something is preloaded or not?
> 
> What does it mean in terms of technology?
> 
> And apart from technology, is it also some kind of
> political signal?

It means that we consider it useful and important enough to have it
always available, including in the code that gets dumped into the bare
vanilla un-customized Emacs.

> I must admit, both points are vague to me. We understand that
> the increase in complexity of Elisp because of the use of
> cl-lib is unrelated to when cl-lib is loaded.

You don't have to understand it.  The maintainers do, but you don't.
If needed, we will point out the implications if they are relevant to
the changes you submit to Emacs.

> Are you keeping it out of some preloaded subset of Elisp?
> Is that "the last stand" vs Common Lisp that Elisp is making?

Don't be ridiculous.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> I must admit, both points are vague to me. We understand that
>> the increase in complexity of Elisp because of the use of
>> cl-lib is unrelated to when cl-lib is loaded.
>
> You don't have to understand it.  The maintainers do, but you don't.
> If needed, we will point out the implications if they are relevant to
> the changes you submit to Emacs.

That is so arrogant, that I can't believe you wrote it.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Alan Mackenzie]

Hello, Gerd.

On Sun, Nov 12, 2023 at 08:22:27 +0100, Gerd Möllmann wrote:
> Eli Zaretskii <eliz@gnu.org> writes:

> >> I must admit, both points are vague to me. We understand that
> >> the increase in complexity of Elisp because of the use of
> >> cl-lib is unrelated to when cl-lib is loaded.

> > You don't have to understand it.  The maintainers do, but you don't.
> > If needed, we will point out the implications if they are relevant to
> > the changes you submit to Emacs.

> That is so arrogant, that I can't believe you wrote it.

You have to understand that Emanuel is not a regular Emacs hacker, and
many of the things he's written in this thread are not conducive to
reaching a conclusion.

In fact, he's been questioning how Eli's been doing his job.  There is
no need for this.  I think Eli is doing an excellent job in difficult
circumstances.

Also, there's a history here.  In the medium past, Emanuel posted quite
a few posts that were less than unhelpful, and he was sanctioned for
this.

So I think it's understandable that Eli finally lost patience with
Emanuel.  I think I would have done the same, only much sooner.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > So I think it's understandable that Eli finally lost patience with
  > Emanuel.  I think I would have done the same, only much sooner.

I managed to control my temper, partly because I came to the conclusion
Berg was trolling us.  One sign of this is that he challenges us
to convince him, but we don't need his agreement for what we do.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>> Cc: joaotavora@gmail.com,  michael_heerdegen@web.de,  emacs-devel@gnu.org
>> Date: Sat, 11 Nov 2023 19:07:59 +0100
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > It isn't like we as a project never made any mistakes, but there's no
>> > reason to assume this decision is one of them until we have ample
>> > evidence.
>> 
>> Seq is 10 years in Emacs
>
> It is preloaded only since a little more than a year ago.

You're saying that the real promotion of seq is only a year old?
And that things will "improve" once the promotion picks up speed?

>> its polymorphism is unused in the tree.
>
> Searching for seq-* in the tree brings more than 590 hits in more than
> 170 Lisp files.

And? The polymorphism isn't used.

>> Joao showed that it's slow.
>
> No, he didn't.

Aha.

>> Dmitry mentioned that it is not a full replacement for what is in
>> cl-lib.
>
> Indeed, it is not a full replacement, and was never meant to be.  I
> believe its name says that much.
>
>> Po Lu mentioned the poor doc strings.
>
> That can be said about a lot of functions in Emacs, unfortunately.
> Including cl-lib, btw.  Which doesn't preclude us from using any of
> those poorly-documented functions.

I guess people like Alan will like that. Not only will he have to cope
with another set of poor doc strings, he will also have to cope more and
more with a new interface, while the old one won't go away.

Anyway, I'm not fighting against seq/map, should someone get that
impression. Live and let live.

But until someone can be bothered to make concrete arguments why generic
functions at this level make sense at all, and especially at this basic
level of the language, I think this is all useless and a mistake.

(And yes this is intentially provocative :-). I've lost hope to hear
something concrete a while ago.)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: joaotavora@gmail.com,  michael_heerdegen@web.de,  emacs-devel@gnu.org
> Date: Sun, 12 Nov 2023 07:59:13 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Seq is 10 years in Emacs
> >
> > It is preloaded only since a little more than a year ago.
> 
> You're saying that the real promotion of seq is only a year old?

Yes.

> And that things will "improve" once the promotion picks up speed?

No.  I'm saying that we are still very far from a point where we have
enough data to decide whether that year-old decision was wrong or not.
Whether we see an improvement or not, time will say.

> >> its polymorphism is unused in the tree.
> >
> > Searching for seq-* in the tree brings more than 590 hits in more than
> > 170 Lisp files.
> 
> And? The polymorphism isn't used.

Then I guess I don't understand what you mean by "polymorphism".

> >> Joao showed that it's slow.
> >
> > No, he didn't.
> 
> Aha.

No, really.  What he showed is that seq.el is in most (though not all)
cases _slower_ than the corresponding cl-lib functions.  Sometimes
much slower, sometimes slightly slower (and in at least one case
faster).  But that doesn't mean seq.el is "slow", enough to make its
use nonsensical.  Because why should we care that some call takes 2
usec instead of just 0.5 usec? both are negligible.  The difference
will only become visible if someone needs to call these in a very
tight loop with a very large number of iterations.

IOW, "slower" is not the same as "slow".  If we cared about "slow", we
would have implemented everything in C.  We didn't because we have
"other considerations", which are important to us and outweigh "slow"
when "slow" is "fast enough".  Exactly like in the case of seq.el vs
cl-lib.  Those "other considerations" in the latter case were
abundantly described and explained up-thread, so I'm sure you know
what they are, even though you disagree.

> I've lost hope to hear something concrete a while ago.

Same here, sadly.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>> Cc: joaotavora@gmail.com,  michael_heerdegen@web.de,  emacs-devel@gnu.org
>> Date: Sun, 12 Nov 2023 07:59:13 +0100
>>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>> >> Seq is 10 years in Emacs
>> >
>> > It is preloaded only since a little more than a year ago.
>>
>> You're saying that the real promotion of seq is only a year old?
>
> Yes.
>
>> And that things will "improve" once the promotion picks up speed?
>
> No.  I'm saying that we are still very far from a point where we have
> enough data to decide whether that year-old decision was wrong or not.
> Whether we see an improvement or not, time will say.

I think here's once difference in our thinking: One of my main critiques
does not depend on time. Please see below

>> >> its polymorphism is unused in the tree.
>> >
>> > Searching for seq-* in the tree brings more than 590 hits in more than
>> > 170 Lisp files.
>>
>> And? The polymorphism isn't used.
>
> Then I guess I don't understand what you mean by "polymorphism".

C'mon :-)

This sub-thread started, when I told why I'm not using seq and will not.
https://lists.gnu.org/archive/html/emacs-devel/2023-11/msg00122.html
I said this (this is all I wrote):

  The reason I will not use seq.el or map.el in the forseeable future is
  quite simple: I haven't ever needed an abstraction over sequence types
  using generic functions, and I never have CPU cycles to give away for
  free.

Neither you nor Richard ever addressed the question why this
polymorphism is needed or even a Good Thing. Richard even read seq.el
and even had his first exposure to pcase etc., but more than pondering
if one could make generic function calls faster was not the result.

>> >> Joao showed that it's slow.
>> >
>> > No, he didn't.
>>
>> Aha.
>
> No, really.  What he showed is that seq.el is in most (though not all)
> cases _slower_ than the corresponding cl-lib functions.  Sometimes
> much slower, sometimes slightly slower (and in at least one case
> faster).  But that doesn't mean seq.el is "slow", enough to make its
> use nonsensical.  Because why should we care that some call takes 2
> usec instead of just 0.5 usec? both are negligible.  The difference
> will only become visible if someone needs to call these in a very
> tight loop with a very large number of iterations.
>
> IOW, "slower" is not the same as "slow".  If we cared about "slow", we
> would have implemented everything in C.  We didn't because we have
> "other considerations", which are important to us and outweigh "slow"
> when "slow" is "fast enough".  Exactly like in the case of seq.el vs
> cl-lib.  Those "other considerations" in the latter case were
> abundantly described and explained up-thread, so I'm sure you know
> what they are, even though you disagree.

I actually got your intention when I wrote aha.

>
>> I've lost hope to hear something concrete a while ago.
>
> Same here, sadly.

Aha.

I doubt that your, and Richard's, intention is to really communicate
over these issues. It has already been decided by Richard and you,
right? The rest is rabulistic. Slow is fast enough, time will show,
maintainers think this or that, it's preloaded, it's not concrete, what
is polymorphism, and so on, and so on. Conspiracy theory is still missing.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: joaotavora@gmail.com,  michael_heerdegen@web.de,  emacs-devel@gnu.org
> Date: Mon, 13 Nov 2023 07:28:54 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> And that things will "improve" once the promotion picks up speed?
> >
> > No.  I'm saying that we are still very far from a point where we have
> > enough data to decide whether that year-old decision was wrong or not.
> > Whether we see an improvement or not, time will say.
> 
> I think here's once difference in our thinking: One of my main critiques
> does not depend on time. Please see below

It is perfectly fine to apply different strategies of assessing Emacs
features, especially since no one can have expert knowledge of every
part in Emacs, and we probably differ in which parts of Emacs are more
familiar to each one of us.  E.g., if we were talking about changes in
bidi.c, I would probably be able to offer definitive opinions and
critiques right there and then.

> >> > Searching for seq-* in the tree brings more than 590 hits in more than
> >> > 170 Lisp files.
> >>
> >> And? The polymorphism isn't used.
> >
> > Then I guess I don't understand what you mean by "polymorphism".
> 
> C'mon :-)
> 
> This sub-thread started, when I told why I'm not using seq and will not.
> https://lists.gnu.org/archive/html/emacs-devel/2023-11/msg00122.html
> I said this (this is all I wrote):
> 
>   The reason I will not use seq.el or map.el in the forseeable future is
>   quite simple: I haven't ever needed an abstraction over sequence types
>   using generic functions, and I never have CPU cycles to give away for
>   free.

This doesn't explain why you say that polymorphism isn't used, it just
states your reasons for not using seq.el.  So I'm still in the dark
here, sorry.

> Neither you nor Richard ever addressed the question why this
> polymorphism is needed or even a Good Thing.

Maybe it isn't strictly necessary, but I don't see why it should be
rejected, once it is there.  Several contributors to Emacs whose
opinions I respect liked the way seq.el is implemented, and wanted to
use it, which is one reason why it is preloaded.

> I doubt that your, and Richard's, intention is to really communicate
> over these issues. It has already been decided by Richard and you,
> right? The rest is rabulistic. Slow is fast enough, time will show,
> maintainers think this or that, it's preloaded, it's not concrete, what
> is polymorphism, and so on, and so on. Conspiracy theory is still missing.

"Rabulistic"? really?  With anyone else I'd take offense.  With you,
I'm just infinitely puzzled, not to say astonished.  Whatever did I do
or say to deserve such denigration?  I do have an opinion on this
issue, but since when is it deemed improper to have an opinion, and
why sticking to that opinion as long as no argument is brought up that
makes me change my mind is considered a vice?  I provided technical
arguments which explain my position; you can disagree, of course, but
please don't treat them as something unworthy, let alone dishonest.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> > Then I guess I don't understand what you mean by "polymorphism".
>> 
>> C'mon :-)
>> 
>> This sub-thread started, when I told why I'm not using seq and will not.
>> https://lists.gnu.org/archive/html/emacs-devel/2023-11/msg00122.html
>> I said this (this is all I wrote):
>> 
>>   The reason I will not use seq.el or map.el in the forseeable future is
>>   quite simple: I haven't ever needed an abstraction over sequence types
>>   using generic functions, and I never have CPU cycles to give away for
>>   free.
>
> This doesn't explain why you say that polymorphism isn't used, it just
> states your reasons for not using seq.el.  So I'm still in the dark
> here, sorry.

Been there already when it was lighter::
  https://lists.gnu.org/archive/html/emacs-devel/2023-11/msg00123.html

You said:
  What prevents us from implementing in seq.el methods for specific
  types of sequences?  AFAIU, the only reason we didn't is because no
  one has yet came up with important situations where the generics in
  seq.el are not performant enough.

>> Neither you nor Richard ever addressed the question why this
>> polymorphism is needed or even a Good Thing.
>
> Maybe it isn't strictly necessary, but I don't see why it should be
> rejected, once it is there.  Several contributors to Emacs whose
> opinions I respect liked the way seq.el is implemented, and wanted to
> use it, which is one reason why it is preloaded.

I never talked about rejecting anything. I said at least twice something
to the effect of "to each his own".

>
>> I doubt that your, and Richard's, intention is to really communicate
>> over these issues. It has already been decided by Richard and you,
>> right? The rest is rabulistic. Slow is fast enough, time will show,
>> maintainers think this or that, it's preloaded, it's not concrete, what
>> is polymorphism, and so on, and so on. Conspiracy theory is still missing.
>
> "Rabulistic"? really? With anyone else I'd take offense. With you, I'm
> just infinitely puzzled, not to say astonished. Whatever did I do or
> say to deserve such denigration?

No personal offence intended. If I caused that, I apologize.

> I do have an opinion on this
> issue, but since when is it deemed improper to have an opinion, and
> why sticking to that opinion as long as no argument is brought up that
> makes me change my mind is considered a vice?  I provided technical
> arguments which explain my position; you can disagree, of course, but
> please don't treat them as something unworthy, let alone dishonest.

As far as I am noncerned, everyone is entitled to his opinion. I can't
stress enough: live and let live. Would saying you're like a piece of
soap under the shower for me be better?

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Neither you nor Richard ever addressed the question why this
  > polymorphism is needed or even a Good Thing.

I have no opinion on that question.  I don't want to argue that we
_need_ seq.el.  (I don't think we _need_ CL functions either.)

Rather, I am comparing these constructs about how much they complicate
or burden the Lisp language.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 11/11/2023 20:07, Gerd Möllmann wrote:
> Seq is 10 years in Emacs, its polymorphism is unused in the tree. Joao
> showed that it's slow. Dmitry mentioned that it is not a full
> replacement for what is in cl-lib.

FWIW, it doesn't seem to me that seq's dynamic dispatch is the main part 
of what makes it "slow" in a number of cases. cl-lib is not that 
well-optimized either.

Just recently (https://debbugs.gnu.org/66806#23) we tested both 
cl-set-difference and seq-difference, and the hand-rolled version was 
much faster.

I think both of these libs could use more performance-oriented 
contributions, and for most other uses (data structures smaller than in 
that example) their speed is largely comparable.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 2658 bytes --]

On Sat, Nov 11, 2023 at 9:49 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
>
> On 11/11/2023 20:07, Gerd Möllmann wrote:
> > Seq is 10 years in Emacs, its polymorphism is unused in the tree. Joao
> > showed that it's slow. Dmitry mentioned that it is not a full
> > replacement for what is in cl-lib.
>
> FWIW, it doesn't seem to me that seq's dynamic dispatch is the main part
> of what makes it "slow" in a number of cases.

Indeed I don't know if dynamic dispatch is the main culprit, but
my measurements show that calling cl-some lots of times on
small lists specifically yields much worse performance than cl-some.

In larger lists, the difference is attenuated.  So that's why
I conjectured that dynamic dispatch played a role, but I
didn't check.

Now that you mention set difference, I find this:

(when nil
  (let ((l2 '(4 5 6 7)))
    (benchmark-run 100000
      (let ((l (list 1 2 3 4 8 9 10 11 12 13 14)))
        (cl-set-difference l l2))))(0.5964884789999999 20 0.38926870999999996)


  (let ((l2 '(4 5 6 7)))
    (benchmark-run 100000
      (let ((l (list 1 2 3 4 8 9 10 11 12 13 14)))
        (cl-nset-difference l l2))))(0.594109028 20 0.37972229299999993)


  (let ((l2 '(4 5 6 7)))
    (benchmark-run 100000
      (let ((l (list 1 2 3 4 8 9 10 11 12 13 14)))
        (seq-difference l l2))))(3.087290361 102 2.020063915)

Eli thinks this doesn't really show seq.el is slower than cl-lib.el
and that may be half-true: it just shows how it is slower in this
micro-benchmark.

But you seem to have a real-world measurable case, which is
of course better, so I'm curious.

> cl-lib is not that well-optimized either.

That's certainly true.  I've been looking at the code and there
are some serious optimization opportunities in cl-lib.el.  The
one you found right away, destructive versions delegating to
overconsing non-destructive ones, but also other opportunities.

I attach a patch that I hope you can try with that bug report.
Very lightly tested here, but seems to show a measurable
improvement.

(when nil
  (let ((l2 '(4 5 6 7)))
    (benchmark-run 100000
      (let ((l (list 1 2 3 4 8 9 10 11 12 13 14)))
        (cl-set-difference l l2))));; (0.480944603 8 0.3380962310000015)

  (let ((l2 '(4 5 6 7)))
    (benchmark-run 100000
      (let ((l (list 1 2 3 4 8 9 10 11 12 13 14)))
        (cl-nset-difference l l2))));; (0.31953939800000003 5
0.21426147399999707)

  (let ((l2 '(4 5 6 7)))
    (benchmark-run 100000
      (let ((l (list 1 2 3 4 8 9 10 11 12 13 14)))
        (seq-difference l l2)))) ;; (2.3330953689999996 41 1.8175730390000027)
  )

João

[-- Attachment #2: 0001-WIP-optimize-cl-nset-difference.patch --]
[-- Type: text/x-patch, Size: 6288 bytes --]

From 205c7e1d9972645cdbdc2dcc85b0a235f337503e Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Jo=C3=A3o=20T=C3=A1vora?= <joaotavora@gmail.com>
Date: Sun, 12 Nov 2023 00:52:18 +0000
Subject: [PATCH] WIP: optimize cl-nset-difference

---
 lisp/emacs-lisp/bytecomp.el          |  2 +-
 lisp/emacs-lisp/cl-seq.el            | 58 ++++++++++++++++++++++++++--
 test/lisp/emacs-lisp/cl-lib-tests.el | 45 ++++++++++++++++++++-
 3 files changed, 100 insertions(+), 5 deletions(-)

diff --git a/lisp/emacs-lisp/bytecomp.el b/lisp/emacs-lisp/bytecomp.el
index cc68db73c9f..e1a11ff9ba3 100644
--- a/lisp/emacs-lisp/bytecomp.el
+++ b/lisp/emacs-lisp/bytecomp.el
@@ -3601,7 +3601,7 @@ byte-compile-form
          (cl-nsubst 3) (cl-nsubst-if 3) (cl-nsubst-if-not 3)
          (cl-nsubstitute 3) (cl-nsubstitute-if 3) (cl-nsubstitute-if-not 3)
          (cl-nsublis 2)
-         (cl-nunion 1 2) (cl-nintersection 1 2) (cl-nset-difference 1 2)
+         (cl-nunion 1 2) (cl-nintersection 1 2) (cl-nset-difference 1)
          (cl-nset-exclusive-or 1 2)
          (cl-nreconc 1)
          (cl-sort 1) (cl-stable-sort 1) (cl-merge 2 3)
diff --git a/lisp/emacs-lisp/cl-seq.el b/lisp/emacs-lisp/cl-seq.el
index 74ca74db679..55c9926d916 100644
--- a/lisp/emacs-lisp/cl-seq.el
+++ b/lisp/emacs-lisp/cl-seq.el
@@ -961,16 +961,68 @@ cl-set-difference
 	  (pop cl-list1))
         (nreverse cl-res)))))
 
+(defmacro cl--list-member-with-keys (e l key test test-not)
+  "Checks if E is in L with standard CL KEY, TEST and TEST-NOT.
+Anaphoric macro! requires cl-test, cl-test-not and cl-key to be bound"
+  `(let ((e ,e) (l ,l) (key ,key) (test ,test) (test-not ,test-not))
+     (cond (test-not
+            (cond (key
+                   (setq e (funcall key e))
+                   (catch 'done
+                     (while l
+                       (unless (funcall test-not e (funcall key (car l)))
+                         (throw 'done t))
+                       (setq l (cdr l)))))
+                  (t
+                   (catch 'done
+                     (while l
+                       (unless (funcall test-not e (car l))
+                         (throw 'done t))
+                       (setq l (cdr l)))))))
+           (test
+            (cond (key
+                   (setq e (funcall key e))
+                   (catch 'done
+                     (while l
+                       (when (funcall test e (funcall key (car l)))
+                         (throw 'done t))
+                       (setq l (cdr l)))))
+                  (t
+                   (catch 'done
+                     (while l
+                       (when (funcall test e (car l))
+                         (throw 'done t))
+                       (setq l (cdr l)))))))
+           (key
+            (setq e (funcall key e))
+            (catch 'done
+              (while l
+                (when (eq e (funcall key (car l)))
+                  (throw 'done t))
+                (setq l (cdr l)))))
+           (t (memql e l)))))
+
 ;;;###autoload
-(defun cl-nset-difference (cl-list1 cl-list2 &rest cl-keys)
+(defun cl-nset-difference (list1 list2 &rest cl-keys)
   "Combine LIST1 and LIST2 using a set-difference operation.
 The resulting list contains all items that appear in LIST1 but not LIST2.
 This is a destructive function; it reuses the storage of LIST1 and LIST2
 whenever possible.
 \nKeywords supported:  :test :test-not :key
 \n(fn LIST1 LIST2 [KEYWORD VALUE]...)"
-  (if (or (null cl-list1) (null cl-list2)) cl-list1
-    (apply 'cl-set-difference cl-list1 cl-list2 cl-keys)))
+  (if (or (null list1) (null list2)) list1
+    (cl--parsing-keywords (:key :test :test-not) ()
+      (let ((res nil))
+	(while (consp list1)
+	  (if ;; (if (or cl-keys (numberp (car list1)))
+	      ;;     (apply 'cl-member (cl--check-key (car list1))
+	      ;;   	 list2 cl-keys)
+	      ;;   (memq (car list1) list2))
+              (cl--list-member-with-keys (car list1)
+                                         list2 cl-key cl-test cl-test-not)
+              (setf list1 (cdr list1))
+	      (cl-shiftf list1 (cdr list1) res list1)))
+        res))))
 
 ;;;###autoload
 (defun cl-set-exclusive-or (cl-list1 cl-list2 &rest cl-keys)
diff --git a/test/lisp/emacs-lisp/cl-lib-tests.el b/test/lisp/emacs-lisp/cl-lib-tests.el
index 0995e71db4e..2a55d982c42 100644
--- a/test/lisp/emacs-lisp/cl-lib-tests.el
+++ b/test/lisp/emacs-lisp/cl-lib-tests.el
@@ -558,5 +558,48 @@ cl-constantly
   (should (equal (mapcar (cl-constantly 3) '(a b c d))
                  '(3 3 3 3))))
 
-
+(ert-deftest cl-set-difference ()
+  ;; our set-difference preserves order, though it is not required to
+  ;; by cl standards, better keep that invariant
+  (should (equal (cl-set-difference '(1 2 3 4) '(3 4 5 6))
+                 '(1 2))))
+
+(ert-deftest cl-nset-difference ()
+  ;; our nset-difference doesn't
+  (let* ((l1 (list 1 2 3 4)) (l2 '(3 4 5 6))
+         (diff (cl-nset-difference l1 l2)))
+    (should (memq 1 diff))
+    (should (memq 2 diff))
+    (should (= (length diff) 2))
+    (should (equal l2 '(3 4 5 6))))
+  (let* ((l1 (list '(a . 1) '(b . 2) '(c . 3) '(d . 4)))
+         (l2 (list '(c . 3) '(d . 4) '(e . 5) '(f . 6)))
+         (diff (cl-nset-difference l1 l2 :key #'car)))
+    (should (member '(a . 1) diff))
+    (should (member '(b . 2) diff))
+    (should (= (length diff) 2)))
+  (let* ((l1 (list '("a" . 1) '("b" . 2) '("c" . 3) '("d" . 4)))
+         (l2 (list '("c" . 3) '("d" . 4) '("e" . 5) '("f" . 6)))
+         (diff (cl-nset-difference l1 l2 :key #'car :test #'string=)))
+    (should (member '("a" . 1) diff))
+    (should (member '("b" . 2) diff))
+    (should (= (length diff) 2))))
+
+
+(when nil
+  (let ((l2 '(4 5 6 7)))
+    (benchmark-run 100000
+      (let ((l (list 1 2 3 4 8 9 10 11 12 13 14)))
+        (cl-set-difference l l2))));; (0.480944603 8 0.3380962310000015)
+
+  (let ((l2 '(4 5 6 7)))
+    (benchmark-run 100000
+      (let ((l (list 1 2 3 4 8 9 10 11 12 13 14)))
+        (cl-nset-difference l l2))));; (0.31953939800000003 5 0.21426147399999707)
+
+  (let ((l2 '(4 5 6 7)))
+    (benchmark-run 100000
+      (let ((l (list 1 2 3 4 8 9 10 11 12 13 14)))
+        (seq-difference l l2)))) ;; (2.3330953689999996 41 1.8175730390000027)
+  )
 ;;; cl-lib-tests.el ends here
-- 
2.39.2

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Sun, 12 Nov 2023 02:36:54 +0000
> Cc: Gerd Möllmann <gerd.moellmann@gmail.com>, 
> 	Eli Zaretskii <eliz@gnu.org>, michael_heerdegen@web.de, emacs-devel@gnu.org
> 
> Eli thinks this doesn't really show seq.el is slower than cl-lib.el

That's not what I said.  Please don't mis-quote me.  I sometimes make
mistakes of writing inaccurate statements, but not in this case.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Sun, Nov 12, 2023 at 6:27 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Sun, 12 Nov 2023 02:36:54 +0000
> > Cc: Gerd Möllmann <gerd.moellmann@gmail.com>,
> >       Eli Zaretskii <eliz@gnu.org>, michael_heerdegen@web.de, emacs-devel@gnu.org
> >
> > Eli thinks this doesn't really show seq.el is slower than cl-lib.el
>
> That's not what I said.  Please don't mis-quote me.  I sometimes make
> mistakes of writing inaccurate statements, but not in this case.

1. I posted some micro-benchmarks about cl-some and seq-some.
2. Gerd wrote "Joao showed that it's slow.".
3. You replied, directly below  "No, he didn't.".

Now you say I misquoted you.  Hope to have rectified that.

So you _do_ think this shows seq.el is slower than cl-lib.el?

Because, to be perfectly honest, I don't!  Not with such little data,
at least.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Sun, 12 Nov 2023 10:21:00 +0000
> Cc: dmitry@gutov.dev, gerd.moellmann@gmail.com, michael_heerdegen@web.de, 
> 	emacs-devel@gnu.org
> 
> On Sun, Nov 12, 2023 at 6:27 AM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > > Eli thinks this doesn't really show seq.el is slower than cl-lib.el
> >
> > That's not what I said.  Please don't mis-quote me.  I sometimes make
> > mistakes of writing inaccurate statements, but not in this case.
> 
> 1. I posted some micro-benchmarks about cl-some and seq-some.
> 2. Gerd wrote "Joao showed that it's slow.".
> 3. You replied, directly below  "No, he didn't.".
> 
> Now you say I misquoted you.  Hope to have rectified that.
> 
> So you _do_ think this shows seq.el is slower than cl-lib.el?

I believe I explained the issue in another message.  It has to do with
the (significant in this case) difference between "slower" and "slow".
I responded to "slow", whereas you said I disagreed with "slower".

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Sun, Nov 12, 2023 at 11:22 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > 1. I posted some micro-benchmarks about cl-some and seq-some.
> > 2. Gerd wrote "Joao showed that it's slow.".
> > 3. You replied, directly below  "No, he didn't.".
> >
> > Now you say I misquoted you.  Hope to have rectified that.
> >
> > So you _do_ think this shows seq.el is slower than cl-lib.el?
>
> I believe I explained the issue in another message.  It has to do with
> the (significant in this case) difference between "slower" and "slow".
> I responded to "slow", whereas you said I disagreed with "slower".

That's clear now, thanks.  I read that email as well.

I agree things can never be taken in absolute terms, and one
has to consider specific jobs.

However, I think it is even more important to note that to
have any useful statement regarding the efficiency  -- relative
or absolute -- of any library, that statement should be falsifiable.
For statements like "micro-benchmark X doesn't matter in the common
case" to be useful we need to characterize the "common case"
well, because if we don't they're not falsifiable.

So I hope that in this discussion we can eventually come up with
some characterization of this "common case". so that we can
design experiments around it. I think performance, despite not being
the sole one, is an important consideration in this discussion.

IME sequence-processing ability and efficiency does play a very
significant role in determining the performance of certain
Lisp programs (including Common Lisp, of course).  This is so
especially when garbage generation is involved.

In certain Common Lisp implementations with more advanced
multi-generational garbage collectors, the results are sometimes
counter-intuitive: generating lots of fast-to-scavenge garbage
ends up being better than long-lived slow-to-scavenge
garbage).  But in Elisp at least, it seems garbage is always bad.

Destructive versions of sequence-processing functions are
in that respect very useful.  cl-lib.el has them but it seems
many of them are missing low-hanging optimization opportunities.
However I worked on that in another patch and it seems to have
provided a good speedup (in micro-benchmarks).  Let's see what
Dmitry comes up with when he takes them to a real-world case.

> Sometimes much slower, sometimes slightly slower (and in at least
> one case faster).

Anyway, I took the "cl-some vs seq-some" measurements again, after
applying this simple patch that I attach after my sig.  The summary:

Small lists:
  cl-some fastest
  seq-some 5.7 times slower, garbage collects

Big lists
  cl-some fastest
  seq-some 1.6 times slower

Small vectors:
  cl-some fastest
  seq-some 5.13 times slower, garbage collects

Big vectors
  cl-some fastest
  seq-some 1.27 times slower

The "in one case faster" is gone.  Also interesting is the
garbage generation footprint of seq.el functions in their current
form.

I'm not sure we can easily dismiss the "small seqs/tight loop" 5x
slowdown that seq.el exhibits in this microbenchmark, because as you
can see in the benchmark detail there's a lot of garbage generation
involved.  So even if the tight loop were to go away, that garbage
would pile up.  I also think that tight sequence-processing loops
are not all that uncommon in Elisp programs.  Finally, a program
may not use these tight loops but do lots of different sequence
processing functions: if it selects slow abstractions for all of
them, it ends up being just as if there was a tight loop.

But as I wrote above  it is best if we find real-world applications
to benchmark the two approaches.  Maybe someone else reading this
can help find one or two.

João


First the benchmarks in full:

(require 'cl-lib)

(defun bench-seq-some (seq)
  (seq-some #'identity seq))

(defun bench-cl-some (seq)
  (cl-some #'identity seq))

(defun bench-cl-loop-list (l)
  ;; checks for some types of improper lists
  (cl-loop for e in l thereis (identity e)))

(defun bench-cl-loop-vec (v)
  (cl-loop for e across v thereis (identity e)))


(when nil
  ;; Small lists
  (let ((l (list nil nil nil nil t)))
    ;; FASTEST (0.23516409500000002 0 0.0)
    (benchmark-run 1000000 (bench-cl-some l)))


  (let ((l (list nil nil nil nil t)))
     ;; 5.7x SLOWER (1.338184149 16 0.8307866220000051)
    (benchmark-run 1000000 (bench-seq-some l)))


  (let ((l (list nil nil nil nil t)))
     ;; 1.14x SLOWER (0.26885113699999996 0 0.0)
    (benchmark-run 1000000 (bench-cl-loop-list l)))


  ;; Big lists
  (let ((l (make-list 10000000 nil)))
    ;; FASTEST (0.266716895 0 0.0)
    (benchmark-run 1 (bench-cl-some l)))


  (let ((l (make-list 10000000 nil)))
    ;; 1.6x SLOWER (0.428996694 0 0.0)
    (benchmark-run 1 (bench-seq-some l)))


  (let ((l (make-list 10000000 nil)))
    ;; 1.05x SLOWER (0.279309231 0 0.0)
    (benchmark-run 1 (bench-cl-loop-list l)))


  ;; Small vectors
  (let ((v (vector nil nil nil nil t)))
    ;; FASTEST (0.257238335 0 0.0)
    (benchmark-run 1000000 (bench-cl-some v)))


  (let ((v (vector nil nil nil nil t)))
    ;; 5.13x SLOWER (1.317641304 16 0.8448574659999935)
    (benchmark-run 1000000 (bench-seq-some v)))


  (let ((v (vector nil nil nil nil t)))
    ;; 1.14x SLOWER (0.29413928100000003 0 0.0)
    (benchmark-run 1000000 (bench-cl-loop-vec v)))


  ;; Big vectors
  (let ((v (make-vector 10000000 nil)))
    ;; FASTEST (0.316211001 0 0.0)
    (benchmark-run 1 (bench-cl-some v)))


  (let ((v (make-vector 10000000 nil)))
    ;; 1.27x SLOWER (0.40362057500000004 0 0.0)
    (benchmark-run 1 (bench-seq-some v)))


  (let ((v (make-vector 10000000 nil)))
    ;; 1.11x SLOWER
    (benchmark-run 1 (bench-cl-loop-vec v)))

  )

And here's the patch I used to make cl-some faster for
the common case of only one non-list sequence passed.

diff --git a/lisp/emacs-lisp/cl-extra.el b/lisp/emacs-lisp/cl-extra.el
index 2ca2d03170a..7c09328eda5 100644
--- a/lisp/emacs-lisp/cl-extra.el
+++ b/lisp/emacs-lisp/cl-extra.el
@@ -206,16 +206,26 @@ cl-some
 non-nil value.

 \n(fn PREDICATE SEQ...)"
-  (if (or cl-rest (nlistp cl-seq))
-      (catch 'cl-some
-        (apply #'cl-map nil
-               (lambda (&rest cl-x)
-                 (let ((cl-res (apply cl-pred cl-x)))
-                   (if cl-res (throw 'cl-some cl-res))))
-       cl-seq cl-rest) nil)
-    (let ((cl-x nil))
-      (while (and cl-seq (not (setq cl-x (funcall cl-pred (pop cl-seq))))))
-      cl-x)))
+  (cond
+   (cl-rest
+    (catch 'cl-some
+      (apply #'cl-map nil
+             (lambda (&rest cl-x)
+               (let ((cl-res (apply cl-pred cl-x)))
+                 (if cl-res (throw 'cl-some cl-res))))
+             cl-seq cl-rest) nil))
+   ((nlistp cl-seq)
+    (let ((cl-x nil)
+          (l (length cl-seq))
+          (i 0))
+      (while (and (< i l)
+                  (prog1 (not (setq cl-x (funcall cl-pred
+                                                  (aref cl-seq i))))
+                    (cl-incf i))))
+      cl-x))
+   (t (let ((cl-x nil))
+        (while (and cl-seq (not (setq cl-x (funcall cl-pred (pop cl-seq))))))
+        cl-x))))

 ;;;###autoload
 (defun cl-every (cl-pred cl-seq &rest cl-rest)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

João Távora <joaotavora@gmail.com> writes:

> In larger lists, the difference is attenuated.  So that's why
> I conjectured that dynamic dispatch played a role, but I
> didn't check.

Also interesting could be how/if performance changes once methods are
defined on generic functions. That would make a difference in PCL, for
instance, because the presence of more than one method would make a
dicriminating function necessary. That's the function dispatching to
methods based on argument types, for example.

Or it could be, of course, that Emacs's doesn't do optimizations on that
level at all.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 12/11/2023 04:36, João Távora wrote:
>> cl-lib is not that well-optimized either.
> That's certainly true.  I've been looking at the code and there
> are some serious optimization opportunities in cl-lib.el.  The
> one you found right away, destructive versions delegating to
> overconsing non-destructive ones, but also other opportunities.
> 
> I attach a patch that I hope you can try with that bug report.
> Very lightly tested here, but seems to show a measurable
> improvement.

Looks like a solid improvement:

(setq cc (all-completions "" obarray))
;; (length cc) => 46002

(defvar list2 '("abc" "abc" "abc" "abc" "abc" "abc"))

(benchmark-run 10 (cl-set-difference cc list2 :test 'equal))
;; => 0.430

(benchmark-run 10 (setq cc (cl-nset-difference cc list2 :test 'equal)))
;; => 0.137 (!)

(benchmark-run 10 (my/set-difference cc list2))
;; => 0.144

(benchmark-run 10 (seq-difference cc list2))
;; => 0.814

(defun my/set-difference (list1 list2)
   (delq 'wrong
         (mapcar
          (lambda (c) (if (member c list2)
                     'wrong
                   c))
          list1)))

So it seems on par with the hand-written version (byte-compiled, too) 
that hardcodes the test function.

Curiously, though, if I make list2 longer (e.g. 12 elements rather than 
6), then my/set-difference starts to win noticeably (0.166 vs 0.200).

I'll keep the real-world case as-is (code in project--vc-list-files) 
both because the result with cl-nset-difference is still a little 
slower, and because it's slower in all released emacsen, but it 
definitely becomes an option.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Sun, Nov 12, 2023 at 10:09 PM Dmitry Gutov <dmitry@gutov.dev> wrote:

> So it seems on par with the hand-written version (byte-compiled, too)
> that hardcodes the test function.
>
> Curiously, though, if I make list2 longer (e.g. 12 elements rather than
> 6), then my/set-difference starts to win noticeably (0.166 vs 0.200).

I have a suspicion, I'll see what I can do.

More importantly, there's an important takeaway here.  Your
results seem to show that regardless of the alternative (cl-lib
or hand-rolled) the solution with Emacs's current recommended
sequence processing library is nearly 6 times slower in a
real-world use-case.

Maybe seq.el can be made faster too?  Who knows, but it seems
difficult without breaking at least some of its defgeneric-based
contract.

> I'll keep the real-world case as-is (code in project--vc-list-files)
> both because the result with cl-nset-difference is still a little
> slower, and because it's slower in all released emacsen, but it
> definitely becomes an option.

Not that I think it's worth it just for this, but isn't cl-lib.el a
:core package?  Theoretically, we could bump the version and depend
on that in project.el, as we do with other such things when one
dependency of a core package is enhanced.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 13/11/2023 01:14, João Távora wrote:
> On Sun, Nov 12, 2023 at 10:09 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
> 
>> So it seems on par with the hand-written version (byte-compiled, too)
>> that hardcodes the test function.
>>
>> Curiously, though, if I make list2 longer (e.g. 12 elements rather than
>> 6), then my/set-difference starts to win noticeably (0.166 vs 0.200).
> 
> I have a suspicion, I'll see what I can do.
> 
> More importantly, there's an important takeaway here.  Your
> results seem to show that regardless of the alternative (cl-lib
> or hand-rolled) the solution with Emacs's current recommended
> sequence processing library is nearly 6 times slower in a
> real-world use-case.

Note that it's still the difference for the case where the "business 
logic" (the filtering predicate or whatever) doesn't do anything. 
Although certain order-of-magnitude differences are worrying.

> Maybe seq.el can be made faster too?  Who knows, but it seems
> difficult without breaking at least some of its defgeneric-based
> contract.

I was wondering whether you tried looking into seq.el's performance 
problems. It being slower on shorter lists is quite expected: if the 
type dispatch has non-negligible overhead, that should be most 
noticeable when the rest of the work is small.

The case with longer lists and other data structures should be possible 
to improve, though. As long as the type dispatch only happens once per 
sequence, and not for each element.

>> I'll keep the real-world case as-is (code in project--vc-list-files)
>> both because the result with cl-nset-difference is still a little
>> slower, and because it's slower in all released emacsen, but it
>> definitely becomes an option.
> 
> Not that I think it's worth it just for this, but isn't cl-lib.el a
> :core package?

It's not: the version in ELPA is a small backward-compatibility shim for 
older emacsen.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

Dmitry Gutov <dmitry@gutov.dev> writes:

>> More importantly, there's an important takeaway here.  Your
>> results seem to show that regardless of the alternative (cl-lib
>> or hand-rolled) the solution with Emacs's current recommended
>> sequence processing library is nearly 6 times slower in a
>> real-world use-case.
>
> Note that it's still the difference for the case where the "business
> logic" (the filtering predicate or whatever) doesn't do
> anything.

OK.  So can you provide an even more realistic case?

> Although certain order-of-magnitude differences are worrying.

You can say that again...  I optimized cl-nset-difference considerably
more.  In the process also found your handrolled version can be sped
up considerably.  Have a look at this:

   (defun joaot/handrolled-nset-difference (list1 list2)
     (if (or (null list1) (null list2)) list1
       (let ((res nil))
         (while (consp list1)
           (if (funcall #'member (car list1) list2)
               (setf list1 (cdr list1))
             (cl-shiftf list1 (cdr list1) res list1)))
         res)))
    
   (defun dmitry/set-difference (list1 list2)
      (delq 'wrong
            (mapcar
             (lambda (c) (if (member c list2)
                        'wrong
                      c))
             list1)))
    
   (setq cc (all-completions "" obarray))
   (setq list2 (make-list 12 "shooveedoowaa"))
    
   (when nil
     ;; FASTEST (0.074594068 0 0.0)
     (benchmark-run 10 (setq cc (joaot/handrolled-nset-difference cc list2)))
    
     ;; FASTEST (0.070370948 0 0.0)
     (benchmark-run 10 (setq cc (cl-nset-difference cc list2 :test #'equal)))
    
     ;; 1.8x SLOWER (0.138797637 1 0.06212087500000507)
     (benchmark-run 10 (cl-set-difference cc list2 :test #'equal))
    
     ;; 3.2x SLOWER (0.22628817199999998 2 0.13694317399999534)
     (benchmark-run 10 (dmitry/set-difference cc list2))
    
     ;; 18x SLOWER  (1.29373404 12 0.7763814810000014) 
     (benchmark-run 10 (seq-difference cc list2 #'equal))
    
     )

Yes, that's _eighteen_.

All my work, including the docstring overhauls Alan requested and some
new tests, now in branch feature/cl-lib-improvements.

>> Maybe seq.el can be made faster too?  Who knows, but it seems
>> difficult without breaking at least some of its defgeneric-based
>> contract.
>
> I was wondering whether you tried looking into seq.el's performance
> problems. It being slower on shorter lists is quite expected: if the
> type dispatch has non-negligible overhead, that should be most
> noticeable when the rest of the work is small.
>
> The case with longer lists and other data structures should be
> possible to improve, though. As long as the type dispatch only happens
> once per sequence, and not for each element.

Maybe it's possible.  But there are two things here: first, you need
non-destructive versions of things in seq.el, because consing is always
a killer.  Second, the generic function interface means the typical
shortcuts I applied in cl-lib.el are difficult.  Maybe even impossible
without breaking the current contract?  I don't know the contract well
enough to tell.  At any rate seems like non-trivial work, but I'm happy
if someone can give it a shot.

Another thing I noticed is that recently cl-lib.el started depending on
seql.el, in its implementation.  Given what I've been seeing, this tells
me there's more low-hanging fruit to optimize in cl-lib.el.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 13/11/2023 03:03, João Távora wrote:
> Dmitry Gutov <dmitry@gutov.dev> writes:
> 
>>> More importantly, there's an important takeaway here.  Your
>>> results seem to show that regardless of the alternative (cl-lib
>>> or hand-rolled) the solution with Emacs's current recommended
>>> sequence processing library is nearly 6 times slower in a
>>> real-world use-case.
>>
>> Note that it's still the difference for the case where the "business
>> logic" (the filtering predicate or whatever) doesn't do
>> anything.
> 
> OK.  So can you provide an even more realistic case?

Probably not. Not sure which big users of cl-lib we can find in the 
core: comp.el uses it, but mostly for the 'loop' macro.

Anyway, my point was these benchmarks are very good for improving each 
of the libs, but not necessarily for making the ultimate choice between 
them (if we really had to).

>> Although certain order-of-magnitude differences are worrying.
> 
> You can say that again...  I optimized cl-nset-difference considerably
> more.  In the process also found your handrolled version can be sped
> up considerably.  Have a look at this:
> 
>     (defun joaot/handrolled-nset-difference (list1 list2)
>       (if (or (null list1) (null list2)) list1
>         (let ((res nil))
>           (while (consp list1)
>             (if (funcall #'member (car list1) list2)
>                 (setf list1 (cdr list1))
>               (cl-shiftf list1 (cdr list1) res list1)))
>           res)))
>      
>     (defun dmitry/set-difference (list1 list2)
>        (delq 'wrong
>              (mapcar
>               (lambda (c) (if (member c list2)
>                          'wrong
>                        c))
>               list1)))
>      
>     (setq cc (all-completions "" obarray))
>     (setq list2 (make-list 12 "shooveedoowaa"))
>      
>     (when nil
>       ;; FASTEST (0.074594068 0 0.0)
>       (benchmark-run 10 (setq cc (joaot/handrolled-nset-difference cc list2)))
>      
>       ;; FASTEST (0.070370948 0 0.0)
>       (benchmark-run 10 (setq cc (cl-nset-difference cc list2 :test #'equal)))
>      
>       ;; 1.8x SLOWER (0.138797637 1 0.06212087500000507)
>       (benchmark-run 10 (cl-set-difference cc list2 :test #'equal))
>      
>       ;; 3.2x SLOWER (0.22628817199999998 2 0.13694317399999534)
>       (benchmark-run 10 (dmitry/set-difference cc list2))
>      
>       ;; 18x SLOWER  (1.29373404 12 0.7763814810000014)
>       (benchmark-run 10 (seq-difference cc list2 #'equal))
>      
>       )

All right, time to roll out the big guns. For your attention, ladies and 
gentlemen, a version in pure, unadulterated Elisp, extracted from my 
unused patch of two weeks ago (https://debbugs.gnu.org/66806#17):

(defun dmitry/set-difference-nocons (list1 list2)
   (let (ref)
     (while (member (car list1) list2)
       (setq list1 (cdr list1)))
     (setq ref list1)
     (while ref
       (if (member (cadr ref) list2)
           (setcdr ref (cddr ref))
         (setq ref (cdr ref))))
     list1))

And the benchmarks (we're so fast, 100 iterations for stable numbers):

(when nil
   ;; (0.38175291299999997 0 0.0) 1.1X SLOWER
   (benchmark-run 100 (setq cc (joaot/handrolled-nset-difference cc list2)))

   ;; (0.9393577359999999 16 0.5063504589999965) NO COMMENTS
   (benchmark-run 100 (dmitry/set-difference cc list2))

   ;; (0.345876673 0 0.0) FASTEST (of course)
   (benchmark-run 100 (dmitry/set-difference-nocons cc list2))
   )

Anyway, it's pretty cool to have cl-nset-difference so "down to the metal".

Although I'm not sure if we should be worried that the funcall overhead 
makes basically no difference (e.g. if I inline the funcall in 
joaot/handrolled-nset-difference). I think it's been said that our 
funcalls are relatively slow.

> All my work, including the docstring overhauls Alan requested and some
> new tests, now in branch feature/cl-lib-improvements.

Nice.

>>> Maybe seq.el can be made faster too?  Who knows, but it seems
>>> difficult without breaking at least some of its defgeneric-based
>>> contract.
>>
>> I was wondering whether you tried looking into seq.el's performance
>> problems. It being slower on shorter lists is quite expected: if the
>> type dispatch has non-negligible overhead, that should be most
>> noticeable when the rest of the work is small.
>>
>> The case with longer lists and other data structures should be
>> possible to improve, though. As long as the type dispatch only happens
>> once per sequence, and not for each element.
> 
> Maybe it's possible.  But there are two things here: first, you need
> non-destructive versions of things in seq.el, because consing is always
> a killer.  Second, the generic function interface means the typical
> shortcuts I applied in cl-lib.el are difficult.  Maybe even impossible
> without breaking the current contract?  I don't know the contract well
> enough to tell.  At any rate seems like non-trivial work, but I'm happy
> if someone can give it a shot.

I hope someone will. And agree about destructive versions.

> Another thing I noticed is that recently cl-lib.el started depending on
> seql.el, in its implementation.  Given what I've been seeing, this tells
> me there's more low-hanging fruit to optimize in cl-lib.el.

Stefan moved a bunch of code there in 2019 (0e4dd67aae8b1003).

Good example to try and see if this actually made anything slower.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 4516 bytes --]

On 13/11/2023 04:43, Dmitry Gutov wrote:
>>> The case with longer lists and other data structures should be
>>> possible to improve, though. As long as the type dispatch only happens
>>> once per sequence, and not for each element.
>>
>> Maybe it's possible.  But there are two things here: first, you need
>> non-destructive versions of things in seq.el, because consing is always
>> a killer.  Second, the generic function interface means the typical
>> shortcuts I applied in cl-lib.el are difficult.  Maybe even impossible
>> without breaking the current contract?  I don't know the contract well
>> enough to tell.  At any rate seems like non-trivial work, but I'm happy
>> if someone can give it a shot.
> 
> I hope someone will. And agree about destructive versions.

Here's an experimental patch that makes seq-difference about as fast as 
your new improved non-destructive cl-set-difference. And some notes.

First of all, the type dispatch _does_ happen more than once per 
sequence in the current master. That doesn't seem to hurt much while the 
method is not specialized (only has the default implementation), but has 
impact as soon as the function gets additional method definitions.

Second, the seq-reduce implementations for the set functions don't seem 
optimal. So, check out the attached with the below continued benchmark:

(defun joaot/handrolled-nset-difference (list1 list2)
   (if (or (null list1) (null list2)) list1
     (let ((res nil))
       (while (consp list1)
         (if (funcall #'member (car list1) list2)
             (setf list1 (cdr list1))
           (cl-shiftf list1 (cdr list1) res list1)))
       res)))

(defun dmitry/set-difference-nocons (list1 list2)
   (let (ref)
     (while (member (car list1) list2)
       (setq list1 (cdr list1)))
     (setq ref list1)
     (while ref
       (if (member (cadr ref) list2)
           (setcdr ref (cddr ref))
         (setq ref (cdr ref))))
     list1))

(setq cc (all-completions "" obarray))
(setq list2 (make-list 12 "shooveedoowaa"))

(when nil
   ;; (0.38175291299999997 0 0.0)
   (benchmark-run 100 (setq cc (joaot/handrolled-nset-difference cc list2)))

   ;; (0.345876673 0 0.0)
   (benchmark-run 100 (dmitry/set-difference-nocons cc list2))

   ;; (1.2209711170000002 38 0.7669010760000001)
   (benchmark-run 100 (cl-set-difference cc list2 :test #'equal))

   ;; (2.10207541 67 1.410268502) NOT THE FASTEST
   (benchmark-run 100 (seq-difference cc list2))

   ;; (1.3434452970000001 33 0.7025866390000006)
   (benchmark-run 100 (seq-difference-2 cc list2))

   ;; (1.243865238 34 0.7060731869999994)
   (benchmark-run 100 (seq-difference-3 cc list2))
   )

seq-difference is the original implementation based on seq-reduce. It's 
much faster here, though, because of the change to seq-contains-p which 
teaches it to use 'member' when it can.

seq-difference-2 is an implementation that just switched to using 
seq-filter.

And seq-difference-3 is the one that makes sure the type dispatch 
happens only once (or twice), and not for every element in SEQUENCE1. 
For that, I defined a new generic seq-contains-pred which returns a 
function.

seq-difference-3 is the fastest among the last three and is about the 
speed of the cl-lib's variant.

The plot twist is that when I tried to extract the sequence type check 
into a separate method (see the commented out "cl-defmethod 
seq-contains-p" line), the performance of seq-difference and 
seq-difference-2 fell by an order of a magnitude (2s -> 9s). So it seems 
like using the new seq-contains-pred is the one way that would keep 
decent performance while supporting generic extensions.

The latter also means all current uses of seq-contains-p inside seq.el 
should be rewritten using seq-contains-pred.

As for seq-some, 1.27x or 1.6x slower for the identity predicate doesn't 
look as bad in comparison. Especially for an implementation this short 
and generic. It incurs an extra funcall in Lisp through the use in 
'seq-doseq', so that might be the cost. It should be easy to add a 
specialization for lists while still keeping the code shorter than 
cl-some, if one were so inclined. Would be cooler to find a more generic 
bottleneck like in the case above, but so far, no luck.

And in other interesting functions, cl-remove-if-not is about 4x faster 
than seq-filter in the best case (e.g. the list is not modified), but 
about the same in the worst case (when the last link is removed). There 
must be some shortcut there too which could be reproduced.

[-- Attachment #2: seq-difference.diff --]
[-- Type: text/x-patch, Size: 2979 bytes --]

diff --git a/lisp/emacs-lisp/seq.el b/lisp/emacs-lisp/seq.el
index 346250c1d35..6869f498e43 100644
--- a/lisp/emacs-lisp/seq.el
+++ b/lisp/emacs-lisp/seq.el
@@ -440,12 +440,43 @@ seq-contains
 (cl-defgeneric seq-contains-p (sequence elt &optional testfn)
   "Return non-nil if SEQUENCE contains an element \"equal\" to ELT.
 \"Equality\" is defined by the function TESTFN, which defaults to `equal'."
+  (cond
+   ((and (listp sequence) (or (null testfn) (eq testfn 'equal)))
+    (member elt sequence))
+   ((and (listp sequence) (eq testfn 'eq))
+    (memq elt sequence))
+   (t
     (catch 'seq--break
-      (seq-doseq (e sequence)
-        (let ((r (funcall (or testfn #'equal) e elt)))
-          (when r
-            (throw 'seq--break r))))
-      nil))
+    (seq-doseq (e sequence)
+      (let ((r (funcall (or testfn #'equal) e elt)))
+        (when r
+          (throw 'seq--break r))))
+    nil)))
+  )
+
+;; (cl-defmethod seq-contains-p ((sequence list) elt &optional testfn)
+;;   (cond
+;;    ((or (null testfn) (eq testfn 'equal))
+;;     (member elt sequence))
+;;    ((eq testfn 'eq)
+;;     (memq elt sequence))
+;;    (t
+;;     (cl-call-next-method))))
+
+(cl-defgeneric seq-contains-pred (_sequence &optional testfn)
+  (cond
+   ((or (null testfn) (eq testfn 'equal))
+    #'member)
+   ((eq testfn 'eq)
+    #'memq)
+   (t
+    (lambda (elt sequence)
+      (catch 'seq--break
+        (seq-doseq (e sequence)
+          (let ((r (funcall testfn e elt)))
+            (when r
+              (throw 'seq--break r))))
+        nil)))))
 
 (cl-defgeneric seq-set-equal-p (sequence1 sequence2 &optional testfn)
   "Return non-nil if SEQUENCE1 and SEQUENCE2 contain the same elements.
@@ -488,9 +519,10 @@ seq-positions
 (cl-defgeneric seq-uniq (sequence &optional testfn)
   "Return a list of the elements of SEQUENCE with duplicates removed.
 TESTFN is used to compare elements, and defaults to `equal'."
-  (let ((result '()))
+  (let ((result '())
+        (pred (seq-contains-pred testfn)))
     (seq-doseq (elt sequence)
-      (unless (seq-contains-p result elt testfn)
+      (unless (funcall pred elt result)
         (setq result (cons elt result))))
     (nreverse result)))
 
@@ -574,6 +606,20 @@ seq-difference
               (seq-reverse sequence1)
               '()))
 
+(cl-defgeneric seq-difference-2 (sequence1 sequence2 &optional testfn)
+  "Return list of all the elements that appear in SEQUENCE1 but not in SEQUENCE2.
+\"Equality\" of elements is defined by the function TESTFN, which
+defaults to `equal'."
+  (seq-filter
+   (lambda (elt) (not (seq-contains-p sequence2 elt testfn)))
+   sequence1))
+
+(cl-defgeneric seq-difference-3 (sequence1 sequence2 &optional testfn)
+  (let ((pred (seq-contains-pred sequence2 testfn)))
+    (seq-filter
+     (lambda (elt) (not (funcall pred elt sequence2)))
+     sequence1)))
+
 ;;;###autoload
 (cl-defgeneric seq-group-by (function sequence)
   "Apply FUNCTION to each element of SEQUENCE.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 3640 bytes --]

On Tue, Nov 14, 2023 at 12:41 AM Dmitry Gutov <dmitry@gutov.dev> wrote:

> On 13/11/2023 04:43, Dmitry Gutov wrote:
> >>> The case with longer lists and other data structures should be
> >>> possible to improve, though. As long as the type dispatch only happens
> >>> once per sequence, and not for each element.
> >>
> >> Maybe it's possible.  But there are two things here: first, you need
> >> non-destructive versions of things in seq.el, because consing is always
> >> a killer.  Second, the generic function interface means the typical
> >> shortcuts I applied in cl-lib.el are difficult.  Maybe even impossible
> >> without breaking the current contract?  I don't know the contract well
> >> enough to tell.  At any rate seems like non-trivial work, but I'm happy
> >> if someone can give it a shot.
> >
> > I hope someone will. And agree about destructive versions.
>
> Here's an experimental patch that makes seq-difference about as fast as
> your new improved non-destructive cl-set-difference. And some notes.


This is all interesting, until one ponders what happens if an existing
seq.el user somewhere has:

(cl-defmethod seq-contains-p ((seq my-voodoo-seq)
                              (elt (eql :secret-voodoo)) &optional _tesfn)
  (invoke-voodoo-priests seq))

making use of seq.el's support for abstract polymorphic sequences.

With seq.el 2.24 a seq-difference operation would consider this user's
method, with seq.el 2.24.dmitry (i.e. your fast seq-difference-3) it
simply won't.  This user's code is clearly broken.

But was the user allowed to do that in the first place?  If not,
why is seq-contains-p a public generic function?

Generic function protocols aren't just a bunch of generic functions
thrown together, they're also precise documentation as to how/when
the generics are called by the framework and what the user may
add to them, desirably making common things easy, difficult things
possible, and mistakes hard.  Normally the framework that calls
into generics isn't exposed to the user customizations, i.e.
it is made of regular defuns.  Of course, you seem to know this
as xref.el has such a practice.  But in seq.el, almost everything
is a generic function, even the entry points, somewhat bizarrely.

What I meant before is that these are non-trivial questions that
must be answered when embarking on these optimizations of seq.el.

So when I say it's non-trivial to optimize, it's not because of
figuring out if seq-filter or seq-reduce is better, or which shortcut
is faster, or if dispatch takes time, it's because if you have made
most everything public and customizable, you have offered an
extremely rich contract to users, so taking any shortcuts is
much more likely break it.

cl-lib.el is easy to optimize because the contract it offers
is mostly clear (it's straight out of the CL standard).

The case in the CL world against generic functions' performance
is often not that the dispatch is slow, but that
employing them too liberally makes framework optimization
hard, because you restrict yourself from optimization
opportunities.

seq.el is 10 years old.  Are people making custom sequences?
To what degree?  In any case, I think a tight contract should
clearly be written down as soon as possible, preferably with all
optimization opportunities like the ones you're making
planned out ahead first.  Maybe demote a bunch of these generic
functions to defuns, and make them internal.

Also as advice future set.el designers, do not just make
everything a generic fucntion, think in terms of protocols.

João

[-- Attachment #2: Type: text/html, Size: 4839 bytes --]

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 14/11/2023 04:27, João Távora wrote:
> 
> On Tue, Nov 14, 2023 at 12:41 AM Dmitry Gutov <dmitry@gutov.dev 
> <mailto:dmitry@gutov.dev>> wrote:
> 
>     On 13/11/2023 04:43, Dmitry Gutov wrote:
>      >>> The case with longer lists and other data structures should be
>      >>> possible to improve, though. As long as the type dispatch only
>     happens
>      >>> once per sequence, and not for each element.
>      >>
>      >> Maybe it's possible.  But there are two things here: first, you need
>      >> non-destructive versions of things in seq.el, because consing is
>     always
>      >> a killer.  Second, the generic function interface means the typical
>      >> shortcuts I applied in cl-lib.el are difficult.  Maybe even
>     impossible
>      >> without breaking the current contract?  I don't know the
>     contract well
>      >> enough to tell.  At any rate seems like non-trivial work, but
>     I'm happy
>      >> if someone can give it a shot.
>      >
>      > I hope someone will. And agree about destructive versions.
> 
>     Here's an experimental patch that makes seq-difference about as fast as
>     your new improved non-destructive cl-set-difference. And some notes.
> 
> 
> This is all interesting, until one ponders what happens if an existing
> seq.el user somewhere has:
> 
> (cl-defmethod seq-contains-p ((seq my-voodoo-seq)
>                                (elt (eql :secret-voodoo)) &optional _tesfn)
>    (invoke-voodoo-priests seq))
> 
> making use of seq.el's support for abstract polymorphic sequences.
> 
> With seq.el 2.24 a seq-difference operation would consider this user's
> method, with seq.el 2.24.dmitry (i.e. your fast seq-difference-3) it
> simply won't.  This user's code is clearly broken.
> 
> But was the user allowed to do that in the first place?  If not,
> why is seq-contains-p a public generic function?

It was allowed, and yes, indeed, it's a breaking change. So we should at 
least examine the existing public code out there and see whether such 
overrides exist.

> Generic function protocols aren't just a bunch of generic functions
> thrown together, they're also precise documentation as to how/when
> the generics are called by the framework and what the user may
> add to them, desirably making common things easy, difficult things
> possible, and mistakes hard.  Normally the framework that calls
> into generics isn't exposed to the user customizations, i.e.
> it is made of regular defuns.  Of course, you seem to know this
> as xref.el has such a practice.  But in seq.el, almost everything
> is a generic function, even the entry points, somewhat bizarrely.

I'm not sure that is a problem in itself, except in the cases like I've 
described, where the type dispatch ends up happening N times instead of 
just once.

> What I meant before is that these are non-trivial questions that
> must be answered when embarking on these optimizations of seq.el.
> 
> So when I say it's non-trivial to optimize, it's not because of
> figuring out if seq-filter or seq-reduce is better, or which shortcut
> is faster, or if dispatch takes time, it's because if you have made
> most everything public and customizable, you have offered an
> extremely rich contract to users, so taking any shortcuts is
> much more likely break it.
> 
> cl-lib.el is easy to optimize because the contract it offers
> is mostly clear (it's straight out of the CL standard).

cl-lib is pretty complex and alien from somebody with zero experience in 
it (and CL), and it's a lot more code, with different macros. So I'm not 
sure I'd call cl-lib easier overall.

> The case in the CL world against generic functions' performance
> is often not that the dispatch is slow, but that
> employing them too liberally makes framework optimization
> hard, because you restrict yourself from optimization
> opportunities.

I don't know if that's a major issue: just like cl-some has different 
branches, seq-some could have three different definitions. The 
approaches to optimization seem transferable, more or less.

> seq.el is 10 years old.  Are people making custom sequences?
> To what degree?

Personally I'd call it a code organization approach: instead of having 
type checks inside functions, you get the cl-generic's infrastructure to 
do it for you. There are some existing cl-defmethod's inside seq.el 
already. And their callers are taking advantage of specialized 
optimizations (e.g. seq-map -> mapcar, seq-do -> mapc). Specializations 
using 'member' and 'mapc' were missing, though (aside from 'seq-uniq').

seq-contains-pred should also have a defmethod for lists and the default 
impl for the rest.

> In any case, I think a tight contract should
> clearly be written down as soon as possible, preferably with all
> optimization opportunities like the ones you're making
> planned out ahead first.

If someone were to sit down and try to figure out the next optimization 
opportunity, the result could imply additional restrictions. Otherwise, 
I don't think we're going to be rewriting it from the ground up.

> Maybe demote a bunch of these generic
> functions to defuns, and make them internal.

I believe the actual value it provides is a list of implementations that 
are quite compact and as such easy to fix/extend/modify. And if cl-lib 
is handy for someone with CL experience, I'll guess that seq.el rings 
more familiar to the Clojure users.

Note that I'm not arguing in favor in one against the other, and the 
latter's still missing destructive versions. Which might also be harder 
to fit in a similar design even if one wanted to (though I'll welcome 
such attempts).

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 7585 bytes --]

[Replying to both of your mails here]

On Tue, Nov 14, 2023 Dmitry Gutov <dmitry@gutov.dev> wrote:

> And if the list has just 1 element (or zero?), the gap would be even
> larger.
>
> Only seeing a 1.6x difference here is a nice surprise, actually.

These two statements together seem contradictory.  Why would
you be happy to see a particular factor for a given
list length if you know the factor is unbounded in general
for small lists?

And it's only in the small lists case.  If I pass my own predicate to
both cl-set-difference and your best seq-difference-3

(defun myequal (a b)
  (equal a b))

(setq cc (make-list 12 "blabla"))
(setq list2 (make-list 12 "shooveedoowaa"))

(when nil
  ;; (0.106080265 4 0.03849865399999963)
  (benchmark-run 10000 (cl-set-difference cc list2 :test #'myequal))
  ;; (0.5504704210000001 39 0.394207416)
  (benchmark-run 10000 (seq-difference-3 cc list2 #'myequal))

   )

I get the 5.5x slowdown again (in one experiment, not shown,
I got a whopping 200x slowdown, but I can't reproduce it right now
from Emacs -Q, maybe I had some method on some crucial seq.el generic)

> > This is all interesting, until one ponders what happens if an existing
> > seq.el user somewhere has:
> >
> > (cl-defmethod seq-contains-p ((seq my-voodoo-seq)
> >                                (elt (eql :secret-voodoo)) &optional
_tesfn)
> >    (invoke-voodoo-priests seq))
> >
> > making use of seq.el's support for abstract polymorphic sequences.
> >
> > With seq.el 2.24 a seq-difference operation would consider this user's
> > method, with seq.el 2.24.dmitry (i.e. your fast seq-difference-3) it
> > simply won't.  This user's code is clearly broken.
> >
> > But was the user allowed to do that in the first place?  If not,
> > why is seq-contains-p a public generic function?
>
> It was allowed, and yes, indeed, it's a breaking change. So we should at
> least examine the existing public code out there and see whether such
> overrides exist.

Not so easy, I'm afraid.  seq-contains-p is not the only such generic,
it's just one I happened to spot first.  seq-do is a generic and in that
group mentioned in the sparse seq.el documentation about mandatory
customization, meaning presumably it is essential for custom sequences.

See https://github.com/search?q=%22%28cl-defmethod+seq-do%22&type=code for
a list of custom sequences that use it.

Your seq-difference-3 doesn't call seq-do like the original seq-difference
does, so the set difference operations for those custom sequences might
well be broken.  What's even more problematic is that it skips seq-do
entirely for certain predicates and not entirely for certain other
predicates.

And this is the type of headache that make-everything-a-gf designs
like seq.el's brings to developers trying to optimize it.

> > Generic function protocols aren't just a bunch of generic functions
> > thrown together, they're also precise documentation as to how/when
> > the generics are called by the framework and what the user may
> > add to them, desirably making common things easy, difficult things
> > possible, and mistakes hard.  Normally the framework that calls
> > into generics isn't exposed to the user customizations, i.e.
> > it is made of regular defuns.  Of course, you seem to know this
> > as xref.el has such a practice.  But in seq.el, almost everything
> > is a generic function, even the entry points, somewhat bizarrely.
>
> I'm not sure that is a problem in itself, except in the cases like I've
> described, where the type dispatch ends up happening N times instead of
> just once.

It's still bizarre.  Here's how I think: if an application is customizing
the entry point directly, why even call the entry point?  OK, so what if
a library is customizing the entry point?   It's presumably because that
library provides a data type for applications to instantiate and use.  But
if the library does that, all calling guarantees of other generic
functions are lost for, say, user method for subtypes of that data type
(or :around, or :after, etc).  So it's also nonsensical.  The only think
where it _could_ make a shred of sense is in a kind of "private library"
where
the application controls both the data type and knows exactly the shortcuts
taken.  But then "private library" is an oxymoron.

> > What I meant before is that these are non-trivial questions that
> > must be answered when embarking on these optimizations of seq.el.
> >
> > So when I say it's non-trivial to optimize, it's not because of
> > figuring out if seq-filter or seq-reduce is better, or which shortcut
> > is faster, or if dispatch takes time, it's because if you have made
> > most everything public and customizable, you have offered an
> > extremely rich contract to users, so taking any shortcuts is
> > much more likely break it.
> >
> > cl-lib.el is easy to optimize because the contract it offers
> > is mostly clear (it's straight out of the CL standard).
>
> cl-lib is pretty complex and alien from somebody with zero experience in
> it (and CL), and it's a lot more code, with different macros. So I'm not
> sure I'd call cl-lib easier overall.

Oh, don't get me wrong: cl-lib.el's implementation details are not pretty
and not easy, likely typical library code (though i've seen much much
worse).  What's fundamentally easier about optimizing cl-lib is that the
contract it offers is much, much more well specified.

Possibly because it comes as a result of careful design in committees of
very capable programmers in the 90's, before the CL winter, that were
already on to this class of difficulties and wanted to make a common
interface.

The interface they made for sequences is not fully generic, but there
_are_ a lot of different implementations for that interface, each CL
compiler has  one.  Most of the tricks, like what I did to optimize
cl-{n}set-difference  are standard stuff in the CL world.   There are even
reference LOOP implementations (likely much more performant than ours,
though possibly not compatible with some non-standard edge cases our
cl-loop has of which I know a couple).

> > The case in the CL world against generic functions' performance
> > is often not that the dispatch is slow, but that
> > employing them too liberally makes framework optimization
> > hard, because you restrict yourself from optimization
> > opportunities.
>
> I don't know if that's a major issue: just like cl-some has different
> branches, seq-some could have three different definitions. The
> approaches to optimization seem transferable, more or less.

Seq-some also calls into a lot of user customizable code, so it'll
suffer from the same class of problems.  Say, are you going to skip
the seq-do generic sometimes there as well? Maybe.  Depends on what
contract we want to uphold, and we have no idea.  At least I don't.

> I believe the actual value it provides is a list of implementations that
> are quite compact and as such easy to fix/extend/modify. And if cl-lib
> is handy for someone with CL experience, I'll guess that seq.el rings
> more familiar to the Clojure users.

Maybe in the superficial user interface, because of names etc.  Or
are you saying seq.el interface is extracted from Clojure's standard much
like cl-lib is from CL standard?  (Do Clojure generic functions work like
ours?)  Then by all means we should rush to study that contract closely,
to know what we can and can't do as optimizers.

João

[-- Attachment #2: Type: text/html, Size: 9378 bytes --]

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 14/11/2023 12:34, João Távora wrote:
> [Replying to both of your mails here]
> 
> On Tue, Nov 14, 2023 Dmitry Gutov <dmitry@gutov.dev 
> <mailto:dmitry@gutov.dev>> wrote:
> 
>  > And if the list has just 1 element (or zero?), the gap would be even
>  > larger.
>  >
>  > Only seeing a 1.6x difference here is a nice surprise, actually.
> 
> These two statements together seem contradictory.  Why would
> you be happy to see a particular factor for a given
> list length if you know the factor is unbounded in general
> for small lists?

I don't think it's unbounded, just high. And 12-element list is a nice 
size for this particular comparison. Programs using tiny list would in 
their majority be tiny as well, so it wouldn't matter.

Though of course there are exceptions: compilers, for example.

> And it's only in the small lists case.  If I pass my own predicate to
> both cl-set-difference and your best seq-difference-3
> 
> (defun myequal (a b)
>    (equal a b))
> 
> (setq cc (make-list 12 "blabla"))
> (setq list2 (make-list 12 "shooveedoowaa"))
> 
> (when nil
>    ;; (0.106080265 4 0.03849865399999963)
>    (benchmark-run 10000 (cl-set-difference cc list2 :test #'myequal))
>    ;; (0.5504704210000001 39 0.394207416)
>    (benchmark-run 10000 (seq-difference-3 cc list2 #'myequal))
> 
>     )
> 
> I get the 5.5x slowdown again (in one experiment, not shown,

Right. This is still for the small lists case. Here we suffer the 
dynamic dispatch at least twice, along with funcall indirection. Again, 
would be great to get these ratios down, but the main scenarios where I 
*did* worry about the performance of a sequence primitive, have always 
been large lists.

> I got a whopping 200x slowdown, but I can't reproduce it right now
> from Emacs -Q, maybe I had some method on some crucial seq.el generic)

That would be helpful to know as well.

>  > > This is all interesting, until one ponders what happens if an existing
>  > > seq.el user somewhere has:
>  > >
>  > > (cl-defmethod seq-contains-p ((seq my-voodoo-seq)
>  > >                                (elt (eql :secret-voodoo)) &optional 
> _tesfn)
>  > >    (invoke-voodoo-priests seq))
>  > >
>  > > making use of seq.el's support for abstract polymorphic sequences.
>  > >
>  > > With seq.el 2.24 a seq-difference operation would consider this user's
>  > > method, with seq.el 2.24.dmitry (i.e. your fast seq-difference-3) it
>  > > simply won't.  This user's code is clearly broken.
>  > >
>  > > But was the user allowed to do that in the first place?  If not,
>  > > why is seq-contains-p a public generic function?
>  >
>  > It was allowed, and yes, indeed, it's a breaking change. So we should at
>  > least examine the existing public code out there and see whether such
>  > overrides exist.
> 
> Not so easy, I'm afraid.  seq-contains-p is not the only such generic,
> it's just one I happened to spot first.  seq-do is a generic and in that
> group mentioned in the sparse seq.el documentation about mandatory
> customization, meaning presumably it is essential for custom sequences.
> 
> See https://github.com/search?q=%22%28cl-defmethod+seq-do%22&type=code 
> <https://github.com/search?q=%22%28cl-defmethod+seq-do%22&type=code> for
> a list of custom sequences that use it.
> 
> Your seq-difference-3 doesn't call seq-do like the original seq-difference
> does, so the set difference operations for those custom sequences might
> well be broken.  What's even more problematic is that it skips seq-do
> entirely for certain predicates and not entirely for certain other
> predicates.

The way I understand this, is any new sequence type has to implement 
seq-do. As soon as that happens, a lot of (probably all) sequence 
functions in seq.el start working on that type.

But the author is also free to provide specialized implementations for 
individual functions (usually for better performance) -- and those 
implementations don't have to use seq-do. Indeed, in most cases the 
optimization would involve cutting out the overhead that seq-do brings 
in the general case.

> It's still bizarre.  Here's how I think: if an application is customizing
> the entry point directly, why even call the entry point?  OK, so what if
> a library is customizing the entry point?   It's presumably because that
> library provides a data type for applications to instantiate and use.  But
> if the library does that, all calling guarantees of other generic
> functions are lost for, say, user method for subtypes of that data type
> (or :around, or :after, etc).  So it's also nonsensical.  The only think
> where it _could_ make a shred of sense is in a kind of "private library" 
> where
> the application controls both the data type and knows exactly the shortcuts
> taken.  But then "private library" is an oxymoron.

An application shouldn't be customizing separate entry points. Separate 
libs (providing new data types) can do that. E.g. if we have a lib 
providing lazy sequences, it can provide its own set of implementations 
for seq.el. And when doing that, make sure the resulting behavior is 
consistent between functions.

I haven't though much about the interaction with :around, :after, etc. 
Maybe they'll be fine, or maybe it would be better to avoid it. Look 
fine on the superficial level, though.

> Oh, don't get me wrong: cl-lib.el's implementation details are not pretty
> and not easy, likely typical library code (though i've seen much much
> worse).  What's fundamentally easier about optimizing cl-lib is that the
> contract it offers is much, much more well specified.
> 
> Possibly because it comes as a result of careful design in committees of
> very capable programmers in the 90's, before the CL winter, that were
> already on to this class of difficulties and wanted to make a common
> interface.
> > The interface they made for sequences is not fully generic, but there
> _are_ a lot of different implementations for that interface, each CL
> compiler has  one.  Most of the tricks, like what I did to optimize
> cl-{n}set-difference  are standard stuff in the CL world.   There are even
> reference LOOP implementations (likely much more performant than ours,
> though possibly not compatible with some non-standard edge cases our
> cl-loop has of which I know a couple).

I do believe it's helpful to have it around.

>  > > The case in the CL world against generic functions' performance
>  > > is often not that the dispatch is slow, but that
>  > > employing them too liberally makes framework optimization
>  > > hard, because you restrict yourself from optimization
>  > > opportunities.
>  >
>  > I don't know if that's a major issue: just like cl-some has different
>  > branches, seq-some could have three different definitions. The
>  > approaches to optimization seem transferable, more or less.
> 
> Seq-some also calls into a lot of user customizable code, so it'll
> suffer from the same class of problems.  Say, are you going to skip
> the seq-do generic sometimes there as well?

I believe so.

>  > I believe the actual value it provides is a list of implementations that
>  > are quite compact and as such easy to fix/extend/modify. And if cl-lib
>  > is handy for someone with CL experience, I'll guess that seq.el rings
>  > more familiar to the Clojure users.
> 
> Maybe in the superficial user interface, because of names etc.  Or
> are you saying seq.el interface is extracted from Clojure's standard much
> like cl-lib is from CL standard?  (Do Clojure generic functions work like
> ours?)  Then by all means we should rush to study that contract closely,
> to know what we can and can't do as optimizers.

It looks very much inspired, both in naming and in the implementation 
approach. But it's not a carbon copy (much farther from it than in 
cl-lib's example), and our VM is also quite different from JVM in 
performance characteristics.

If someone were to write up a direct comparison, that would be great, of 
course.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Tue, Nov 14, 2023 at 11:47 AM Dmitry Gutov <dmitry@gutov.dev> wrote:

> I don't think it's unbounded, just high. And 12-element list is a nice
> size for this particular comparison. Programs using tiny list would in
> their majority be tiny as well, so it wouldn't matter.
>
> Though of course there are exceptions: compilers, for example.

Yes, of course, of which there are many in Lisp: every macro
is one.  And most of the uses for set difference operations that I
see in core seem to be for relatively small lists.  Just because
for large sets (at least for large list2), lists are inherently
not good anyway because quadratic ;-)

> > (defun myequal (a b)
> >    (equal a b))
> >
> > (setq cc (make-list 12 "blabla"))
> > (setq list2 (make-list 12 "shooveedoowaa"))
> >
> > (when nil
> >    ;; (0.106080265 4 0.03849865399999963)
> >    (benchmark-run 10000 (cl-set-difference cc list2 :test #'myequal))
> >    ;; (0.5504704210000001 39 0.394207416)
> >    (benchmark-run 10000 (seq-difference-3 cc list2 #'myequal))
> >
> >     )
> >
> > I get the 5.5x slowdown again (in one experiment, not shown,
>
> Right. This is still for the small lists case.

Not sure.  I don't have time to test now, but the custom #'myequal
predicates negates your seq-do shortcut (which is probably illegal anyway,
but let's pretend it isn't).  If that shortcut is negated, things
become slower than in the pred=#'equal case, no?

> > Your seq-difference-3 doesn't call seq-do like the original seq-difference
> > does, so the set difference operations for those custom sequences might
> > well be broken.  What's even more problematic is that it skips seq-do
> > entirely for certain predicates and not entirely for certain other
> > predicates.
>
> The way I understand this, is any new sequence type has to implement
> seq-do. As soon as that happens, a lot of (probably all) sequence
> functions in seq.el start working on that type.

Right.  But seq-difference-3 doesn't call into seq-do anymore,
at least not always and not in the same way.  So today, _before_
your seq-difference "skips-the-seq-do" optimization, a given person's
set difference operations would work just fine for their custom
data type.  The person is happy and doesn't care about seq.el's
performance.

One day they upgrade seq.el which comes with your optimization to
seq-difference, and that code is mysteriously broken.  Worse  for some
equality predicates it is broken  in a given way, for other equality
predicates it's a different error.

Can't you see the fundamental problem?

> > the application controls both the data type and knows exactly the shortcuts
> > taken.  But then "private library" is an oxymoron.
>
> An application shouldn't be customizing separate entry points.

Right.  We agree.  It makes no sense.

> Separate
> libs (providing new data types) can do that. E.g. if we have a lib
> providing lazy sequences, it can provide its own set of implementations
> for seq.el. And when doing that, make sure the resulting behavior is
> consistent between functions.

Right, but then if they fail the customization of the entry point
generic like not calling the right generics at the right time, then
_other_  customizations of _other_ generics won't be called, or will
be called too much.  Just like in your seq-difference-3 optimization
the user code breaks.

> > cl-{n}set-difference  are standard stuff in the CL world.   There are even
> > reference LOOP implementations (likely much more performant than ours,
> > though possibly not compatible with some non-standard edge cases our
> > cl-loop has of which I know a couple).
>
> I do believe it's helpful to have it around.

What is?

> > Seq-some also calls into a lot of user customizable code, so it'll
> > suffer from the same class of problems.  Say, are you going to skip
> > the seq-do generic sometimes there as well?
>
> I believe so.

Then my seq-do methods will just stop being called.

> It looks very much inspired, both in naming and in the implementation
> approach. But it's not a carbon copy (much farther from it than in
> cl-lib's example), and our VM is also quite different from JVM in
> performance characteristics.

OK, but naming and implementation are exactly the two least
important things that don't matter much here.  What matters are the
interfaces, the contracts for the implementor's interface and user's
interface: which would specify exactly how those generic functions
are expected to cooperate with each other.  This is what has to be
written down in a generic protocol.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 14/11/2023 14:14, João Távora wrote:
>> The way I understand this, is any new sequence type has to implement
>> seq-do. As soon as that happens, a lot of (probably all) sequence
>> functions in seq.el start working on that type.
> Right.  But seq-difference-3 doesn't call into seq-do anymore,
> at least not always and not in the same way.  So today,_before_
> your seq-difference "skips-the-seq-do" optimization, a given person's
> set difference operations would work just fine for their custom
> data type.  The person is happy and doesn't care about seq.el's
> performance.

seq-difference-3 calls seq-filter which calls seq-map, which in turn 
delegates to seq-do (or a specialized implementation).

There is a bug in my prototype patch -- the absence of the type check 
inside seq-contains-pred. I think it should be a specialized impl for 
seq-contains-pred anyway (for when sequence is a list). Since 
seq-contains-pred is only called once, that shouldn't have any 
measurable effect on performance, aside from very tiny lists.

> One day they upgrade seq.el which comes with your optimization to
> seq-difference, and that code is mysteriously broken.  Worse  for some
> equality predicates it is broken  in a given way, for other equality
> predicates it's a different error.
> 
> Can't you see the fundamental problem?
> 
> Right, but then if they fail the customization of the entry point
> generic like not calling the right generics at the right time, then
> _other_   customizations of_other_  generics won't be called, or will
> be called too much.  Just like in your seq-difference-3 optimization
> the user code breaks.
>>> cl-{n}set-difference  are standard stuff in the CL world.   There are even
>>> reference LOOP implementations (likely much more performant than ours,
>>> though possibly not compatible with some non-standard edge cases our
>>> cl-loop has of which I know a couple).
>> I do believe it's helpful to have it around.
> What is?

cl-lib.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Tue, Nov 14, 2023 at 12:20 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
>
> On 14/11/2023 14:14, João Távora wrote:
> >> The way I understand this, is any new sequence type has to implement
> >> seq-do. As soon as that happens, a lot of (probably all) sequence
> >> functions in seq.el start working on that type.
> > Right.  But seq-difference-3 doesn't call into seq-do anymore,
> > at least not always and not in the same way.  So today,_before_
> > your seq-difference "skips-the-seq-do" optimization, a given person's
> > set difference operations would work just fine for their custom
> > data type.  The person is happy and doesn't care about seq.el's
> > performance.
>
> seq-difference-3 calls seq-filter which calls seq-map, which in turn
> delegates to seq-do

OK, but not always in the same ways as seq-difference did,
and not always in the same way for different predicates.
And and not for sequences of type sequence and list, so if I have
special method on seq-do for my FUNCTION argument, it will
not be called.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 793 bytes --]

On Tue, Nov 14, 2023 at 2:27 AM João Távora <joaotavora@gmail.com> wrote:

>
>
> The case in the CL world against generic functions' performance
> is often not that the dispatch is slow, but that
>


That said, it still matters, of course.  Here's your non-destructive
seq-difference-3 but with many small lists:

(setq cc (make-list 12 "blabla"))
(setq list2 (make-list 12 "shooveedoowaa"))

;; (4.225398191 31 2.3103362619999928)
(benchmark-run 1000000 (cl-set-difference cc list2 :test #'equal))

;; (6.959332908 56 4.235225837999991)
(benchmark-run 1000000 (seq-difference-3 cc list2))

And of course the destructive version still wins by 10x

;; (0.676608019 4 0.26997238299998116)
(benchmark-run 1000000 (cl-nset-difference cc list2 :test #'equal))

João

[-- Attachment #2: Type: text/html, Size: 1367 bytes --]

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 14/11/2023 04:44, João Távora wrote:
> 
> On Tue, Nov 14, 2023 at 2:27 AM João Távora <joaotavora@gmail.com 
> <mailto:joaotavora@gmail.com>> wrote:
> 
> 
> 
>     The case in the CL world against generic functions' performance
>     is often not that the dispatch is slow, but that
> 
> 
> That said, it still matters, of course.  Here's your non-destructive
> seq-difference-3 but with many small lists:
> 
> (setq cc (make-list 12 "blabla"))
> (setq list2 (make-list 12 "shooveedoowaa"))

> ;; (4.225398191 31 2.3103362619999928)
> (benchmark-run 1000000 (cl-set-difference cc list2 :test #'equal))
> 
> ;; (6.959332908 56 4.235225837999991)
> (benchmark-run 1000000 (seq-difference-3 cc list2))

And if the list has just 1 element (or zero?), the gap would be even 
larger. Yes, generic dispatch has a cost.

Only seeing a 1.6x difference here is a nice surprise, actually.

> And of course the destructive version still wins by 10x
> 
> ;; (0.676608019 4 0.26997238299998116)
> (benchmark-run 1000000 (cl-nset-difference cc list2 :test #'equal))

No doubt.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Michael Heerdegen]

João Távora <joaotavora@gmail.com> writes:

> This is all interesting, until one ponders what happens if an existing
> seq.el user somewhere has:
>
> (cl-defmethod seq-contains-p ((seq my-voodoo-seq)
>                               (elt (eql :secret-voodoo)) &optional _tesfn)
>   (invoke-voodoo-priests seq))
>
> making use of seq.el's support for abstract polymorphic sequences.
>
> With seq.el 2.24 a seq-difference operation would consider this user's
> method, with seq.el 2.24.dmitry (i.e. your fast seq-difference-3) it 
> simply won't.  This user's code is clearly broken.

This is something to keep in mind indeed.  But this problem only appears
when a new generic is added to seq.el.  Then the programmers defining
implementations of the generics needs to update definitions to include a
definition for the new generic - s?he probably wants to do that anyway.

We will surely not have to add a new generic very often.

> But was the user allowed to do that in the first place?  If not,
> why is seq-contains-p a public generic function?

We have lost our way a little more, because cl-lib would support your
my-voodoo-seq not very well.

Michael.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 14/11/2023 04:27, João Távora wrote:
> This is all interesting, until one ponders what happens if an existing
> seq.el user somewhere has:
> 
> (cl-defmethod seq-contains-p ((seq my-voodoo-seq)
>                                (elt (eql :secret-voodoo)) &optional _tesfn)
>    (invoke-voodoo-priests seq))
> 
> making use of seq.el's support for abstract polymorphic sequences.
> 
> With seq.el 2.24 a seq-difference operation would consider this user's
> method, with seq.el 2.24.dmitry (i.e. your fast seq-difference-3) it
> simply won't.  This user's code is clearly broken.
> 
> But was the user allowed to do that in the first place?  If not,
> why is seq-contains-p a public generic function?

Allow me to try to answer this again: special voodoo is probably not 
allowed. What the specialized methods are supposed to do is implement 
the same logic (described in the generic function's docstring) but in a 
more efficient manner.

So the author of the new data type (who works on the implementation of 
seq-do and specialized methods) is responsible for making sure the new 
definition is equivalent but faster.

This is not documented, but seems to hold for all the code in the 
package. If someone found a case where violating the above rule would be 
helpful for a certain practical purpose, we could discuss it.

Note that all the functions in seq.el are currently non-destructive or 
perhaps even "pure", so it's not like there could be something special 
done to the input data behind the scenes, aside from what's in the 
contract (data in + data out).

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Tue, Nov 14, 2023 at 11:11 PM Dmitry Gutov <dmitry@gutov.dev> wrote:

> Allow me to try to answer this again: special voodoo is probably not
> allowed. What the specialized methods are supposed to do is implement
> the same logic (described in the generic function's docstring) but in a
> more efficient manner.

But isn't my method just that? Let's say the methods's is simpler,
like this:

(cl-defmethod seq-contains-p ((seq my-voodoo-seq)
                                (elt (eql :secret-voodoo)) &optional _tesfn)
    ;; my-voodoo-seq always has :secret-voodoo.
    t)

Hard to be more efficient than that, right?  Then I could use

(seq-difference '(1 2 3 :secret-voodoo 5) (make-a-voodoo-seq))

And get any voodoo out.

> This is not documented,

You don't say? :-)  i think we should get started on documenting all
these ad-hoc rules (sorry but in this case they are quite literally
ad-hoc).  Better than nothing, I guess, since as to actually enforcing
them, I think that train has left the station a long time ago and is
now in a voodoo swamp in monkey island.

Anyway, I'll be back with more custom sequences soon.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 15/11/2023 01:52, João Távora wrote:
> On Tue, Nov 14, 2023 at 11:11 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
> 
>> Allow me to try to answer this again: special voodoo is probably not
>> allowed. What the specialized methods are supposed to do is implement
>> the same logic (described in the generic function's docstring) but in a
>> more efficient manner.
> 
> But isn't my method just that?

It's a good style, but any specific project's requirements might differ, 
and libraries often have to get defensive against bad practices, too.

> Let's say the methods's is simpler,
> like this:
> 
> (cl-defmethod seq-contains-p ((seq my-voodoo-seq)
>                                  (elt (eql :secret-voodoo)) &optional _tesfn)
>      ;; my-voodoo-seq always has :secret-voodoo.
>      t)
> 
> Hard to be more efficient than that, right?  Then I could use
> 
> (seq-difference '(1 2 3 :secret-voodoo 5) (make-a-voodoo-seq))
> 
> And get any voodoo out.

Okay... so a voodoo seq contains the :secret-voodoo keyword (maybe once, 
maybe several times, but not any other element).

Then it must have a seq-do implementation that yields :secret-voodoo one 
or several times. And then the seq-contains-pred implementation will do 
right by it (the default one, not the specialization for lists).

>> This is not documented,
> 
> You don't say? :-)  i think we should get started on documenting all
> these ad-hoc rules (sorry but in this case they are quite literally
> ad-hoc).  Better than nothing, I guess, since as to actually enforcing
> them, I think that train has left the station a long time ago and is
> now in a voodoo swamp in monkey island.

Not sure "enforcing them" is the right phrase: I wonder if there's ever 
been a change proposed to seq.el which would break them.

Anyway, documenting them sounds best. As soon as we can arrive on a 
phrasing that sounds clear to all interested parties.

> Anyway, I'll be back with more custom sequences soon.

Please do.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Wed, Nov 15, 2023 at 12:46 AM Dmitry Gutov <dmitry@gutov.dev> wrote:

> Then it must have a seq-do implementation that yields :secret-voodoo one
> or several times.

Not really.  Only if I use my seq in contexts where I need that
generic, and being second argument to seq-difference doesn't count.
So why should I bother implementing that when my code is working
fine today?

But say I did that seq-do, then what is the seq-contains-p generic good
for then?  Why some many redundant generics that the user learns the
hard way have to be consistent with each other?

> And then the seq-contains-pred implementation will do
> right by it (the default one, not the specialization for lists).
>
> >> This is not documented,
> >
> > You don't say? :-)  i think we should get started on documenting all
> > these ad-hoc rules (sorry but in this case they are quite literally
> > ad-hoc).  Better than nothing, I guess, since as to actually enforcing
> > them, I think that train has left the station a long time ago and is
> > now in a voodoo swamp in monkey island.
>
> Not sure "enforcing them" is the right phrase: I wonder if there's ever
> been a change proposed to seq.el which would break them.

I think it's pretty easy to do, especially if these optimziations
want going to go all the way to cl-lib's easier-to-get optimal
performance.  What about this?  Is this toy implementation of
compressed space-saving lists legal?

(cl-defmethod seq-do (function (l (head :m6-sparse)))
  (mapc (lambda (e)
          (if (eq e :double-o-seven)
              (progn (funcall function 0)
                     (funcall function 0)
                     (funcall function 7))
            (funcall function e)))
        (cdr l)))

(cl-defmethod seq-reverse ((l (head :m6-sparse)))
  (let (res)
    (seq-do (lambda (e) (push e res)) l)
    res))

(seq-difference '(:m6-sparse 1 2 :double-o-seven 4) '(7)) ;; => (1 2 0 0 4)

Again, not implementing all the generics, just the ones I need
to get my desired effect, which in this case is passing a
custom sequence type as a first argument to seq-difference.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Wed, Nov 15, 2023 at 1:07 AM João Távora <joaotavora@gmail.com> wrote:

> Not really.  Only if I use my seq in contexts where I need that
> generic, and being second argument to seq-difference doesn't count.
> So why should I bother implementing that when my code is working
> fine today?
>
> But say I did that seq-do, then what is the seq-contains-p generic good
> for then?

Let me answer that for you... It could be, maybe even likely, that the
code was trying to get out of seq-do for some types of operation, because in
the case of lazy lists, seq-do would mandate the list be completely expanded.

So, while the code works, you seq-difference-3 now calls seq-do on the
second argument, and kills my laziness, naughty naughty.

Actually for this point to make sense, it doesn't even have to be
a bizarre voodoo list.  The lazy list may just be expanded enough already
for me to know in seq-contains-p that there's a given element
there without having to call the generator for more list elements.

So seq-contains-p was indeed an optimization, in more or less
the sense you described, which is now lost when you tried to optimize
for bare Lisp sequences.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 15/11/2023 03:17, João Távora wrote:
> On Wed, Nov 15, 2023 at 1:07 AM João Távora <joaotavora@gmail.com> wrote:
> 
>> Not really.  Only if I use my seq in contexts where I need that
>> generic, and being second argument to seq-difference doesn't count.
>> So why should I bother implementing that when my code is working
>> fine today?
>>
>> But say I did that seq-do, then what is the seq-contains-p generic good
>> for then?
> 
> Let me answer that for you... It could be, maybe even likely, that the
> code was trying to get out of seq-do for some types of operation, because in
> the case of lazy lists, seq-do would mandate the list be completely expanded.
> 
> So, while the code works, you seq-difference-3 now calls seq-do on the
> second argument, and kills my laziness, naughty naughty.

Indeed, it might as well be that a type had a specialized fast 
implementation for seq-contains-p and doesn't have one for 
seq-contains-pred, and the switchover would tank the performance of its 
callers.

As long as the observable behavior stays the same (aside from 
performance), that seems fine. The type's author will need to implement 
seq-contains-pred. Hopefully such migrations will remain rare enough.

> Actually for this point to make sense, it doesn't even have to be
> a bizarre voodoo list.  The lazy list may just be expanded enough already
> for me to know in seq-contains-p that there's a given element
> there without having to call the generator for more list elements.
> 
> So seq-contains-p was indeed an optimization, in more or less
> the sense you described, which is now lost when you tried to optimize
> for bare Lisp sequences.

Right. The author will now need to optimize seq-contains-pred, getting 
even better performance than previously in return.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 15/11/2023 03:07, João Távora wrote:
> On Wed, Nov 15, 2023 at 12:46 AM Dmitry Gutov <dmitry@gutov.dev> wrote:
> 
>> Then it must have a seq-do implementation that yields :secret-voodoo one
>> or several times.
> 
> Not really.  Only if I use my seq in contexts where I need that
> generic, and being second argument to seq-difference doesn't count.
> So why should I bother implementing that when my code is working
> fine today?

Commentary at the top of seq.el says:

;; seq.el can be extended to support new type of sequences.  Here are
;; the generic functions that must be implemented by new seq types:
;; - `seq-elt'
;; - `seq-length'
;; - `seq-do'
;; - `seqp'
;; - `seq-subseq'
;; - `seq-into-sequence'
;; - `seq-copy'
;; - `seq-into'

> But say I did that seq-do, then what is the seq-contains-p generic good
> for then?  Why some many redundant generics that the user learns the
> hard way have to be consistent with each other?

So any new type gets a lot of functions implemented very easily, and 
could add specialized implementations for performance.

>> And then the seq-contains-pred implementation will do
>> right by it (the default one, not the specialization for lists).
>>
>>>> This is not documented,
>>>
>>> You don't say? :-)  i think we should get started on documenting all
>>> these ad-hoc rules (sorry but in this case they are quite literally
>>> ad-hoc).  Better than nothing, I guess, since as to actually enforcing
>>> them, I think that train has left the station a long time ago and is
>>> now in a voodoo swamp in monkey island.
>>
>> Not sure "enforcing them" is the right phrase: I wonder if there's ever
>> been a change proposed to seq.el which would break them.
> 
> I think it's pretty easy to do, especially if these optimziations
> want going to go all the way to cl-lib's easier-to-get optimal
> performance.  What about this?  Is this toy implementation of
> compressed space-saving lists legal?
> 
> (cl-defmethod seq-do (function (l (head :m6-sparse)))
>    (mapc (lambda (e)
>            (if (eq e :double-o-seven)
>                (progn (funcall function 0)
>                       (funcall function 0)
>                       (funcall function 7))
>              (funcall function e)))
>          (cdr l)))
> 
> (cl-defmethod seq-reverse ((l (head :m6-sparse)))
>    (let (res)
>      (seq-do (lambda (e) (push e res)) l)
>      res))
> 
> (seq-difference '(:m6-sparse 1 2 :double-o-seven 4) '(7)) ;; => (1 2 0 0 4)
> 
> Again, not implementing all the generics, just the ones I need
> to get my desired effect, which in this case is passing a
> custom sequence type as a first argument to seq-difference.

Very clever. I'm not thinking too well this time of night, but the first 
part of the reply in this email probably answers the question (there is 
a certain set of generics that must be implemeneted).

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

[replying to both subthreads here]

On Wed, Nov 15, 2023 at 1:28 AM Dmitry Gutov <dmitry@gutov.dev> wrote:

> > But say I did that seq-do, then what is the seq-contains-p generic good
> > for then?  Why some many redundant generics that the user learns the
> > hard way have to be consistent with each other?
> So any new type gets a lot of functions implemented very easily, and
> could add specialized implementations for performance.

Fair enough.  Except, as you noted, those optimizations can be
completely nullified overnight, when trying to optimize something
else.

> the switchover would tank the performance of its callers.
> As long as the observable behavior stays the same (aside from
> performance), that seems fine.

I don't think this is "fine" at all to tank user code performance,
not from any library, much less such a basic foundation library.

> > (seq-difference '(:m6-sparse 1 2 :double-o-seven 4) '(7)) ;; => (1 2 0 0 4)

> Very clever. I'm not thinking too well this time of night, but the first
> part of the reply in this email probably answers the question (there is
> a certain set of generics that must be implemeneted).

Alright, if you must have them, they're all very trivial and I don't
think it'll add much to your analysis.  Here's the full
compressed/sparse/lazy list impl:

(cl-defmethod seq-do (function (l (head :m6-sparse)))
  (mapc (lambda (e)
          (if (eq e :double-o-seven) (mapc function '(0 0 7))
            (funcall function e)))
        (cdr l)))

(cl-defmethod seq-reverse ((l (head :m6-sparse)))
  (let (res) (seq-do (lambda (e) (push e res)) l) res))

(cl-defmethod seq-elt ((l (head :m6-sparse)) n) (elt (nreverse
(seq-reverse l)) n))
(cl-defmethod seq-length ((l (head :m6-sparse))) (length (nreverse
(seq-reverse l))))
(cl-defmethod seqp ((_l (head :m6-sparse))) t)
(cl-defmethod seq-into-sequence ((l (head :m6-sparse))) (nreverse
(seq-reverse l)))
(cl-defmethod seq-subseq ((l (head :m6-sparse)) start &optional end)
  (cl-subseq (nreverse (seq-reverse l)) start end))
(cl-defmethod seq-copy ((l (head :m6-sparse))) (nreverse (seq-reverse l)))
(cl-defmethod seq-into ((l (head :m6-sparse)) type)
  (cl-call-next-method (nreverse (seq-reverse l)) type))

(defvar my-compressed-list '(:m6-sparse 1 2 :double-o-seven 4
:double-o-seven 9 10 11))

(seq-difference my-compressed-list '(7)) ;; => (1 2 0 0 4 0 0 9 10 11)
(seq-elt my-compressed-list 0)           ;; => 1
(seq-elt my-compressed-list 4)           ;; => 7
(seq-elt my-compressed-list 5)           ;; => 4
(seq-elt my-compressed-list 10)          ;; => 10
(seq-elt my-compressed-list 1)           ;; => 2
(seq-elt my-compressed-list 3)           ;; => 0
(seq-elt my-compressed-list 13)          ;; => nil
(seq-length my-compressed-list)          ;; => 12
(seqp my-compressed-list)                ;; => t
(seq-subseq my-compressed-list 2 5)      ;; => (0 0 7)
(seq-copy my-compressed-list)            ;; => (1 2 0 0 7 4 0 0 7 9 10 11)
(seq-into my-compressed-list 'vector)    ;; => [1 2 0 0 7 4 0 0 7 9 10 11]
(seq-into-sequence my-compressed-list)   ;; => (1 2 0 0 7 4 0 0 7 9 10 11)


;; non consing version
(cl-defmethod seq-elt ((l (head :m6-sparse)) n)
  (cl-loop for e in (cdr l)
           for diff = (- n i)
           while (cl-plusp diff)
           sum (if (eq e :double-o-seven) 3 1) into i
           finally return
           (cond ((eq e :double-o-seven) 0)
                 ((cl-minusp diff) (elt '(7 0 0) (- (1+ diff))))
                 ((zerop diff) e)
                 (t nil))))
;; non consing version
(cl-defmethod seq-length ((l (head :m6-sparse)))
  (cl-loop for e in (cdr l)
           sum (if (eq e :double-o-seven) 3 1)))

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 15/11/2023 14:14, João Távora wrote:
>>> But say I did that seq-do, then what is the seq-contains-p generic good
>>> for then?  Why some many redundant generics that the user learns the
>>> hard way have to be consistent with each other?
>> So any new type gets a lot of functions implemented very easily, and
>> could add specialized implementations for performance.
> Fair enough.  Except, as you noted, those optimizations can be
> completely nullified overnight, when trying to optimize something
> else.

I don't see where they would be "completely nullified". The gains seem 
consistent, even if they don't extend as much to certain scenarios, like 
custom test-fn. Even that one should be improved, though.

More importantly, I'm not seeing any meaningful regressions, in any 
related or different scenario.

>> the switchover would tank the performance of its callers.
>> As long as the observable behavior stays the same (aside from
>> performance), that seems fine.
> I don't think this is "fine" at all to tank user code performance,
> not from any library, much less such a basic foundation library.

If that is the only drawback we can find, it is an unavoidable one.

But that is the price one has to pay for correcting a design mistake. 
We're not going to do this every Tuesday.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Wed, Nov 15, 2023 at 2:05 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
>
> On 15/11/2023 14:14, João Távora wrote:
> >>> But say I did that seq-do, then what is the seq-contains-p generic good
> >>> for then?  Why some many redundant generics that the user learns the
> >>> hard way have to be consistent with each other?
> >> So any new type gets a lot of functions implemented very easily, and
> >> could add specialized implementations for performance.
> > Fair enough.  Except, as you noted, those optimizations can be
> > completely nullified overnight, when trying to optimize something
> > else.
>
> I don't see where they would be "completely nullified".

You wrote "tank" :-)  I was talking about those kinds of optimisations,
not the ones you're making.  What I meant is in order to optimize the
list/sequence/#equal/#eql/#eq case you tank the user's optimizations
to custom sequences.

> The gains seem
> consistent, even if they don't extend as much to certain scenarios, like
> custom test-fn.

In that particular corner, both your (the library dev's) optimisations
and the user optimisations are tanked, yes.

> Even that one should be improved, though.

The only way you are eventually going to get to equal performance
with cl-lib's fastest variants (but only the non-destructive, of course)
is if you start every seq.el entry point with sth like

   (if (sequence-p seq) (call-equivalent-cl-lib-version seq) ...)

But then I don't see how that fits in with the cl-defgeneric,
which does dispatching before that if.  The only way I see this
is to invent a new kind of specializer in cl-generic.el like
'definitely-not-sequence'.  There could be some other solution,
maybe Michael as a clue.

Anyway we do it, it's much easier than trying to wrangle entry-point
by entry-point by adding more generics and finding pitfalls like this
one about seq-contains-p.  And this I think, is what Michael is
suggesting: fair enough.

But of course we will be outlawing every extension like the :m6-sparse
example I gave.

And also, IMHO, we end up with a much poorer protocol in terms
of versatility (no find-by-key :key, no :from-end, no :start/:end).
But this part is not about performance, rather interface, and that's
not the topic of this subthread.

> If that is the only drawback we can find, it is an unavoidable one.

There is still the other example I gave you.  I think your only
way out is to outlaw my :m6-sparse extension.  You basically have
to outlaw everything (:around, :before, specializers) that
cl-defgeneric gives the user when it comes to the sequence types
for which you want to have acceptable performance.

And even outlaw more stuff.  For example, these generics even have
specializers on all mandatory arguments?

For example, why does seq-do have FUNCTION as a required argument???
It should be &optional or a &key, with a default value.  Which
cl-degeneric obviously supports. That way specializations on FUNCTION
become impossible,  or at least become much harder and there's less
risk of tanking user code.  Design mistake IMO.

> But that is the price one has to pay for correcting a design mistake.
> We're not going to do this every Tuesday.

Sure, but that price increases manyfold if we start suggesting
seq.el as a replacement for all your sequence processing needs.
We just shouldn't.

Why working on the :m6-sparse extension, I noticed Emacs becomes
noticeably slower, and I suspect that's because while I was editing,
all the seq  functions I was writing where being probed for
applicability, while core things like completion, buffer-switching
etc are calling seq-foo generics.

I find this performance aspect very bad.  Maybe it can be obviated,
but only if you drop the '(if (sequence-p seq)' bomb into seq.el
somehow.  I don't see how we can avoid that one.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

Sorry for all the typos, there's probably more but I found these:

> maybe Michael as a clue.
                ^^
"has"

> Why working on the :m6-sparse extension, I noticed Emacs becomes
  ^^^
"While"

> noticeably slower, and I suspect that's because while I was editing,
> all the seq  functions I was writing where being probed for
                                       ^^^^^
"were also"

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 15/11/2023 17:02, João Távora wrote:
> On Wed, Nov 15, 2023 at 2:05 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
>>
>> On 15/11/2023 14:14, João Távora wrote:
>>>>> But say I did that seq-do, then what is the seq-contains-p generic good
>>>>> for then?  Why some many redundant generics that the user learns the
>>>>> hard way have to be consistent with each other?
>>>> So any new type gets a lot of functions implemented very easily, and
>>>> could add specialized implementations for performance.
>>> Fair enough.  Except, as you noted, those optimizations can be
>>> completely nullified overnight, when trying to optimize something
>>> else.
>>
>> I don't see where they would be "completely nullified".
> 
> You wrote "tank" :-) I was talking about those kinds of optimisations,
> not the ones you're making.  What I meant is in order to optimize the
> list/sequence/#equal/#eql/#eq case you tank the user's optimizations
> to custom sequences.

Only until the author of the third-party data type updates their code to 
include the method seq-contains-pred.

And if that code is not inside Emacs, its time-to-market measures in 
days, not months or years.

>> Even that one should be improved, though.
> 
> The only way you are eventually going to get to equal performance
> with cl-lib's fastest variants (but only the non-destructive, of course)
> is if you start every seq.el entry point with sth like
> 
>     (if (sequence-p seq) (call-equivalent-cl-lib-version seq) ...)
> 
> But then I don't see how that fits in with the cl-defgeneric,
> which does dispatching before that if.  The only way I see this
> is to invent a new kind of specializer in cl-generic.el like
> 'definitely-not-sequence'.  There could be some other solution,
> maybe Michael as a clue.
> 
> Anyway we do it, it's much easier than trying to wrangle entry-point
> by entry-point by adding more generics and finding pitfalls like this
> one about seq-contains-p.  And this I think, is what Michael is
> suggesting: fair enough.

I don't immediately see that new generics are needed. But if someone 
finds a workable approach (with hard numbers), good.

> But of course we will be outlawing every extension like the :m6-sparse
> example I gave.
> 
> And also, IMHO, we end up with a much poorer protocol in terms
> of versatility (no find-by-key :key, no :from-end, no :start/:end).
> But this part is not about performance, rather interface, and that's
> not the topic of this subthread.

cl-lib is more flexible in one aspect (its additional keywords 
vocabulary which basically multiplies the provided interface by 100x), 
but it's more rigid in the data it works with.

>> If that is the only drawback we can find, it is an unavoidable one.
> 
> There is still the other example I gave you.  I think your only
> way out is to outlaw my :m6-sparse extension.  You basically have
> to outlaw everything (:around, :before, specializers) that
> cl-defgeneric gives the user when it comes to the sequence types
> for which you want to have acceptable performance.

Could you try to explain what I should find in the second example? What 
do you mean by "outlaw"? Does causing worse performance for a short time 
constitute "outlawing" something?

> And even outlaw more stuff.  For example, these generics even have
> specializers on all mandatory arguments?
> 
> For example, why does seq-do have FUNCTION as a required argument???

Because FUNCTION is applied to SEQUENCE?

> It should be &optional or a &key, with a default value.  Which
> cl-degeneric obviously supports. That way specializations on FUNCTION
> become impossible,  or at least become much harder and there's less
> risk of tanking user code.  Design mistake IMO.

I'm reasonably sure nobody expects function to be anything but a 
straight function (symbol or a lambda), because that's how 'seq-do' is 
used throughout the code.

>> But that is the price one has to pay for correcting a design mistake.
>> We're not going to do this every Tuesday.
> 
> Sure, but that price increases manyfold if we start suggesting
> seq.el as a replacement for all your sequence processing needs.

We can first fix the mistake and then go on to continue "suggesting it 
as a replacement". Or not.

I don't exactly see it that way, though. And you give an impression of 
arguing for the opposite: toward never using it at all.

> Why working on the :m6-sparse extension, I noticed Emacs becomes
> noticeably slower, and I suspect that's because while I was editing,
> all the seq  functions I was writing where being probed for
> applicability, while core things like completion, buffer-switching
> etc are calling seq-foo generics.

It could be helpful to do some profiling and see where the slowdown came 
from. Could it come exactly from the set operations?

> I find this performance aspect very bad.  Maybe it can be obviated,
> but only if you drop the '(if (sequence-p seq)' bomb into seq.el
> somehow.  I don't see how we can avoid that one.

I don't quite see the need. And it's unlikely to be of reliable help: my 
observations say that method dispatch simply becomes slower as soon as a 
generic function gets a second implementation. And that implementation 
might arrive from any third-party code.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Dmitry Gutov <dmitry@gutov.dev> writes:

> my observations say that method dispatch simply becomes slower as soon
> as a generic function gets a second implementation. And that
> implementation might arrive from any third-party code.

To me that observation makes a lot of sense, extrapolating from PCL. It
means that Emacs does an optimization that PCL also does. PCL being the
CLOS implementationof CMUCL/SBCL.

In the most general case, the function definition of a generic function
is a discriminating function which dispatches to effective method
functions based on actual argument types etc. Each effective method
function then calls the applicable method functions.

In the case that there is only 1 method, Emacs seems to arrange for that
method function to be used as function definition of the generic
function (I'm a bit simplifying here). So it avoids both using a
discrimating functions, and an effective method function.

In the case of more than one method, using a discrimating function
cannot possibly be avoided, but I'd expect that Emacs is still able to
avoid effective method functions. The emfs cannot possibly be avoided
when there is more than 1 applicable method.

Pretty impressive, actually. Hats off to Stefan!

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 15/11/2023 21:56, Gerd Möllmann wrote:
> In the case that there is only 1 method, Emacs seems to arrange for that
> method function to be used as function definition of the generic
> function (I'm a bit simplifying here). So it avoids both using a
> discrimating functions, and an effective method function.

That makes a lot of sense indeed: since the function slot can be 
modified directly (and wouldn't break any inline caches, for example), 
why not do that when there's only one definition.

The only downside is that the performance characteristics of callers can 
change unexpectedly as more definitions arrive (which the authors of the 
caller code might not have tested for).

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Wed, Nov 15, 2023 at 11:25 PM Dmitry Gutov <dmitry@gutov.dev> wrote:

> The only downside is that the performance characteristics of callers can
> change unexpectedly as more definitions arrive (which the authors of the
> caller code might not have tested for).

Yes, I think so too.  Expect more benchmarks soon.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

João Távora <joaotavora@gmail.com> writes:

> On Wed, Nov 15, 2023 at 11:25 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
>
>> The only downside is that the performance characteristics of callers can
>> change unexpectedly as more definitions arrive (which the authors of the
>> caller code might not have tested for).
>
> Yes, I think so too.  Expect more benchmarks soon.

It's probably not important, but since you mention benchmarks, let me
just quickly add that, in PCL, the way discriminating functions are
implemented, the algorithm for dispatching, can also vary, depending on
various things. I don't have the details in my head anymore.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 9237 bytes --]

Dmitry Gutov <dmitry@gutov.dev> writes:

> I don't immediately see that new generics are needed. But if someone
> finds a workable approach (with hard numbers), good.

See benchmarks.

>> But of course we will be outlawing every extension like the :m6-sparse
>> example I gave.
>> And also, IMHO, we end up with a much poorer protocol in terms
>> of versatility (no find-by-key :key, no :from-end, no :start/:end).
>> But this part is not about performance, rather interface, and that's
>> not the topic of this subthread.
>
> cl-lib is more flexible in one aspect (its additional keywords
> vocabulary which basically multiplies the provided interface by 100x),
> but it's more rigid in the data it works with.

Exactly.  It's the CL tradeoff, and it give you superior performance.

CL was never meant for custom sequences.  One should use seq.el for
that, and IMO, _only_ for that.

The seq.el tradeoff gives you inferior performance and poorer interface.

But it's not only a matter of performance.  Take the
'seq-remove-at-position' generic.  Presumably someone thought that
operationq was common enough to merit a separate entry point. In CL,
this very operation can be done (probably faster) with cl-remove using
keyword arguments.

But say you want to remove two elements at two consecutive positions?
Two calls with two slowdowns?  What about n positions?  cl-remove can do
all of that in the same call with :COUNT and :START for example.

If you want to cut that time down in seq.el you either have to introduce
'seq-remove-at-consecutive-positions' generic or pay what is probably a
tremedous price by using `seq-drop-while` which calls the seq-elt
generic in a hot loop.

And so, in general, the seq.el interface either bloats up, or is slow,
or is inherently inflexible.  I think seq.el can obviate any one of
these problems at best, but not two or three simultaneously.

cl-lib obviates all three of them at the cost of one drawback: custom
sequence support.

So in general, different tools, different jobs.

>>> If that is the only drawback we can find, it is an unavoidable one.

> Could you try to explain what I should find in the second example?
> What do you mean by "outlaw"? 

What I meant is that the only way to get the same performance out of
seq.el is to have early #'sequencep checks that bypass the generics
completely, and this makes custom sequences based on sequences
impossible.

> Does causing worse performance for a short time constitute "outlawing"
> something?

But you should also find in that "m6sparse" example that the logic is
broken -- not only in terms of performance -- until its author
implements seq-contains-pred.  So this is pure "Incompatible Lisp
changes" material (which I also think tanked performance should be btw.)

But in fact is is already broken by all the "list optimzations" in
seq.el. Optimizations, before yours, that caused whatever the contract
was to be violated.  To be able to use `seq-drop-while` for my m6sparse
sequence, I have to add implementations to all those generic entry
points, which is just akward.

>> And even outlaw more stuff.  For example, these generics even have
>> specializers on all mandatory arguments?
>> For example, why does seq-do have FUNCTION as a required argument???
>
> Because FUNCTION is applied to SEQUENCE?

>
>> It should be &optional or a &key, with a default value.  Which
>> cl-degeneric obviously supports. That way specializations on FUNCTION
>> become impossible,  or at least become much harder and there's less
>> risk of tanking user code.  Design mistake IMO.
>
> I'm reasonably sure nobody expects function to be anything but a
> straight function (symbol or a lambda), because that's how 'seq-do' is
> used throughout the code.

Yes, but putting as a required argument in the arglist means users can
specialize for it, and that isn't needed.  Not sure anyone does it or
even if that makes the generic even slower.

>>> But that is the price one has to pay for correcting a design mistake.
>>> We're not going to do this every Tuesday.
>> Sure, but that price increases manyfold if we start suggesting
>> seq.el as a replacement for all your sequence processing needs.
>
> We can first fix the mistake and then go on to continue "suggesting it
> as a replacement". Or not.
>
> I don't exactly see it that way, though. And you give an impression of
> arguing for the opposite: toward never using it at all.

Not at all.  Maybe you missed some of my previous messages.  I think
seq.el's support for polymorphic sequences, though a little bit flawed
in some respects, is very useful.  For example, I've been pondering
using it in eglot.el to try to speed up JSON parsing.  If some kind of
seq-plist-get and seq-plist-destructuring-bind can be designed, I might
be able to skip consing much of the useless elements of a gigantic JSON
blob and parse just the parts I need.  Of course, not easy, but I think
seq.el is the tool for that.

>> Why working on the :m6-sparse extension, I noticed Emacs becomes
>> noticeably slower, and I suspect that's because while I was editing,
>> all the seq  functions I was writing where being probed for
>> applicability, while core things like completion, buffer-switching
>> etc are calling seq-foo generics.
>
> It could be helpful to do some profiling and see where the slowdown
> came from. Could it come exactly from the set operations?

Not sure.  It might have come from tracing seq.el functions, for
example.  You might say that it's my fault I was tracing them, but
should I be punished in Emacs usability just for trying to use Emacs to
iteratively develop a seq.el extension?

Anyway tracing basic staples such as seq-do and seq-let gives some
insight as to where they are used and what shape of arguments they are
called with in your normal programming activities.  Small lists seem to
appear a lot more often.  But expect a massive slowdown while tracing:
even with modest seq.el usage in current core, these generics are called
a lot already.

>> I find this performance aspect very bad.  Maybe it can be obviated,
>> but only if you drop the '(if (sequence-p seq)' bomb into seq.el
>> somehow.  I don't see how we can avoid that one.
>
> I don't quite see the need. And it's unlikely to be of reliable help:
> my observations say that method dispatch simply becomes slower as soon
> as a generic function gets a second implementation. And that
> implementation might arrive from any third-party code.

Exactly.  The entry point generics probably can never be avoided.  I
think we agreed that noone -- user or library -- should add
implementations to them.  That's why I think not making them defuns was
another design mistake.  But other intermediary generics _can_ be
skipped and would bring a performance boost to seq.el.

Alright.  That all said, here's the latest results, which I gathered
using the attached sequence-benchmarks.el are also attached in
results.txt.

I gathered each set of timings by running these two things

   src/emacs -Q -nw sequence-benchmarks.el -f emacs-lisp-byte-compile-and-load

and then

  src/emacs -Q -nw -l m6sparse.el sequence-benchmarks.el -f emacs-lisp-byte-compile-and-load

And I also attach m6sparse.el.

The branch I used was feature/cl-lib-improvements where I also pushed
your seq-difference-3 patch.

This email is long enough, so take your conclusions.  I don't think any
results are exactly flattering to seq.el, especially -- but not only --
when you compare to the destructive versions of some utils that cl-lib
offers.

But this one stands out.  I hope you can read this macro more or less.

   (joaot/with-benchmark-group "destructuring"
     ((list1 . (make-list 3 0)))
     100000
     (joaot/bench (pcase-let ((`(,a ,b ,c) list1)) (+ a b c)))
     (joaot/bench (cl-destructuring-bind (a b c) list1 (+ a b c)))
     (joaot/bench (seq-let (a b c) list1 (+ a b c))))

Running in Emacs -Q:

  ("destructuring"
    (cl-destructuring-bind "FASTEST" 0.010702393 0 0.0)
    (pcase-let "1.3x SLOWER" 0.014360937999999998 0 0.0)
    (seq-let "3.5x (rel 2.6x) SLOWER" 0.03706726 0 0.0))

Where after loading m6sparse.el we go to this:

  (("destructuring" (cl-destructuring-bind "FASTEST" 0.010157632 0 0.0)
    (pcase-let "1.3x SLOWER" 0.013152518000000002 0 0.0)
    (seq-let "14.8x (rel 11.4x) SLOWER" 0.15057331499999999
      6 0.04785139399999849))

You may try to optimize this, but you'll probably have to introduce
yet another generic.

And this seq-let thing should also remind us that this should never only
be about "fast enough".  Eli was suggesting seq-let as an alternative to
pcase-let the other day.  Let-like forms do appear in tight loops (and
tight loops aplenty do exist), I expect a let-like form on a list to
expand to little more than some car, cadr, etc calls, not some immensely
slow generic call by comparison.

So, in summary.  YES to seq.el for custom sequences, we need more of
those (probably in core even) and NO to seq.el as a drop-in
general-purpose sequence processing library.

João


[-- Attachment #2: m6sparse.el --]
[-- Type: text/plain, Size: 3086 bytes --]

;;; m6-sparse -*- lexical-binding: t -*-
(cl-defmethod seq-do (function (l (head :m6-sparse)))
  (mapc (lambda (e)
          (if (eq e :double-o-seven) (mapc function '(0 0 7))
            (funcall function e)))
        (cdr l)))

(cl-defmethod seq-contains-p ((l (head :m6-sparse)) elt &optional test-fn)
  (cl-member elt (nreverse (seq-reverse l)) :test test-fn))

(cl-defmethod seq-contains-pred ((_l (head :m6-sparse)) &optional test-fn)
  (lambda (elt sequence)
    (cl-member elt (nreverse (seq-reverse sequence)) :test test-fn)))

(cl-defmethod seq-reverse ((l (head :m6-sparse)))
  (let (res) (seq-do (lambda (e) (push e res)) l) res))

(cl-defmethod seq-elt ((l (head :m6-sparse)) n) (elt (nreverse (seq-reverse l)) n))

(cl-defmethod seq-length ((l (head :m6-sparse))) (length (nreverse (seq-reverse l))))
(cl-defmethod seqp ((_l (head :m6-sparse))) t)
(cl-defmethod seq-into-sequence ((l (head :m6-sparse))) (nreverse (seq-reverse l)))
(cl-defmethod seq-subseq ((l (head :m6-sparse)) start &optional end) (cl-subseq (nreverse (seq-reverse l)) start end))
(cl-defmethod seq-copy ((l (head :m6-sparse))) (nreverse (seq-reverse l)))
(cl-defmethod seq-into ((l (head :m6-sparse)) type)
  (cl-call-next-method (nreverse (seq-reverse l)) type))
(cl-defmethod seq-map (function (l (head :m6-sparse)))
  (let ((res))
    (seq-do (lambda (e)
              (push (funcall function e) res))
            l)
    (nreverse res)))

(defvar my-compressed-list '(:m6-sparse 1 2 :double-o-seven 4
                                        :double-o-seven 9 10 11))

(seq-difference my-compressed-list '(7)) ;; => (1 2 0 0 4 0 0 9 10 11)
(seq-difference  '(42 42 42 0 0 7 42 42) my-compressed-list) ;; => (42 42 42 42 42)
(seq-difference-3  '(42 42 42 0 0 7 42 42) my-compressed-list) ;; => (42 42 42 42 42)
(seq-elt my-compressed-list 0)           ;; => 1
(seq-elt my-compressed-list 4)           ;; => 7
(seq-elt my-compressed-list 5)           ;; => 4
(seq-elt my-compressed-list 10)          ;; => 10
(seq-elt my-compressed-list 1)           ;; => 2
(seq-elt my-compressed-list 3)           ;; => 0
(seq-elt my-compressed-list 13)          ;; => nil
(seq-length my-compressed-list)          ;; => 12
(seqp my-compressed-list)                ;; => t
(seq-subseq my-compressed-list 2 5)      ;; => (0 0 7)
(seq-copy my-compressed-list)            ;; => (1 2 0 0 7 4 0 0 7 9 10 11)
(seq-into my-compressed-list 'vector)    ;; => [1 2 0 0 7 4 0 0 7 9 10 11]
(seq-into-sequence my-compressed-list)   ;; => (1 2 0 0 7 4 0 0 7 9 10 11)

;; non consing version
(cl-defmethod seq-elt ((l (head :m6-sparse)) n)
  (cl-loop for e in (cdr l)
           for diff = (- n i)
           while (cl-plusp diff)
           sum (if (eq e :double-o-seven) 3 1) into i
           finally return
           (cond ((eq e :double-o-seven) 0)
                 ((cl-minusp diff) (elt '(7 0 0) (- (1+ diff))))
                 ((zerop diff) e)
                 (t nil))))
;; non consing version
(cl-defmethod seq-length ((l (head :m6-sparse)))
  (cl-loop for e in (cdr l)
           sum (if (eq e :double-o-seven) 3 1)))

[-- Attachment #3: results.txt --]
[-- Type: text/plain, Size: 5906 bytes --]

BEFORE LOADING m6sparse

(("destructuring" (cl-destructuring-bind "FASTEST" 0.010702393 0 0.0)
  (pcase-let "1.3x SLOWER" 0.014360937999999998 0 0.0)
  (seq-let "3.5x (rel 2.6x) SLOWER" 0.03706726 0 0.0))
 ("\"some\" operation, small lists"
  (cl-some "FASTEST" 0.034695811 0 0.0)
  (seq-some "3.9x SLOWER" 0.135977155 10 0.0755767589999996))
 ("\"some\" operation, big lists"
  (cl-some "FASTEST" 0.29491865700000003 0 0.0)
  (seq-some "1.5x SLOWER" 0.44417032 0 0.0))
 ("set difference, small lists, custom pred"
  (cl-nset-difference "FASTEST" 0.19550686 15 0.11099579500000001)
  (cl-set-difference "5.1x SLOWER" 1.000326248 39 0.2885333050000005)
  (seq-difference-3 "13.2x (rel 2.6x) SLOWER" 2.575436995 192
		    1.4284019790000002)
  (seq-difference "14.4x (rel 1.1x) SLOWER" 2.8172440610000002 212
		  1.5752595139999999))
 ("set difference, big lists, custom pred"
  (cl-nset-difference "FASTEST" 0.011390392 0 0.0)
  (cl-set-difference "5.5x SLOWER" 0.062250377999999995 1
		     0.007866447000000054)
  (seq-difference-3 "15.9x (rel 2.9x) SLOWER" 0.18127607499999998 12
		    0.09376674499999993)
  (seq-difference "19.0x (rel 1.2x) SLOWER" 0.21642625100000001 15
		  0.1219803650000002))
 ("set difference, small lists, #'eql pred"
  (cl-nset-difference "FASTEST" 0.025416642 0 0.0)
  (cl-set-difference "10.6x SLOWER" 0.268309301 24 0.17514854700000004)
  (seq-difference-3 "19.3x (rel 1.8x) SLOWER" 0.491795223 47
		    0.3400537589999999)
  (seq-difference "101.8x (rel 5.3x) SLOWER" 2.5877423690000003 212
		  1.5652519980000001))
 ("set difference, big lists, #'eql pred"
  (cl-nset-difference "FASTEST" 0.001086087 0 0.0)
  (cl-set-difference "31.2x SLOWER" 0.033846905999999996 3
		     0.022589776999999867)
  (seq-difference-3 "38.0x (rel 1.2x) SLOWER" 0.041256314 3
		    0.024156587999999868)
  (seq-difference "363.2x (rel 9.6x) SLOWER" 0.39448685299999997 30
		  0.24322218400000017))
 ("set difference, small lists, #'equal pred"
  (joaot/handrolled-nset-difference "FASTEST" 0.015404288 0 0.0)
  (cl-nset-difference "3.9x SLOWER" 0.060033165 4 0.027628342)
  (dmitry/set-difference-nocons "7.8x (rel 2.0x) SLOWER" 0.120780652 0
				0.0)
  (cl-set-difference "23.6x (rel 3.0x) SLOWER" 0.362814037 28
		     0.201481361)
  (seq-difference-3 "36.5x (rel 1.6x) SLOWER" 0.5629666289999999 47
		    0.33870848)
  (seq-difference "48.9x (rel 1.3x) SLOWER" 0.753127336 68
		  0.47948227499999996))
 ("set difference, big lists, #'equal pred"
  (joaot/handrolled-nset-difference "FASTEST" 0.002013658 0 0.0)
  (cl-nset-difference "1.7x SLOWER" 0.0033535120000000003 0 0.0)
  (dmitry/set-difference-nocons "8.1x (rel 4.9x) SLOWER" 0.016377504 0
				0.0)
  (cl-set-difference "21.6x (rel 2.7x) SLOWER" 0.043477418 3
		     0.021467216000000004)
  (seq-difference-3 "25.2x (rel 1.2x) SLOWER" 0.050788065 3
		    0.022892674000000002)
  (seq-difference "39.0x (rel 1.5x) SLOWER" 0.07861690199999999 6
		  0.04483213700000001)))

AFTER loading it

(("destructuring" (cl-destructuring-bind "FASTEST" 0.010157632 0 0.0)
  (pcase-let "1.3x SLOWER" 0.013152518000000002 0 0.0)
  (seq-let "14.8x (rel 11.4x) SLOWER" 0.15057331499999999
    6 0.04785139399999849))
 ("\"some\" operation, small lists"
  (cl-some "FASTEST" 0.030627893999999996 0 0.0)
  (seq-some "5.1x SLOWER" 0.156528706 10 0.08218940500000116))
 ("\"some\" operation, big lists"
  (cl-some "FASTEST" 0.253589884 0 0.0)
  (seq-some "1.7x SLOWER" 0.42605076199999997 0 0.0))
 ("set difference, small lists, custom pred"
  (cl-nset-difference "FASTEST" 0.201229483 15 0.11990391199999983)
  (cl-set-difference "5.0x SLOWER" 1.002034638 39 0.31673883500000066)
  (seq-difference-3 "15.6x (rel 3.1x) SLOWER" 3.136544874 194
		    1.7686539850000003)
  (seq-difference "18.9x (rel 1.2x) SLOWER" 3.795912411 260
		  2.140581182000002))
 ("set difference, big lists, custom pred"
  (cl-nset-difference "FASTEST" 0.011600632 0 0.0)
  (cl-set-difference "5.4x SLOWER" 0.062588482 1 0.008457714000000394)
  (seq-difference-3 "18.1x (rel 3.4x) SLOWER" 0.21003662699999998 12
		    0.10525734000000142)
  (seq-difference "25.5x (rel 1.4x) SLOWER" 0.29535328699999996 19
		  0.16559472099999972))
 ("set difference, small lists, #'eql pred"
  (cl-nset-difference "FASTEST" 0.023517136 0 0.0)
  (cl-set-difference "11.5x SLOWER" 0.271050456 24 0.18550450299999888)
  (seq-difference-3 "24.1x (rel 2.1x) SLOWER" 0.567758752 50
		    0.39358252499999935)
  (seq-difference "148.5x (rel 6.1x) SLOWER" 3.491236757 260
		  2.072315871999999))
 ("set difference, big lists, #'eql pred"
  (cl-nset-difference "FASTEST" 0.0010804710000000002 0 0.0)
  (cl-set-difference "33.0x SLOWER" 0.035650159 3 0.024719922999999255)
  (seq-difference-3 "39.0x (rel 1.2x) SLOWER" 0.042105057 3
		    0.025471970999999982)
  (seq-difference "520.3x (rel 13.4x) SLOWER" 0.5621415289999999 39
		  0.34187674300000026))
 ("set difference, small lists, #'equal pred"
  (joaot/handrolled-nset-difference "FASTEST" 0.014848083 0 0.0)
  (cl-nset-difference "4.4x SLOWER" 0.064930546 4 0.03191499300000089)
  (dmitry/set-difference-nocons "8.2x (rel 1.9x) SLOWER"
				0.12118037799999999 0 0.0)
  (cl-set-difference "25.4x (rel 3.1x) SLOWER" 0.377759915 28
		     0.22155792999999946)
  (seq-difference-3 "42.5x (rel 1.7x) SLOWER" 0.630364124 50
		    0.3947469649999995)
  (seq-difference "93.6x (rel 2.2x) SLOWER" 1.390502927 116
		  0.9019907159999985))
 ("set difference, big lists, #'equal pred"
  (cl-nset-difference "FASTEST" 0.002104955 0 0.0)
  (joaot/handrolled-nset-difference "1.2x SLOWER" 0.002503964 0 0.0)
  (dmitry/set-difference-nocons "7.9x (rel 6.6x) SLOWER"
				0.016558740000000002 0 0.0)
  (cl-set-difference "22.2x (rel 2.8x) SLOWER" 0.046786407 3
		     0.02427869799999982)
  (seq-difference-3 "26.0x (rel 1.2x) SLOWER" 0.054804263 3
		    0.02563775999999862)
  (seq-difference "93.5x (rel 3.6x) SLOWER" 0.196867405 15
		  0.12648469499999848)))




[-- Attachment #4: sequence-benchmarks.el --]
[-- Type: text/plain, Size: 6036 bytes --]

;;; -*- lexical-binding: t -*-

(defun joaot/handrolled-nset-difference (list1 list2)
   (if (or (null list1) (null list2)) list1
     (let ((res nil))
       (while (consp list1)
         (if (funcall #'member (car list1) list2)
             (setf list1 (cdr list1))
           (cl-shiftf list1 (cdr list1) res list1)))
       res)))

(defun dmitry/set-difference-nocons (list1 list2)
   (let (ref)
     (while (member (car list1) list2)
       (setq list1 (cdr list1)))
     (setq ref list1)
     (while ref
       (if (member (cadr ref) list2)
           (setcdr ref (cddr ref))
         (setq ref (cdr ref))))
     list1))

(defvar joaot/bench-group-name nil)
(defvar joaot/setup-form nil)
(defvar joaot/timings-alist nil)
(defvar joaot/repetitions nil)

(defmacro joaot/with-benchmark-group (bench-name
                                      setup-form
                                      repetitions
                                      &rest body)
  (declare (indent 1))
  (let ((f `(alist-get ,bench-name joaot/timings-alist nil nil #'equal)))
    `(let ((joaot/bench-group-name ,bench-name)
           (joaot/setup-form ',setup-form)
           (joaot/repetitions ,repetitions))
       (setf ,f (list))
       ,@body
       (cl-loop with sorted = (setf ,f (cl-sort ,f #'< :key #'cadr))
                with fastest = (cadr (car sorted))
                for i from 0
                for res in sorted
                for prev-time = time
                for (_n time . more) = res
                do (setcdr res
                           (cl-list* (if (eq time fastest) "FASTEST"
                                       (format "%2.1fx %sSLOWER"
                                               (/ time fastest)
                                               (if (= i 1) ""
                                                 (format "(rel %2.1fx) "
                                                         (/ time prev-time)))))
                                     time
                                     more)))
       joaot/timings-alist)))

(defmacro joaot/bench (form)
  `(cl-progv
       (mapcar #'car joaot/setup-form)
       (mapcar #'eval (mapcar #'cdr joaot/setup-form))
     (let ((res
            (benchmark-call (prog1
                                (,(if (native-comp-available-p)
                                      'native-compile
                                    'byte-compile)
                                 '(lambda () ,form))
                              (garbage-collect))
                            joaot/repetitions))
           (group-name joaot/bench-group-name)
           (bench-name ',(car form)))
       (push (cons bench-name res)
             (alist-get group-name joaot/timings-alist nil nil #'equal)))))

(joaot/with-benchmark-group "set difference, big lists, #'equal pred"
  ((list1 . (all-completions "" obarray))
   (list2 . (make-list 12 "shooveedoowaa")))
  10
  (joaot/bench (joaot/handrolled-nset-difference list1 list2))
  (joaot/bench (dmitry/set-difference-nocons list1 list2))
  (joaot/bench (cl-nset-difference list1 list2 :test #'equal))
  (joaot/bench (cl-set-difference list1 list2 :test #'equal))
  (joaot/bench (seq-difference list1 list2))
  (joaot/bench (seq-difference-3 list1 list2)))

(joaot/with-benchmark-group "set difference, small lists, #'equal pred"
  ((list1 . (make-list 12 "bla"))
   (list2 . (make-list 12 "shooveedoowaa")))
  100000
  (joaot/bench (joaot/handrolled-nset-difference list1 list2))
  (joaot/bench (dmitry/set-difference-nocons list1 list2))
  (joaot/bench (cl-nset-difference list1 list2 :test #'equal))
  (joaot/bench (cl-set-difference list1 list2 :test #'equal))
  (joaot/bench (seq-difference list1 list2))
  (joaot/bench (seq-difference-3 list1 list2)))

(joaot/with-benchmark-group "set difference, big lists, #'eql pred"
  ((list1 . (all-completions "" obarray))
   (list2 . (make-list 12 "shooveedoowaa")))
  10
  (joaot/bench (cl-nset-difference list1 list2))
  (joaot/bench (cl-set-difference list1 list2))
  (joaot/bench (seq-difference list1 list2 #'eql))
  (joaot/bench (seq-difference-3 list1 list2 #'eql)))

(joaot/with-benchmark-group "set difference, small lists, #'eql pred"
  ((list1 . (make-list 12 "bla"))
   (list2 . (make-list 12 "shooveedoowaa")))
  100000
  (joaot/bench (cl-nset-difference list1 list2))
  (joaot/bench (cl-set-difference list1 list2))
  (joaot/bench (seq-difference list1 list2 #'eql))
  (joaot/bench (seq-difference-3 list1 list2 #'eql)))

(joaot/with-benchmark-group "set difference, big lists, custom pred"
  ((list1 . (all-completions "" obarray))
   (list2 . (make-list 12 "shooveedoowaa")))
  5
  (byte-compile (defun myequal (a b) (equal a b)))
  (joaot/bench (cl-nset-difference list1 list2 :test #'myequal))
  (joaot/bench (cl-set-difference list1 list2 :test #'myequal))
  (joaot/bench (seq-difference-3 list1 list2 #'myequal))
  (joaot/bench (seq-difference list1 list2 #'myequal)))

(joaot/with-benchmark-group "set difference, small lists, custom pred"
  ((list1 . (make-list 12 "bla"))
   (list2 . (make-list 12 "shooveedoowaa")))
  100000
  (byte-compile (defun myequal (a b) (equal a b)))
  (joaot/bench (cl-nset-difference list1 list2 :test #'myequal))
  (joaot/bench (cl-set-difference list1 list2 :test #'myequal))
  (joaot/bench (seq-difference-3 list1 list2 #'myequal))
  (joaot/bench (seq-difference list1 list2 #'myequal)))

(joaot/with-benchmark-group "\"some\" operation, big lists"
  ((list1 . (make-list 100000 nil)))
  100
  (joaot/bench (cl-some #'identity list1))
  (joaot/bench (seq-some #'identity list1)))

(joaot/with-benchmark-group "\"some\" operation, small lists"
  ((list1 . (make-list 10 nil)))
  100000
  (joaot/bench (cl-some #'identity list1))
  (joaot/bench (seq-some #'identity list1)))

(joaot/with-benchmark-group "destructuring"
  ((list1 . (make-list 3 0)))
  100000
  (joaot/bench (pcase-let ((`(,a ,b ,c) list1)) (+ a b c)))
  (joaot/bench (cl-destructuring-bind (a b c) list1 (+ a b c)))
  (joaot/bench (seq-let (a b c) list1 (+ a b c))))

Re: What's missing in ELisp that makes people want to use cl-lib?

[Michael Heerdegen]

João Távora <joaotavora@gmail.com> writes:

> But this one stands out.  I hope you can read this macro more or less.
>
>    (joaot/with-benchmark-group "destructuring"
>      ((list1 . (make-list 3 0)))
>      100000
>      (joaot/bench (pcase-let ((`(,a ,b ,c) list1)) (+ a b c)))
>      (joaot/bench (cl-destructuring-bind (a b c) list1 (+ a b c)))
>      (joaot/bench (seq-let (a b c) list1 (+ a b c))))
>
> Running in Emacs -Q:
>
>   ("destructuring"
>     (cl-destructuring-bind "FASTEST" 0.010702393 0 0.0)
>     (pcase-let "1.3x SLOWER" 0.014360937999999998 0 0.0)
>     (seq-let "3.5x (rel 2.6x) SLOWER" 0.03706726 0 0.0))
>
> Where after loading m6sparse.el we go to this:
>
>   (("destructuring" (cl-destructuring-bind "FASTEST" 0.010157632 0 0.0)
>     (pcase-let "1.3x SLOWER" 0.013152518000000002 0 0.0)
>     (seq-let "14.8x (rel 11.4x) SLOWER" 0.15057331499999999
>       6 0.04785139399999849))

Isn't seq-let directly translated into a pcase-let call (it's a
simplistic macro)?  How is this possible?

Michael.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Thu, Nov 16, 2023 at 4:02 PM Michael Heerdegen
<michael_heerdegen@web.de> wrote:
>
> João Távora <joaotavora@gmail.com> writes:

> >
> >   (("destructuring" (cl-destructuring-bind "FASTEST" 0.010157632 0 0.0)
> >     (pcase-let "1.3x SLOWER" 0.013152518000000002 0 0.0)
> >     (seq-let "14.8x (rel 11.4x) SLOWER" 0.15057331499999999
> >       6 0.04785139399999849))
>
> Isn't seq-let directly translated into a pcase-let call (it's a
> simplistic macro)?  How is this possible?

Yes, but look closely at the expansion ;-)

And you can also think.  How can that pcase-let call know at at
runtime whether it's dealing with a bare list, a vector/string,
or a custom sequence.  Only through runtime indirection
obviously!  Based on what?  Generic functions no less, seq-elt
in this case, which become slower when added to.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

João Távora <joaotavora@gmail.com> writes:

> AFTER loading it
>
> ...

The magnitude of the difference when additional methods are defined I
find surprising. I take it as a strong indicator that cl-generic.el
indeed works completely differently than PCL. Assuming it is not a bug
af some sort. That the manual nowhere uses the term "discriminating
function" might also be a hint.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Thu, Nov 16, 2023 at 5:44 PM Gerd Möllmann <gerd.moellmann@gmail.com> wrote:
>
> João Távora <joaotavora@gmail.com> writes:
>
> > AFTER loading it
> >
> > ...
>
> The magnitude of the difference when additional methods are defined I
> find surprising.

Speaking of that, be sure to reproduce my results.   These are results
from an Emacs -Q run, and fortunately they are consistent across runs
in the same machine.  In my normal work session, I sometimes get much
more dramatic slowdowns with these very same benchmarks, and I don't
know why, maybe some garbage collection effect?

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

João Távora <joaotavora@gmail.com> writes:

> On Thu, Nov 16, 2023 at 5:44 PM Gerd Möllmann <gerd.moellmann@gmail.com> wrote:
>>
>> João Távora <joaotavora@gmail.com> writes:
>>
>> > AFTER loading it
>> >
>> > ...
>>
>> The magnitude of the difference when additional methods are defined I
>> find surprising.
>
> Speaking of that, be sure to reproduce my results.   These are results
> from an Emacs -Q run, and fortunately they are consistent across runs
> in the same machine.  In my normal work session, I sometimes get much
> more dramatic slowdowns with these very same benchmarks, and I don't
> know why, maybe some garbage collection effect?

Yeah, please see my other mail, I've looked into cl-generics a bit. Not
that I could explain anything now :-)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> João Távora <joaotavora@gmail.com> writes:
>
>> On Thu, Nov 16, 2023 at 5:44 PM Gerd Möllmann <gerd.moellmann@gmail.com> wrote:
>>>
>>> João Távora <joaotavora@gmail.com> writes:
>>>
>>> > AFTER loading it
>>> >
>>> > ...
>>>
>>> The magnitude of the difference when additional methods are defined I
>>> find surprising.
>>
>> Speaking of that, be sure to reproduce my results.   These are results
>> from an Emacs -Q run, and fortunately they are consistent across runs
>> in the same machine.  In my normal work session, I sometimes get much
>> more dramatic slowdowns with these very same benchmarks, and I don't
>> know why, maybe some garbage collection effect?
>
> Yeah, please see my other mail, I've looked into cl-generics a bit. Not
> that I could explain anything now :-)

Hm, that's actually not true.

One thing that one can conclude, I think, is that the very first
execution of a generic function with a given argument type combination
(i.e. an effective method) does not count, so to say, because that
computes the effective memthod and memoizes it.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> João Távora <joaotavora@gmail.com> writes:
>
>> AFTER loading it
>>
>> ...
>
> The magnitude of the difference when additional methods are defined I
> find surprising. I take it as a strong indicator that cl-generic.el
> indeed works completely differently than PCL. Assuming it is not a bug
> af some sort. That the manual nowhere uses the term "discriminating
> function" might also be a hint.

Looking a bit at cl-generics.el, at least the discriminating functions
part is indeed like nothing in PCL. And discriminating functions are not
in the manual, because there are none.

Disclaimer: I do not know cl-generics.el, and I've just looked enough to
get an impression.

So: I would describe the difference between PCL and cl-generic something
like static vs. dynamic, perhaps.

PCL goes to great lengths computing applicable methods etc. constructing
an as optimal as possible discriminating function. Once this has all
been done, nothing more has to be done at runtime. When methods are
changed, added etc. the whole thing is done from scratch again.

Cl-generics, in contrast, I'd say relies on runtime computation of
applicable methods and so on, plus memoization. If someone wants to see
what a generic function looks like, see cl-generic-define ->
cl--generic-make-function -> cl--generic-make-next-function ->
cl--generic-get-dispatcher. There we see

      (funcall
       cl--generic-compiler
       `(lambda (generic dispatches-left methods)
          (let ((method-cache (make-hash-table :test #'eql)))
            (lambda (,@fixedargs &rest args)
              (let ,bindings
                (apply (with-memoization
                           (gethash ,tag-exp method-cache)
                         (cl--generic-cache-miss
                          generic ',dispatch-arg dispatches-left methods
                          ,(if (cdr typescodes)
                               `(append ,@typescodes) (car typescodes))))
                       ,@fixedargs args)))))))))

The hash-table is a cache, the inner lambda is the function definition
of the gf, the apply is the execution of an effective method, AFAIU. If
we hit an argument combination not in the cache, we compute applicable
methods at runtime, I believe.

A bit strange is that cl--generic-next-function seems to be called
recursively in the process, which I think could create another such
hash-table. Or I'm reading that simply wrong, as I mentioned I just
wanted to see if cl-generic is so different, so I didn't spend much time
on this.

The question how that leads to such-and-such performance effects I can't
answer. I haven't seen such an implementation before.

And I'm not saying it's bad! There are very very very (did I say very
enough?) good reasons to try and avoid the incredible complexity of
something like PCL.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

Thanks, Gerd.

I think Stefan is not subscribed to this list anymore, so I'll Cc him in 
case he has something to add (or considers worth improving).

But cl-defmethod with just one method resulting in a simple function 
definition in an experimental fact (evaluate (cl-defmethod abc () 345); 
then (cl-defmethod abc () 345) returns (lambda nil (progn 345))), so 
that must be faster, without any computation of applicable methods.

The comment above the code you quoted mentions "generic functions with a 
single method"; maybe it was written before the above optimization was made.

On 16/11/2023 22:09, Gerd Möllmann wrote:
> Gerd Möllmann<gerd.moellmann@gmail.com>  writes:
> 
>> João Távora<joaotavora@gmail.com>  writes:
>>
>>> AFTER loading it
>>>
>>> ...
>> The magnitude of the difference when additional methods are defined I
>> find surprising. I take it as a strong indicator that cl-generic.el
>> indeed works completely differently than PCL. Assuming it is not a bug
>> af some sort. That the manual nowhere uses the term "discriminating
>> function" might also be a hint.
> Looking a bit at cl-generics.el, at least the discriminating functions
> part is indeed like nothing in PCL. And discriminating functions are not
> in the manual, because there are none.
> 
> Disclaimer: I do not know cl-generics.el, and I've just looked enough to
> get an impression.
> 
> So: I would describe the difference between PCL and cl-generic something
> like static vs. dynamic, perhaps.
> 
> PCL goes to great lengths computing applicable methods etc. constructing
> an as optimal as possible discriminating function. Once this has all
> been done, nothing more has to be done at runtime. When methods are
> changed, added etc. the whole thing is done from scratch again.
> 
> Cl-generics, in contrast, I'd say relies on runtime computation of
> applicable methods and so on, plus memoization. If someone wants to see
> what a generic function looks like, see cl-generic-define ->
> cl--generic-make-function -> cl--generic-make-next-function ->
> cl--generic-get-dispatcher. There we see
> 
>        (funcall
>         cl--generic-compiler
>         `(lambda (generic dispatches-left methods)
>            (let ((method-cache (make-hash-table :test #'eql)))
>              (lambda (,@fixedargs &rest args)
>                (let ,bindings
>                  (apply (with-memoization
>                             (gethash ,tag-exp method-cache)
>                           (cl--generic-cache-miss
>                            generic ',dispatch-arg dispatches-left methods
>                            ,(if (cdr typescodes)
>                                 `(append ,@typescodes) (car typescodes))))
>                         ,@fixedargs args)))))))))
> 
> The hash-table is a cache, the inner lambda is the function definition
> of the gf, the apply is the execution of an effective method, AFAIU. If
> we hit an argument combination not in the cache, we compute applicable
> methods at runtime, I believe.
> 
> A bit strange is that cl--generic-next-function seems to be called
> recursively in the process, which I think could create another such
> hash-table. Or I'm reading that simply wrong, as I mentioned I just
> wanted to see if cl-generic is so different, so I didn't spend much time
> on this.
> 
> The question how that leads to such-and-such performance effects I can't
> answer. I haven't seen such an implementation before.
> 
> And I'm not saying it's bad! There are very very very (did I say very
> enough?) good reasons to try and avoid the incredible complexity of
> something like PCL.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Stefan Monnier]

> But cl-defmethod with just one method resulting in a simple function
>  definition in an experimental fact (evaluate (cl-defmethod abc () 345);
>  then (cl-defmethod abc () 345) returns (lambda nil (progn 345))), so that
> must be faster, without any computation of applicable methods.

"one method" is necessary but not sufficient for this optimization.
This one method has to have no specializer (aka only the `t` specializer).

> The comment above the code you quoted mentions "generic functions with
> a single method"; maybe it was written before the above optimization
> was made.

No, that comment refers to the case where there's a single method but
with a non-t specializer, so we can't just always call that one method
without first testing that the arg matches the specializer.


        Stefan

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 17/11/2023 04:44, Stefan Monnier wrote:
>> But cl-defmethod with just one method resulting in a simple function
>>   definition in an experimental fact (evaluate (cl-defmethod abc () 345);
>>   then (cl-defmethod abc () 345) returns (lambda nil (progn 345))), so that
>> must be faster, without any computation of applicable methods.
> "one method" is necessary but not sufficient for this optimization.
> This one method has to have no specializer (aka only the `t` specializer).
> 
>> The comment above the code you quoted mentions "generic functions with
>> a single method"; maybe it was written before the above optimization
>> was made.
> No, that comment refers to the case where there's a single method but
> with a non-t specializer, so we can't just always call that one method
> without first testing that the arg matches the specializer.

Makes sense. Thanks!

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> Date: Fri, 17 Nov 2023 04:47:41 +0200
> Cc: Gerd Möllmann <gerd.moellmann@gmail.com>,
>  João Távora <joaotavora@gmail.com>,
>  Eli Zaretskii <eliz@gnu.org>, michael_heerdegen@web.de, emacs-devel@gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 17/11/2023 04:44, Stefan Monnier wrote:
> >> But cl-defmethod with just one method resulting in a simple function
> >>   definition in an experimental fact (evaluate (cl-defmethod abc () 345);
> >>   then (cl-defmethod abc () 345) returns (lambda nil (progn 345))), so that
> >> must be faster, without any computation of applicable methods.
> > "one method" is necessary but not sufficient for this optimization.
> > This one method has to have no specializer (aka only the `t` specializer).
> > 
> >> The comment above the code you quoted mentions "generic functions with
> >> a single method"; maybe it was written before the above optimization
> >> was made.
> > No, that comment refers to the case where there's a single method but
> > with a non-t specializer, so we can't just always call that one method
> > without first testing that the arg matches the specializer.
> 
> Makes sense. Thanks!

If this warrants some improvements of the commentary, please feel free
to install such improvements.  TIA.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Dmitry Gutov <dmitry@gutov.dev> writes:

> I think Stefan is not subscribed to this list anymore, so I'll Cc him
> in case he has something to add (or considers worth improving).

Oops, I didn't realize that, sorry. Hi Stefan :-).

> But cl-defmethod with just one method resulting in a simple function
> definition in an experimental fact (evaluate (cl-defmethod abc ()
> 345); then (cl-defmethod abc () 345) returns (lambda nil (progn
> 345))), so that must be faster, without any computation of applicable
> methods.
>
> The comment above the code you quoted mentions "generic functions with
> a single method"; maybe it was written before the above optimization
> was made.

That might be a slight misunderstanding. I was not wondering why there
is a performance difference between the 1 method case and the rest. I
was wondering why it is so large, or can be so large, which I didn't
expect from a PCL-like implementation. And the reason for that was my
false assumption that it is PCL-like. Sorry for leading you and Joao on
wrong tracks by extrapolating from PCL, BTW.

Anyway - as I already mentioned I have absolutely no intuition about the
performance characteristics of cl-generic's dispatch mechanism.

Maybe, Stefan, you can help with that? Possibly in the context of
Dmitry's and Joao's case of using seq.el's generic function interface,
how do I say it, to replace the sequences part of cl-lib in Emacs?

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: João Távora <joaotavora@gmail.com>,  Stefan Monnier
>  <monnier@IRO.UMontreal.CA>,  Eli Zaretskii <eliz@gnu.org>,
>   michael_heerdegen@web.de,  emacs-devel@gnu.org
> Date: Fri, 17 Nov 2023 07:08:19 +0100
> 
> Dmitry Gutov <dmitry@gutov.dev> writes:
> 
> > I think Stefan is not subscribed to this list anymore, so I'll Cc him
> > in case he has something to add (or considers worth improving).
> 
> Oops, I didn't realize that, sorry. Hi Stefan :-).

You didn't have to: Dmitry misremembered -- Stefan _is_ subscribed to
emacs-devel.  It is bug-gnu-emacs that Stefan isn't subscribed to.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Stefan Monnier]

> Anyway - as I already mentioned I have absolutely no intuition about the
> performance characteristics of cl-generic's dispatch mechanism.

I don't have a good understanding of it either.
I did investigate a performance issue in `cl-print.el` where it seemed
clear that a significant part of the overhead comes from memory
allocations (and associated GC load) coming from `&rest`.

For this reason, I've been toying (in my mind only, so far) with how we
could have something like `&rest` but which doesn't allocate memory
from the heap for the specific sub-case of

    (lambda (.. &rest args)
      ...
      (apply ... args)
      ...)


-- Stefan

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Anyway - as I already mentioned I have absolutely no intuition about the
>> performance characteristics of cl-generic's dispatch mechanism.
>
> I don't have a good understanding of it either.

Thanks, Stefan. Then I guess no-one knows.

And of course we're not a millimeter closer to answering the question
why seq.el should replace cl-lib.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Augusto Stoffel]

On Sat, 18 Nov 2023 at 06:58, Gerd Möllmann wrote:

> And of course we're not a millimeter closer to answering the question
> why seq.el should replace cl-lib.

I think the original question was, why do you need an extra library of
basic list processing functions in a LISt Processing language...  Also,
assuming performance is indeed crucial, shouldn't they be implemented in
C?

The vocabulary missing from the built-ins is pretty small.  The
following are the fundamental ones IMO:

- filter aka remove-if-not
- some

(reduce is also academically fundamental, but I guess dolist is the more
idiomatic replacement.)

The following I would count as nice-to-haves:

- drop (already exists under a funny name)
- drop-while (basically like `member' with a different TESTFN)
- take-while
- perhaps find and position (those are small variations on `some')

The rest of seq I would count as "sporadically handy" (seq-partition,
seq-mapn, etc) or "pretty much unnecessary" (seq-difference,
seq-intersection, etc).

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
> A bit strange is that cl--generic-next-function seems to be called
> recursively in the process, which I think could create another such
> hash-table. Or I'm reading that simply wrong, as I mentioned I just
> wanted to see if cl-generic is so different, so I didn't spend much time
> on this.

Perhaps someone could verify if I'm reading this wrong?

Start in cl--generic-get-dispatcher again:

      (funcall
       cl--generic-compiler
       `(lambda (generic dispatches-left methods)
          (let ((method-cache (make-hash-table :test #'eql)))
            (lambda (,@fixedargs &rest args)
              (let ,bindings
                (apply (with-memoization
                           (gethash ,tag-exp method-cache)
                         (cl--generic-cache-miss
                          generic ',dispatch-arg dispatches-left methods
                          ,(if (cdr typescodes)
                               `(append ,@typescodes) (car typescodes))))
                       ,@fixedargs args)))))))))

The hash table is the cache, and so on, as before.

We have a cache miss, so -> cl--generic-cache-miss ->
cl--generic-make-next-function -> cl--generic-get-dispatcher, and we
procuce a function like the above, with a a method-cache of its own?

I suspect that I'm taking a wrong turn here, somewhere.

PCL [Re: What's missing in ELisp that makes people want to use cl-lib?

[Madhu]

* Gerd M?llmann <m2r0kptq7l.fsf @Pro.fritz.box> :
Wrote on Thu, 16 Nov 2023 21:09:50 +0100:
> Looking a bit at cl-generics.el, at least the discriminating functions
> part is indeed like nothing in PCL. And discriminating functions are not
> in the manual, because there are none.
>
> Disclaimer: I do not know cl-generics.el, and I've just looked enough to
> get an impression.
>
> So: I would describe the difference between PCL and cl-generic something
> like static vs. dynamic, perhaps.
>
> PCL goes to great lengths computing applicable methods etc. constructing
> an as optimal as possible discriminating function. Once this has all
> been done, nothing more has to be done at runtime. When methods are
> changed, added etc. the whole thing is done from scratch again.
>
> Cl-generics, in contrast, I'd say relies on runtime computation of
> applicable methods and so on, plus memoization. If someone wants to see
> what a generic function looks like, see cl-generic-define ->
> cl--generic-make-function -> cl--generic-make-next-function ->
> cl--generic-get-dispatcher. There we see
>
>       (funcall
>        cl--generic-compiler
>        `(lambda (generic dispatches-left methods)
>           (let ((method-cache (make-hash-table :test #'eql)))
>             (lambda (,@fixedargs &rest args)
>               (let ,bindings
>                 (apply (with-memoization
>                            (gethash ,tag-exp method-cache)
>                          (cl--generic-cache-miss
>                           generic ',dispatch-arg dispatches-left methods
>                           ,(if (cdr typescodes)
>                                `(append ,@typescodes) (car typescodes))))
>                        ,@fixedargs args)))))))))
>
> The hash-table is a cache, the inner lambda is the function definition
> of the gf, the apply is the execution of an effective method, AFAIU. If
> we hit an argument combination not in the cache, we compute applicable
> methods at runtime, I believe.
>
> A bit strange is that cl--generic-next-function seems to be called
> recursively in the process, which I think could create another such
> hash-table. Or I'm reading that simply wrong, as I mentioned I just
> wanted to see if cl-generic is so different, so I didn't spend much time
> on this.
>
> The question how that leads to such-and-such performance effects I can't
> answer. I haven't seen such an implementation before.
>
> And I'm not saying it's bad! There are very very very (did I say very
> enough?) good reasons to try and avoid the incredible complexity of
> something like PCL.

There are advantages in the complexity of PCL. For instance, AFAIK there
is no way to undefine a method defined with cl-defmethod after defining
it without restartimg emacs, which I think misses the fundamental point
about Lisp.

I remember making made a few posts in 2016 on comp.lang.lisp about the
design and use of I faced with cl-generics, where I pointed the issues
I hit.

Re: PCL [Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Mon, Nov 20, 2023 at 2:31 PM Madhu <enometh@meer.net> wrote:

> > And I'm not saying it's bad! There are very very very (did I say very
> > enough?) good reasons to try and avoid the incredible complexity of
> > something like PCL.
>
> There are advantages in the complexity of PCL. For instance, AFAIK there
> is no way to undefine a method defined with cl-defmethod after defining
> it without restartimg emacs, which I think misses the fundamental point
> about Lisp.

Well, you can unintern the symbol and define all the methods again.
Not great but at least doesn't restart Emacs.

João

Re: PCL [Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Madhu <enometh@meer.net> writes:

> * Gerd M?llmann <m2r0kptq7l.fsf @Pro.fritz.box> :
>> And I'm not saying it's bad! There are very very very (did I say very
>> enough?) good reasons to try and avoid the incredible complexity of
>> something like PCL.
>
> There are advantages in the complexity of PCL.

Techincally, 100%.

> For instance, AFAIK there
> is no way to undefine a method defined with cl-defmethod after defining
> it without restartimg emacs, which I think misses the fundamental point
> about Lisp.

I would have guessed fmakunbound + defgeneric would do the trick. So
that the gf gets a new, empty, cache. Haven't tried, though.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 16/11/2023 16:30, João Távora wrote:

> But it's not only a matter of performance.  Take the
> 'seq-remove-at-position' generic.  Presumably someone thought that
> operationq was common enough to merit a separate entry point. In CL,
> this very operation can be done (probably faster) with cl-remove using
> keyword arguments.

Probably. Like I said, it looks more like Clojure's stdlib, but due to 
the absence of efficient "persistent data structures" the performance 
profile is different.

>>>> If that is the only drawback we can find, it is an unavoidable one.
> 
>> Could you try to explain what I should find in the second example?
>> What do you mean by "outlaw"?
> 
> What I meant is that the only way to get the same performance out of
> seq.el is to have early #'sequencep checks that bypass the generics
> completely, and this makes custom sequences based on sequences
> impossible.

If we optimize for short lists -- maybe. But for longer lists the 
dynamic dispatch (if performed once) shouldn't be a problem.

>> Does causing worse performance for a short time constitute "outlawing"
>> something?
> 
> But you should also find in that "m6sparse" example that the logic is
> broken -- not only in terms of performance -- until its author
> implements seq-contains-pred.  So this is pure "Incompatible Lisp
> changes" material (which I also think tanked performance should be btw.)
> 
> But in fact is is already broken by all the "list optimzations" in
> seq.el. Optimizations, before yours, that caused whatever the contract
> was to be violated.  To be able to use `seq-drop-while` for my m6sparse
> sequence, I have to add implementations to all those generic entry
> points, which is just akward.

Were those the 'list' type specializations?

>>> And even outlaw more stuff.  For example, these generics even have
>>> specializers on all mandatory arguments?
>>> For example, why does seq-do have FUNCTION as a required argument???
>>
>> Because FUNCTION is applied to SEQUENCE?
> 
>>
>>> It should be &optional or a &key, with a default value.  Which
>>> cl-degeneric obviously supports. That way specializations on FUNCTION
>>> become impossible,  or at least become much harder and there's less
>>> risk of tanking user code.  Design mistake IMO.
>>
>> I'm reasonably sure nobody expects function to be anything but a
>> straight function (symbol or a lambda), because that's how 'seq-do' is
>> used throughout the code.
> 
> Yes, but putting as a required argument in the arglist means users can
> specialize for it, and that isn't needed.  Not sure anyone does it or
> even if that makes the generic even slower.

And what would happen if FUNCTION were optional and omitted by the caller?

>>>> But that is the price one has to pay for correcting a design mistake.
>>>> We're not going to do this every Tuesday.
>>> Sure, but that price increases manyfold if we start suggesting
>>> seq.el as a replacement for all your sequence processing needs.
>>
>> We can first fix the mistake and then go on to continue "suggesting it
>> as a replacement". Or not.
>>
>> I don't exactly see it that way, though. And you give an impression of
>> arguing for the opposite: toward never using it at all.
> 
> Not at all.  Maybe you missed some of my previous messages.  I think
> seq.el's support for polymorphic sequences, though a little bit flawed
> in some respects, is very useful.  For example, I've been pondering
> using it in eglot.el to try to speed up JSON parsing.  If some kind of
> seq-plist-get and seq-plist-destructuring-bind can be designed, I might
> be able to skip consing much of the useless elements of a gigantic JSON
> blob and parse just the parts I need.  Of course, not easy, but I think
> seq.el is the tool for that.

plist like Elisp plist? It's difficult to write a type predicate for.

>>> Why working on the :m6-sparse extension, I noticed Emacs becomes
>>> noticeably slower, and I suspect that's because while I was editing,
>>> all the seq  functions I was writing where being probed for
>>> applicability, while core things like completion, buffer-switching
>>> etc are calling seq-foo generics.
>>
>> It could be helpful to do some profiling and see where the slowdown
>> came from. Could it come exactly from the set operations?
> 
> Not sure.  It might have come from tracing seq.el functions, for
> example.  You might say that it's my fault I was tracing them, but
> should I be punished in Emacs usability just for trying to use Emacs to
> iteratively develop a seq.el extension?

I'm not sure I understand. If you added tracing to 'car', it would also 
slow Emacs down, wouldn't it? Is that a problem with the design of 'car'?

> Anyway tracing basic staples such as seq-do and seq-let gives some
> insight as to where they are used and what shape of arguments they are
> called with in your normal programming activities.  Small lists seem to
> appear a lot more often.  But expect a massive slowdown while tracing:
> even with modest seq.el usage in current core, these generics are called
> a lot already.

That both sounds like a compliment and a pressing motivation to iron out 
the most apparent performance pitfalls.

>>> I find this performance aspect very bad.  Maybe it can be obviated,
>>> but only if you drop the '(if (sequence-p seq)' bomb into seq.el
>>> somehow.  I don't see how we can avoid that one.
>>
>> I don't quite see the need. And it's unlikely to be of reliable help:
>> my observations say that method dispatch simply becomes slower as soon
>> as a generic function gets a second implementation. And that
>> implementation might arrive from any third-party code.
> 
> Exactly.  The entry point generics probably can never be avoided.  I
> think we agreed that noone -- user or library -- should add
> implementations to them.

Um, no. I'm saying that seq.el function should be written in a way that 
causes the dynamic dispatch to happen only once (or perhaps a few 
times), as opposed to doing it N times (for every element of a sequence) 
or more.

seq-let might suffer for a similar problem.

Having overrides for entry points, OTOH, should be fine enough, and 
whatever added cost the dispatch itself incurs, it still happens once, 
and can be made up for by the more efficient specialized implementation 
of the function's logic.

> That's why I think not making them defuns was
> another design mistake.  But other intermediary generics _can_ be
> skipped and would bring a performance boost to seq.el.

Other intermediate generics, in general, can be skipped by defining 
different methods at earlier points.

> Alright.  That all said, here's the latest results, which I gathered
> using the attached sequence-benchmarks.el are also attached in
> results.txt.
> 
> I gathered each set of timings by running these two things
> 
>     src/emacs -Q -nw sequence-benchmarks.el -f emacs-lisp-byte-compile-and-load

First of all, when I try to run the above command (on your branch, with 
your attachments from the last email), it ends with:

Compiling file 
/home/dgutov/examples/cl-lib-vs-seq/sequence-benchmarks.el at Fri Nov 17 
03:57:31 2023
sequence-benchmarks.el:73:2: Error: Wrong type argument: listp, more

So before I proceed further, could you make sure that you ran the tests 
with these exact files? Superficially, it looks like just (require 
'cl-lib) is missing, but maybe something else inside was also different?

I also tried to produce some pretty-printed comparisons, but the above 
command (even with "fixed" require statement) doesn't produce anything 
to *Messages* or stdout when ran with --batch.

I can evaluate individual joaot/with-benchmark-group forms, and they 
print -- how'd you call it -- minified Lisp data to messages, but it's 
not formatted the same way as in your results.txt, nor is it easy to get 
out of the -nw session because the clipboard is naturally not shared. Is 
-nw needed? I expected it'd be used to print something to stdout.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 1477 bytes --]

On 17/11/2023 04:09, Dmitry Gutov wrote:
> I can evaluate individual joaot/with-benchmark-group forms, and they 
> print -- how'd you call it -- minified Lisp data to messages, but it's 
> not formatted the same way as in your results.txt, nor is it easy to get 
> out of the -nw session because the clipboard is naturally not shared. Is 
> -nw needed? I expected it'd be used to print something to stdout.

OK, I used just that, as well as 'benchmark-run-compiled' inside 
'joaot/bench' instead of the complex expression which chose whether to 
compile a form.

Here are some results.

0) "mine" - just running the tests on your branch.

1) "mine-proper" - is me splitting of the typecheck into a separate 
method. Not an optimization, just better code.

2) "mine-proper+astoff" - including Augusto's optimization as well. 
Notably, seq-some gets ahead on big lists by about 1.2x. Not sure how 
reliable this number is, though.

3) "astoff" - just Augusto's patch. In some cases it's faster than 2 
because of no extra dispatch (or multimorphic methods, I guess), but the 
order-of-magnitude difference is still there in many set-difference 
examples.

Loading m6sparse.el (using (load (expand-file-name "../m6sparse.el" 
load-file-name)) at the top) didn't seem to affect the results of (2) 
too much, though it brought -- predictably -- the performance of 
seq-difference (the original version) further down. Though the results 
fluctuate a lot, so I might have missed some.

[-- Attachment #2: results-astoff.txt --]
[-- Type: text/plain, Size: 2278 bytes --]

(("destructuring"
  (cl-destructuring-bind "FASTEST" 0.008716133000000001 0 0.0)
  (pcase-let "1.2x SLOWER" 0.010243486 0 0.0)
  (seq-let "3.1x (rel 2.7x) SLOWER" 0.027214243 0 0.0))
 ("\"some\" operation, small lists"
  (seq-some "FASTEST" 0.02428174 0 0.0)
  (cl-some "1.0x SLOWER" 0.0244231 0 0.0))
 ("\"some\" operation, big lists"
  (seq-some "FASTEST" 0.191672023 0 0.0)
  (cl-some "1.0x SLOWER" 0.197579632 0 0.0))
 ("set difference, small lists, custom pred"
  (cl-nset-difference "FASTEST" 0.102954774 5 0.03422299700000003)
  (cl-set-difference "6.3x SLOWER" 0.650314872 14 0.08941152499999994)
  (seq-difference "7.9x (rel 1.2x) SLOWER" 0.8126014229999999 22 0.13833590900000003))
 ("set difference, big lists, custom pred"
  (cl-nset-difference "FASTEST" 0.009105017 0 0.0)
  (cl-set-difference "4.9x SLOWER" 0.044808576999999995 0 0.0)
  (seq-difference "6.3x (rel 1.3x) SLOWER" 0.057444879000000004 1 0.006419620000000015))
 ("set difference, small lists, #'eql pred"
  (cl-nset-difference "FASTEST" 0.019827222 0 0.0)
  (cl-set-difference "6.4x SLOWER" 0.12714725799999999 9 0.05405195400000001)
  (seq-difference "31.7x (rel 4.9x) SLOWER" 0.62896921 21 0.130310427))
 ("set difference, big lists, #'eql pred"
  (cl-nset-difference "FASTEST" 0.000826303 0 0.0)
  (cl-set-difference "18.8x SLOWER" 0.015559618999999998 1 0.0064666000000000445)
  (seq-difference "98.1x (rel 5.2x) SLOWER" 0.081062122 1 0.0074823319999999804))
 ("set difference, small lists, #'equal pred"
  (joaot/handrolled-nset-difference "FASTEST" 0.011843948 0 0.0)
  (cl-nset-difference "3.3x SLOWER" 0.039604176 2 0.013306504999999996)
  (dmitry/set-difference-nocons "8.0x (rel 2.4x) SLOWER" 0.09519276600000001 0 0.0)
  (cl-set-difference "15.4x (rel 1.9x) SLOWER" 0.182937625 10 0.061539529999999995)
  (seq-difference "57.1x (rel 3.7x) SLOWER" 0.676095982 21 0.129525192))
 ("set difference, big lists, #'equal pred"
  (cl-nset-difference "FASTEST" 0.0030955889999999997 0 0.0)
  (joaot/handrolled-nset-difference "2.1x SLOWER" 0.006550018 0 0.0)
  (dmitry/set-difference-nocons "4.9x (rel 2.3x) SLOWER" 0.015023027000000001 0 0.0)
  (cl-set-difference "8.2x (rel 1.7x) SLOWER" 0.025435355 1 0.007049931999999995)
  (seq-difference "28.2x (rel 3.4x) SLOWER" 0.087276368 1 0.006623073)))

[-- Attachment #3: results-mine.txt --]
[-- Type: text/plain, Size: 2772 bytes --]

(("destructuring" (cl-destructuring-bind "FASTEST" 0.007508548 0 0.0)
  (pcase-let "1.3x SLOWER" 0.010040264 0 0.0)
  (seq-let "3.3x (rel 2.5x) SLOWER" 0.025065294000000002 0 0.0)
 ("\"some\" operation, small lists"
  (cl-some "FASTEST" 0.023633029 0 0.0)
  (seq-some "2.2x SLOWER" 0.052465596 1 0.007674551000000029)
 ("\"some\" operation, big lists"
  (cl-some "FASTEST" 0.195993925 0 0.0)
  (seq-some "1.6x SLOWER" 0.315841508 0 0.0)
 ("set difference, small lists, custom pred"
  (cl-nset-difference "FASTEST" 0.100909502 5 0.03441776400000007)
  (cl-set-difference "6.4x SLOWER" 0.6473928059999999 14 0.090176123)
  (seq-difference-3 "13.6x (rel 2.1x) SLOWER" 1.3704889370000002 68 0.42959640599999993)
  (seq-difference "14.1x (rel 1.0x) SLOWER" 1.42464227 75 0.4613546180000001))
 ("set difference, big lists, custom pred"
  (cl-nset-difference "FASTEST" 0.008583963 0 0.0)
  (cl-set-difference "6.7x SLOWER" 0.057137517 1 0.007286566000000105)
  (seq-difference-3 "10.3x (rel 1.6x) SLOWER" 0.088811108 3 0.02027387699999994)
  (seq-difference "10.9x (rel 1.1x) SLOWER" 0.09363935100000001 3 0.020642897000000104))
 ("set difference, small lists, #'eql pred"
  (cl-nset-difference "FASTEST" 0.019733991 0 0.0)
  (cl-set-difference "6.2x SLOWER" 0.122059607 8 0.04862236800000003)
  (seq-difference-3 "11.5x (rel 1.9x) SLOWER" 0.22615767 17 0.10414085099999992)
  (seq-difference "63.1x (rel 5.5x) SLOWER" 1.2446914809999998 75 0.44716219399999996))
 ("set difference, big lists, #'eql pred"
  (cl-nset-difference "FASTEST" 0.000827584 0 0.0)
  (cl-set-difference "18.8x SLOWER" 0.015517249 1 0.006894151000000015)
  (seq-difference-3 "23.9x (rel 1.3x) SLOWER" 0.019756616 1 0.006745060999999997)
  (seq-difference "200.8x (rel 8.4x) SLOWER" 0.16621615199999998 7 0.04840139199999999))
 ("set difference, small lists, #'equal pred"
  (joaot/handrolled-nset-difference "FASTEST" 0.01179042 0 0.0)
  (cl-nset-difference "2.8x SLOWER" 0.032970384 1 0.006517913)
  (dmitry/set-difference-nocons "7.9x (rel 2.8x) SLOWER" 0.09332390900000001 0 0.0)
  (cl-set-difference "15.8x (rel 2.0x) SLOWER" 0.18612888100000002 10 0.061874161)
  (seq-difference-3 "23.5x (rel 1.5x) SLOWER" 0.276987626 17 0.10811353499999998)
  (seq-difference "32.0x (rel 1.4x) SLOWER" 0.376752902 25 0.152489869))
 ("set difference, big lists, #'equal pred"
  (joaot/handrolled-nset-difference "FASTEST" 0.003132761 0 0.0)
  (cl-nset-difference "1.2x SLOWER" 0.003833 0 0.0)
  (dmitry/set-difference-nocons "5.6x (rel 4.6x) SLOWER" 0.017451134 0 0.0)
  (cl-set-difference "8.3x (rel 1.5x) SLOWER" 0.025889626 1 0.006706274999999998)
  (seq-difference-3 "9.2x (rel 1.1x) SLOWER" 0.028935888 1 0.006479378000000001)
  (seq-difference "11.4x (rel 1.2x) SLOWER" 0.035722551 1 0.0070541059999999975))
 )

[-- Attachment #4: results-mine-proper.txt --]
[-- Type: text/plain, Size: 2817 bytes --]

(("destructuring"
  (cl-destructuring-bind "FASTEST" 0.008374391 0 0.0)
  (pcase-let "1.3x SLOWER" 0.0104752 0 0.0)
  (seq-let "3.1x (rel 2.5x) SLOWER" 0.026114335 0 0.0))
 ("\"some\" operation, small lists"
  (cl-some "FASTEST" 0.023994479 0 0.0)
  (seq-some "2.9x SLOWER" 0.07007991599999999 3 0.02121758500000004))
 ("\"some\" operation, big lists"
  (cl-some "FASTEST" 0.20129617100000002 0 0.0)
  (seq-some "1.7x SLOWER" 0.350107408 0 0.0))
 ("set difference, small lists, custom pred"
  (cl-nset-difference "FASTEST" 0.103420572 5 0.03438092000000026)
  (cl-set-difference "6.6x SLOWER" 0.677639723 17 0.11333318999999986)
  (seq-difference "15.5x (rel 2.4x) SLOWER" 1.6034870689999998 91 0.6029158240000001)
  (seq-difference-3 "15.6x (rel 1.0x) SLOWER" 1.610416625 88 0.6408485689999999))
 ("set difference, big lists, custom pred"
  (cl-nset-difference "FASTEST" 0.008586518999999999 0 0.0)
  (cl-set-difference "5.7x SLOWER" 0.049010411999999996 0 0.0)
  (seq-difference-3 "12.0x (rel 2.1x) SLOWER" 0.103116948 4 0.028720732999999887)
  (seq-difference "12.4x (rel 1.0x) SLOWER" 0.106261259 4 0.029437464999999996))
 ("set difference, small lists, #'eql pred"
  (cl-nset-difference "FASTEST" 0.019229662 0 0.0)
  (cl-set-difference "7.6x SLOWER" 0.145943298 11 0.07134210400000007)
  (seq-difference-3 "16.9x (rel 2.2x) SLOWER" 0.32409309399999997 27 0.17366687899999977)
  (seq-difference "74.4x (rel 4.4x) SLOWER" 1.430963487 90 0.610517504))
 ("set difference, big lists, #'eql pred"
  (cl-nset-difference "FASTEST" 0.0008175820000000001 0 0.0)
  (cl-set-difference "18.7x SLOWER" 0.01528772 1 0.0060815740000000895)
  (seq-difference-3 "24.4x (rel 1.3x) SLOWER" 0.019946676 1 0.006836014000000001)
  (seq-difference "208.7x (rel 8.6x) SLOWER" 0.170651892 7 0.05150274999999993))
 ("set difference, small lists, #'equal pred"
  (joaot/handrolled-nset-difference "FASTEST" 0.013188440999999999 0 0.0)
  (cl-nset-difference "3.0x SLOWER" 0.039793513 2 0.013435085999999999)
  (dmitry/set-difference-nocons "7.2x (rel 2.4x) SLOWER" 0.09443695199999999 0 0.0)
  (cl-set-difference "16.1x (rel 2.2x) SLOWER" 0.21211317799999999 12 0.08097277)
  (seq-difference-3 "29.4x (rel 1.8x) SLOWER" 0.38735803599999996 28 0.18473822899999998)
  (seq-difference "32.3x (rel 1.1x) SLOWER" 0.426180881 29 0.19561955699999997))
 ("set difference, big lists, #'equal pred"
  (cl-nset-difference "FASTEST" 0.002728576 0 0.0)
  (joaot/handrolled-nset-difference "5.0x SLOWER" 0.013517241 1 0.008937456)
  (dmitry/set-difference-nocons "8.7x (rel 1.8x) SLOWER" 0.023723545999999998 0 0.0)
  (seq-difference-3 "10.4x (rel 1.2x) SLOWER" 0.028304957 1 0.006144501000000004)
  (cl-set-difference "10.4x (rel 1.0x) SLOWER" 0.028483389 1 0.007728865)
  (seq-difference "15.9x (rel 1.5x) SLOWER" 0.043508337999999994 2 0.014429581)))

[-- Attachment #5: results-mine-proper+astoff.txt --]
[-- Type: text/plain, Size: 2829 bytes --]

(("destructuring"
  (cl-destructuring-bind "FASTEST" 0.008805193 0 0.0)
  (pcase-let "1.2x SLOWER" 0.010528567 0 0.0)
  (seq-let "3.0x (rel 2.5x) SLOWER" 0.026212175 0 0.0))
 ("\"some\" operation, small lists"
  (cl-some "FASTEST" 0.024282236999999998 0 0.0)
  (seq-some "1.1x SLOWER" 0.025573062 0 0.0))
 ("\"some\" operation, big lists"
  (seq-some "FASTEST" 0.197265253 0 0.0)
  (cl-some "1.2x SLOWER" 0.23777484799999998 0 0.0))
 ("set difference, small lists, custom pred"
  (cl-nset-difference "FASTEST" 0.108415098 5 0.03888447500000003)
  (cl-set-difference "6.3x SLOWER" 0.681953283 17 0.10882586699999997)
  (seq-difference "8.2x (rel 1.3x) SLOWER" 0.889311564 26 0.1774222139999999)
  (seq-difference-3 "8.8x (rel 1.1x) SLOWER" 0.958568638 33 0.2262132990000001))
 ("set difference, big lists, custom pred"
  (cl-nset-difference "FASTEST" 0.008485322 0 0.0)
  (cl-set-difference "6.8x SLOWER" 0.057843906 1 0.008032880999999992)
  (seq-difference-3 "7.0x (rel 1.0x) SLOWER" 0.059558123 1 0.006597587000000016)
  (seq-difference "7.3x (rel 1.0x) SLOWER" 0.061867936 1 0.007283522000000042))
 ("set difference, small lists, #'eql pred"
  (cl-nset-difference "FASTEST" 0.019067578999999998 0 0.0)
  (cl-set-difference "6.9x SLOWER" 0.131699003 10 0.05825310500000003)
  (seq-difference-3 "16.2x (rel 2.3x) SLOWER" 0.308024268 27 0.15978121099999998)
  (seq-difference "35.4x (rel 2.2x) SLOWER" 0.6750600449999999 26 0.15592904399999996))
 ("set difference, big lists, #'eql pred"
  (cl-nset-difference "FASTEST" 0.000829391 0 0.0)
  (cl-set-difference "17.9x SLOWER" 0.014861217 1 0.00603653799999998)
  (seq-difference-3 "23.6x (rel 1.3x) SLOWER" 0.019564613999999998 1 0.00658552200000001)
  (seq-difference "106.0x (rel 4.5x) SLOWER" 0.087925567 2 0.012859830999999988))
 ("set difference, small lists, #'equal pred"
  (joaot/handrolled-nset-difference "FASTEST" 0.012052723000000001 0 0.0)
  (cl-nset-difference "3.2x SLOWER" 0.038597212000000006 2 0.012354418999999998)
  (dmitry/set-difference-nocons "7.7x (rel 2.4x) SLOWER" 0.09330716600000001 0 0.0)
  (cl-set-difference "16.0x (rel 2.1x) SLOWER" 0.192478013 12 0.069325267)
  (seq-difference "28.6x (rel 1.8x) SLOWER" 0.344323734 26 0.152115906)
  (seq-difference-3 "30.6x (rel 1.1x) SLOWER" 0.369063511 27 0.16940377700000003))
 ("set difference, big lists, #'equal pred"
  (cl-nset-difference "FASTEST" 0.0027295080000000003 0 0.0)
  (dmitry/set-difference-nocons "5.2x SLOWER" 0.014315372 0 0.0)
  (joaot/handrolled-nset-difference "6.1x (rel 1.2x) SLOWER" 0.016597400000000002 1 0.010671791)
  (cl-set-difference "8.8x (rel 1.5x) SLOWER" 0.024141194999999997 1 0.0063431120000000014)
  (seq-difference-3 "10.2x (rel 1.1x) SLOWER" 0.027760892 1 0.006199146000000003)
  (seq-difference "14.0x (rel 1.4x) SLOWER" 0.038263214000000004 2 0.012877529999999998)))

Re: What's missing in ELisp that makes people want to use cl-lib?

[Michael Heerdegen]

João Távora <joaotavora@gmail.com> writes:

> The only way you are eventually going to get to equal performance
> with cl-lib's fastest variants (but only the non-destructive, of course)
> is if you start every seq.el entry point with sth like
>
>    (if (sequence-p seq) (call-equivalent-cl-lib-version seq) ...)
>
> But then I don't see how that fits in with the cl-defgeneric,
> which does dispatching before that if.  The only way I see this
> is to invent a new kind of specializer in cl-generic.el like
> 'definitely-not-sequence'.  There could be some other solution,
> maybe Michael as a clue.

I'm a bit lost I must admit.  I don't understand the use of your
`sequence-p' test (all we use seq.el for are sequences...?).

Sorry if I missed something obvious,

Michael.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1339 bytes --]

On Thu, Nov 16, 2023 at 1:23 PM Michael Heerdegen <michael_heerdegen@web.de>
wrote:

> João Távora <joaotavora@gmail.com> writes:
>
> > The only way you are eventually going to get to equal performance
> > with cl-lib's fastest variants (but only the non-destructive, of course)
> > is if you start every seq.el entry point with sth like
> >
> >    (if (sequence-p seq) (call-equivalent-cl-lib-version seq) ...)
> >
> > But then I don't see how that fits in with the cl-defgeneric,
> > which does dispatching before that if.  The only way I see this
> > is to invent a new kind of specializer in cl-generic.el like
> > 'definitely-not-sequence'.  There could be some other solution,
> > maybe Michael as a clue.
>
> I'm a bit lost I must admit.  I don't understand the use of your
> `sequence-p' test (all we use seq.el for are sequences...?).


Yes, but custom sequences don't respond t to sequencep.  It is a
C built-in that only returns t for lists and arrays and strings.
And those are the types that the sequence functions in cl-lib.el
works with, and why it is faster than seq.el, because it only
dispatches between them with ifs.  If seq.el wants to be just
as fast, it must to that much simpler kind of dispatch, and
this has of course has implications for its generic function
interface.

João

[-- Attachment #2: Type: text/html, Size: 1927 bytes --]

Re: What's missing in ELisp that makes people want to use cl-lib?

[Michael Heerdegen]

João Távora <joaotavora@gmail.com> writes:

>>  I'm a bit lost I must admit.  I don't understand the use of your
>>  `sequence-p' test (all we use seq.el for are sequences...?).
>
> Yes, but custom sequences don't respond t to sequencep.

Ok, I got confused by the hyphen you had added to the predicate name.

If adding new methods slows down the generic functions significantly
this is something to be investigated.  Maybe there is a problem, AFAIR
parts of cl-generic are relatively new.

Michael.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Thu, Nov 16, 2023 at 3:24 PM Michael Heerdegen
<michael_heerdegen@web.de> wrote:
>
> João Távora <joaotavora@gmail.com> writes:
>
> >>  I'm a bit lost I must admit.  I don't understand the use of your
> >>  `sequence-p' test (all we use seq.el for are sequences...?).
> >
> > Yes, but custom sequences don't respond t to sequencep.
>
> Ok, I got confused by the hyphen you had added to the predicate name.

Yeah, my bad, sorry.

> If adding new methods slows down the generic functions significantly
> this is something to be investigated.  Maybe there is a problem, AFAIR
> parts of cl-generic are relatively new.

I think cl-generic is already pretty awesomely optimized.  But maybe
who knows?

Re: What's missing in ELisp that makes people want to use cl-lib?

[Michael Heerdegen]

João Távora <joaotavora@gmail.com> writes:

> This is all interesting, until one ponders what happens if an existing
> seq.el user somewhere has:
>
> (cl-defmethod seq-contains-p ((seq my-voodoo-seq)
>                               (elt (eql :secret-voodoo)) &optional _tesfn)
>   (invoke-voodoo-priests seq))
>
> making use of seq.el's support for abstract polymorphic sequences.
>
> With seq.el 2.24 a seq-difference operation would consider this user's
> method, with seq.el 2.24.dmitry (i.e. your fast seq-difference-3) it 
> simply won't.  This user's code is clearly broken.

It is debatable whether the result of such a seq difference is even well
defined with the information you provided to Elisp:

  - the difference of two voodoo sequences should be a voodoo sequence
    so it will contain :secret-voodoo
  - since both voodoo sequences contain :secret-voodoo, the result
    should _not_ contain it.

AFAIR it was not among the design ideas of seq.el to provide an
interface for tweaking seq.el implementations for special cases.

That you can define new implementations doesn't mean you have to, like
the fact that you can advice any functions doesn't mean it's a good idea
to do that.

The functions are generics mainly for the purpose to allow the support
new sequence types.  When you want to add a new type, it's expected
that you provide a consistent set of implementations for all generics.

In some cases, as you mentioned, some arguments will probably never be
specialized on (sorry if this is not correct English...).  This could be
avoided but one of the main goals was to keep the code simple and
developers didn't expect people to shoot themselves in the foot on
purpose.

But yes, a bit more documentation about how this lib is supposed to be
used would definitely not harm.  I can't even tell whether it was a good
idea to use generic functions but the author and Stefan M. did so I
think that this had been thought about.

Michael.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> João Távora <joaotavora@gmail.com> writes:
>
>> This is all interesting, until one ponders what happens if an existing
>> seq.el user somewhere has:
>>
>> (cl-defmethod seq-contains-p ((seq my-voodoo-seq)
>>                               (elt (eql :secret-voodoo)) &optional _tesfn)
>>   (invoke-voodoo-priests seq))
>>
>> making use of seq.el's support for abstract polymorphic sequences.
>>
>> With seq.el 2.24 a seq-difference operation would consider this user's
>> method, with seq.el 2.24.dmitry (i.e. your fast seq-difference-3) it 
>> simply won't.  This user's code is clearly broken.
>
> It is debatable whether the result of such a seq difference is even well
> defined with the information you provided to Elisp:

Yes, of course: that's why I called it "voodoo".  It was just for the
sake of argument, and made Dmitry aware of a serious performance bug
that happens in much more legitimate* cases. The fix to that bug, while
it exists to a great extent (it's seq-difference-3 in Dmitry's patch or
branch feature/cl-lib-improvements), requires incompatible lisp changes,
hence proving my point that a bunch of generics thrown together does not
a good library make.

Anyway, read my latest mail to Dmitry.

João

[*] For some meaning of "legitimate", of course, but I think sparse
sequences are a good legitimate, for example.  seq.el's documentation is
where?  I don't see it.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Michael Heerdegen]

João Távora <joaotavora@gmail.com> writes:

> seq.el's documentation is where?  I don't see it.

It's currently the file header and the rest of the file (the code).

Michael.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Thu, Nov 16, 2023 at 3:16 PM Michael Heerdegen
<michael_heerdegen@web.de> wrote:
>
> João Távora <joaotavora@gmail.com> writes:
>
> > seq.el's documentation is where?  I don't see it.
>
> It's currently the file header and the rest of the file (the code).

Precisely, as I learned the hard way, much of that code already
means that the custom sequence idea expressed very sparsely
at the top isn't quite so.  IMO it'd be very nice to have a contract
we can stick to for a long time.  At least for such a fundamental
thing in a Lisp language.  It'll be a lot of work to go
through the corner cases and write that contract, but better
late than never, I guess.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Date: Thu, 16 Nov 2023 16:16:34 +0100
> 
> João Távora <joaotavora@gmail.com> writes:
> 
> > seq.el's documentation is where?  I don't see it.
> 
> It's currently the file header and the rest of the file (the code).

?? There's a whole section in the ELisp manual called "Sequence
Functions", which documents the seq-* functions.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> > > seq.el's documentation is where?  I don't see it.
> > 
> > It's currently the file header and the rest of the file (the code).
>
> ?? There's a whole section in the ELisp manual called "Sequence
> Functions", which documents the seq-* functions.

Argh, I should have checked before writing that.  I'm sorry for my
nonsense reply.

Michael.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1657 bytes --]

On Thu, Nov 16, 2023 at 4:42 PM Eli Zaretskii <eliz@gnu.org> wrote:

> ?? There's a whole section in the ELisp manual called "Sequence
> Functions", which documents the seq-* functions.

Yes, but from a custom sequence implementors perspective,
the "best" documentation is still that bit in the header,
which describes "you must implement these to get a working
custom sequence".

So while the Elisp manual addresses custom sequences
(very briefly -- it just tells the user to look up generic
functions) it spends most time describing the generic
functions one by one, and while that's important, it's
just not enough.

It doesn't talk about the guarantees that the framework
offers with respect to when they are called, or if they
are called at all. For example, while I didn't read the
whole thing, I don't think it is stated that custom sequences
based on Lisp lists need many more gf implementations
than others, else they simply won't work (because of
list-specific shortcutting optimizations only found
when reading the seq.el code).  Nor is it stated that, for
any representation,  :around methods are probably a bad
idea. Nor is it stated that if you just implement the bare
minimum (which is described in seq.el's header) you will
likely get very poor performance.

Such a manual is where one would find sentences like
"to make a working custom sequence that is accepted by
any seq-* call, the user must add implementations to
the following N generics.  To get good performance
you must also add implementations to these other M
generics".  (Dmitry's proposed seq-contains-pred would
be in this set).

João

[-- Attachment #2: Type: text/html, Size: 2616 bytes --]

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Thu, 16 Nov 2023 21:58:51 +0000
> Cc: Michael Heerdegen <michael_heerdegen@web.de>, emacs-devel <emacs-devel@gnu.org>
> 
> On Thu, Nov 16, 2023 at 4:42 PM Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > ?? There's a whole section in the ELisp manual called "Sequence
> > Functions", which documents the seq-* functions.
> 
> Yes, but from a custom sequence implementors perspective, 
> the "best" documentation is still that bit in the header, 
> which describes "you must implement these to get a working 
> custom sequence".
> 
> So while the Elisp manual addresses custom sequences
> (very briefly -- it just tells the user to look up generic
> functions) it spends most time describing the generic 
> functions one by one, and while that's important, it's 
> just not enough.

Feel free to suggest improvements and additions to what we have.  Such
improvements are always welcome here.  I just answered your original
question, viz.:

> > seq.el's documentation is where?  I don't see it.

which seemed to imply that you didn't know at all that we have seq.el
functions described in the manual.

> It doesn't talk about the guarantees that the framework 
> offers with respect to when they are called, or if they 
> are called at all. For example, while I didn't read the 
> whole thing, I don't think it is stated that custom sequences
> based on Lisp lists need many more gf implementations 
> than others, else they simply won't work (because of 
> list-specific shortcutting optimizations only found 
> when reading the seq.el code).  Nor is it stated that, for 
> any representation,  :around methods are probably a bad
> idea. Nor is it stated that if you just implement the bare
> minimum (which is described in seq.el's header) you will 
> likely get very poor performance.

It says this:

     All functions defined in this library are free of side-effects; i.e.,
  they do not modify any sequence (list, vector, or string) that you pass
  as an argument.  Unless otherwise stated, the result is a sequence of
  the same type as the input.  For those functions that take a predicate,
  this should be a function of one argument.

     The ‘seq.el’ library can be extended to work with additional types of
  sequential data-structures.  For that purpose, all functions are defined
  using ‘cl-defgeneric’.  *Note Generic Functions::, for more details
  about using ‘cl-defgeneric’ for adding extensions.

If this is not enough, once again, please feel free to suggest
improvements and extensions.

> Such a manual is where one would find sentences like 
> "to make a working custom sequence that is accepted by
> any seq-* call, the user must add implementations to 
> the following N generics.  To get good performance 
> you must also add implementations to these other M 
> generics".

Do you see something like that in other Lisp-related manuals we have?
Does cl.info has this information regarding cl-lib?

I think the answer is NO, because we generally don't say such things
in our manuals, except in rare exceptional cases (which the seq.el one
isn't, IMNSHO).  Perhaps we should, but then we have a lot of work to
do on our manuals, and that work is not only the necessary writeups,
but also non-trivial research and analysis.  Look how much effort was
needed to perform a similar analysis on seq.el.  Volunteers are
welcome to do the same in other places, and suggest additions to the
manuals for any significant findings.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

Eli Zaretskii <eliz@gnu.org> writes:

> Such improvements are always welcome here.  I just answered your
> original question, viz.:
>
>> > seq.el's documentation is where?  I don't see it.
>
> which seemed to imply that you didn't know at all that we have seq.el
> functions described in the manual.

Sorry, I should have been clearer.  I was talking for some while in this
thread about API implementor's documentation, a description on how to
_extend_ seq.el to define new sequence types, not how to _use_ it in
programs to process sequences.

> Feel free to suggest improvements and additions to what we have.

I've been doing that.  But it still feels like some people are not
convinced that that documentation is essential at all (you seem to be
one of them, reading further).  And before this manual is written down,
we have to come to an agreement of exactly what guarantees seq.el gives
the user who wants to extend it for new sequence types.

> Do you see something like that in other Lisp-related manuals we have?

Yes, for my part see Flymake, Eglot and JSONrpc manuals.  They're not
great, but they each have a section on how to extend the library to
provide new diagnostic backends, new request handlers and new JSONRPC
transports, respectively.

> Does cl.info has this information regarding cl-lib? I think the answer
> is NO

Of course, and that's obvious.  cl-lib is not extensible.

>, because we generally don't say such things in our manuals,

Of course we do.  See 21.6.7

    * Programmed Completion::  Writing your own completion function.

It tells you how to _write_ a completion backend.  The instructions on
how to _use_ completion functions to complete things are elsewhere.

I use this part of the manual all the time, and so does everyone else
writing a backend.  We've been upholding that contract very well for a
number of Emacs releases so, we don't break backends willy-nilly.  I
recently added an optimization (completion-lazy-hilit) that doesn't
break anything (AFAI and other reviewers could tell).

> except in rare exceptional cases

Huh, is Flymake rare and exceptional?  I see a lot of backends popping
up that (correctly) take inspiration from the manual describing how to
extend Flymake.

> (which the seq.el one isn't, IMNSHO).

As far as I could gather from this discussion, seq.el is to be
recommended to users as the preferred way to process sequences in their
Lisp programs, most any program, including basic facilities in the core.

I have shown that performance aspects of this library, even when applied
to primitive data types such as lists and vectors, vary in accordance
with the presence or absence of other custom sequences, because of the
GF-based nature of seq.el.  The nature of these custom sequences changes
the magnitude of the impact.  So IMO it is wise to explain to users
exactly what custom sequences are and aren't supported/recommended.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

Eli Zaretskii <eliz@gnu.org> writes:

> Such improvements are always welcome here.  I just answered your
> original question, viz.:
>
>> > seq.el's documentation is where?  I don't see it.
>
> which seemed to imply that you didn't know at all that we have seq.el
> functions described in the manual.

Sorry, I should have been clearer.  I was talking for some while in this
thread about API implementor's documentation, a description on how to
_extend_ seq.el to define new sequence types, not how to _use_ it in
programs to process sequences.

> Feel free to suggest improvements and additions to what we have.

I've been doing that.  But it still feels like some people are not
convinced that that documentation is essential at all (you seem to be
one of them, reading further).  And before this manual is written down,
we have to come to an agreement of exactly what guarantees seq.el gives
the user who wants to extend it for new sequence types.

> Do you see something like that in other Lisp-related manuals we have?

Yes, for my part see Flymake, Eglot and JSONrpc manuals.  They're not
great, but they each have a section on how to extend the library to
provide new diagnostic backends, new request handlers and new JSONRPC
transports, respectively.

> Does cl.info has this information regarding cl-lib? I think the answer
> is NO

Of course, and that's obvious.  cl-lib is not extensible.

>, because we generally don't say such things in our manuals,

Of course we do.  See 21.6.7

    * Programmed Completion::  Writing your own completion function.

It tells you how to _write_ a completion backend.  The instructions on
how to _use_ completion functions to complete things are elsewhere.

I use this part of the manual all the time, and so does everyone else
writing a backend.  We've been upholding that contract very well for a
number of Emacs releases so, we don't break backends willy-nilly.  I
recently added an optimization (completion-lazy-hilit) that doesn't
break anything (AFAI and other reviewers could tell).

> except in rare exceptional cases

Huh, is Flymake rare and exceptional?  I see a lot of backends popping
up that (correctly) take inspiration from the manual describing how to
extend Flymake.

> (which the seq.el one isn't, IMNSHO).

As far as I could gather from this discussion, seq.el is to be
recommended to users as the preferred way to process sequences in their
Lisp programs, most any program, including basic facilities in the core.

I have shown that performance aspects of this library, even when applied
to primitive data types such as lists and vectors, vary in accordance
with the presence or absence of other custom sequences, because of the
GF-based nature of seq.el.  The nature of these custom sequences changes
the magnitude of the impact.  So IMO it is wise to explain to users
exactly what custom sequences are and aren't supported/recommended.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Cc: michael_heerdegen@web.de,  emacs-devel@gnu.org
> Date: Fri, 17 Nov 2023 10:47:29 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Feel free to suggest improvements and additions to what we have.
> 
> I've been doing that.  But it still feels like some people are not
> convinced that that documentation is essential at all (you seem to be
> one of them, reading further).

Show me the docs changes you think should be installed, and let's take
it from there.  Until then, this kind of hand-waving and vague
accusations are not useful.

In general, I suggest to be less "combative" in this and similar
discussions.  It doesn't help arriving to a consensus.  But I digress.

> And before this manual is written down, we have to come to an
> agreement of exactly what guarantees seq.el gives the user who wants
> to extend it for new sequence types.

The code is there, so AFAIU any "agreements" boil down to
understanding what it does and what it does not.  In which case,
perhaps bringing the author on board of this discussion would be
helpful?

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 17, 2023 at 12:23 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > But it still feels like some people are not
> > convinced that that documentation is essential at all (you seem to be
> > one of them, reading further).
> Show me the docs changes you think should be installed, and let's take
> it from there.  Until then, this kind of hand-waving and vague
> accusations are not useful.

You can't make this stuff up.  Didn't you just write 30 minutes ago

  because we generally don't say such things
  in our manuals, except in rare exceptional cases (which the
  seq.el one isn't, IMNSHO)

?? Is this not crystal clear evidence that IYNSHO this type
of documentation for seq.el isn't needed?  Are you sure _I'm_ the
vague accuser??

I hope that by sniping the rest of the message you at least
acknowledge that we _do_ say such things in our Elisp manual.

> In general, I suggest to be less "combative" in this and similar
> discussions.  It doesn't help arriving to a consensus.  But I digress.

Indeed.  And likewise.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Fri, 17 Nov 2023 13:14:11 +0000
> Cc: michael_heerdegen@web.de, emacs-devel@gnu.org
> 
> On Fri, Nov 17, 2023 at 12:23 PM Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > > But it still feels like some people are not
> > > convinced that that documentation is essential at all (you seem to be
> > > one of them, reading further).
> > Show me the docs changes you think should be installed, and let's take
> > it from there.  Until then, this kind of hand-waving and vague
> > accusations are not useful.
> 
> You can't make this stuff up.  Didn't you just write 30 minutes ago
> 
>   because we generally don't say such things
>   in our manuals, except in rare exceptional cases (which the
>   seq.el one isn't, IMNSHO)
> 
> ?? Is this not crystal clear evidence that IYNSHO this type
> of documentation for seq.el isn't needed?

No, it states the facts as I know them.  It doesn't mean I necessarily
think we should keep doing that.

> Are you sure _I'm_ the vague accuser??

Show me that I was wrong by submitting the docs you think should be
added, and let's take it from there.

> I hope that by sniping the rest of the message you at least
> acknowledge that we _do_ say such things in our Elisp manual.

No, it just means that I have nothing useful to say about that, and
prefer to say nothing.

> > In general, I suggest to be less "combative" in this and similar
> > discussions.  It doesn't help arriving to a consensus.  But I digress.
> 
> Indeed.  And likewise.

Yeah, right.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 17, 2023 at 3:01 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > You can't make this stuff up.  Didn't you just write 30 minutes ago
> >
> >   because we generally don't say such things
> >   in our manuals, except in rare exceptional cases (which the
> >   seq.el one isn't, IMNSHO)
> >
> > ?? Is this not crystal clear evidence that IYNSHO this type
> > of documentation for seq.el isn't needed?
>
> No, it states the facts as I know them.  It doesn't mean I necessarily
> think we should keep doing that.

This gives me serious vibes of Monty Python's "Not necessarily: I could be
arguing in my spare time".

> > Are you sure _I'm_ the vague accuser??
>
> Show me that I was wrong by submitting the docs you think should be
> added, and let's take it from there.

OK.  Noting that there's no documentation of a particular kind for a
given library without producing such documentation immediately merits
the super-friendly epithet of "handwaver" and "vague accuser".  So
now I know.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

João Távora <joaotavora@gmail.com> writes:

> On Fri, Nov 17, 2023 at 3:01 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
>> > You can't make this stuff up.  Didn't you just write 30 minutes ago
>> >
>> >   because we generally don't say such things
>> >   in our manuals, except in rare exceptional cases (which the
>> >   seq.el one isn't, IMNSHO)
>> >
>> > ?? Is this not crystal clear evidence that IYNSHO this type
>> > of documentation for seq.el isn't needed?
>>
>> No, it states the facts as I know them.  It doesn't mean I necessarily
>> think we should keep doing that.
>
> This gives me serious vibes of Monty Python's "Not necessarily: I could be
> arguing in my spare time".

You made me spill my coffee :-) :-) :-).
Thanks for reminding me of that.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> In which case, perhaps bringing the author on board of this discussion
> would be helpful?

In the past Nicolas had been quite busy most of the time and responded
only sporadically.  I'm not even sure his email address is still valid.

The co-author had been Stefan M, and to some small amount, I had been
involved, among others.

Michael.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Date: Fri, 17 Nov 2023 15:49:16 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > In which case, perhaps bringing the author on board of this discussion
> > would be helpful?
> 
> In the past Nicolas had been quite busy most of the time and responded
> only sporadically.  I'm not even sure his email address is still valid.

That's fine, but the worst that could happen here is that Nicolas will
not respond, in which case we lose nothing.  But maybe he will respond...

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > ?? There's a whole section in the ELisp manual called "Sequence
  > > Functions", which documents the seq-* functions.

  > Yes, but from a custom sequence implementors perspective,
  > the "best" documentation is still that bit in the header,
  > which describes "you must implement these to get a working
  > custom sequence".

  > So while the Elisp manual addresses custom sequences
  > (very briefly -- it just tells the user to look up generic
  > functions) it spends most time describing the generic
  > functions one by one, and while that's important, it's
  > just not enough.

We need both of these two kinds of documentation.
The Emacs Lisp Reference Manual should say how to _use_ the seq functions,
and comments in seq.el should say how to understand and maintain it.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Dmitry Gutov <dmitry@gutov.dev> writes:

> First of all, the type dispatch _does_ happen more than once per
> sequence in the current master. That doesn't seem to hurt much while
> the method is not specialized (only has the default implementation),
> but has impact as soon as the function gets additional method
> definitions.

Thanks for taking a look az the effect of having more than one method.
It's pretty much what is to expect, I guess. It's not bad though.

Next hit is then probably in the case where effective method functions
have to be constructed. Emfs being the result of combining applicable
methods. I think in Emacs that boils down to combining a main method
with around, after, before methods.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Michael Heerdegen]

João Távora <joaotavora@gmail.com> writes:

> Maybe seq.el can be made faster too?  Who knows, but it seems
> difficult without breaking at least some of its defgeneric-based
> contract.

Of course can it, and it is not difficult in most cases.

For example, `cl-some' has two code paths, while `seq-some' has only one
that corresponds to the more general and slower one in `cl-some'.
Nothing forbids us to add the same optimization to the `seq-some'
algorithm.

These kinds of benchmarks are more or less irrelevant.  seq.el is new
while cl-lib has been tuned and optimized for decades.  Nobody has so
far ever decidedly tried to optimize seq.el with respect to efficiency.
Not because it would be hard (it isn't) - just because nobody did the
job yet.  seq.el is in a state where most functions work reliably and
most bugs of the initial code have been fixed, but improvements are
still low relatively hanging fruits.  Unless we decide that seq.el
should intentionally have a more simple design.  But then, this would
have to be discussed again, it is a minor question compared to this one
(about cl-lib usage), IMO.

So please let's avoid such benchmarking contests here unless you want to
work on seq.el (then a bug report is a better place for discussion).
Because it has little relevance here.

Michael.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Mon, Nov 13, 2023 at 8:34 AM Michael Heerdegen
<michael_heerdegen@web.de> wrote:
>
> João Távora <joaotavora@gmail.com> writes:
>
> > Maybe seq.el can be made faster too?  Who knows, but it seems
> > difficult without breaking at least some of its defgeneric-based
> > contract.
>
> Of course can it, and it is not difficult in most cases.

Go right ahead, presuming you understand the pros and cons.

> For example, `cl-some' has two code paths, while `seq-some' has only one
> that corresponds to the more general and slower one in `cl-some'.
> Nothing forbids us to add the same optimization to the `seq-some'
> algorithm.

Sure you could have seq.el require 'cl-lib.el' ;-)

Now really, sure?  What about those generic functions that you
presumably have to shortcut?  Won't it break an existing contract to users
that all of the sudden you're not calling them anymore for certain
types of sequence?  And how are you going to do this without introducing
non-destructive versions of the seq.el utils?  Isn't non-mutability
in the contract?  And wasn't it a goal here to  end up with a smaller
dictionary of such utils?  Won't it bloat up comparibly to cl-lib while
still being less versatile?  These questions might have answers, of
course, but they're not answers as trivial as you make them sound.

> These kinds of benchmarks are more or less irrelevant.

I beg to differ, I think a recommended sequence processing
function that needlessly conses and takes orders of magnitude more
time to do its job is really quite relevant.  Elisp is still a
"LISt Processing" language by and large.

But also, I have to ask: what _would_ be a relevant benchmark
to you?

Just for cl-set-{n}difference, I'm seeing significant usage
in core right now, direct _and_ indirect.  I'm not sure all
the authors, let alone users, of these packages would like
to learn that these parts of their code have suddenly become
a full order of magnitude slower, let alone users using such
things, by virtue of some cl-lib.el -> seq.el operation
as is being suggested here.

At the very least, if this idea is to go ahead, answering
all those questions about seq.el's performance and measuring
everything twice should be a prerequisite.

> seq.el is new
> while cl-lib has been tuned and optimized for decades.

No, the point is that it hasn't.  Some of the destructive versions
weren't even destructive at all!  if you take a look at my code you'll
notice I optimized cl-lib considerably in very few cases.
There's a lot more that can be optimized there.

Also it  seems that the optional keyword arguments in these
functions are  not a  cause for performance concerns at all,
at least in these cases.  So just for learning that, these
benchmarks are useful.

> Nobody has so
> far ever decidedly tried to optimize seq.el with respect to efficiency.

The seq.el file has "optimized implementation for lists section" by
tweaking the generics for lists.  I find plausible the designer of
seq.el noticed that it is still much slower to do this but wanted to
keep a specific generic function contract anyway.

IOW, there's nothing wrong with declaring "library X is slow, but
powerful".  It's just that IMO it's bad to tell people to use just
that library all the time for a class of jobs.

> So please let's avoid such benchmarking contests here unless you want to
> work on seq.el (then a bug report is a better place for discussion).
> Because it has little relevance here.

Last I checked, the title of this thread is still "What's missing
in Elisp that makes people want to use cl-lib?".

One answer among several: fast sequence functions.

So, uncomfortable as the results may be (for now at least)
for proponents of "seq.el only" or "seq.el first", as long as
there's talk of deprecating cl-lib in favour of these
alternatives, I think concrete evidence and hard numbers very
much belong to this discussion.  Surely much more useful than
many ideological considerations about what complexity is.

If they lead to an improvement of seq.el in terms of speed,
so much the better.  If they lead to improvement of seq.el in
terms of versatility (like allowing backward traversal, key functions),
even better.

Finally, please note I'm not trying to bash seq.el.  I think it
has its place -- polymorphic sequences are great, if I ever
need them -- , but its use should _also_ be discriminate, not
indiscriminate.  Same for your set.el idea: IMHO it's very welcome,
but I think it will suffer (and gain, of course) from the same
kind of tradeoffs as seq.el.


João Távora

Re: What's missing in ELisp that makes people want to use cl-lib?

[Michael Heerdegen]

João Távora <joaotavora@gmail.com> writes:

> Now really, sure?  What about those generic functions that you
> presumably have to shortcut?  Won't it break an existing contract to
> users that all of the sudden you're not calling them anymore for
> certain types of sequence?

If you don't make a mess, no.  It's of course possible to choose
different implementations (algorithms) for different types of sequences.
There a few technical reasons why seq.el should be slower.

> And how are you going to do this without introducing non-destructive
> versions of the seq.el utils?

That's a thing we probably can't do, yes.  But this shouldn't lead to
algorithms reaching a different complexity class, or running slower by a
factor N with large N.

Do we have places in the Emacs Elisp sources where this would make a
significant difference?

>  And wasn't it a goal here to end up with a smaller dictionary of such
> utils?  Won't it bloat up comparibly to cl-lib while still being less
> versatile?

The result will still be much smaller than cl-lib.  The idea is of
course not to use nonsense algorithms just to spare some dozens of
Bytes.

> > These kinds of benchmarks are more or less irrelevant.
>
> I beg to differ, I think a recommended sequence processing
> function that needlessly conses and takes orders of magnitude more
> time to do its job is really quite relevant.  Elisp is still a
> "LISt Processing" language by and large.
>
> But also, I have to ask: what _would_ be a relevant benchmark
> to you?

You misunderstood.  Of course do these benchmarks matter very much, but
as I said, please, this thread is not a seq.el bug report.  What I said
is that benchmarks do not much tell us about how our future treatment of
cl-lib should look like as long as seq.el can be improved and sped up
and you argument with seq.el efficiency.

> Just for cl-set-{n}difference, I'm seeing significant usage in core
> right now, direct _and_ indirect.  I'm not sure all the authors, let
> alone users, of these packages would like to learn that these parts of
> their code have suddenly become a full order of magnitude slower, let
> alone users using such things, by virtue of some cl-lib.el -> seq.el
> operation as is being suggested here.

These actually do not really fit in seq.el but would rather be a natural
part of set.el.

> No, the point is that it hasn't.  Some of the destructive versions
> weren't even destructive at all!  if you take a look at my code you'll
> notice I optimized cl-lib considerably in very few cases.
> There's a lot more that can be optimized there.

Other parts were optimized in the time, of course.

> The seq.el file has "optimized implementation for lists section" by
> tweaking the generics for lists.  I find plausible the designer of
> seq.el noticed that it is still much slower to do this but wanted to
> keep a specific generic function contract anyway.

Again, such a contract does _not_ per se have implications about
efficiency.  A seq.el implementation of a generic function for lists can
find the type is 'list' and then use exactly the same algorithm as
cl-lib.  All you have to do additionally is one check (`listp').
This is currently not the case everywhere in seq.el, some algorithms
perform repeated, strictly speaking redundant, tests, but that can (and
should be, where it matters) avoided.

> Finally, please note I'm not trying to bash seq.el.  I think it
> has its place -- polymorphic sequences are great, if I ever
> need them -- , but its use should _also_ be discriminate, not
> indiscriminate.  Same for your set.el idea: IMHO it's very welcome,
> but I think it will suffer (and gain, of course) from the same
> kind of tradeoffs as seq.el.

The only relevant question for me in this message is: is an improved
seq.el good enough to replace the sequence functions in cl-lib?  I think
it is.

To convince Eli that it is not, he would need something like a real-life
case where the seq.el functions make some program significantly slower,
and where we know that this can't be improved in seq.el (something like
sorting a million three-element lists in a row is not a good example).


Michael.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> The only relevant question for me in this message is: is an improved
> seq.el good enough to replace the sequence functions in cl-lib?  I think
> it is.

The real issue is hidden in there: why do "we" need to replace cl-lib
again? IOW, I think the "seq.el question" is only part of something
else going on.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Tue, Nov 14, 2023 at 2:46 PM Michael Heerdegen
<michael_heerdegen@web.de> wrote:
>
> João Távora <joaotavora@gmail.com> writes:
>
> > Now really, sure?  What about those generic functions that you
> > presumably have to shortcut?  Won't it break an existing contract to
> > users that all of the sudden you're not calling them anymore for
> > certain types of sequence?
>
> If you don't make a mess, no.  It's of course possible to choose
> different implementations (algorithms) for different types of sequences.
> There a few technical reasons why seq.el should be slower.

Beg to differ.  See my latest benchmarks sent to Dmitry and his
updated code in feature/cl-lib-improvements.  Sure, it can be argues
that you can make it not as horribly slow as it is in some cases,
but never quite as fast.  And you'll be breaking compatibility all
the way, the further you go.

> > And how are you going to do this without introducing non-destructive
> > versions of the seq.el utils?
>
> That's a thing we probably can't do, yes.  But this shouldn't lead to
> algorithms reaching a different complexity class, or running slower by a
> factor N with large N.

Of course, algorithms don't change complexity :-) That would have
been too terrible.  But when you compare non-destructive versions of
some seq operations to cl-lib destructive operations "that we
probably can't do", you will see large N.  A snippet from my results
(again, see message to Dmitry with these files).

("set difference, small lists, custom pred"
  (cl-nset-difference "FASTEST" 0.201229483 15 0.11990391199999983)
  (cl-set-difference "5.0x SLOWER" 1.002034638 39 0.31673883500000066)
  (seq-difference-3 "15.6x (rel 3.1x) SLOWER" 3.136544874 194
   1.7686539850000003)
  (seq-difference "18.9x (rel 1.2x) SLOWER" 3.795912411 260
 2.140581182000002))
 ("set difference, big lists, custom pred"
  (cl-nset-difference "FASTEST" 0.011600632 0 0.0)
  (cl-set-difference "5.4x SLOWER" 0.062588482 1 0.008457714000000394)
  (seq-difference-3 "18.1x (rel 3.4x) SLOWER" 0.21003662699999998 12
   0.10525734000000142)
  (seq-difference "25.5x (rel 1.4x) SLOWER" 0.29535328699999996 19
 0.16559472099999972))

Where seq-difference-3's is what I think is Dmitry's best attempt
so far.

> Do we have places in the Emacs Elisp sources where this would make a
> significant difference?
>
> >  And wasn't it a goal here to end up with a smaller dictionary of such
> > utils?  Won't it bloat up comparibly to cl-lib while still being less
> > versatile?
>
> The result will still be much smaller than cl-lib.

It's completely unfair to compare seq.el to cl-lib.el in general.
You should be comparing to the cl-lib's sequence dictionary.

This is such a common (and annoying I must say) misconception

Here's the sequences dictionary summarized directly from the
CL Hyperspec.

   https://www.lispworks.com/documentation/lw70/CLHS/Body/c_sequen.htm

Notice how neatly organized it is.  Compare it to the seq.el shortdoc.
I don't see a big difference in size, do you?  When compared directly
to the cl-lib.el versions seq.el are almost always less versatile
and often greatly less versatile (not to mention slow as heck at least
int the current form).

Sure it has interesting things like seq-group, but it is missing
other interesting things like the "mismatch" operations, and
super-important things like destructive versions of things.

> These actually do not really fit in seq.el but would rather be a natural
> part of set.el.

But they're there.  Are they bloat?  I think it's pretty good to
have these "set difference" operations on lists, even if it obviously
the naming is off (as it is in mnay other languages by the way).

Anyway naming aside sometimes it's just what you want.  Of course,
if you also want fast, use cl-lib.el.

> > No, the point is that it hasn't.  Some of the destructive versions
> > weren't even destructive at all!  if you take a look at my code you'll
> > notice I optimized cl-lib considerably in very few cases.
> > There's a lot more that can be optimized there.
>
> Other parts were optimized in the time, of course.

And so were seq's

>
> > The seq.el file has "optimized implementation for lists section" by
> > tweaking the generics for lists.  I find plausible the designer of
> > seq.el noticed that it is still much slower to do this but wanted to
> > keep a specific generic function contract anyway.
>
> Again, such a contract does _not_ per se have implications about
> efficiency.

Of course it does.  The "list optimizations" I pointed to, are
designed for efficiency, to state the obvious.  And they already break
the contract laid out in the very spare seq.el documentation
that I only need to implement those 6 or 7 generics to have
a fully functioning custom seq.  I need only make something based
on lists or arrays to be bitten in the tail by those optimizations.
And even without this "based on lists" thing, the seq-contains-p
performance hog had to be circumvented by changing the contract,
changing the imaginary lines where it said

   "the user agrees to implement seq-contains-p"

to

   "the user agrees to implement seq-contains-pred"

> > Finally, please note I'm not trying to bash seq.el.  I think it
> > has its place -- polymorphic sequences are great, if I ever
> > need them -- , but its use should _also_ be discriminate, not
> > indiscriminate.  Same for your set.el idea: IMHO it's very welcome,
> > but I think it will suffer (and gain, of course) from the same
> > kind of tradeoffs as seq.el.
>
> The only relevant question for me in this message is: is an improved
> seq.el good enough to replace the sequence functions in cl-lib?  I think
> it is.

An "improved" seq.el?  Sure, maybe.  As we say in my country,
"if my grandmother had wheels whe should be school bus".  Let's see
what wheels you put on seq.el.

Currently, please no.  DO use seq.el for custom sequences, or when
there is good suspicion that custom sequences might appear.  Do NOT
use seq.el for macro expansions, bread and butter list processing, etc.
Do NOT use seq-let for example in anything that might eventually
be called many times.

> To convince Eli that it is not, he would need something like a real-life
> case where the seq.el functions make some program significantly slower,
> and where we know that this can't be improved in seq.el (something like
> sorting a million three-element lists in a row is not a good example).

I think we should be cautious instead, and take those benchmarks
seriously (like Dmitry is taking them).

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > But when you need to process plists, for example, it becomes
  > very handy.  This construct, which does require knowing a bit
  > of the mini-language that is cl-loop,

  >   (cl-loop for (k v) on plist by #'cddr collect (cons k v))

It is certainly concise.  If I saw that code I would not understand it,
but if you said what job it does I could see how it might do that.

That's because cl-loop defines its own little language, which I don't know.

The question of readability here is a combination of two questions
that work against each other.  First, how readable are the programs ou
can write in that little language?  Second, how readable are they to
those who don't know the little language?

This is a rather extreme example of how the frequent use of the CL
constructs, so that they become part of typical Emacs Lisp code, adds
to what people need to know in order to understand typical Emacs Lisp
code.
-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Sat, 11 Nov 2023 21:57:02 -0500
> 
>   > But when you need to process plists, for example, it becomes
>   > very handy.  This construct, which does require knowing a bit
>   > of the mini-language that is cl-loop,
> 
>   >   (cl-loop for (k v) on plist by #'cddr collect (cons k v))
> 
> It is certainly concise.  If I saw that code I would not understand it,
> but if you said what job it does I could see how it might do that.
> 
> That's because cl-loop defines its own little language, which I don't know.

Btw, the above is a very simple use of cl-loop.  We have quite a few
of much more complex ones.  For example:

  (cl-loop
   with comp-ctxt = (make-comp-cstr-ctxt)
   with h = (make-hash-table :test #'eq)
   for (f type-spec) in comp-known-type-specifiers
   for cstr = (comp-type-spec-to-cstr type-spec)
   do (puthash f cstr h)
   finally return h)

or this:

            (cl-loop for i = 0 then (+ i (if (eq (nth i list) ?\\) 4 1))
                     for var = (nth i list)
                     while (< i size)
                     if (eq var ?\\)
                     collect (string-to-number
                              (concat (cl-subseq list (+ i 1) (+ i 4))) 8)
                     else
                     collect var))

or this:

  (cl-loop
   named loop
   with above
   for lane in (comp-cstr-ctxt-typeof-types comp-ctxt)
   do (let ((x (memq type lane)))
        (cond
         ((null x) nil)
         ((eq x lane) (cl-return-from loop x)) ;A base type: easy case.
         (t (setq above
                  (if above (comp--intersection x above) x)))))
   finally return above))

It isn't an accident that the reference documentation of cl-loop in
cl.info takes a whopping 700(!) lines.

RE: [External] : Re: What's missing in ELisp that makes people want to use cl-lib?

[Drew Adams]

> Btw, the above is a very simple use of cl-loop.  We have quite a few
> of much more complex ones.  For example:
> 
>   (cl-loop
>    with comp-ctxt = (make-comp-cstr-ctxt)
>    with h = (make-hash-table :test #'eq)
>    for (f type-spec) in comp-known-type-specifiers
>    for cstr = (comp-type-spec-to-cstr type-spec)
>    do (puthash f cstr h)
>    finally return h)
> 
> or this:
> 
>             (cl-loop for i = 0 then (+ i (if (eq (nth i list) ?\\) 4 1))
>                      for var = (nth i list)
>                      while (< i size)
>                      if (eq var ?\\)
>                      collect (string-to-number
>                               (concat (cl-subseq list (+ i 1) (+ i 4))) 8)
>                      else
>                      collect var))
> 
> or this:
> 
>   (cl-loop
>    named loop
>    with above
>    for lane in (comp-cstr-ctxt-typeof-types comp-ctxt)
>    do (let ((x (memq type lane)))
>         (cond
>          ((null x) nil)
>          ((eq x lane) (cl-return-from loop x)) ;A base type: easy case.
>          (t (setq above
>                   (if above (comp--intersection x above) x)))))
>    finally return above))
> 
> It isn't an accident that the reference documentation of cl-loop in
> cl.info takes a whopping 700(!) lines.

FWIW:

"Don't Loop, Iterate":
https://iterate.common-lisp.dev/doc/Don_0027t-Loop-Iterate.html

"The Iterate Manual":
https://iterate.common-lisp.dev/iterate-manual.pdf

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Btw, the above is a very simple use of cl-loop.  We have quite a few
  > of much more complex ones.  For example:

  >   (cl-loop
  >    with comp-ctxt = (make-comp-cstr-ctxt)
  >    with h = (make-hash-table :test #'eq)
  >    for (f type-spec) in comp-known-type-specifiers
  >    for cstr = (comp-type-spec-to-cstr type-spec)
  >    do (puthash f cstr h)
  >    finally return h)

Boy that is hard to understand.  I suggest we rewrite those cl-loop calls
into ordinary Lisp, avoiding use of cl-loop.

I looked at the manual for the iter construct ("iterate").  The added
parentheses make it fit better into Lisp.

However, I have to ask how the complexity of the usage of iter.
Does iter offer a corresponding construct for each cl-loop construct?

Are the iter constructs in 1-1 correspondence with cl-loop constructs?
If so, replacing c-loop with iter might not reduce the complexity that
either one adds to Emacs Lisp.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Sebastián Monía]

Richard Stallman <rms@gnu.org> writes:

> Boy that is hard to understand.  I suggest we rewrite those cl-loop calls
> into ordinary Lisp, avoiding use of cl-loop.

I find that loop very readable. I am not an expert Lisper. Beginner at
CL, maybe mid-level if being generous. I did write a couple Emacs
packages.

I have been following this thread somewhat closely. My opinion maybe
isn't worth much because I am not a regular contributor. But maybe it
helps as one of the angles discussed earlier was related to the size of
Elisp and complexity for newcomers.

1. This kind of thread is more "scary" to potential Emacs contributors
than any degree of complexity in cl-lib. I can confirm that.

2. There's a lot of vague arguments for and against the library that, as
an semi-outsider, read more like stylistic preference. Sometimes
familiarity, or lack of it. Few objective reasons for/against.

3. Anyone interested in writing code for Emacs has to make the leap to a
Lisp (elisp, in this case) which is already quite alien to a large
population. For non-programmers, the problem is indistinguishable from
learning any other language. Even for most people familiar with software
development, cl-lib is the least of their difficulties.

I hope that was in some way helpful.
Thank you,
Sebastián

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Sebastián Monía <code@sebasmonia.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  joaotavora@gmail.com,  emacs-devel@gnu.org
> Date: Mon, 13 Nov 2023 22:31:59 -0500
> 
> 1. This kind of thread is more "scary" to potential Emacs contributors
> than any degree of complexity in cl-lib. I can confirm that.

There's nothing to be scared of.  People express their opinions,
sometimes in strong and even harsh terms.  Nothing special, it happens
here all the time.  People are people.  Show me another development
forum with many very different participants that doesn't exhibit such
behavior from time to time.

> 2. There's a lot of vague arguments for and against the library that, as
> an semi-outsider, read more like stylistic preference. Sometimes
> familiarity, or lack of it. Few objective reasons for/against.

This, too, is quite usual in forums that discuss coding and styles.
Many people don't possess basic abilities to stay focused and conduct
a rational, constructive, and meaningful discussion, and we are no
different here.  Just open any TV channel where a few people discuss
something non-trivial, and you will see it more often than not.

> 3. Anyone interested in writing code for Emacs has to make the leap to a
> Lisp (elisp, in this case) which is already quite alien to a large
> population. For non-programmers, the problem is indistinguishable from
> learning any other language.

I came to Emacs Lisp from very different languages, and don't remember
that as being a leap, just an adjustment of how I think about code and
data.

> Even for most people familiar with software development, cl-lib is
> the least of their difficulties.

Not my experience, FWIW.  Maybe not the most of my difficulties, but
it certainly isn't negligible.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Sebastián Monía wrote:

> 1. This kind of thread is more "scary" to potential Emacs
>    contributors than any degree of complexity in cl-lib.
>    I can confirm that.

Very true! It is alienating to newcomers but also to
experienced people who have solved problem with cl-lib in
Emacs and saw no problem with that, on the contrary, they used
what they thought was the best way to do it.

> 2. There's a lot of vague arguments for and against the
>    library that, as an semi-outsider, read more like
>    stylistic preference. Sometimes familiarity, or lack of
>    it. Few objective reasons for/against.

Again, true. In particular, what is lacking is whatever the
alternative would be. If cl-lib has increased the general
complexity, fine, maybe that is a problem to some. But surely
it is totally impractical to kick it out and for the vast
majority of the places where it is used we can't make the case
not using it will improve the quality of that code on
a case-by-case basis.

> 3. Anyone interested in writing code for Emacs has to make
>    the leap to a Lisp (elisp, in this case) which is already
>    quite alien to a large population. For non-programmers,
>    the problem is indistinguishable from learning any other
>    language. Even for most people familiar with software
>    development, cl-lib is the least of their difficulties.

Very true, there is no huge leap from Elisp not including
cl-lib to Elisp including cl-lib. If used well, cl-lib will on
the contrary reduce the complexity and make the code cleaner.
And in very complicated cases, as we have seen with some
examples of `cl-loop', it is not clear in any way that
re-writing that in any other style would reduce complexity.
On the contrary `cl-loop' can be seen as a handy frame where
all that complexity, when encountered, is neatly organized
and accessible.

At the very least, the people who used cl-lib and cl-loop and
every thing else cl- that they used, they didn't agree that
use was detrimental to anything, style or functionality.
And they solved their problems! In general, we should trust
them, and in particular, we should absolutely not alienate
them or anyone else for no reason other than style issues that
other people have on their contributions to Emacs.

We shipped the library, people used it, that's the situation.

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[T.V Raman]

Emanuel Berg <incal@dataswamp.org> writes:

And to make things concrete, take a look for example at some of the good
uses of cl-loop  within the emacs codebase itself, eg eww.el.

Many of those idiomatic uses would I suspect be  not as elegant  (for
some measure of elegance) without cl-loop

> Sebastián Monía wrote:
>
>> 1. This kind of thread is more "scary" to potential Emacs
>>    contributors than any degree of complexity in cl-lib.
>>    I can confirm that.
>
> Very true! It is alienating to newcomers but also to
> experienced people who have solved problem with cl-lib in
> Emacs and saw no problem with that, on the contrary, they used
> what they thought was the best way to do it.
>
>> 2. There's a lot of vague arguments for and against the
>>    library that, as an semi-outsider, read more like
>>    stylistic preference. Sometimes familiarity, or lack of
>>    it. Few objective reasons for/against.
>
> Again, true. In particular, what is lacking is whatever the
> alternative would be. If cl-lib has increased the general
> complexity, fine, maybe that is a problem to some. But surely
> it is totally impractical to kick it out and for the vast
> majority of the places where it is used we can't make the case
> not using it will improve the quality of that code on
> a case-by-case basis.
>
>> 3. Anyone interested in writing code for Emacs has to make
>>    the leap to a Lisp (elisp, in this case) which is already
>>    quite alien to a large population. For non-programmers,
>>    the problem is indistinguishable from learning any other
>>    language. Even for most people familiar with software
>>    development, cl-lib is the least of their difficulties.
>
> Very true, there is no huge leap from Elisp not including
> cl-lib to Elisp including cl-lib. If used well, cl-lib will on
> the contrary reduce the complexity and make the code cleaner.
> And in very complicated cases, as we have seen with some
> examples of `cl-loop', it is not clear in any way that
> re-writing that in any other style would reduce complexity.
> On the contrary `cl-loop' can be seen as a handy frame where
> all that complexity, when encountered, is neatly organized
> and accessible.
>
> At the very least, the people who used cl-lib and cl-loop and
> every thing else cl- that they used, they didn't agree that
> use was detrimental to anything, style or functionality.
> And they solved their problems! In general, we should trust
> them, and in particular, we should absolutely not alienate
> them or anyone else for no reason other than style issues that
> other people have on their contributions to Emacs.
>
> We shipped the library, people used it, that's the situation.

-- 

RE: [External] : Re: What's missing in ELisp that makes people want to use cl-lib?

[Drew Adams]

> I looked at the manual for the iter construct ("iterate").  The added
> parentheses make it fit better into Lisp.
> 
> However, I have to ask how the complexity of the usage of iter.
> Does iter offer a corresponding construct for each cl-loop construct?
> 
> Are the iter constructs in 1-1 correspondence with cl-loop constructs?
> If so, replacing c-loop with iter might not reduce the complexity that
> either one adds to Emacs Lisp.

I don't know the answer.  I last read the manual
and paper over 30 years ago.  I don't think there's
any attempt to correspond to CL `loop', 1:1 or
otherwise.

But someone else, who has the time and wants to
read it over now, can maybe answer your question
authoritatively.  Preferably someone who is very
familiar with `loop' and interested in the pursuit.

You might also try contacting the author about it
or asking in some general Lisp forum.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Tomas Hlavaty]

On Mon 13 Nov 2023 at 21:58, Richard Stallman <rms@gnu.org> wrote:
>   >   (cl-loop
>   >    with comp-ctxt = (make-comp-cstr-ctxt)
>   >    with h = (make-hash-table :test #'eq)
>   >    for (f type-spec) in comp-known-type-specifiers
>   >    for cstr = (comp-type-spec-to-cstr type-spec)
>   >    do (puthash f cstr h)
>   >    finally return h)
> Boy that is hard to understand.

It is ugly.

> I suggest we rewrite those cl-loop calls
> into ordinary Lisp, avoiding use of cl-loop.

That would be something along the lines of:

(let ((comp-ctxt (make-comp-cstr-ctxt))
      (h (make-hash-table :test #'eq))
      f)
  (while (setq f (pop comp-known-type-specifiers))
    (puthash f (comp-type-spec-to-cstr type-spec) h))
  h)

comp-ctxt does not seem to be used.
Does emacs-lisp compiler not warn about unused variables?

> I looked at the manual for the iter construct ("iterate").  The added
> parentheses make it fit better into Lisp.

I find the idea of iterate even uglier that cl-loop
and I think that it leads people completely astray.
It is not ordinary lisp code or macro, it is a compiler.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 327 bytes --]

On Tue, Nov 14, 2023 at 8:13 AM Tomas Hlavaty <tom@logand.com> wrote:

>
> I find the idea of iterate even uglier that cl-loop
> and I think that it leads people completely astray.
> It is not ordinary lisp code or macro, it is a compiler.
>

Every macro is a compiler: it translates arbitrary forms to Lisp
forms.

[-- Attachment #2: Type: text/html, Size: 625 bytes --]

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> Every macro is a compiler: it translates arbitrary forms to Lisp 
> forms.

That's quibbling.  The transformations carried out by cl-loop are by a
long shot more consequential to the structure of their input than those
by most any other macro there is, which is a demerit and, I think, is
what Thomas meant to communicate.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Tue, Nov 14, 2023 at 10:55 AM Po Lu <luangruo@yahoo.com> wrote:

> The transformations carried out by cl-loop are by a
> long shot more consequential to the structure of their input than those
> by most any other macro there is

That's true.

> , which is a demerit

That's not.

LOOP, pcase, etc, like any other macros, are mini-languages.  Like
any higher-level language, the more powerful they are, the more
transformations they will do on your input.  LOOP is pretty powerful,
and may or may not be suitable for the task at hand, like any other
language.

At any rate Tomas' translation of the loop is just plain
wrong.  Where is 'type-spec' being bound?  It needs to be
destructured: LOOP is a language which is very good
at expressing this fact concisely in a self-documenting way:

  for (f type-spec) in comp-known-type-specifiers

Here from this one line, without ever having looked at Andrea's
code I learn that comp-known-type-specifiers is a list of
2-element lists, where each such tuple is at least made of
an 'f' and a 'type-spec'.

It would take longer (for me!) to learn this fact if
'car' and 'cadr' had been used in subsequent forms away from
detached from a hypothetical (pop comp-known-type-specifiers).

But again, that's just for me, and that's because I have learned
the LOOP language, because I thought it was worth learning
just for reading other people's code.  And it sure was.

But if you don't want to learn it, for whatever personal reason,
open up the macroexpansion, or step through the function with
edebug.  To each his own.

Anyway Tomas' example, if it were to be corrected, would become
"ugly" perhaps "uglier" (as if "ugly" were even an objective
thing)

Here, I think dolist + pcase-let/cl-destructuring-case/seq-let
would also be appropriate.  Though I'm sure someone out there
would find it insufferably ugly or anti-Elisp as well.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> That's not.
>
> LOOP, pcase, etc, like any other macros, are mini-languages.  Like
> any higher-level language, the more powerful they are, the more
> transformations they will do on your input.  LOOP is pretty powerful,
> and may or may not be suitable for the task at hand, like any other
> language.

Four (by some counts five) individuals, all Emacs users, that is to say
the constituency who will ultimately judge these constructs, have taken
issue with these macros on the grounds that they are difficult to
understand.

When all's said and done, the factor that contributes to whether using a
feature is meritorious is not the amount of use that can be gotten out
of them alone.  Rather, it is that amount considered in tandem with the
complexity introduced into its users as perceived from "20,000
ft. views" and the extent to which its users cannot be recast in terms
of different, simpler constructs.  The examples that have been
circulated in support of cl-loop thus far have not demonstrated this
extent to be anything except nil, so it's fair to conclude that most
calls to cl-loop are frivolous, and its complexity is a great failing.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 690 bytes --]

On Tue, Nov 14, 2023 at 2:04 PM Po Lu <luangruo@yahoo.com> wrote:

>
> extent to be anything except nil, so it's fair to conclude that most
> calls to cl-loop are frivolous,


You want to believe that, obviously, what can I do? you're preaching
to your choir of converts.  IMO this just a very poor generatlization
on far-fetched  conceptions of 20.000 feet views and vague defition of
merits.  If you would show me code, I could tell you if I think a given
cl-loop  is frivolous or not like I did there.  But t would still be my
opinion, and we'd descend into this ridiculous thesaurus-fueled
rambling again.

So take care and learn loop or don't, I don't care.

João

[-- Attachment #2: Type: text/html, Size: 2430 bytes --]

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> You want to believe that, obviously, what can I do? you're preaching
> to your choir of converts.  IMO this just a very poor generatlization
> on far-fetched conceptions of 20.000 feet views and vague defition of
> merits.  If you would show me code, I could tell you if I think a
> given cl-loop is frivolous or not like I did there.  But t would still
> be my opinion, and we'd descend into this ridiculous thesaurus-fueled
> rambling again.
>
> So take care and learn loop or don't, I don't care.

Then there's no point in continuing, is there?  Because my intent was to
arrive at a consensus agreeable to me and others who resent the
pervasiveness of cl-loop, and that of like contrivances in general.

If you're insular enough to never accept anything in that vein, then no
consensus can be established, since I'm well aware I too am hidebound to
my own beliefs, formed and tempered by time, such that I will not
concede in this regard.  Thus instead of assenting to a series of
procedures that preclude the introduction of cryptic code, the stage
will be set for more bickering to erupt each time one of us chances upon
code he needlessly cannot understand, consequent on the refusal of
Common Lisp sectarians like you to write simple loops with constructs
native to *this* language.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Tue, Nov 14, 2023 at 3:03 PM Po Lu <luangruo@yahoo.com> wrote:
>
> João Távora <joaotavora@gmail.com> writes:

> > So take care and learn loop or don't, I don't care.
>
> Then there's no point in continuing, is there?  Because my intent was to
> arrive at a consensus agreeable to me and others who resent the
> pervasiveness of cl-loop, and that of like contrivances in general.
>
> If you're insular enough to never accept anything in that vein, then no
> consensus can be established,

If there's anything "frivolous" and "insular" here, it's this rococo arguing
of style matters, full of unfounded adjectivation, not even comparing to
any actual code or working alternatives. This is Lisp, it has macros.  Like C++
has templates.  I use both when I think there is an advantage.  C programmers
naturally don't like any of those, some of them think the preprocessor is truly
the chef's kiss.  I don't judge them, it's useless.  In the other thread even
the very idea of a sequence processing library was aberrant to you.  What can
I say?  Just that it's not for me to judge, not even worth my time

Again: take care and learn loop, cl-position, seq.el, pcase, or don't.
Though I don't use your code, I think you're doing important contributions
to Emacs, keep it up.  I won't be a jury in the silly code beauty pageant.
Really I won't, and neither should you if you want a piece of advice.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> If there's anything "frivolous" and "insular" here, it's this rococo
> arguing of style matters, full of unfounded adjectivation, not even
> comparing to any actual code or working alternatives. This is Lisp, it
> has macros.  Like C++ has templates.  I use both when I think there is
> an advantage.  C programmers naturally don't like any of those, some
> of them think the preprocessor is truly the chef's kiss.  I don't
> judge them, it's useless.  In the other thread even the very idea of a
> sequence processing library was aberrant to you.  What can I say?
> Just that it's not for me to judge, not even worth my time

Of course, that's your choice.  Up until it begins to affect _others_
who must read the code produced by such a choice, at which point their
views become the last word.

So use cl-lib if you want.  Just keep it out of Emacs.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Wed, Nov 15, 2023 at 12:13 AM Po Lu <luangruo@yahoo.com> wrote:

> So use cl-lib if you want.  Just keep it out of Emacs.

The day you become Emacs maintainer, you can issue that order, at
which point I will just leave the project, go back to GitHub or
whatever, no hard feelings.  Until then, stop the rhetorical
buffoonery and your bullying (or don't, but I will just ignore
you).

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

On November 15, 2023 12:28:44 PM GMT+02:00, "João Távora" <joaotavora@gmail.com> wrote:
>
>  Until then, stop the rhetorical
> buffoonery and your bullying

Please keep such personal attacks out of this forum.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Wed, Nov 15, 2023 at 10:41 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> On November 15, 2023 12:28:44 PM GMT+02:00, "João Távora" <joaotavora@gmail.com> wrote:
> >
> >  Until then, stop the rhetorical
> > buffoonery and your bullying
>
> Please keep such personal attacks out of this forum.

Sure thing, and if you want to second Po's order
(not bullish at all, never!) just say so.

In that case I will leave right now, no hard feelings.  Else,
I'll take your advice, step away from this subthread,
keep on using cl-lib.el when I feel it's fit, be open to
feedback about each and every specific use, but not to
mind-numbing religious theories.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Wed, 15 Nov 2023 10:56:01 +0000
> Cc: emacs-devel@gnu.org
> 
> > Please keep such personal attacks out of this forum.
> 
> Sure thing, and if you want to second Po's order
> (not bullish at all, never!) just say so.

My remark has nothing to do with the technical issue at hand, and
doesn't express any opinion on it.  It is only about the words you
used.  I'm asking you and others to please make a point of keeping the
discussions here civilized in tone and wording.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> The day you become Emacs maintainer, you can issue that order, at
> which point I will just leave the project, go back to GitHub or
> whatever, no hard feelings.  Until then, stop the rhetorical
> buffoonery and your bullying

In other words, "it's my way or the highway."  Since that's how you've
been behaving from the outset, I can't be taxed for responding in equal
measure.

> (or don't, but I will just ignore you).

I'll hold you to that.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Wed, Nov 15, 2023 at 1:15 PM Po Lu <luangruo@yahoo.com> wrote:
>
> João Távora <joaotavora@gmail.com> writes:
>
> > The day you become Emacs maintainer, you can issue that order, at
> > which point I will just leave the project, go back to GitHub or
> > whatever, no hard feelings.  Until then, stop the rhetorical
> > buffoonery and your bullying
>
> In other words, "it's my way or the highway."  Since that's how you've
> been behaving from the outset, I can't be taxed for responding in equal
> measure.

Nothing of the sort.  It's my highway, I'm not telling _you_ to go
anywhere or change anything.  My guess is you'll be quite happy
in a cl-lib.el free Emacs, so when you become maintainer, you'll
have it.

> > (or don't, but I will just ignore you).
>
> I'll hold you to that.

OK.  You're now in my MUA ignore list, direct to the spam folder.
It's sad I won't be reading your contributions to other matters,
but if you don't want to stop this little theater, so be it.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Tomas Hlavaty <tom@logand.com>,  rms@gnu.org,  Eli Zaretskii
>  <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Wed, 15 Nov 2023 21:15:01 +0800
> 
> João Távora <joaotavora@gmail.com> writes:
> 
> > The day you become Emacs maintainer, you can issue that order, at
> > which point I will just leave the project, go back to GitHub or
> > whatever, no hard feelings.  Until then, stop the rhetorical
> > buffoonery and your bullying
> 
> In other words, "it's my way or the highway."  Since that's how you've
> been behaving from the outset, I can't be taxed for responding in equal
> measure.

Yes, you can.  Please try not to respond unkindly when someone attacks
you.

> > (or don't, but I will just ignore you).
> 
> I'll hold you to that.

That was unnecessary.  Please try to avoid that in the future.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Tomas Hlavaty]

On Tue 14 Nov 2023 at 11:47, João Távora <joaotavora@gmail.com> wrote:
> On Tue, Nov 14, 2023 at 10:55 AM Po Lu <luangruo@yahoo.com> wrote:
> LOOP, pcase, etc, like any other macros, are mini-languages.  Like
> any higher-level language, the more powerful they are, the more
> transformations they will do on your input.  LOOP is pretty powerful,
> and may or may not be suitable for the task at hand, like any other
> language.

The question is if those are "nice" mini-languages or expressing the
code in simple emacs-lisp is nicer.

> At any rate Tomas' translation of the loop is just plain
> wrong.  Where is 'type-spec' being bound?  It needs to be
> destructured:

Thanks, you found a bug in the sketch.  It should have been:

    (puthash (pop f) (comp-type-spec-to-cstr (pop f)) h)

or maybe

    (puthash (car f) (comp-type-spec-to-cstr (cadr f)) h)

> LOOP is a language which is very good
> at expressing this fact concisely in a self-documenting way:
>
>   for (f type-spec) in comp-known-type-specifiers

or

(dolist (x comp-known-type-specifiers)
  (destructuring-bind (f type-spec) x ...))

if emacs-lisp had destructuring-bind.

Note that in this particular example, the variables are not needed at
all.

Also note that comp-known-type-specifiers contains unnecessary cons cell
for each element.

> Here from this one line, without ever having looked at Andrea's
> code I learn that comp-known-type-specifiers is a list of
> 2-element lists, where each such tuple is at least made of
> an 'f' and a 'type-spec'.

Even better feature of destructuring-bind is: when
comp-known-type-specifiers changes structure, destructuring-bind will
fail laudly.

If cl-loop supported keywords, it could be even more strict:

(destructuring-bind (&key f type-spec) comp-known-type-specifiers ...)

Or maybe comp-known-type-specifiers should be a structure?

How strict do we want to be?

> Here, I think dolist + pcase-let/cl-destructuring-case/seq-let
> would also be appropriate.  Though I'm sure someone out there
> would find it insufferably ugly or anti-Elisp as well.

I think dolist would be the right choice.

>   >   (cl-loop
>   >    with comp-ctxt = (make-comp-cstr-ctxt)
>   >    with h = (make-hash-table :test #'eq)
>   >    for (f type-spec) in comp-known-type-specifiers
>   >    for cstr = (comp-type-spec-to-cstr type-spec)
>   >    do (puthash f cstr h)
>   >    finally return h)

That would be something along the lines of:

(let ((h (make-hash-table :test #'eq)))
  (dolist (f comp-known-type-specifiers h)
    (puthash (pop f) (comp-type-spec-to-cstr (pop f)) h)))

which seems the most concise translation of the cl-loop code above

or

(let ((h (make-hash-table :test #'eq)))
  (dolist (f comp-known-type-specifiers h)
    (puthash (car f) (comp-type-spec-to-cstr (cadr f)) h)))

> Anyway Tomas' example, if it were to be corrected, would become
> "ugly" perhaps "uglier" (as if "ugly" were even an objective
> thing)

So I removed the unused variable comp-ctxt, pulled the hash-table out of
cl-loop, and puthash stuff for each element of
comp-known-type-specifiers.

The only ugly thing remaining is destructuring the elements of
comp-known-type-specifiers which plain emacs-lisp does not have a good
way of expressing:

(let ((h (make-hash-table :test #'eq)))
  (dolist (x comp-known-type-specifiers h)
    (destructuring-bind (f type-spec) x
      (puthash f (comp-type-spec-to-cstr type-spec) h))))

or (if emacs-lisp had &key)

(let ((h (make-hash-table :test #'eq)))
  (dolist (x comp-known-type-specifiers h)
    (destructuring-bind (&key f type-spec) x
      (puthash f (comp-type-spec-to-cstr type-spec) h))))

or in plain emacs-lisp:

(let ((h (make-hash-table :test #'eq)))
  (dolist (x comp-known-type-specifiers h)
    (apply (lambda (f type-spec)
             (puthash f (comp-type-spec-to-cstr type-spec) h))
           x)))

or

(defun mapcly (fn seq)
  (mapc (lambda (x) (apply fn x)) seq))

(let ((h (make-hash-table :test #'eq)))
  (mapcly (lambda (f type-spec)
            (puthash f (comp-type-spec-to-cstr type-spec) h))
          comp-known-type-specifiers)
  h)

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Tue, Nov 14, 2023 at 5:40 PM Tomas Hlavaty <tom@logand.com> wrote:
>
> On Tue 14 Nov 2023 at 11:47, João Távora <joaotavora@gmail.com> wrote:
> > On Tue, Nov 14, 2023 at 10:55 AM Po Lu <luangruo@yahoo.com> wrote:
> > LOOP, pcase, etc, like any other macros, are mini-languages.  Like
> > any higher-level language, the more powerful they are, the more
> > transformations they will do on your input.  LOOP is pretty powerful,
> > and may or may not be suitable for the task at hand, like any other
> > language.
>
> The question is if those are "nice" mini-languages or expressing the
> code in simple emacs-lisp is nicer.
>
> > At any rate Tomas' translation of the loop is just plain
> > wrong.  Where is 'type-spec' being bound?  It needs to be
> > destructured:
>
> Thanks, you found a bug in the sketch.  It should have been:
>
>     (puthash (pop f) (comp-type-spec-to-cstr (pop f)) h)
>
> or maybe
>
>     (puthash (car f) (comp-type-spec-to-cstr (cadr f)) h)

Yeah, this one's better.  The pop is just confusing.  As is
the setq in the while condition, which my common lisp priests
would not approve.  But the dolist+destructuring-bind is pretty
nice and solves all that.

> (dolist (x comp-known-type-specifiers)
>   (destructuring-bind (f type-spec) x ...))
>
> if emacs-lisp had destructuring-bind.

But it does!  in cl-lib.el no less :-)

> (let ((h (make-hash-table :test #'eq)))
>   (dolist (x comp-known-type-specifiers h)
>     (apply (lambda (f type-spec)
>              (puthash f (comp-type-spec-to-cstr type-spec) h))
>            x)))
>

This one is kinda hard to read for me, at least I couldn't tell
if it was correct right way, putting a lambda in there confuses
things (also if I not mistaken 'apply' instead of directly accessing
the elements is slower).

Anyway, I think

(cl-loop with h = (make-hash-table :test #'eq)
         for (f type-spec) in comp-known-type-specifiers
         do (puthash f (puthash f (comp-type-spec-to-cstr type-spec) h)
         finally return h)

is the comparable loop and I don't think it's so bad.
It's one line less and less complex sexp tree.
But all your solutions are nice too, so I guess I'm pretty
tolerant.  I don't know if I would write this loop myself
because while the destructuring is nice, the return h spoils
it, and the dolist gives you for free (if you know the
mini-language ;-) )

Loop is really good (IMHO!) for complex iteration with
repeat, counts, maximizations, early returns.  I like loop's
self-documenting properties.
Complex iteration happens in programs, in all languages! Some
things are hard regardless of the idiom, unless you learn the
problem domain and the code is documented.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Tomas Hlavaty]

On Tue 14 Nov 2023 at 18:55, Po Lu <luangruo@yahoo.com> wrote:
> João Távora <joaotavora@gmail.com> writes:
>
>> Every macro is a compiler: it translates arbitrary forms to Lisp 
>> forms.
>
> That's quibbling.  The transformations carried out by cl-loop are by a
> long shot more consequential to the structure of their input than those
> by most any other macro there is, which is a demerit and, I think, is
> what Thomas meant to communicate.

Yes, thanks, that was the idea.

Claiming "every macro is a compiler" is not useful for understanding,
why cl-loop or iterate are ugly.

cl-loop is ugly because it invents kind of flat (at the list level)
syntax and rewrites the top flat part.  But it does not change deeper
parts of the input.

iterate is another level of ugly.  It does not have flat syntax, but
rewrites the input all the way to the bottom of the form.  And it needs
to understand every form it walks.

I usually prefer using the simplest tool for the job.

In this example the choices could be:

   while|dolist|mapcar < cl-loop < iterate

I would use while or dolist or mapcar.
There is no point using cl-loop.  It is bearable if somebody insist.
iterate should not exist.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > I suggest we rewrite those cl-loop calls
  > > into ordinary Lisp, avoiding use of cl-loop.

  > That would be something along the lines of:

  > (let ((comp-ctxt (make-comp-cstr-ctxt))
  >       (h (make-hash-table :test #'eq))
  >       f)
  >   (while (setq f (pop comp-known-type-specifiers))
  >     (puthash f (comp-type-spec-to-cstr type-spec) h))
  >   h)

That is simpler than the cl-loop construct that you rewrote.
Though it will alter the ambient valu eof comp-type-spec-to-cstr,
which might be undesirable.

To avoid that, we could use

   (dolist (f comp-known-type-specifiers)
     (puthash f (comp-type-spec-to-cstr type-spec) h))

and avoid binding f in the let.  That would be one line less.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Plesae forgive my delay in getting to this.

  > Btw, the above is a very simple use of cl-loop.  We have quite a few
  > of much more complex ones.  For example:

  >   (cl-loop
  >    with comp-ctxt = (make-comp-cstr-ctxt)
  >    with h = (make-hash-table :test #'eq)
  >    for (f type-spec) in comp-known-type-specifiers
  >    for cstr = (comp-type-spec-to-cstr type-spec)
  >    do (puthash f cstr h)
  >    finally return h)

  ...

  > It isn't an accident that the reference documentation of cl-loop in
  > cl.info takes a whopping 700(!) lines.

This suggests an idea to me:
we can define something in the byte compiler to distinguish simple
uses of cl-loop from non-simle ones, and warn about the latter.

The distinction could be a matter of whether it uses cl-loop features
that are less frequently used.  Or perhaps how MANY cl-loop features
are used.  (There is no need to try to make this precise or quibble
about the details.)

We could enable the warning for source files in the Emacs distribution.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Wed, Jan 3, 2024 at 4:11 AM Richard Stallman <rms@gnu.org> wrote:

> The distinction could be a matter of whether it uses cl-loop features
> that are less frequently used.  Or perhaps how MANY cl-loop features
> are used.

Whatever becomes of this (sorry Richard, I hope not much), I do hope
this doesn't go into the byte-compiler proper, but some special
linter.  Give this special linter the ability to detect lisp sexps
trees deeper than a given threshold or longer than a certain number
of lines.  That's likely much simpler to agree on and implement.

Make this linter optional, like M-x checkdoc.

> (There is no need to try to make this precise or quibble
> about the details.)

Of course there is.  cl-loop users should be consulted about
which features they use to make something happen concisely
and readably.

What you're proposing amounts to making that linter mostly
ignorant about exactly what makes cl-loop useful. A linter that
flags a construct and can't suggest a universally accepted better
alternative -- like the byte-compiler does today -- is just
unneeded nuisance.

I'll be very clear: It's totally legitimate for the maintainers
to decide these kinds of things if they think it's better, and if
this were a paid job I might go along with it out of necessity,
but it isn't so i'd just stop pushing to any repo that starts
enforcing this kind of stuff, wish you all good luck and no hard
feelings.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Wed, 3 Jan 2024 11:55:52 +0000
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> 
> I'll be very clear: It's totally legitimate for the maintainers
> to decide these kinds of things if they think it's better, and if
> this were a paid job I might go along with it out of necessity,
> but it isn't so i'd just stop pushing to any repo that starts
> enforcing this kind of stuff, wish you all good luck and no hard
> feelings.

Noted.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: joaotavora@gmail.com, emacs-devel@gnu.org
> Date: Tue, 02 Jan 2024 23:11:27 -0500
> 
> This suggests an idea to me:
> we can define something in the byte compiler to distinguish simple
> uses of cl-loop from non-simle ones, and warn about the latter.
> 
> The distinction could be a matter of whether it uses cl-loop features
> that are less frequently used.  Or perhaps how MANY cl-loop features
> are used.  (There is no need to try to make this precise or quibble
> about the details.)
> 
> We could enable the warning for source files in the Emacs distribution.

If someone wants to work on this, it's fine by me.  The actual
decision needs to be taken only when and if patches to that effect are
submitted.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Michael Heerdegen]

João Távora <joaotavora@gmail.com> writes:

> Also, I know you don't like cl-loop but in terms of efficiency
> it's sometimes a very good choice.

The problem of `cl-loop' is more one of expressiveness and readability.
Even some Common Lisp people don't like it and have developed
alternatives - like "iterate" (https://iterate.common-lisp.dev/).

Dunno if something like that would be more acceptable to Po Lu, but it
could be an improvement compared to `cl-loop'.


Michael.

RE: [External] : Re: What's missing in ELisp that makes people want to use cl-lib?

[Drew Adams]

> > Also, I know you don't like cl-loop but in terms of efficiency
> > it's sometimes a very good choice.
> 
> The problem of `cl-loop' is more one of expressiveness and readability.
> Even some Common Lisp people don't like it and have developed
> alternatives - like "iterate"
> (https://urldefense.com/v3/__https://iterate.common-
> lisp.dev/__;!!ACWV5N9M2RV99hQ!Ol1EyLyV-Fu5xj6jaaLCQzvsrtW0qRWo9zcY6-
> y5VTHNrTf2yPZwm6ZqfQhREJo0Nb4qd0I2EZyuSpRF37bfzRpUEkRL$ ).
> 
> Dunno if something like that would be more acceptable to Po Lu, but it
> could be an improvement compared to `cl-loop'.

+1.

Hadn't seen your msg when I sent a similar one,
for the same reason.  IF there's some attempt
to add something akin to `cl-loop' to Elisp at
some point, maybe consider `iterate' as a more
lispy alternative.  Not so much a "foreign
language".

Re: [External] : Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Sun, Nov 12, 2023 at 7:48 PM Drew Adams <drew.adams@oracle.com> wrote:
>
> > > Also, I know you don't like cl-loop but in terms of efficiency
> > > it's sometimes a very good choice.
> >
> > The problem of `cl-loop' is more one of expressiveness and readability.
> > Even some Common Lisp people don't like it and have developed
> > alternatives - like "iterate"
> > (https://urldefense.com/v3/__https://iterate.common-
> > lisp.dev/__;!!ACWV5N9M2RV99hQ!Ol1EyLyV-Fu5xj6jaaLCQzvsrtW0qRWo9zcY6-
> > y5VTHNrTf2yPZwm6ZqfQhREJo0Nb4qd0I2EZyuSpRF37bfzRpUEkRL$ ).
> >
> > Dunno if something like that would be more acceptable to Po Lu, but it
> > could be an improvement compared to `cl-loop'.
>
> +1.
>
> Hadn't seen your msg when I sent a similar one,
> for the same reason.  IF there's some attempt
> to add something akin to `cl-loop' to Elisp at
> some point, maybe consider `iterate' as a more
> lispy alternative.  Not so much a "foreign
> language".

As far as I know (I used or had to read it once or twice)
it's just cl-loop with more  parenthesis...  Or is there
some fundamental difference in ease of use?

I expect it to have a substantial manual too, as any
powerful language.

Anyway, I hope whatever is it, that it is better optimized
than our cl-loop which seems to create blocks needlessly
and almost always puts an nreverse at the end when
collecting stuff in order.  Other 'loop' implementations
(outside Elisp)  don't do that and are usually as fast or
faster than the sequence functions beside them.  But ours
isn't (not by much tho).

João

RE: [External] : Re: What's missing in ELisp that makes people want to use cl-lib?

[Drew Adams]

> I expect it to have a substantial manual

I already pointed you to "The Iterate Manual":

https://iterate.common-lisp.dev/iterate-manual.pdf

Judge for yourself whether you consider it "substantial".

Appendix A of the manual was also published as a paper, "Don't Loop, Iterate".  From Appendix A's intro:

"But all’s not joy in Consville. For—I beg your pardon, but—there really is no good way to _iterate_ in Lisp. Now, some are happy to map their way about, whether for real with `mapcar' and friends, or with the make-believe of Series; others are so satisfied with `do' it’s a wonder they’re not C hackers. Still others have gotten by with `loop', but are getting tired of looking up the syntax in the manual over and over again. And in the elegant schemes of some, only tail recursion and lambdas figure. But that still leaves a sizeable majority of folk—well, me, at least—who would simply like to _iterate_, thank you, but in a way that provides nice abstractions, is extensible, and looks like honest-to-God Lisp.

In what follows I describe a macro package, called `iterate', that provides the power and convenient abstractions of `loop' but in a more syntactically palatable way. `iter' also has many features that `loop' lacks, like generators and better support for nested loops. `iterate' generates inline code, so it’s more efficient than using the higher-order function approach. And `iterate' is also extensible—it’s easy to add new clauses to its vocabulary in order to express new patterns of iteration in a convenient way.

A Common Lisp programmer who wonders what’s lacking with present-day iteration features would do well to consider `setf'. Of course, `setf' doesn’t iterate, but it has some other nice properties. It’s easy to use, for one thing. It’s extensible—you can define new `setf' methods very easily, so that `setf' will work with new forms. `setf' is also efficient, turning into code that’s as good as anyone could write by hand. Arguably, `setf' provides a nice abstraction: it allows you to view value-returning forms, like `(car ...)' or `(get ...)' as locations that can be stored into. Finally and most obviously, `setf' _looks_ like Lisp; it’s got a syntax right out of `setq'.

`iterate' attempts to provide all of these properties..."

Re: [External] : Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Sun, Nov 12, 2023 at 10:43 PM Drew Adams <drew.adams@oracle.com> wrote:

> In what follows I describe a macro package, called `iterate', that provides the power and convenient abstractions of `loop' but in a more syntactically palatable way.

Ah right, so LOOP with more parens :-)

But don't get me wrong, I like reading that abstract/preface
and it looks good, very good even.  Didn't know it supported
generators, that's awesome.

João

RE: [External] : Re: What's missing in ELisp that makes people want to use cl-lib?

[Drew Adams]

> > In what follows I describe a macro package, called `iterate', that
> provides the power and convenient abstractions of `loop' but in a more
> syntactically palatable way.
> 
> Ah right, so LOOP with more parens :-)
> 
> But don't get me wrong, I like reading that abstract/preface
> and it looks good, very good even.  Didn't know it supported
> generators, that's awesome.

FWIW, I've never used it.  I've been interested
since I read about it many moon ago.  Maybe the
devil's in the details (e.g. implementation),
but the approach/design sounds sound to me.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Michael Heerdegen wrote:

> The problem of `cl-loop' is more one of expressiveness
> and readability.

It depends how you use it, it can be used in a very clean and
clear way.

Because it has been used so much (2nd on the CL list in the
Emacs source, 1004 times in 192 files), doesn't that mean it
is either good, or the alternatives are worse?

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Augusto Stoffel]

On Sat, 11 Nov 2023 at 13:09, João Távora wrote:

> seq.el strength over cl-lib.el is that it is a generic sequence
> processing library.  IMO it _should_ be used if there is the suspicion
> that the sequences being in that data being processed by a given
> function can be custom seq.el sequences.  Or when there is good
> reason to suspect that one they might be such custom sequences.
>
> I think that seq.el _should_ be used in those situations because
> it expresses exactly this fact, so it becomes self-documenting.
>
> I think the argument that Gerd is making in his economic
> contributions is that very often seq.el is overkill.
>
> For example, in its current form should _not_ be used when
> there is certainty that the sequences are Lisp listsm because as I
> showed elsewhere here, seq-some is significantly slower than
> cl-some and cl-loop when processing lots of small lists.

FWIW, I don't really use seq for its polymorphism.  Rather, I just care
about miscellaneous operations such as seq-some, seq-filter, seq-uniq
where a list-only built-in is not available.

From what I see in other people's code, this is by far the most frequent
reason to use seq- functions.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Wed, Nov 15, 2023 at 2:41 PM Augusto Stoffel <arstoffel@gmail.com> wrote:

> FWIW, I don't really use seq for its polymorphism.

Few people do, in core 0.

> From what I see in other people's code, this is by far the most frequent
> reason to use seq- functions.

That and the fact they aren't aware they're anywhere from 5x to
10x slower, sometimes more (I measured 18x for decently realistic),
than what they could be.

Also the fact they don't know cl-seq.el provides all that with
a much more versatile interface, and frequently much better
performance.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Augusto Stoffel]

On Wed, 15 Nov 2023 at 14:51, João Távora wrote:

> That and the fact they aren't aware they're anywhere from 5x to
> 10x slower, sometimes more (I measured 18x for decently realistic),
> than what they could be.

seq can surely be optimized, and I'm skeptical the issue lies in the
generic method dispatch (which would be a fundamental problem).  It
seems more likely due to the fact that the default implementations wrap
predicate functions and macro bodies into lambdas.

I tried one of the your benchmarks, with results consistent with yours:

  (setq cc (all-completions "" obarray))
  (setq list2 (make-list 12 "shooveedoowaa"))

  (when nil
     (benchmark-run 100 (cl-set-difference cc list2 :test #'equal))
     ;(11.517403284999999 45 4.082223424999995)

     (benchmark-run 100 (seq-difference cc list2))
     ;(18.507667340999998 119 10.999239876000019)
     )

Then I tried the first optimization that comes to mind, which is to
redefine seq-doseq as

  (defmacro seq-doseq (spec &rest body)
    "Loop over a SEQUENCE, evaluating BODY with VAR bound to each of its elements.

  Similar to `dolist' but can be applied to lists, strings, and vectors.

  \(fn (VAR SEQUENCE) BODY...)"
    (declare (indent 1) (debug ((symbolp form &optional form) body)))
    `(let ((var1234 ,(cadr spec)))
       (cond
        ((listp var1234)
         (dolist ,spec ,@body))
        (t
         (seq-do (lambda (,(car spec))
                   ,@body)
                 var1234)))))

And now I get:

  (when nil
     (benchmark-run 100 (cl-set-difference cc list2 :test #'equal))
     ;(11.517403284999999 45 4.082223424999995)

     (benchmark-run 100 (seq-difference cc list2))
     ;(7.308574982 29 2.6960850209999876)
     )

> Also the fact they don't know cl-seq.el provides all that with
> a much more versatile interface, and frequently much better
> performance.

I for one like the simple aesthetics of seq.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Wed, Nov 15, 2023 at 9:21 PM Augusto Stoffel <arstoffel@gmail.com> wrote:
>
> On Wed, 15 Nov 2023 at 14:51, João Távora wrote:
>
> > That and the fact they aren't aware they're anywhere from 5x to
> > 10x slower, sometimes more (I measured 18x for decently realistic),
> > than what they could be.
>
> seq can surely be optimized, and I'm skeptical the issue lies in the
> generic method dispatch (which would be a fundamental problem).  It
> seems more likely due to the fact that the default implementations wrap
> predicate functions and macro bodies into lambdas.
>
> I tried one of the your benchmarks, with results consistent with yours:
>
>   (setq cc (all-completions "" obarray))
>   (setq list2 (make-list 12 "shooveedoowaa"))
>
>   (when nil
>      (benchmark-run 100 (cl-set-difference cc list2 :test #'equal))
>      ;(11.517403284999999 45 4.082223424999995)
>
>      (benchmark-run 100 (seq-difference cc list2))
>      ;(18.507667340999998 119 10.999239876000019)
>      )
>
> Then I tried the first optimization that comes to mind, which is to
> redefine seq-doseq as
>
>   (defmacro seq-doseq (spec &rest body)
>     "Loop over a SEQUENCE, evaluating BODY with VAR bound to each of its elements.
>
>   Similar to `dolist' but can be applied to lists, strings, and vectors.
>
>   \(fn (VAR SEQUENCE) BODY...)"
>     (declare (indent 1) (debug ((symbolp form &optional form) body)))
>     `(let ((var1234 ,(cadr spec)))
>        (cond
>         ((listp var1234)
>          (dolist ,spec ,@body))

I don't think you just skip the seq-do generic like that :-) That breaks
any sequence type based on lists.  This is exactly the problem I
was raising with too-many-generic interfaces, btw.

Try Dmitry's patch instead, the one containing seq-difference-3.  That's
more consequential (but still not fully consequential, though,
also breaks things).

> And now I get:
>
>   (when nil
>      (benchmark-run 100 (cl-set-difference cc list2 :test #'equal))
>      ;(11.517403284999999 45 4.082223424999995)
>
>      (benchmark-run 100 (seq-difference cc list2))
>      ;(7.308574982 29 2.6960850209999876)
>      )

Did you try my feature/cl-lib-improvements branch?
Give cl-nset-difference a try there.

> > Also the fact they don't know cl-seq.el provides all that with
> > a much more versatile interface, and frequently much better
> > performance.

> I for one like the simple aesthetics of seq.

Are you familiar with the cl-lib.el "aesthetics"?
The simple versions argumentless versions are no uglier than seq's,

cl-some vs seq-some
cl-every vs set-every-p
cl-set-difference vs seq-intersection

What specific thing do you like in seq.el that you don't
find in the sequence functions of cl-lib.el?

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Augusto Stoffel]

On Thu, 16 Nov 2023 at 00:28, João Távora wrote:

> I don't think you just skip the seq-do generic like that :-)

Of course you don't.  This _is not_ a patch submission :-).

What I'm pointing out is that the seq-doseq macro is probably the most
consequential bit regarding performance.  If you use a mapc and closures
where you could use a simple dolist, performance will obviously suffer.

There exist extensible macros in Emacs (such as pcase), so adding some
kind dynamic dispatch for seq-doseq should be possible.  Barring that,
one would have to define list-specific methods for 8 generic functions
that use seq-doseq, which is also not unfeasible.

> That breaks any sequence type based on lists.  This is exactly the
> problem I was raising with too-many-generic interfaces, btw.

Good point.  IMO you just can't have both lists and "types based on
lists" to work together properly; if we defined a method for
seq-difference on lists, the issue would be the same (I guess that's the
caveat you raised regarding Dmitri's patch).

I don't see much appeal in user-defined seq types (as opposed to dealing
uniformly with lists, vectors and strings, which is mildly useful).  In
any case, the user always has the options to wrap a struct or class
around their custom list types.  So "sequence types based on lists"
could just be disallowed (I suppose this was part of those long
discussions, which I admit I didn't follow closely.)

> Try Dmitry's patch instead, the one containing seq-difference-3.

That approach is fine, but it only addresses one function at a time.
(If we go that way, I suggest starting by the most useful ones:
seq-filter, seq-some and seq-reduce.)

>> I for one like the simple aesthetics of seq.
>
> Are you familiar with the cl-lib.el "aesthetics"?

Yes.

> The simple versions argumentless versions are no uglier than seq's,

As far as names are concerned, cl-remove-if-not is uglier than
seq-filter.

> cl-some vs seq-some
> cl-every vs set-every-p
> cl-set-difference vs seq-intersection

cl-set-difference is an operation on lists, not sets...

> What specific thing do you like in seq.el that you don't
> find in the sequence functions of cl-lib.el?

Like other people already said, it's more about what you _don't_ find in
seq.  The seq API is much more straightforward than cl-lib's.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Thu, Nov 16, 2023 at 2:36 PM Augusto Stoffel <arstoffel@gmail.com> wrote:
>
> On Thu, 16 Nov 2023 at 00:28, João Távora wrote:
>
> > I don't think you just skip the seq-do generic like that :-)
>
> Of course you don't.  This _is not_ a patch submission :-).
>
> What I'm pointing out is that the seq-doseq macro is probably the most
> consequential bit regarding performance.  If you use a mapc and closures
> where you could use a simple dolist, performance will obviously suffer.
>
> There exist extensible macros in Emacs (such as pcase), so adding some
> kind dynamic dispatch for seq-doseq should be possible.  Barring that,
> one would have to define list-specific methods for 8 generic functions
> that use seq-doseq, which is also not unfeasible.
>
> > That breaks any sequence type based on lists.  This is exactly the
> > problem I was raising with too-many-generic interfaces, btw.
>
> Good point.  IMO you just can't have both lists and "types based on
> lists" to work together properly; if we defined a method for
> seq-difference on lists, the issue would be the same (I guess that's the
> caveat you raised regarding Dmitri's patch).
>
> I don't see much appeal in user-defined seq types (as opposed to dealing
> uniformly with lists, vectors and strings, which is mildly useful).


User-defined sequences _are_ useful in certain problem domains.
Generators, streams, lazy sequences, these are all great things.

But in bread-and-butter Lisp (LISt Processing ;-) ) we don't
need them indeed.  And we should absolutely use the fastest
library we have.  seq.el can hardly  be it, even if you do that
shortcut (and effectively outlaw a big class of custom sequences,
presumably the thing seq.el is good at).

>   In
> any case, the user always has the options to wrap a struct or class
> around their custom list types.  So "sequence types based on lists"
> could just be disallowed (I suppose this was part of those long
> discussions, which I admit I didn't follow closely.)

Yes, exactly.  And sure yes, but in many cases the lingering
performance overhead of generic functions still remains, unless
you go all the way to the entry point and break compatibility
there too.  See my latest benchmarks i sent in the Dmitry thread.

> > Try Dmitry's patch instead, the one containing seq-difference-3.
>
> That approach is fine, but it only addresses one function at a time.
> (If we go that way, I suggest starting by the most useful ones:
> seq-filter, seq-some and seq-reduce.)

Feel free to start your branch from feature/cl-lib-improvements
and keep posting benchmarks.

> As far as names are concerned, cl-remove-if-not is uglier than
> seq-filter.

Ha, this one haunts me from the early 2000's

"filter" for me personally has, for as long as I remember, been
a pain in the brain.  I can never tell if is is meant to keep
things that verify a predicate or throw away such things.  I
still can't, I always have to look it up.

like: "filtering coffee" leaves the bad stuff behind.
"filtering Yukon river water" leaves the good stuff behind (gold).

So while I have to agree that cl-remove-if-not is a pain to type,
it's very clear as to what it does when you read it.   Have long
argued it should have been called "keep-if" though.

> > cl-some vs seq-some
> > cl-every vs set-every-p
> > cl-set-difference vs seq-intersection
>
> cl-set-difference is an operation on lists, not sets...

So is seq-intersection.  Also not on sets.

> > What specific thing do you like in seq.el that you don't
> > find in the sequence functions of cl-lib.el?
> Like other people already said, it's more about what you _don't_ find in
> seq.  The seq API is much more straightforward than cl-lib's.

See my message to Dmitry.  You think the super-special
seq-remove-at-position is a good replacement for
the much more versatile cl-remove?  Bah.

But hey, if it weren't for the performance aspect  I would
absolutely surely 1000% not mind you using seq.el in
your code or in Emacs.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 16/11/2023 16:36, Augusto Stoffel wrote:
>> That breaks any sequence type based on lists.  This is exactly the
>> problem I was raising with too-many-generic interfaces, btw.
> Good point.  IMO you just can't have both lists and "types based on
> lists" to work together properly; if we defined a method for
> seq-difference on lists, the issue would be the same (I guess that's the
> caveat you raised regarding Dmitri's patch).

Actually, we might not allow custom sequences based on lists to begin 
with. E.g. if there are two definitions

  (cl-defmethod seq-length ((l (head :m6-sparse)))

and

  (cl-defmethod seq-length ((l list))

...is there a well-defined ordering between them which would determine 
the one that will be used for (cons :m6-sparse nil)?

If not, the optimization proposed by Augusto should be quite valid. And 
whatever custom sequences are added later, should use cl-defstruct or 
similar.

>> Try Dmitry's patch instead, the one containing seq-difference-3.
> That approach is fine, but it only addresses one function at a time.
> (If we go that way, I suggest starting by the most useful ones:
> seq-filter, seq-some and seq-reduce.)

My patch should help all the set operations (difference, union, 
intersection). And seq-uniq too.

Anyway, there is no reason not to use both optimizations together.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Thu, Nov 16, 2023 at 3:43 PM Dmitry Gutov <dgutov@yandex.ru> wrote:
>
> On 16/11/2023 16:36, Augusto Stoffel wrote:
> >> That breaks any sequence type based on lists.  This is exactly the
> >> problem I was raising with too-many-generic interfaces, btw.
> > Good point.  IMO you just can't have both lists and "types based on
> > lists" to work together properly; if we defined a method for
> > seq-difference on lists, the issue would be the same (I guess that's the
> > caveat you raised regarding Dmitri's patch).
>
> Actually, we might not allow custom sequences based on lists to begin
> with. E.g. if there are two definitions
>
>   (cl-defmethod seq-length ((l (head :m6-sparse)))
>
> and
>
>   (cl-defmethod seq-length ((l list))
>
> ...is there a well-defined ordering between them which would determine
> the one that will be used for (cons :m6-sparse nil)?

The former is more specific and kicks in first.  So it's an implicit
interface already even if it not described anywhere (but it probably
is, I didn't check).

And then there's the :around, :before, etc methods.  Those are perfectly
well defined.

> If not, the optimization proposed by Augusto should be quite valid. And
> whatever custom sequences are added later, should use cl-defstruct or
> similar.

Looking forward to see benchmarks (and maybe some unit tests) based
on the latest branch.  If you can, please use my joaot/bench macro
it makes all this data easier to digest.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Eli Zaretskii wrote:

> We use keyword arguments when there are many of them, or
> when a few of them are obscure and rarely used. From where
> I stand, the dispute is not about banishing then -- that
> will not happen -- the dispute is about how _much_ to
> use them.

But doesn't that mean there has to be rules for every
individual library? And what kind of rule, for example for
cl-lib? "You are allowed to use it - but not too much"?
How are you going to formulate and enforce such rules?

Well, anyway, then yes - it is not the way I think about
programming and libraries. If there is a library, one should
be allowed to use it as much or little as is needed to improve
one's code.

If everyone is instead supposed to write their own things
instead of using the library solution then it means there will
be code everywhere doing the same thing, almost, and in
a slightly new way for every time it is implemented.

I see that as more complexity and more time spent trying to
understand other people's code, not less.

But yes, the Emacs maintainers have to be familiar with the
libraries being shipped with and included in Emacs. There is
no way around it.

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Sat, 11 Nov 2023 16:23:18 +0100
> 
> Eli Zaretskii wrote:
> 
> > We use keyword arguments when there are many of them, or
> > when a few of them are obscure and rarely used. From where
> > I stand, the dispute is not about banishing then -- that
> > will not happen -- the dispute is about how _much_ to
> > use them.
> 
> But doesn't that mean there has to be rules for every
> individual library? And what kind of rule, for example for
> cl-lib? "You are allowed to use it - but not too much"?
> How are you going to formulate and enforce such rules?

It's a judgment call, to be made when reviewing patches.  Like many
other similar issues.  Not everything could or should be codified.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Eli Zaretskii wrote:

>> But doesn't that mean there has to be rules for every
>> individual library? And what kind of rule, for example for
>> cl-lib? "You are allowed to use it - but not too much"?
>> How are you going to formulate and enforce such rules?
>
> It's a judgment call, to be made when reviewing patches.
> Like many other similar issues. Not everything could or
> should be codified.

Sure, but it will be a lot of work for you having to think
about it and doing that judgment call every time.

You also risk alienating people who do drive-by fixes and are
unaware of or do not agree with the approach to cl-lib.
Even explaining to them in the best of friendly terms will
not guarantee they will return with a new patch. So when they
don't, it'll be even more work for you if you intend to remove
the cl-lib from their code, so it can be used.

But okay, this is how you will keep it out then.

What will happen to the code already in? Are you gonna kick
that code out?

That will again be a lot of work for you, time that could have
been spent elsewhere.

But obviously it is up to you how to spend your time and what
efforts in Emacs you prioritize.

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Sat, 11 Nov 2023 17:41:20 +0100
> 
> But obviously it is up to you how to spend your time and what
> efforts in Emacs you prioritize.

Indeed.

RE: [External] : Re: What's missing in ELisp that makes people want to use cl-lib?

[Drew Adams]

> > My goal would be like this:
> >  - keep cl-lib available to it's current extent for end users
> >  - try to get unnecessary cl-lib uses out of the code base
> >  - make useful cl ideas available in the core language
> 
> These are exactly our goals.  We've been doing this for years: that's
> how stuff like 'when', 'unless', 'push', 'pop', and others got into
> the core.

Great.  But I think this happened without any
particular, explicit program/goal to
discover/discuss/understand which things are
the candidate "useful cl ideas" to consider.

Nothing wrong with that lack of explicit goal,
per se.  But it speaks a bit to the situation
or problem you raise of some people being too
eager to use/include whatever they personally
want ("unconstrained liberty of using all or most"):

> The problem (at least my problem) is that some of the people in this
> discussion don't agree with this cautious and slow approach.  They
> want the unconstrained liberty of using all or most of cl-lib and its
> extra libraries everywhere, and they consider any request to avoid
> using these facilities an unacceptable pressure.  They basically want
> cl-lib, cl-extra, etc. preloaded and always available for use,
> regardless of the aspects you mention above and without any
> restrictions.

I think that Michael's follow-on point/goal,
which you didn't comment on, would help with
that problem you raised, because it would be
less likely to always remain a question of
all-or-nothing change or personal preference: 

  And: one main problem we have is that cl-lib
  is more or less a huge monolithic block (as
  a `require'able library).  Why not split it up?
 
> We use keyword arguments when there are many of them, or when a few of
> them are obscure and rarely used.  From where I stand, the dispute is
> not about banishing then -- that will not happen -- the dispute is
> about how _much_ to use them.

I suggested we start with considering more use
of :key and :test, IOW, look for places where
those might simplify things.  (And no, that
doesn't mean we need to add them here & there
willy nilly, e.g. get rid of `memq'.)

RE: [External] : Re: What's missing in ELisp that makes people want to use cl-lib?

[Drew Adams]

There's much (maybe most) of what you say here,
Michael, that I agree with - in particular,
that the devil is in the details and blanket
opinions aren't very helpful.

I'll reply mainly to this, saying that I think
it makes a lot of sense:

> My goal would be like this:
> 
>  - keep cl-lib available to it's current extent for end users
>  - try to get unnecessary cl-lib uses out of the code base
>  - make useful cl ideas available in the core language
> 
> And: one main problem we have is that cl-lib is more
> or less a huge monolithic block (as a `require'able library).

Indeed.  That's what comes, I think, from its
creation as an attempt at emulating most of
what is, in effect, a huge, other, language:
Common Lisp.

(I'm glad that we _have_ such a wide-ranging
emulation, but it shouldn't be so monolithic
to use, understand, and debug.)

> Why not split it up?

100% agreement.  Just what split to end up with
might be more difficult and more controversial
to decide.  But worth discussion, at least.

And whatever the split in terms of supported
functions/features, maybe split off the defs of
macros needed for those split-off functions and
features.

We used to be able to just load `cl-macs.el' at
compile time, to get support for some `cl.el'
features without loading the latter.  Now,
`cl-macs' requires `cl-lib anyway.  (But see
previous paragraph - it might be good to split
up the macro defs as well, instead of having
all the macros for all of our emulation of CL
in one file.)

> I don't see why it is necessary that when I
> want to use a small useful cl thing that
> cl-loop must be made available.  Such things
> could go to its own library, name it cl-obscure
>  when you want.

Yes.  But put `cl-loop' in its own file - it's
a monster (essentially another language, on its
own).

And it's not just `cl-loop' - split `cl-lib'
into multiple files, according to some useful
criteria (maybe including arcaneness, lack of
overlap with other Elisp constructs, etc.).
(I can imagine there would be varying opinions
about the splitting vs lumping. ;-))

> cl-lib is big.  I would like us to try to
> factor it when the size is a problem (is it
> a practical, objective, or more or less mainly
> only a psychological problem btw?

I think it's probably a problem in multiple ways.

It's a _practical_ problem both (1) for users
(who don't want to load everything just to get
something) and (2) for Elisp maintenance and
evolution (a monolithic monster means discussion
and decision are more difficult; the scope
quickly becomes/grows too wide for each question
that might come up).

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Po Lu wrote:

>> So it's not true what you said originally that everytime
>> you find a cl-list* it can be cleanly replaced by nconc.
>> Not if you understand or even appreciate
>> function composition.
>
> I don't appreciate "function combination," whatever that is.

Composition - it is

  y = f(g(x))

or what is sometimes called set, aggregate or most often
perhaps higher-order functions, HOFs. [1]

Yes, those are indispensible - and I think present in C as
well, actually, by means of function pointers. But they are
much easier to handle in Lisp.

Writing such functions manually with loops etc is very much
not recommended whenever there are HOFs that do that already.

[1] https://en.wikipedia.org/wiki/Higher-order_function

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

Emanuel Berg <incal@dataswamp.org> writes:

> Yes, those are indispensible - and I think present in C as
> well, actually, by means of function pointers. But they are
> much easier to handle in Lisp.

And by virtue of being ungainly in C, they are absent from programs
written in that language.  Unequivocally a change for the better.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Sat, Nov 11, 2023 at 1:50 PM Po Lu <luangruo@yahoo.com> wrote:
>
> Emanuel Berg <incal@dataswamp.org> writes:
>
> > Yes, those are indispensible - and I think present in C as
> > well, actually, by means of function pointers. But they are
> > much easier to handle in Lisp.
>
> And by virtue of being ungainly in C, they are absent from programs
> written in that language.

Wooot?  I don't dispute they're "ungainly", but clearly, you've
never used qsort of stdlib.h or so, so many other examples.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> Wooot?  I don't dispute they're "ungainly", but clearly, you've
> never used qsort of stdlib.h or so, so many other examples.

qsort doesn't chain multiple functions together or rewrite the natural
control flow of an expression, it accepts a single callback used to
compare two elements of an array.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Po Lu wrote:

> There is no reason to require a 21 KB file for a six line
> function. None.

Yes, one: If that six line function is useful and is defined
in a readily available library of 21 KB.

Sure, one can break up that library do decrease its modules'
sizes, but this is up to the people at the library's side, not
the person writing code and using the library, because s/he is
using it as intended.

Especially if the library is readily available, like cl-lib,
there is no hesitation. If it has to be downloaded from the
web or by means of some package manager etc then the practical
reasons not to do it appear, so maybe it makes sense in one
sense, but isn't reasonable practically to do, for just
six lines.

But again, that isn't the case with cl-lib. It is already
there and it is very easy to use to improve your code.
Even for a single file and a single use case, what would be
the reason no to do it?

-- 
underground experts united
https://dataswamp.org/~incal

RE: [External] : Re: What's missing in ELisp that makes people want to use cl-lib?

[Drew Adams]

> > There is no reason to require a 21 KB file for a six line
> > function. None.
> 
> Yes, one: If that six line function is useful and is defined
> in a readily available library of 21 KB.
> 
> Sure, one can break up that library do decrease its modules'
> sizes, but this is up to the people at the library's side, not
> the person writing code and using the library, because s/he is
> using it as intended.
> 
> Especially if the library is readily available, like cl-lib,
> there is no hesitation. If it has to be downloaded from the
> web or by means of some package manager etc then the practical
> reasons not to do it appear, so maybe it makes sense in one
> sense, but isn't reasonable practically to do, for just
> six lines.
> 
> But again, that isn't the case with cl-lib. It is already
> there and it is very easy to use to improve your code.
> Even for a single file and a single use case, what would be
> the reason no to do it?

I agree with you - the logic there makes sense.
That's what often happens.

And this is a reason it would help (users and
maintainers) if `cl-lib.el' were split up.

Just how it should be split is debatable and
would deserve discussion.  And yes, if split
poorly that might hurt in some areas, not help.

Re: [External] : Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

Drew Adams <drew.adams@oracle.com> writes:

>> > There is no reason to require a 21 KB file for a six line
>> > function. None.
>> 
>> Yes, one: If that six line function is useful and is defined
>> in a readily available library of 21 KB.
>> 
>> Sure, one can break up that library do decrease its modules'
>> sizes, but this is up to the people at the library's side, not
>> the person writing code and using the library, because s/he is
>> using it as intended.

cl-lib was never meant to be used in such a carefree manner.

>> Especially if the library is readily available, like cl-lib,
>> there is no hesitation. If it has to be downloaded from the
>> web or by means of some package manager etc then the practical
>> reasons not to do it appear, so maybe it makes sense in one
>> sense, but isn't reasonable practically to do, for just
>> six lines.

I think that particular line is better drawn at "whether it is loaded
into /usr/bin/emacs" than at "whether it must be downloaded from the
Internet."  By which I mean the line governing whether we should resign
ourselves to its proliferation, not that governing whether its
proliferation is all that advantageous.

>> But again, that isn't the case with cl-lib. It is already
>> there and it is very easy to use to improve your code.
>> Even for a single file and a single use case, what would be
>> the reason no to do it?
>
> I agree with you - the logic there makes sense.
> That's what often happens.

It's not preloaded and makes for an alluring invitation for others to
produce more cryptic code with it, like virtually any caller of cl-loop,
as so vividly demonstrated by some other posters in this thread.

Re: [External] : Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Po Lu wrote:

>>> But again, that isn't the case with cl-lib. It is already
>>> there and it is very easy to use to improve your code.
>>> Even for a single file and a single use case, what would
>>> be the reason no to do it?
>>
>> I agree with you - the logic there makes sense. That's what
>> often happens.
>
> It's not preloaded

It is loaded when it is used. When it is loaded is
a technicality at that point, unless the preloaded or not
distinction has some hidden meaning I'm unaware of.

> and makes for an alluring invitation for others to produce
> more cryptic code with it, like virtually any caller of
> cl-loop, as so vividly demonstrated by some other posters in
> this thread.

"cl-loop" appears 1004 times in the Emacs source so a lot of
people put it do good use. I also think `cl-loop' is
very useful.

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

Emanuel Berg <incal@dataswamp.org> writes:

> It is loaded when it is used. When it is loaded is
> a technicality at that point, unless the preloaded or not
> distinction has some hidden meaning I'm unaware of.

Preloaded Lisp is present within the dumped emacs binary, other Lisp is
not.  I don't think any distinction could be more marked than that.

> "cl-loop" appears 1004 times in the Emacs source so a lot of
> people put it do good use. I also think `cl-loop' is
> very useful.

Those 1004 instances are precisely what we are trying to prevent from
growing, and then reduce.  They are incomprehensible to Emacs Lisp
programmers and therefore antithetical to how Emacs Lisp should be
written.

Re: [External] : Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Po Lu wrote:

>> It is loaded when it is used. When it is loaded is
>> a technicality at that point, unless the preloaded or not
>> distinction has some hidden meaning I'm unaware of.
>
> Preloaded Lisp is present within the dumped emacs binary,
> other Lisp is not. I don't think any distinction could be
> more marked than that.

Yes, but what does it matter if cl-lib is not loaded there?
It is loaded everywhere else and that is where it is used.
You can of course keep it out of some subset but that isn't
reducing its use everywhere else, which is what has been
identified as the problem.

>> "cl-loop" appears 1004 times in the Emacs source so a lot
>> of people put it do good use. I also think `cl-loop' is
>> very useful.
>
> Those 1004 instances are precisely what we are trying to
> prevent from growing, and then reduce. They are
> incomprehensible to Emacs Lisp programmers and therefore
> antithetical to how Emacs Lisp should be written.

Those who wrote it disagrees, they didn't see anything wrong
with using a library provided by Emacs. And I don't, either.

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

Emanuel Berg <incal@dataswamp.org> writes:

> Yes, but what does it matter if cl-lib is not loaded there?

It means cl-lib is something we don't, and have never, wanted to be
widespread.

> Those who wrote it disagrees, they didn't see anything wrong
> with using a library provided by Emacs. And I don't, either.

They are not the only people who are charged with its upkeep, and you
are not such at all.

Re: [External] : Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Po Lu wrote:

>> Yes, but what does it matter if cl-lib is not loaded there?
>
> It means cl-lib is something we don't, and have never,
> wanted to be widespread.

Well, the opposite happened.

People used it in the correct way, to improve their code.

Now maintainers don't like it?

Too bad, but it is still completely impractical to do anything
about it. Keep it not preloaded by all means, it is still used
everywhere else and there is no way getting it out.

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

João Távora wrote:

> But I can agree it's not worth requiring cl-lib.el just for
> cl-list* alone.

Some people will say one should write as good code as
possible, and `require' whatever necessary for that.

I can agree bringing in external packages and such should be
avoided if one can alter just some detail so to not have to do
that, but cl-lib.el is not external in any way. In this
directory, where the cl-lib files are

  /usr/local/share/emacs/30.0.50/lisp/emacs-lisp

are 102 .el files.

Are we gonna have individual rules for 102 files? What files
are to be used and what aren't?

I think there should be only one rule, write as good code you
can and 'require' any file you need to achieve this.
These files are there because they are useful and there is no
harm interconnecting them.

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> Are _you_ the suit?  I missed the point of most of your very
> elegant text, but isn't policy-making exactly what you (and I) are
> trying to do, i.e. discourage some things and encourage others?

I'm not advocating partitioning Elisp into compartments, and to give
countenance to some of them on the grounds of the stature of their
designers.  That's your proposal.  My synthesis is roughly as follows:

  1. cl-lib is a comparatively large file.

  2. cl-lib is frequently loaded to provide trivial functions
     which could effortlessly be replaced by built-in constructs.

  3. cl-lib also defines a number of functions imported from a different
     programming language, the beliefs underlying whose design don't
     align with those of Emacs.

Therefore, we should discourage cl-lib from being introduced in the
first place, because its proliferation will render programmers more
eager to use it without considering its implications.

> What's the gripe with compartments?  We don't have all of Elisp in
> a single library?  I was talking about namespaces by the way,
> if you didn't catch the reference (admittedly obscure)

Because Elisp isn't naturally partitioned into "compartments."  The
basic unit into which our code is divided is the file, and all
considerations must be framed on that basis.

> Also, is really cl-set-difference to take the difference
> of two sets worse than writing a "hallowed" multi-line
> construct??  Maybe I'm not seeing what the construct is, so
> what is your preferred Elisp hallowed way to take the
> difference of two sets, say make a list  that contains
> all items that appear in LIST1 but not LIST2?  Or to find
> the index of a certain element of a sequence...

dolist with a catch/throw, or dolist, member and push?

> These comparisons, taken one by one, are the hard evidence of that
> allows us to determine what is "alien" or not, not theological
> consecrations of personal tastes of a given group, _any_ group.
> For example, I don't think your 'nconc' form is more readable
> than 'cl-list*'.  It's slightly less readable IMO.  We could alias
> `list*` to nconc though, even better.  But I can agree it's not
> worth requiring cl-lib.el just for cl-list* alone.
>
> Much like you prefer setcar to rplacd. Which I also do, btw.  But
> I prefer (setf (car ...))  even more.  But you'll frequently see
> me using setcar and setcdr (or even rplacwahtevercannevertypeit)
> if that's the style of the file I'm working on.  Because I too
> have my personal tastes, but contrary to others I can let go of
> them without too much fuss.

My point was that comparing rplaca to cl-lib is much the same as
comparing apples to oranges, since rplaca is an _alias_ to setcar.

> Stefan M's pcase in particular is great.  Its docstring is tough,
> I agree, and could use some examples (I always end up grepping
> for examples).  But that macro is a lifesaver.
>
> The way you talk about seq.el and pcase.el makes me think you
> want to ban _all the things_, not just cl-lib.  I'm confused about
> what you want Elisp code in core to look like.

Well, touch-screen.el is a good example of that.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 10, 2023 at 12:32 AM Po Lu <luangruo@yahoo.com> wrote:
>
> João Távora <joaotavora@gmail.com> writes:
>
> > Are _you_ the suit?  I missed the point of most of your very
> > elegant text, but isn't policy-making exactly what you (and I) are
> > trying to do, i.e. discourage some things and encourage others?
>
> I'm not advocating partitioning Elisp into compartments, and to give
> countenance to some of them on the grounds of the stature of their
> designers.  That's your proposal.

What the heck?  What I said, is that if Elisp _had_ better
ways of making compartments, such as namespaces, managing which
compartments are preferred in certain parts of code would be easier.
I stated multiple times that I tend to trust developers to pick
the tools they find most appropriate for the job.  I simply stated
that it's funny and ironic that the language whose influence you're
trying to exorcize is precisely the one with one of the best
tools to do that exorcizing.

Maybe you didn't understand this because you're not familiar with
CL packages, so I can explain.  In CL you can actually write a program
in a package that doesn't ':USE' the :COMMON-LISP package at all, or
that only imports a handful of symbols from it (maybe the sacred
Elisp subset).  Or you can make your own packages with just the
symbols you want and maybe use that.  Meaning you'd be effectively
insulated from CL or at least the parts you don't like and could
write your perfect purist Lisp program.   Come on, isn't that funny?

>   3. cl-lib also defines a number of functions imported from a different
>      programming language, the beliefs underlying whose design don't
>      align with those of Emacs.

There are no "beliefs" in Common Lisp's, neither should there be
in Elisp.  I don't think it helps your case to repeatedly evoke
religion.

> Therefore, we should discourage cl-lib from being introduced in the
> first place, because its proliferation will render programmers more
> eager to use it without considering its implications.

Every other choice also has "implications".  Most people favourable
to cl-lib in this discussion know why they chose to use its functions.
I wouldn't be so condescending.

> > Also, is really cl-set-difference to take the difference
> > of two sets worse than writing a "hallowed" multi-line
> > construct??  Maybe I'm not seeing what the construct is, so
> > what is your preferred Elisp hallowed way to take the
> > difference of two sets, say make a list  that contains
> > all items that appear in LIST1 but not LIST2?  Or to find
> > the index of a certain element of a sequence...
>
> dolist with a catch/throw, or dolist, member and push?

Hmmm, a bit vague, no?  Humor me: if it's really so easy and so
readable please write out your preferred equivalent of, say

   (cl-set-difference l1 l2) ; hopefully obvious, like l1 - l2 in python

and

   ;; iterate through someseq and find index of first element
   ;; whose car matches probe according to 'probe-equals'.
   (cl-position probe someseq :key #'car :test #'probe-equals)

Shouldn't take long.  I can try myself, of course, but I think it's
fair to give the proponent of the alternate approach a shot before
comparing the two approaches.

> My point was that comparing rplaca to cl-lib is much the same as
> comparing apples to oranges, since rplaca is an _alias_ to setcar.

Some get confused by cl-labels, some by archaic things like rplaca,
some by new stuff like static-if.  Doesn't matter what is alias to
what, when you see both these things  in code you don't know what
they do until you look them up.  And many times you'll think "there's
a much better way".

> Well, touch-screen.el is a good example of that.

OK.  So, basically, your code :-) '(if x (progn ...))' instead
of 'cond', and lots of handrolled catch/throw with while t. OK fine.

I have a feeling I think you'd love cl-loop, maybe even too much :-)

Also quite some long functions there.  300 lines? Maybe typing
"touch-screen" each time to define and call functions
discourages you from modularizing more?  Or maybe it doesn't
bother you at all, so fine.

Don't get me wrong, I am very thankful for your contributions,
I really am.  Write Elisp code in the manner you find most
comfortable.  But there are programmers working on problem domains
in the core other than touch-screen.el, maybe with different
non-C backgrounds.  Many of them are historical contributors to
Emacs.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> What the heck?  What I said, is that if Elisp _had_ better
> ways of making compartments, such as namespaces, managing which
> compartments are preferred in certain parts of code would be easier.
> I stated multiple times that I tend to trust developers to pick
> the tools they find most appropriate for the job.  I simply stated
> that it's funny and ironic that the language whose influence you're
> trying to exorcize is precisely the one with one of the best
> tools to do that exorcizing.

Which is tantamount to, or at least comes off as, arguing in favor of
that language's influence.

> Maybe you didn't understand this because you're not familiar with
> CL packages, so I can explain.  In CL you can actually write a program
> in a package that doesn't ':USE' the :COMMON-LISP package at all, or
> that only imports a handful of symbols from it (maybe the sacred
> Elisp subset).  Or you can make your own packages with just the
> symbols you want and maybe use that.  Meaning you'd be effectively
> insulated from CL or at least the parts you don't like and could
> write your perfect purist Lisp program.   Come on, isn't that funny?

I know how packages function under Common Lisp, but they are not
salvation.  My objective is not to write code free of Common Lisp
influence; this I already do.  It is for everyone else to write them, or
provide a 95% effective guarantee that I (and many others) will never
have to read them, which is easier said than done; a leitmotif in Emacs
development is for active developers to vanish overnight, leaving a
small corps of developers with reponsibility for their upkeep in the
years that follow.

Granted, the foregoing is something of an overstatement.  But the
alacrity with which programmers seize at opportunities to employ cl-lib
does call for one.

> There are no "beliefs" in Common Lisp's, neither should there be
> in Elisp.  I don't think it helps your case to repeatedly evoke
> religion.

Really?  Here is one example to the contrary: keyword arguments.  No
Emacs Lisp built-in takes them, yet it is scarcely possible to name a
Common Lisp list manipulation function that does not.

> Every other choice also has "implications".  Most people favourable
> to cl-lib in this discussion know why they chose to use its functions.
> I wouldn't be so condescending.

Yet few of these people consider the impact their decision makes to
others, who will in due course have the responsibility for reading this
code pawned off them.

> Hmmm, a bit vague, no?  Humor me: if it's really so easy and so
> readable please write out your preferred equivalent of, say
>
>    (cl-set-difference l1 l2) ; hopefully obvious, like l1 - l2 in python

(let ((list nil))
  (dolist (x l1)
    (unless (member x l2)
      (push x list)))
  (dolist (x l2)
    (unless (member x l1)
      (push x list)))
  list)

> and
>
>    ;; iterate through someseq and find index of first element
>    ;; whose car matches probe according to 'probe-equals'.
>    (cl-position probe someseq :key #'car :test #'probe-equals)
>
> Shouldn't take long.  I can try myself, of course, but I think it's
> fair to give the proponent of the alternate approach a shot before
> comparing the two approaches.

(catch 'tag
  (let ((index 0))
    (dolist (tem someseq)
      (when (eq (car tem) probe)
        (throw 'tag index))
      (setq index (1+ index)))))

> Some get confused by cl-labels, some by archaic things like rplaca,
> some by new stuff like static-if.  Doesn't matter what is alias to
> what, when you see both these things  in code you don't know what
> they do until you look them up.  And many times you'll think "there's
> a much better way".

There's not much of a point in arguing against such patent absurdities
as considering the operation of setcar as complex as that of cl-labels,
is there?

> OK.  So, basically, your code :-) '(if x (progn ...))' instead
> of 'cond', and lots of handrolled catch/throw with while t. OK fine.
>
> I have a feeling I think you'd love cl-loop, maybe even too much :-)

I think not.

> Also quite some long functions there.  300 lines? Maybe typing
> "touch-screen" each time to define and call functions
> discourages you from modularizing more?  Or maybe it doesn't
> bother you at all, so fine.

It doesn't bother me that functions for whom there are no reasons to use
their constituents elsewhere have yet to be balkanized, no.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 10, 2023 at 3:17 AM Po Lu <luangruo@yahoo.com> wrote:
>
> João Távora <joaotavora@gmail.com> writes:
>
> > What the heck?  What I said, is that if Elisp _had_ better
> > ways of making compartments, such as namespaces, managing which
> > compartments are preferred in certain parts of code would be easier.
> > I stated multiple times that I tend to trust developers to pick
> > the tools they find most appropriate for the job.  I simply stated
> > that it's funny and ironic that the language whose influence you're
> > trying to exorcize is precisely the one with one of the best
> > tools to do that exorcizing.
>
> Which is tantamount to, or at least comes off as, arguing in favor of
> that language's influence.

No, not mounting any tantas, I was just making a joke.  But yes, of course
I like Common Lisp.  I like C++ too (not as much, tho) and it pisses off
C people, too.

> Granted, the foregoing is something of an overstatement.  But the
> alacrity with which programmers seize at opportunities to employ cl-lib
> does call for one.

But that leitmotif is everywhere (and us much less strong in open-source,
btw).  Surely if you've ever worked at a large enough company you'll see
bizarre code from people not working there anymore.  And that goes for
everything.  You think everyone will forever be grateful to read your
300-line inline-all-the-things functions in touch-screen.el when you're
not around to explain what they do?  They won't.  It makes no sense
to single out any given library for this evil, especially not cl-lib.el
which is a well-understood stable piece of kit.  In Emacs core and
just as strongly outside it.

> > There are no "beliefs" in Common Lisp's, neither should there be
> > in Elisp.  I don't think it helps your case to repeatedly evoke
> > religion.
>
> Really?  Here is one example to the contrary: keyword arguments.  No
> Emacs Lisp built-in takes them, yet it is scarcely possible to name a
> Common Lisp list manipulation function that does not.

Many examples of Emacs Lisp functions take them.  Even Richard seems
to have come around to their usefulness.  CL functions take them in
consistent ways, but you can use them without keyword arguments if
you're stubborn enough.  Anyway, it's not a "belief", it's just a
good way to provide versatility for functions.

> > Every other choice also has "implications".  Most people favourable
> > to cl-lib in this discussion know why they chose to use its functions.
> > I wouldn't be so condescending.
>
> Yet few of these people consider the impact their decision makes to
> others, who will in due course have the responsibility for reading this
> code pawned off them.

Why do you exempt yourself from this irresponsibility?  Can't you see
this is all subjective and it seems a bit arrogant to say "my code is
so responsible"??  All code is bad.

> > Hmmm, a bit vague, no?  Humor me: if it's really so easy and so
> > readable please write out your preferred equivalent of, say
> >
> >    (cl-set-difference l1 l2) ; hopefully obvious, like l1 - l2 in python
>
> (let ((list nil))
>   (dolist (x l1)
>     (unless (member x l2)
>       (push x list)))
>   (dolist (x l2)
>     (unless (member x l1)
>       (push x list)))
>   list)

>
> (catch 'tag
>   (let ((index 0))
>     (dolist (tem someseq)
>       (when (eq (car tem) probe)
>         (throw 'tag index))
>       (setq index (1+ index)))))

Nuff said :-)  .  So if multiple set difference operations or multiple
index-finding operations are needed you write these choo-choo trains
again and again.  That sure explains the 300 lines.

> > Some get confused by cl-labels, some by archaic things like rplaca,
> > some by new stuff like static-if.  Doesn't matter what is alias to
> > what, when you see both these things  in code you don't know what
> > they do until you look them up.  And many times you'll think "there's
> > a much better way".
>
> There's not much of a point in arguing against such patent absurdities
> as considering the operation of setcar as complex as that of cl-labels,
> is there?

I guess not for human versions of --finline-functions who think,
or rather "believe" that such common practices for code reuse
as subroutine encapsulation is "balkanization".  It's probably "patently absurd"
for them yes.

João Távora

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> But that leitmotif is everywhere (and us much less strong in open-source,
> btw).  Surely if you've ever worked at a large enough company you'll see
> bizarre code from people not working there anymore.

Where I work, our practices and policies prevent these difficulties from
ever materializing.  Think a standardized coding style, mandatory quotas
for documentation length, expositions written and, hmm, anthologized
independently of the code itself, and review by individuals who have,
needless to say, never seen the code before in their lives, and will not
so much as bat an eyelid before striking down code they cannot easily
understand.  The first and third are what's being proposed here for
Emacs, unless I'm greatly mistaken.

> And that goes for everything.  You think everyone will forever be
> grateful to read your 300-line inline-all-the-things functions in
> touch-screen.el when you're not around to explain what they do?  They
> won't.

It's well-documented and in line with other Emacs Lisp code, so I think
they will.  Indeed one individual is already implementing the
recognition of panning and zoom gestures with it as a basis, without any
counsel from me besides referring him to the file itself.

> It makes no sense to single out any given library for this evil,
> especially not cl-lib.el which is a well-understood stable piece of
> kit.  In Emacs core and just as strongly outside it.

See what I explained earlier.

> Many examples of Emacs Lisp functions take them.

Like?  Which list manipulation functions do?
Contrast cl-member to member.

> Even Richard seems to have come around to their usefulness.  CL
> functions take them in consistent ways, but you can use them without
> keyword arguments if you're stubborn enough.  Anyway, it's not a
> "belief", it's just a good way to provide versatility for functions.

A "belief" in this sense is a model or general scheme that is adhered to
when designing a function, of course, so let's not descend into
quibbling.

> Why do you exempt yourself from this irresponsibility?  Can't you see
> this is all subjective and it seems a bit arrogant to say "my code is
> so responsible"??

Thus far there have been no complaints that code _without_ cl-lib,
pcase, seq or elt is unreadable.  Most of our code (this time measured
by lines) falls squarely within that category.

> All code is bad.

With that outlook, there's not much of a purpose in writing any code for
Emacs, is there?

>> > Hmmm, a bit vague, no?  Humor me: if it's really so easy and so
>> > readable please write out your preferred equivalent of, say
>> >
>> >    (cl-set-difference l1 l2) ; hopefully obvious, like l1 - l2 in python
>>
>> (let ((list nil))
>>   (dolist (x l1)
>>     (unless (member x l2)
>>       (push x list)))
>>   (dolist (x l2)
>>     (unless (member x l1)
>>       (push x list)))
>>   list)
>
>>
>> (catch 'tag
>>   (let ((index 0))
>>     (dolist (tem someseq)
>>       (when (eq (car tem) probe)
>>         (throw 'tag index))
>>       (setq index (1+ index)))))
>
> Nuff said :-)  .  So if multiple set difference operations or multiple
> index-finding operations are needed you write these choo-choo trains
> again and again.  That sure explains the 300 lines.

Sure, but catch, let, dolist, when, throw, setq and member are
elementary operations under Emacs Lisp everyone learns.  They are also
few in number.

> I guess not for human versions of --finline-functions who think, or
> rather "believe" that such common practices for code reuse as
> subroutine encapsulation is "balkanization".  It's probably "patently
> absurd" for them yes.

But the Emacs Lisp compiler doesn't inline functions, does it?  At any
rate you have conflated one of my statements with the other, so read
again.

Thanks.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 10, 2023 at 1:14 PM Po Lu <luangruo@yahoo.com> wrote:

> needless to say, never seen the code before in their lives, and will not
> so much as bat an eyelid before striking down code they cannot easily
> understand.

Wow, sounds like a fantastic healthy work environment.  Do people
ever leave the company?  Like, alive?

> > Many examples of Emacs Lisp functions take them.
>
> Like?  Which list manipulation functions do?
> Contrast cl-member to member.

You didn't say "list manipulation function" . You wrote "No Emacs
built-in takes them".  That's false.  Take make-hash-table.  Oh noes,
looks just like CL.  Guess you're going to hand-roll hash tables
now, right? :-)

Take make-process or json-serialize.  There are more.  Keyword
arguments are in other Non-lisp languages too, as you probably
know, they're called named parameters sometimes.  Heck C++
has been pining for them for so long it's sad.

> > All code is bad.
>
> With that outlook, there's not much of a purpose in writing any code for
> Emacs, is there?

There is, because bad as it is, code gets jobs done that humans can't do.

> But the Emacs Lisp compiler doesn't inline functions, does it?

No, but you do, in actual source code!  That was my point!  You
have these fine abstractions to common operations like intersecting
sets, finding indices, and so much more, and you prefer to handroll
your expanded versions each time instead of going through a function
of an established library.

Very rarely, IME, are there for doing this manual inlining
even more rarely in Elisp which has a compiler-macro mechanism.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> Wow, sounds like a fantastic healthy work environment.  Do people
> ever leave the company?  Like, alive?

Sarcasm aside, most people have no trouble assenting to these
procedures.  Not when they yield good code whose meaning remains clear
for generations to come.

Dismissing the intrinsic merit to readable code should be a no-starter
hereabouts.

> You didn't say "list manipulation function" . You wrote "No Emacs
> built-in takes them".  That's false.  Take make-hash-table.  Oh noes,
> looks just like CL.  Guess you're going to hand-roll hash tables
> now, right? :-)

Hash tables don't count, because they were designed to be compatible
with the old CL hash table functions, just all that faster.

> Take make-process or json-serialize.  There are more.  Keyword
> arguments are in other Non-lisp languages too, as you probably
> know, they're called named parameters sometimes.  Heck C++
> has been pining for them for so long it's sad.

I don't know why make-process was introduced, but it enjoys the excuse
of accepting plenty of arguments.  json-serialize does not: its two
optional parameters don't even encounter the 7-argument limit for C
DEFUNs.

It is lamentable that the keyword argument illness has spread into C,
but that doesn't palliate its deficiencies.

> There is, because bad as it is, code gets jobs done that humans can't
> do.

I don't think anyone seriously believes what you claim here, in essence
that it is OK to install dismal code while being well aware of its
nature as such.

> No, but you do, in actual source code!  That was my point!  You
> have these fine abstractions to common operations like intersecting
> sets, finding indices, and so much more, and you prefer to handroll
> your expanded versions each time instead of going through a function
> of an established library.

These "fine abstractions" are also abstruse, while the set of primitives
you never hesitate to impugn have carried Emacs through its entire
existence.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Sat, Nov 11, 2023 at 12:38 AM Po Lu <luangruo@yahoo.com> wrote:

> > No, but you do, in actual source code!  That was my point!  You
> > have these fine abstractions to common operations like intersecting
> > sets, finding indices, and so much more, and you prefer to handroll
> > your expanded versions each time instead of going through a function
> > of an established library.
>
> These "fine abstractions" are also abstruse, while the set of primitives
> you never hesitate to impugn have carried Emacs through its entire
> existence.

Where in heck did I "impugn" Emacs's Lisp primitives?? I use them
all the time!!  I just think it's a really really bad idea to
use them exclusively, almost religiously, to inline existing utils
in write-only code.  When given the opportunity to demonstrate the
superiority of this approach to substitute cl-set-difference, you
failed.  It's just my opinion, of course, but that hard patent fact
that is on display in this thread, along with the hard patent fact
that I simply failed to notice your mistake -- because reading that
code takes much more effort -- is a much better demonstration
of the relative "abstrusity" of the practice you so energetically
recommend.

Like Dmitry, I don't think it's because you don't have
coding ability or intelligence -- which you most clearly do --
just that your practice lends itself to these problems _more often
than_ other more eclectic and pondered practices such as using
more programming techniques.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> Where in heck did I "impugn" Emacs's Lisp primitives?? I use them
> all the time!!  I just think it's a really really bad idea to
> use them exclusively, almost religiously, to inline existing utils
> in write-only code.  When given the opportunity to demonstrate the
> superiority of this approach to substitute cl-set-difference, you
> failed.  It's just my opinion, of course, but that hard patent fact
> that is on display in this thread, along with the hard patent fact
> that I simply failed to notice your mistake -- because reading that
> code takes much more effort -- is a much better demonstration
> of the relative "abstrusity" of the practice you so energetically
> recommend.

That failure belies the alleged advantages of cl-lib, and vindicates all
Alan and Richard have said thus far: for all their augustness, cl-lib's
designers did not select an unambiguous English word for a comparatively
elementary set operation.  Instead devolving responsibility for second
guessing their intent to such as I and they.

> Like Dmitry, I don't think it's because you don't have
> coding ability or intelligence -- which you most clearly do --
> just that your practice lends itself to these problems _more often
> than_ other more eclectic and pondered practices such as using
> more programming techniques.

Actually, a problem would have arisen had I used cl-set-difference while
under the impression that it calculated the disjunctive union of LIST1
with LIST2, as the signification of the word "difference" suggests.

Writing such code with only primitives sidesteps such pitfalls
altogether, since what is thought is directly set down.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Sat, Nov 11, 2023 at 1:04 PM Po Lu <luangruo@yahoo.com> wrote:

> That failure belies the alleged advantages of cl-lib, and vindicates all
> Alan and Richard have said thus far: for all their augustness, cl-lib's
> designers did not select an unambiguous English word for a comparatively
> elementary set operation.  Instead devolving responsibility for second
> guessing their intent to such as I and they.

First, that's such a weak pillar which remains on which to base all your
cl-lib.el fury.   Hair-splitting over a word and and bringing up a
dictionary definition written more than 100 years ago when programming
was basically non-existent.

And did you see my reply to Tomas where I listed that at least 5 other
programming languages use the same word (or at least the '-'
operator) for the same operation?  There is indeed such a thing
as set theory in mathematics and the literature uses that term
to refer to that operation.  What a bad idea it would be for Elisp to
stray from that practice just so your argument can stay afloat.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> First, that's such a weak pillar which remains on which to base all your
> cl-lib.el fury.   Hair-splitting over a word and and bringing up a
> dictionary definition written more than 100 years ago when programming
> was basically non-existent.
>
> And did you see my reply to Tomas where I listed that at least 5 other
> programming languages use the same word (or at least the '-'
> operator) for the same operation?  There is indeed such a thing
> as set theory in mathematics and the literature uses that term
> to refer to that operation.  What a bad idea it would be for Elisp to
> stray from that practice just so your argument can stay afloat.

That's one example out of many, scarcely the "only pillar on which to
lash out at cl-lib."

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Po Lu wrote:

> These "fine abstractions" are also abstruse, while the set
> of primitives you never hesitate to impugn have carried
> Emacs through its entire existence.

The abstractions elevate Elisp to new levels, and from there,
they make it easier for everyone else writing code not having
to do on the level of very basic building blocks. Relying on
wizards to juggle and combine basics like crazy to achieve
wonders isn't the way computer technology has been moving
forward, on the contrary that has been done by moving
complexity into modules and providing neat interfaces so
people only have to master the interfaces, not the details
what happens below that to carry out the desired task.

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

Emanuel Berg <incal@dataswamp.org> writes:

> The abstractions elevate Elisp to new levels, and from there,
> they make it easier for everyone else writing code not having
> to do on the level of very basic building blocks. Relying on
> wizards to juggle and combine basics like crazy to achieve
> wonders isn't the way computer technology has been moving
> forward, on the contrary that has been done by moving
> complexity into modules and providing neat interfaces so
> people only have to master the interfaces, not the details
> what happens below that to carry out the desired task.

When such interfaces number greater than the finer details, that proves
so much more challenging than expressing these details with the
elementary operations which constitute them.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Po Lu wrote:

>> The abstractions elevate Elisp to new levels, and from
>> there, they make it easier for everyone else writing code
>> not having to do on the level of very basic building
>> blocks. Relying on wizards to juggle and combine basics
>> like crazy to achieve wonders isn't the way computer
>> technology has been moving forward, on the contrary that
>> has been done by moving complexity into modules and
>> providing neat interfaces so people only have to master the
>> interfaces, not the details what happens below that to
>> carry out the desired task.
>
> When such interfaces number greater than the finer details,
> that proves so much more challenging than expressing these
> details with the elementary operations which
> constitute them.

As we have seen here and in the stats I provided earlier in
the thread how much cl-lib is used, it follows a lot of people
have been able to put it into good use. It is not too
complicated for anyone, and often on the contrary makes the
code shorter and easier to read. If a bunch of guys take on
the entire Emacs source and for that reason want a more simple
Elisp, then that's their opinion based on their particular
point of view. But cl-lib is still awesome and makes the code
better, not worse.

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

Emanuel Berg <incal@dataswamp.org> writes:

> As we have seen here and in the stats I provided earlier in
> the thread how much cl-lib is used, it follows a lot of people
> have been able to put it into good use. It is not too
> complicated for anyone, and often on the contrary makes the
> code shorter and easier to read.

For you.  Not for Alan, Richard, Eli, or me, or for scores of others.

If four active Emacs developers have expressed their discontent at an
undoubtedly popular feature, there there must be more users scratching
their heads in the face of cl-lib code among the ranks of the
disenfranchised Emacs users deterred from voicing their concerns as to
the Emacs development process by our allegedly unfamiliar development
practices, or so the frequently-cited truism goes.  The self-anointed
warriors who generally take every opportunity to fight for their
representation are uncharacteristically taciturn in this regard.  But
that is by the by, since ultimately the practices we adopt are subject
to the concerns of people who must read this code; and alas, very few of
our users do, for largely unrelated reasons.

> If a bunch of guys take on the entire Emacs source and for that reason
> want a more simple Elisp, then that's their opinion based on their
> particular point of view. But cl-lib is still awesome and makes the
> code better, not worse.

You are entitled to that view, but when this "bunch of guys" have to
tread through such code, they are, in like fashion, entitled to a say in
how it is written.  Correct?

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Po Lu wrote:

> You are entitled to that view, but when this "bunch of guys"
> have to tread through such code, they are, in like fashion,
> entitled to a say in how it is written. Correct?

Yes, but all the people who have used it then, whose code are
now all over Emacs? Don't their view count as well?

cl-lib is included with Emacs. They have used a library that
is shipped with Emacs because they thought that was the best
way to solve whatever situation they had. Now those solutions
are in place, working.

This speaks very strongly in favor of cl-lib.

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

Emanuel Berg <incal@dataswamp.org> writes:

> Yes, but all the people who have used it then, whose code are
> now all over Emacs? Don't their view count as well?

Nobody said theirs doesn't, but it is within their power not to write
such code as well.  Thus, it is not reasonable to tax us for attempting
to exact concessions they can easily grant, and will ultimately make
life easier for us.

> cl-lib is included with Emacs. They have used a library that
> is shipped with Emacs because they thought that was the best
> way to solve whatever situation they had. Now those solutions
> are in place, working.
>
> This speaks very strongly in favor of cl-lib.

It does not.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Po Lu wrote:

>> cl-lib is included with Emacs. They have used a library
>> that is shipped with Emacs because they thought that was
>> the best way to solve whatever situation they had.
>> Now those solutions are in place, working.
>>
>> This speaks very strongly in favor of cl-lib.
>
> It does not.

It does and any anti cl-lib policy will just alienate people
using it for now reason.

Telling people it is better to hand-roll with basic Elisp than
to use libraries is bad technology and a very unwise
alienating move for no reason.

Especially since there is nothing to do about it, the
situation is the way it is. So even less reason.

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

João Távora wrote:

> No, but you do, in actual source code! That was my point!
> You have these fine abstractions to common operations like
> intersecting sets, finding indices, and so much more, and
> you prefer to handroll your expanded versions each time
> instead of going through a function of an
> established library.

Yes, this is exactly what will happen if one doesn't rely on
libraries. Everyone will handroll their versions of the same
thing. Just looking at that it is easy to say "nice code,
I know all these functions, that code is easy to maintain" but
actually all those versions aren't quite the same, and they
will start to drift even more afar. Some dude will think, "oh,
I understand this from somewhere else already" but it isn't
exactly the same, just something close to what he previously
saw. Replace all those with one library function - yes,
everyone will have to learn that function but it will be much
better code and also easier to maintain since it is much more
reliable, a neat module that always does what it does.

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]



  > You didn't say "list manipulation function" . You wrote "No Emacs
  > built-in takes them".  That's false.  Take make-hash-table.  Oh noes,
  > looks just like CL.  Guess you're going to hand-roll hash tables
  > now, right? :-)

The tone of your message is harsh and contemptuous.  You can disagree
with him, nut please disagree respectfully.

It is not unusual for people who have basically valid points to
overstate them and thus make an incorrect assertion.  We all make such
minor mistakes like that.  To keep the discussion civil, and aim for a
useful conclusion, we should not exaggerate the importance of those
minor mistakes.

The usefuk goal is to see what the real difference is -- even if it
is not a simple absolute rule without exceptions.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 10/11/2023 15:14, Po Lu wrote:
>> Why do you exempt yourself from this irresponsibility?  Can't you see
>> this is all subjective and it seems a bit arrogant to say "my code is
>> so responsible"??
> Thus far there have been no complaints that code_without_  cl-lib,
> pcase, seq or elt is unreadable.  Most of our code (this time measured
> by lines) falls squarely within that category.

Without having a specific enemy to blame, when the function is long, 
people just either admit out loud they find the code difficult or 
silently avoid changing it.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

Dmitry Gutov <dmitry@gutov.dev> writes:

> Without having a specific enemy to blame, when the function is long,
> people just either admit out loud they find the code difficult or
> silently avoid changing it.

The length of the function plays a far lesser role in one's perception
of a function's complexity than the quantity (or scarcity) of commentary
and the number of unfamiliar constructs it employs do.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Dmitry Gutov wrote:

>>> Why do you exempt yourself from this irresponsibility?
>>> Can't you see this is all subjective and it seems a bit
>>> arrogant to say "my code is so responsible"??
>>
>> Thus far there have been no complaints that code_without_
>> cl-lib, pcase, seq or elt is unreadable. Most of our code
>> (this time measured by lines) falls squarely within
>> that category.
>
> Without having a specific enemy to blame, when the function
> is long, people just either admit out loud they find the
> code difficult or silently avoid changing it.

The way people have been writing Elisp in functions that are so
long it is insane always puzzled me. I mean, not because they
were so long - just look at the Gnus source for example - but
why they were written like that to begin with?

Also the files themselves are so long? (Maybe because they are
made up of so long functions LOL.)

In my own Elisp I never had that problem of almost endless
functions. But I'm unsure why, maybe people solved much more
difficult problems then? *impressed* *envious*

But I think cl-lib and pcase should make the code more compact
(i.e., shorter lines for the same job).

So yeah, it is less complexity in the sense fewer lines and
more standard library usage.

But also more complexity in the sense people have to learn the
basics of cl-lib and pcase, at the very least, including their
mini-languages (e.g. `cl-loop'), and further not be frustrated
by what they can't learn by heart the way they once mastered
Elisp but must look up in docstrings when encountered.

One can also hope that the most common use cases of cl-lib and
pcase will in time be idiomatic as well - so even tho you
don't know the whole library or construct by heart, the way
they are usually applied, that will be familiar enough to the
trained eyes of Emacs maintainers and contributors.

For example, I'm sure many people are fond of their scientific
calculators. And are quite fluent with those. But no one knows
what all these functions and buttons, every single one,
actually do, right?

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 10/11/2023 05:17, Po Lu wrote:
>> Hmmm, a bit vague, no?  Humor me: if it's really so easy and so
>> readable please write out your preferred equivalent of, say
>>
>>     (cl-set-difference l1 l2) ; hopefully obvious, like l1 - l2 in python
> (let ((list nil))
>    (dolist (x l1)
>      (unless (member x l2)
>        (push x list)))
>    (dolist (x l2)
>      (unless (member x l1)
>        (push x list)))
>    list)

Just to take this example, there are about 2 major bugs: 1) the list 
will come out reversed, 2) the code does XOR rather than subtraction.

To be clear, this is not to take a dig at your coding ability (which I'm 
convinced is quite high), but rather to make a point about the potential 
for problems when one uses lower-level tools and has to reinvent the wheel.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

Dmitry Gutov <dmitry@gutov.dev> writes:

> Just to take this example, there are about 2 major bugs: 1) the list
> will come out reversed, 2) the code does XOR rather than subtraction.

Hmm, reading its doc string, I think a more illustrative name for
cl-set-difference would be cl-set-complement.  In this case, the second
loop can be removed.

If the order is significant (it seldom is), the list can be nreversed
afterwards.  Or alternatively a local variable can be maintained holding
the last cdr of the list, and insertion performed on that instead.

Re: What's missing in ELisp that makes people want to use cl-lib?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 569 bytes --]

On Sat, Nov 11, 2023 at 09:23:26AM +0800, Po Lu wrote:
> Dmitry Gutov <dmitry@gutov.dev> writes:
> 
> > Just to take this example, there are about 2 major bugs: 1) the list
> > will come out reversed, 2) the code does XOR rather than subtraction.
> 
> Hmm, reading its doc string, I think a more illustrative name for
> cl-set-difference would be cl-set-complement.  In this case, the second
> loop can be removed.

Yes, "difference" is, surprisingly, ambiguous. Your implementation is
usually called "symmetric difference" to avoid that.

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Sat, Nov 11, 2023 at 6:12 AM <tomas@tuxteam.de> wrote:
>
> On Sat, Nov 11, 2023 at 09:23:26AM +0800, Po Lu wrote:
> > Dmitry Gutov <dmitry@gutov.dev> writes:
> >
> > > Just to take this example, there are about 2 major bugs: 1) the list
> > > will come out reversed, 2) the code does XOR rather than subtraction.
> >
> > Hmm, reading its doc string, I think a more illustrative name for
> > cl-set-difference would be cl-set-complement.  In this case, the second
> > loop can be removed.
>
> Yes, "difference" is, surprisingly, ambiguous.

I wouldn't say "ambiguous", but this page has a brief review of
different ways the operation is referred to in literature.

   https://proofwiki.org/wiki/Definition:Set_Difference/Also_known_as

In programming languages "set difference" seems to be the
overwhelmingly established term

  https://cplusplus.com/reference/algorithm/set_difference/
  http://clhs.lisp.se/Body/f_set_di.htm
  https://docs.python.org/3/library/stdtypes.html#set
  https://ruby-doc.org/core-1.9.3/Array.html#method-i-2D
  https://doc.rust-lang.org/std/collections/struct.HashSet.html#method.difference

> Your implementation is
> usually called "symmetric difference" to avoid that.

Correct.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Michael Heerdegen]

Po Lu <luangruo@yahoo.com> writes:

> Dmitry Gutov <dmitry@gutov.dev> writes:
>
> > Just to take this example, there are about 2 major bugs: 1) the list
> > will come out reversed, 2) the code does XOR rather than subtraction.
>
> Hmm, reading its doc string, I think a more illustrative name for
> cl-set-difference would be cl-set-complement.  In this case, the second
> loop can be removed.

BTW, a "set.el" library is on our wishlist, complementing seq.el and
map.el.

Michael.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Michael Heerdegen wrote:

>> Hmm, reading its doc string, I think a more illustrative
>> name for cl-set-difference would be cl-set-complement.
>> In this case, the second loop can be removed.
>
> BTW, a "set.el" library is on our wishlist, complementing
> seq.el and map.el.

Good idea - but are you allowed to use cl-lib from it?

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Michael Heerdegen]

Emanuel Berg <incal@dataswamp.org> writes:

> > BTW, a "set.el" library is on our wishlist, complementing
> > seq.el and map.el.
>
> Good idea - but are you allowed to use cl-lib from it?

That would not make much sense.

Michael.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Michael Heerdegen wrote:

>>> BTW, a "set.el" library is on our wishlist, complementing
>>> seq.el and map.el.
>>
>> Good idea - but are you allowed to use cl-lib from it?
>
> That would not make much sense.

Why not? Because then cl-lib would have to be loaded and we
are not allowed to do that?

But even so, tell me then, how will additional functions
dealing with sets, supposedly added to set.el, how would they
not contribute to the increased complexity of Emacs Lisp the
same way cl-lib has?

A computer program is a bunch of files, this whole strategy is
putting a limit how good an individual file can be, because
that increases the overall complexity over a large amount of
such files, that a few maintainers have taken on to maintain.

That is a problem on the human side. And not all humans side,
even, as all the people who have used cl-lib in their source
and in their source for Emacs are, of course, humans as well.
So there are two different human perspectives.

On the technology side alone, if a file can be improved by
using a library, how is that not the right thing?

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Michael Heerdegen]

Emanuel Berg <incal@dataswamp.org> writes:

> >> Good idea - but are you allowed to use cl-lib from it?
> >
> > That would not make much sense.
>
> Why not? Because then cl-lib would have to be loaded and we
> are not allowed to do that?

cl-lib has not really good support for set operations.  I has some
limited support for performing set operations on lists when interpreting
these as set specifications.  There is little sense in requiring this
large suite just for these three functions.

> But even so, tell me then, how will additional functions
> dealing with sets, supposedly added to set.el, how would they
> not contribute to the increased complexity of Emacs Lisp the
> same way cl-lib has?

cl-lib only implements a small subset on what is needed.  Sets are
useful mostly to provide a test "is some thing in the set?".  A natural
way to implement this feature efficiently is using hash tables.  cl-lib
has literally nothing to offer here.  There are only the functions for
lists.

And "set" can be a useful abstraction to improve readability, so having
a dedicated library providing efficient implementations would be a good
thing.


Michael.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Mon, Nov 13, 2023 at 8:53 AM Michael Heerdegen
<michael_heerdegen@web.de> wrote:
>
> Emanuel Berg <incal@dataswamp.org> writes:
>
> > >> Good idea - but are you allowed to use cl-lib from it?
> > >
> > > That would not make much sense.
> >
> > Why not? Because then cl-lib would have to be loaded and we
> > are not allowed to do that?
>
> cl-lib has not really good support for set operations.  I has some
> limited support for performing set operations on lists when interpreting
> these as set specifications.  There is little sense in requiring this
> large suite just for these three functions.

C.f. my benchmarks.  But I agree it would be a good thing to split it up.
into, say cl-list-set.el, I've been working on this code, it's not particularly
difficult.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Michael Heerdegen wrote:

> There is little sense in requiring this large suite just for
> these three functions.

The sense is: then you don't have to write these three
functions again, locally :)

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Sat, Nov 11, 2023 at 1:23 AM Po Lu <luangruo@yahoo.com> wrote:
>
> Dmitry Gutov <dmitry@gutov.dev> writes:
>
> > Just to take this example, there are about 2 major bugs: 1) the list
> > will come out reversed, 2) the code does XOR rather than subtraction.
>
> Hmm, reading its doc string, I think a more illustrative name for
> cl-set-difference would be cl-set-complement.

No, it's called "set difference" in CL and in other languages for a reason:

   https://proofwiki.org/wiki/Definition:Set_Difference

> If the order is significant (it seldom is),

You don't know that, of course.

> Or alternatively a local variable can be maintained holding
> the last cdr of the list, and insertion performed on that instead.

choo-choo!

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> No, it's called "set difference" in CL and in other languages for a reason:
>
>    https://proofwiki.org/wiki/Definition:Set_Difference

With Emacs, Webster's 1913 calls the shots to a far greater extent than
that does:

   1. The act of differing; the state or measure of being
      different or unlike; distinction; dissimilarity;
      unlikeness; variation; as, a difference of quality in
      paper; a difference in degrees of heat, or of light; what
      is the difference between the innocent and the guilty?
      [1913 Webster]

The difference between the innocent and the guilty is the culpability of
the latter.

   (cl-set-difference '(innocent) '(innocent culpable))

> You don't know that, of course.

Yes, I do: I've written much Emacs Lisp over the years.

> choo-choo!

Sure, if that's your preferred sobriquet for the customary practice as
regards the creation of linked lists.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

Po Lu <luangruo@yahoo.com> writes:

> Respect for volunteers is no doubt laudable, but it is very much an
> position to be considered alone which everything you've presented above
> does not impact and nobody takes exception to.  From it, and even
> supposing for the sake of argument that everything else you've said is
> incontrovertible fact, it doesn't really follow that we should not seek
> to reduce the incidence of cl-lib usage in core code.  Now I can't be
> certain that is the object of your militating, if you will, though
> promoting cl-set-difference and cl-position as alternatives to hallowed
> Emacs Lisp constructs does suggest so, but the principle argument
                                                 ---------

Principal, sorry.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Po Lu <luangruo@yahoo.com> writes:

> Respect for volunteers is no doubt laudable, but it is very much an
> position to be considered alone which everything you've presented above
> does not impact and nobody takes exception to.  From it, and even
> supposing for the sake of argument that everything else you've said is
> incontrovertible fact, it doesn't really follow that we should not seek
> to reduce the incidence of cl-lib usage in core code.  Now I can't be
> certain that is the object of your militating, if you will, though
> promoting cl-set-difference and cl-position as alternatives to hallowed
> Emacs Lisp constructs does suggest so, but the principle argument
> against these constructs is that they are alien to the small group of
> individuals charged with trudging through code where people want to make
> use of them.  So no contentions resting on their virtues and
> distinctions rather than their prevalence among us can hold water...

(Sorry if I don't cite the whole mail. Please re-add what you find
important.)

Could you perhaps formulate a plan for what should be done? I personally
find something concrete much easier to handle than a back-and-forth of
arguments.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Could you perhaps formulate a plan for what should be done? I personally
> find something concrete much easier to handle than a back-and-forth of
> arguments.

First off, I think people should make it clear that new code saturated
with calls to cl-lib will not be accepted.  Once people are coaxed out
of this habit, it will be possible to start removing calls to cl-lib
from existing code.

This will also afford us opportunities to move frequently used cl-lib
constructs that aren't all that adventitious to subr.el.  If all goes
well, it will be feasible to use Emacs without loading cl-lib at all.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 10, 2023 at 10:42 AM Po Lu <luangruo@yahoo.com> wrote:
>
> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
> > Could you perhaps formulate a plan for what should be done? I personally
> > find something concrete much easier to handle than a back-and-forth of
> > arguments.
>
> First off, I think people should make it clear that new code saturated
> with calls to cl-lib will not be accepted.  Once people are coaxed out
> of this habit, it will be possible to start removing calls to cl-lib
> from existing code.
>
> This will also afford us opportunities to move frequently used cl-lib
> constructs that aren't all that adventitious to subr.el.  If all goes
> well, it will be feasible to use Emacs without loading cl-lib at all.

Just a quick question.  Does all of this go for seq.el, map.el and
pcase.el as well, or just cl-lib?

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> Just a quick question.  Does all of this go for seq.el, map.el and
> pcase.el as well, or just cl-lib?

It's not a requirement for pcase to be loaded in code which makes use of
it, and the latter two are (I believe) preloaded.  So only cl-lib.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Po Lu <luangruo@yahoo.com> writes:

> João Távora <joaotavora@gmail.com> writes:
>
>> Just a quick question.  Does all of this go for seq.el, map.el and
>> pcase.el as well, or just cl-lib?
>
> It's not a requirement for pcase to be loaded in code which makes use of
> it, and the latter two are (I believe) preloaded.  So only cl-lib.

I don't understand that. If cl-lib were preloaded, it would also be no
problem? 

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> I don't understand that. If cl-lib were preloaded, it would also be no
> problem? 

That's an invitation to use it, which will lead to more cryptic cl-lib
code.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Po Lu <luangruo@yahoo.com> writes:

> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
>> I don't understand that. If cl-lib were preloaded, it would also be no
>> problem? 
>
> That's an invitation to use it, which will lead to more cryptic cl-lib
> code.

Aha, but seq/map are okay because they are preloaded.
I guess I don't have to understand everything.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Aha, but seq/map are okay because they are preloaded.
> I guess I don't have to understand everything.

I don't think they are, but their incursion into our code is
accomplished fact.  So we can only bow to fate...

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: João Távora <joaotavora@gmail.com>,  Alan Mackenzie
>  <acm@muc.de>,
>  Dmitry Gutov <dmitry@gutov.dev>,  Björn Bidar
>  <bjorn.bidar@thaodan.de>, emacs-devel <emacs-devel@gnu.org>
> Date: Sat, 11 Nov 2023 07:34:49 +0100
> 
> Po Lu <luangruo@yahoo.com> writes:
> 
> > Gerd Möllmann <gerd.moellmann@gmail.com> writes:
> >
> >> I don't understand that. If cl-lib were preloaded, it would also be no
> >> problem? 
> >
> > That's an invitation to use it, which will lead to more cryptic cl-lib
> > code.
> 
> Aha, but seq/map are okay because they are preloaded.

map.el isn't preloaded.  Only seq.el is.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Augusto Stoffel]

On Sat, 11 Nov 2023 at 10:01, Eli Zaretskii wrote:

> map.el isn't preloaded.  Only seq.el is.

By the way, map.el has some serious flaws that should be fixed before it
becomes more widespread (I've filed a couple of bugs some time ago).

The main issue is that it treats maps as mutable objects rather than
values, so it only feels natural when used on hash tables.  It's
cumbersome when operating on alists and plists.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Just a quick question.  Does all of this go for seq.el, map.el and
  > pcase.el as well, or just cl-lib?

Those cases have similarities, but also differences.  The cl functions
cover a much bigger range of activities than seq.e or map.el.

pcase is just one construct -- but it is a very confusing one that I
can't grasp from its doc string.  The basic idea is a natural thing to
want, but the details are so unnatural that I can't figure them out.

I am sure we can find a cleaner way to express the idea of trying a
series of patterns against one or more objects until one matches.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Po Lu <luangruo@yahoo.com> writes:

> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
>> Could you perhaps formulate a plan for what should be done? I personally
>> find something concrete much easier to handle than a back-and-forth of
>> arguments.
>
> First off, I think people should make it clear that new code saturated
> with calls to cl-lib will not be accepted.  Once people are coaxed out
> of this habit, it will be possible to start removing calls to cl-lib
> from existing code.
>
> This will also afford us opportunities to move frequently used cl-lib
> constructs that aren't all that adventitious to subr.el.  If all goes
> well, it will be feasible to use Emacs without loading cl-lib at all.

Thanks!

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

João Távora <joaotavora@gmail.com> writes:

> On Thu, Nov 9, 2023 at 10:05 AM Alan Mackenzie <acm@muc.de> wrote:
>
>> >    If it needed any confirmation, I too like cl-lib and I too help
>> >    maintain other people's code in the Emacs core tree as well as
>> >    maintaining a number of libraries I have authored.
>>
>> There's a difference between liking cl-lib and advocating its
>> indiscriminate use.  I don't think you've done the latter in this (and
>> related) threads.
>
> Yes, you're right.  Indeed I don't' advocate for its indiscriminate use,
> just as I don't advocate for indiscriminate use of anything, except
> perhaps drinking water and brushing teeth.
>
>> Nobody who likes cl-lib has yet addressed the point made by Richard and
>> (less eloquently) by me, namely that the incorporation and use of cl-lib
>> swells the size and complexity of Emacs Lisp to the point of making
>> maintenance difficult.  What is your view on this important point?
>
> That it doesn't make maintenance any more difficult than any other
> Elisp construct, be it very old and curiously named like 'rplacd' or
> much, much newer like `seq-do` or `pcase-lambda`.
>
> My specific view on cl-lib.el is that it brings us a small part of
> the results of  non-trivial design work put in when important figures
> in the Lisp world met  regularly for many years to deliver what has
> proved to become excellent,  battle-tested, widely understood and
> impeccably documented programming abstractions.
>
> What I'm reading so far in this long discussion is that the argument
> of its detractors isn't really that cl-lib isn't good, but that
> it is superfluous and that learning it is a burden on maintainers.
> Well, it's just as superfluous as all of Elisp apart from two handfuls
> of primitives, I guess.  Or any programming language for that matter, if
> you know enough machine code.  Or any other programming abstraction I
> happen not to be familiar with.
>
> Also I seem to hear that Elisp has some kind of ideal hard-to-define
> identity or fingerprint and that it shouldn't try to become anything
> else.  But this argument is very strange given Elisp is, like any
> decent language, easy to extend.  Not to mention I struggle to see
> the technical advantage in uniqueness for uniqueness sake.  A good
> part of Elisp is about a special purpose function: editing buffers,
> and an the equally important part is about doing things with bits in
> memory, there's no reason to advocate for singularity here.
>
> I also hear Elisp shouldn't become Common Lisp, but not only is the
> use of cl-lib.el nowhere a step in that direction, but also -- somewhat
> ironically -- if Elisp _were_ Common Lisp, then that hard-to-define
> identity would be much easier to define and language extension would be
> much easier to organize into compartments to facilitate policy-making.
>
> Again, the only thing that has brought Elisp any closer to Common
> Lisp significantly, was the introduction of lexical binding some 10
> years ago.  Elisp looks a lot different now than it did in the 90's.
> Closures everywhere, higher-order functions!  Shudder!
>
> There's even talk of a continuation-passing style library, to
> be called future.el or promise.el or something.  Oh dear, what
> will be of us that we will have to evolve like any other language
> in the world!
>
> So I propose we let programmers use their judgement.  Really
> respect people who write code for no money and give the copyright
> away to the FSF.  Maybe suggest guidelines such as not introduce
> cl-loop where a dolist would do the job just as economically and
> elegantly. Don't use higher-order functions just because it looks
> cool. But maybe do suggest to use cl-position with a :key and a
> :test instead of a complex while loop with an auxiliary variable.
> Or cl-set-difference instead of nested loops.  Suggest to express
> intent, use abstractions.   Suggest to be consistent, be scrutinous,
> be "discriminate", be mindful of the style of the current area
> you are working on.
>
> But don't suggest anything too hard, especially if it's not
> modifying code you have authored.  Don't use arguments of authority
> when you can point to specific things.  Be generally respectful of
> people putting in any good work even if you don't like the style,
> and try to learn a new thing or two every now and then.
>
> BTW here are some nice generic suggestions from the Lisp world,
> written by two fenomenal programmers.  I love reading this from
> time to time:
>
>    https://www.cs.umd.edu/~nau/cmsc421/norvig-lisp-style.pdf

Thanks, that was well written, João. I agree.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Alan Mackenzie]

Hello, João.

On Thu, Nov 09, 2023 at 11:06:02 +0000, João Távora wrote:
> On Thu, Nov 9, 2023 at 10:05 AM Alan Mackenzie <acm@muc.de> wrote:

> > >    If it needed any confirmation, I too like cl-lib and I too help
> > >    maintain other people's code in the Emacs core tree as well as
> > >    maintaining a number of libraries I have authored.
> >
> > There's a difference between liking cl-lib and advocating its
> > indiscriminate use.  I don't think you've done the latter in this (and
> > related) threads.

> Yes, you're right.  Indeed I don't' advocate for its indiscriminate use,
> just as I don't advocate for indiscriminate use of anything, except
> perhaps drinking water and brushing teeth.

> > Nobody who likes cl-lib has yet addressed the point made by Richard and
> > (less eloquently) by me, namely that the incorporation and use of cl-lib
> > swells the size and complexity of Emacs Lisp to the point of making
> > maintenance difficult.  What is your view on this important point?

> That it doesn't make maintenance any more difficult than any other
> Elisp construct, be it very old and curiously named like 'rplacd' or
> much, much newer like `seq-do` or `pcase-lambda`.

OK, thanks for answering the question.  Please bare in mind that other
people's experience of maintaining code using cl-* is very different.

> My specific view on cl-lib.el is that it brings us a small part of
> the results of  non-trivial design work put in when important figures
> in the Lisp world met  regularly for many years to deliver what has
> proved to become excellent,  battle-tested, widely understood and
> impeccably documented programming abstractions.

> What I'm reading so far in this long discussion is that the argument
> of its detractors isn't really that cl-lib isn't good, but that
> it is superfluous and that learning it is a burden on maintainers.
> Well, it's just as superfluous as all of Elisp apart from two handfuls
> of primitives, I guess.

No, it's completely superfluous, even if it can be convenient.  Learning
it is a burden because its objects and abstractions are, well, very
abstract as well as being badly documented.  They have little
relationship to anything concrete in Emacs, and they are typically
complicated with many seemingly random details in a way that Emacs
primitives are not.

> Or any programming language for that matter, if you know enough
> machine code.  Or any other programming abstraction I happen not to be
> familiar with.

I am familiar with machine code.  It is perhaps pertinent to remark that
modern processors tend to the reduced instruction set model - that the
complicated "do it in one single complicated instruction" notion doesn't
help the writing of good compilers, or the running of programs
efficiently.  I suspect that analagous logic applies to programming
languages too.

[ .... ]

> João

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 10, 2023 at 1:16 PM Alan Mackenzie <acm@muc.de> wrote:

> > Or any programming language for that matter, if you know enough
> > machine code.  Or any other programming abstraction I happen not to be
> > familiar with.
>
> I am familiar with machine code.  It is perhaps pertinent to remark that
> modern processors tend to the reduced instruction set model - that the
> complicated "do it in one single complicated instruction" notion doesn't
> help the writing of good compilers, or the running of programs
> efficiently.  I suspect that analagous logic applies to programming
> languages too.

That's a clever and interesting perspective actually, but I would
tend to say no.  Programming languages are still written and read
by humans.  Human brains aren't designed the same way processors
are.  For one they aren't "designed" at all.  And we don't consume
information the same way a metal likes to, we don't unroll loops,
and at least I don't prefetch all of the supermarket shelf when
someone tells me to get olives, etc, etc.  Plus I'm horrible at
parallelism.  And we don't speak with reduced instruction sets,
we have many words with nuanced meanings.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Manuel Giraud via Emacs development discussions.]

João Távora <joaotavora@gmail.com> writes:

[...]

> and at least I don't prefetch all of the supermarket shelf when
> someone tells me to get olives, etc, etc.

😂 you should!  It sure takes some space at home but then you really
save time.
-- 
Manuel Giraud

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> That's a clever and interesting perspective actually, but I would
> tend to say no.  Programming languages are still written and read
> by humans.  Human brains aren't designed the same way processors
> are.  For one they aren't "designed" at all.  And we don't consume
> information the same way a metal likes to, we don't unroll loops,
> and at least I don't prefetch all of the supermarket shelf when
> someone tells me to get olives, etc, etc.  Plus I'm horrible at
> parallelism.  And we don't speak with reduced instruction sets,
> we have many words with nuanced meanings.

No, but humans are less still adapted for processing dozens of
obliquely-named constructs which wreck havoc on the control and data
flow of many a form.

Suppose that to retrieve a shopping cart, it were mandatory to connect
half a dozen unmarked pulleys and pneumatic tubes to one another in a
particular arrangement before depositing your coin in one end of the
contrivance, rather than to merely insert the coin in a small
receptacle.  Would you not be so deterred from availing yourself of the
gratis trolley rentals?

Re: What's missing in ELisp that makes people want to use cl-lib?

[Michael Heerdegen]

Dmitry Gutov <dmitry@gutov.dev> writes:

> I like cl-lib (it could be optimized better still, but it's good for
> certain uses), and this discussion doesn't look like it will reach any
> constructive conclusion, so it didn't seem worth it to spend time on
> it.

I also find it a bit exaggerating: cl-lib is large, but the majority of
its uses in Emacs are from a small subset.  In cases where it's
objectively beneficial, using its stuff can make the code actually
better readable.  But everybody should, as always, just select the
simplest tools when coding.  Please let's avoid a discussion about what
different opinions about readability and simplicity we have.

But of course: when we continue to migrate the most useful stuff in
cl-lib out to other libraries (like we did with generic functions,
generalized variables, sequence functions, ...), answering this question
might become harder in the future.

Michael.

Re: What's missing in ELisp that makes people want to use cl-lib?

[T.V Raman]

Michael Heerdegen <michael_heerdegen@web.de> writes:


One more fan of cl, and cl-libs is a good solution and a nice compromise
 between having the expressiveness of cl, without loading all of cl at
 runtime.

Creating and migrating to cl-libs was a large effort:

A. For the creators of cl-libs who carefully did the refactoring,
   and for
   B: Package authors who paid attention to the "cl used at runtime "
   warning to properly namespace-qualify cl calls,  and remove  (require
   'cl) 


So I dont quite understand the apparent desire to negate all of that work?
   
>
>> I like cl-lib (it could be optimized better still, but it's good for
>> certain uses), and this discussion doesn't look like it will reach any
>> constructive conclusion, so it didn't seem worth it to spend time on
>> it.
>
> I also find it a bit exaggerating: cl-lib is large, but the majority of
> its uses in Emacs are from a small subset.  In cases where it's
> objectively beneficial, using its stuff can make the code actually
> better readable.  But everybody should, as always, just select the
> simplest tools when coding.  Please let's avoid a discussion about what
> different opinions about readability and simplicity we have.
>
> But of course: when we continue to migrate the most useful stuff in
> cl-lib out to other libraries (like we did with generic functions,
> generalized variables, sequence functions, ...), answering this question
> might become harder in the future.
>
> Michael.
>
>

-- 

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Alan Mackenzie <acm@muc.de> writes:

> That is the current reality.

Maybe you could elaborate on what the plan then could look like?
Or is it not about a plan, but something else? 

Re: What's missing in ELisp that makes people want to use cl-lib?

[Alan Mackenzie]

On Thu, Nov 09, 2023 at 08:24:14 +0100, Gerd Möllmann wrote:
> Alan Mackenzie <acm@muc.de> writes:

>>Just take one look at this "reality" you're so supportive of: the
>>widespread use of cl-lib, not just in people's own projects, but
>>throughout the core of Emacs, has multiplied the size of Lisp language
>>part of Emacs by a factor of around 3.   This is a gross increase in
>>complexity for maintainers that is not justified by the slight increase
>>in facility that cl-lib (along with things like seq.el and oclosures)
>>gives.

>>Throughout this long discussion, this indiscriminate use of cl-lib has
>>been supported only by occasional contributers.  Those who actually
>>maintain other people's code, apart from (I think) Eli, Richard and
>>myself, have been conspicuously silent.  None of us three have favoured
>>such use of cl-lib.

>>Occasional contributors may be fascinated by cl-lib, and learn enough
>>of it to use random bits of it in their code.  The trouble is, each
>>such contributor uses a different piece of cl-lib, with the result that
>>those who end up maintaining it need to know a far greater part of it
>>just to cope.

>>This factor of 3 is, I believe, a significant barrier to new
>>programmers coming into Emacs; Elisp is just that much more difficult
>>than it was in the past.  And it isn't just for newcomers that it is
>>more difficult.  I spend a significant amount of debugging time having
>>to look up doc strings and manual pages for obscure cl-lib (etc.)
>>functions.

> > That is the current reality.

> Maybe you could elaborate on what the plan then could look like?
> Or is it not about a plan, but something else? 

You stripped all the context.  I've put it back again.

As a concrete plan, I would propose the following for discussion:

We should deprecate those functions/macros/variables in cl-lib that have
no doc string, or a substandard one.  This includes "internal" functions,
too.  Also to be deprecated are obscure functions/m/v (such as
cl-labels).

Having done this, we recode code currently using those deprecated f/m/v.

(Here a "substandard" doc string is contrasted with an adequate one,
which does all of the following:
(i) It says what the function/macro _does_, or what the variable _is_.
(ii) It describes the form and meaning of each parameter, and its
  relationship to (i).
(iii) If the return value is significant, it describes this.
(iv) It describes all effects on the global state, such as where it
  writes results to, and suchlike.)

This would reduce the size of cl-lib to something more manageable than
what we have at the moment.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 09/11/2023 12:34, Alan Mackenzie wrote:
> As a concrete plan, I would propose the following for discussion:
> 
> We should deprecate those functions/macros/variables in cl-lib that have
> no doc string, or a substandard one.  This includes "internal" functions,
> too.  Also to be deprecated are obscure functions/m/v (such as
> cl-labels).

I'm not sure we can remove cl-labels -- it's useful enough for the cases 
where it does help, and there are callers inside and outside of Emacs.

Whatever other removals, would probably need to be discussed case-by-case.

> Having done this, we recode code currently using those deprecated f/m/v.
> 
> (Here a "substandard" doc string is contrasted with an adequate one,
> which does all of the following:
> (i) It says what the function/macro_does_, or what the variable_is_.
> (ii) It describes the form and meaning of each parameter, and its
>    relationship to (i).
> (iii) If the return value is significant, it describes this.
> (iv) It describes all effects on the global state, such as where it
>    writes results to, and suchlike.)

Improving cl-lib's documentation would be a welcome effort.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Thu, Nov 9, 2023 at 11:49 AM Dmitry Gutov <dmitry@gutov.dev> wrote:

> Improving cl-lib's documentation would be a welcome effort.

For sure, and not a hard one as well, as all those functions and
macros are pretty good, often flawless emulations of CL functions that
are impeccably documented in

  http://www.lispworks.com/documentation/HyperSpec/Front/

Which is of free access (though not of a compatible license, I
think).  But if people can point to the 5 most confusing functions
they think are poorly documented, I volunteer to rewrite the
docstrings for them.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Alan Mackenzie]

Hello, João.

On Thu, Nov 09, 2023 at 12:40:24 +0000, João Távora wrote:
> On Thu, Nov 9, 2023 at 11:49 AM Dmitry Gutov <dmitry@gutov.dev> wrote:

> > Improving cl-lib's documentation would be a welcome effort.

> For sure, and not a hard one as well, as all those functions and
> macros are pretty good, often flawless emulations of CL functions that
> are impeccably documented in

>   http://www.lispworks.com/documentation/HyperSpec/Front/

> Which is of free access (though not of a compatible license, I
> think).  But if people can point to the 5 most confusing functions
> they think are poorly documented, I volunteer to rewrite the
> docstrings for them.

How much are you prepared to do?  I don't have a list of the _most_
confusing doc strings, there are too many to chose from.  But starting
at the start of cl-macs.el, we have:

(i) cl--compiler-macro-list*; completely undocumented.
(ii) cl--simple-expr-p: Talks about "side effects", but not what they
  are side effects of.  Doesn't describe it's parameters or return
  value.  It's unclear what it is that "executes quickly".
(iii) cl--expr-contains: It's unclear what X and Y are, and what "refers
  to" means.
(iv) cl--expr-contains-any; completely undocumented.
(v) cl--expr-depends-p: It's unclear what X and Y are, though Y appears
  to be some sort of container of symbols.  It's unclear what sort of
  "dependency" the function handles, or what "may" means in the context.

There are many more.

> João

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Thu, Nov 9, 2023 at 1:36 PM Alan Mackenzie <acm@muc.de> wrote:
>
> Hello, João.
>
> On Thu, Nov 09, 2023 at 12:40:24 +0000, João Távora wrote:
> > On Thu, Nov 9, 2023 at 11:49 AM Dmitry Gutov <dmitry@gutov.dev> wrote:
>
> > > Improving cl-lib's documentation would be a welcome effort.
>
> > For sure, and not a hard one as well, as all those functions and
> > macros are pretty good, often flawless emulations of CL functions that
> > are impeccably documented in
>
> >   http://www.lispworks.com/documentation/HyperSpec/Front/
>
> > Which is of free access (though not of a compatible license, I
> > think).  But if people can point to the 5 most confusing functions
> > they think are poorly documented, I volunteer to rewrite the
> > docstrings for them.
>
> How much are you prepared to do?  I don't have a list of the _most_
> confusing doc strings, there are too many to chose from.  But starting
> at the start of cl-macs.el, we have:
>
> (i) cl--compiler-macro-list*; completely undocumented.
> (ii) cl--simple-expr-p: Talks about "side effects", but not what they
>   are side effects of.  Doesn't describe it's parameters or return
>   value.  It's unclear what it is that "executes quickly".
> (iii) cl--expr-contains: It's unclear what X and Y are, and what "refers
>   to" means.
> (iv) cl--expr-contains-any; completely undocumented.
> (v) cl--expr-depends-p: It's unclear what X and Y are, though Y appears
>   to be some sort of container of symbols.  It's unclear what sort of
>   "dependency" the function handles, or what "may" means in the context.
>
> There are many more.

These are all internal functions and implementation details.  They're
not necessary at all for users of cl-lib.el, only for its developers.
What problem are you trying to solve by enhancing these docstrings?  I thought
the problem here was code that _used_ cl-lib.el, not hacking on
cl-lib.el itself.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

João Távora wrote:

> These are all internal functions and implementation details.
> They're not necessary at all for users of cl-lib.el, only
> for its developers. What problem are you trying to solve by
> enhancing these docstrings? I thought the problem here was
> code that _used_ cl-lib.el, not hacking on cl-lib.el itself.

It is still nuch better than removing or rewriting it, so
let's do that.

We can use checkdoc to find out what docstrings are not
submitted or not correctly formated. Those should be pretty
quick fixes, to get away with those warnings.

After we have done that, if someone says some specific
docstring is still lacking we can focus on such individual
cases in more detail.

Here, to get the list from, e.g., cl-lib.el

(defun check-package-style ()
  (interactive)
  (let ((msg "Style check..."))
    (message msg)
    (checkdoc-current-buffer t)
    (message "%sdone" msg) ))

And the list is

cl-lib.el:133: Argument ‘keys’ should appear (as KEYS) in the doc string
cl-lib.el:160: All variables and subroutines might as well have a documentation string
cl-lib.el:166: All variables and subroutines might as well have a documentation string
cl-lib.el:197: All variables and subroutines might as well have a documentation string
cl-lib.el:197: All variables and subroutines might as well have a documentation string
cl-lib.el:226: Arguments occur in the doc string out of order
cl-lib.el:258: Argument ‘specs’ should appear (as SPECS) in the doc string
cl-lib.el:357: Argument ‘cl-func’ should appear (as CL-FUNC) in the doc string
cl-lib.el:450: Argument ‘rest’ should appear (as REST) in the doc string
cl-lib.el:488: Argument ‘cl-item’ should appear (as CL-ITEM) in the doc string
cl-lib.el:501: Argument ‘cl-new’ should appear (as CL-NEW) in the doc string
cl-lib.el:510: All variables and subroutines might as well have a documentation string
cl-lib.el:544: All variables and subroutines might as well have a documentation string
cl-lib.el:582: Probably "returns" should be imperative "return"

Here are all the cl-lib files

cl-extra.el
cl-generic.el
cl-indent.el
cl-lib.el
cl-loaddefs.el
cl-macs.el
cl-preloaded.el
cl-print.el
cl-seq.el

GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, cairo version 1.16.0)
of 2023-11-01 [commit
8eb9b82ce58a8323af21f52625a401e19e279a9c]

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Alan Mackenzie]

Hello, João.

On Thu, Nov 09, 2023 at 13:41:48 +0000, João Távora wrote:
> On Thu, Nov 9, 2023 at 1:36 PM Alan Mackenzie <acm@muc.de> wrote:

> > Hello, João.

> > On Thu, Nov 09, 2023 at 12:40:24 +0000, João Távora wrote:
> > > On Thu, Nov 9, 2023 at 11:49 AM Dmitry Gutov <dmitry@gutov.dev> wrote:

> > > > Improving cl-lib's documentation would be a welcome effort.

> > > For sure, and not a hard one as well, as all those functions and
> > > macros are pretty good, often flawless emulations of CL functions that
> > > are impeccably documented in

> > >   http://www.lispworks.com/documentation/HyperSpec/Front/

> > > Which is of free access (though not of a compatible license, I
> > > think).  But if people can point to the 5 most confusing functions
> > > they think are poorly documented, I volunteer to rewrite the
> > > docstrings for them.

> > How much are you prepared to do?  I don't have a list of the _most_
> > confusing doc strings, there are too many to chose from.  But starting
> > at the start of cl-macs.el, we have:

> > (i) cl--compiler-macro-list*; completely undocumented.
> > (ii) cl--simple-expr-p: Talks about "side effects", but not what they
> >   are side effects of.  Doesn't describe it's parameters or return
> >   value.  It's unclear what it is that "executes quickly".
> > (iii) cl--expr-contains: It's unclear what X and Y are, and what "refers
> >   to" means.
> > (iv) cl--expr-contains-any; completely undocumented.
> > (v) cl--expr-depends-p: It's unclear what X and Y are, though Y appears
> >   to be some sort of container of symbols.  It's unclear what sort of
> >   "dependency" the function handles, or what "may" means in the context.

> > There are many more.

> These are all internal functions and implementation details.  They're
> not necessary at all for users of cl-lib.el, only for its developers.

So, you're not prepared to rewrite their doc strings as you promised
yesterday, any more.

> What problem are you trying to solve by enhancing these docstrings?

Being able to debug Emacs problems.

> I thought the problem here was code that _used_ cl-lib.el, not hacking
> on cl-lib.el itself.

cl-lib uses itself, and needs debugging too.  Debugging code that uses
abstractions often requires penetrating these abstractions and checking
their innards.

Still, if you're not prepared to fix doc strings in internal functions,
here are five external functions with inadequate doc strings:

(i) cl-fill: The doc string misnames some of the parameters.  It is
unclear what "fill" means, and the forms of CL-SEQ and CL-ITEM aren't
specified.  Is each element of CL-SEQ "filled" with the entirety of
CL-ITEM, or what?  Finally, there is no documentation of CL-KEYS, beyond
stating that two particular keywords are "supported", whatever that
means.  What do :start and :end actually do?  The return value is not
specified.

(ii) cl-replace: Much the same problems as for cl-fill.

(iii) cl-remove: Same again.  Additionally, it is not specified what an
"occurrence" is; this presumably involves some equality test, and which
one is used is not specified.  Again the effect of the keywords is
unspecified.  There are keywords used in its code which aren't listed in
the doc string, namely :if and :if-not.

(iv) cl-remove-if: Much the same again.

(v) cl-remove-if-not: ... and again.

I found these simply by starting at the beginning of cl-seq and listing
pretty much each external function I found in a simple search.  There are
a lot more like these.

Still, even having just those five things properly documented would be
worthwhile.  Thanks for volunteering to do this!

> João

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 10, 2023 at 12:13 PM Alan Mackenzie <acm@muc.de> wrote:
>
> Hello, João.
>
> On Thu, Nov 09, 2023 at 13:41:48 +0000, João Távora wrote:
> > On Thu, Nov 9, 2023 at 1:36 PM Alan Mackenzie <acm@muc.de> wrote:
>
> > > Hello, João.
>
> > > On Thu, Nov 09, 2023 at 12:40:24 +0000, João Távora wrote:
> > > > On Thu, Nov 9, 2023 at 11:49 AM Dmitry Gutov <dmitry@gutov.dev> wrote:
>
> > > > > Improving cl-lib's documentation would be a welcome effort.
>
> > > > For sure, and not a hard one as well, as all those functions and
> > > > macros are pretty good, often flawless emulations of CL functions that
> > > > are impeccably documented in
>
> > > >   http://www.lispworks.com/documentation/HyperSpec/Front/
>
> > > > Which is of free access (though not of a compatible license, I
> > > > think).  But if people can point to the 5 most confusing functions
> > > > they think are poorly documented, I volunteer to rewrite the
> > > > docstrings for them.
>
> > > How much are you prepared to do?  I don't have a list of the _most_
> > > confusing doc strings, there are too many to chose from.  But starting
> > > at the start of cl-macs.el, we have:
>
> > > (i) cl--compiler-macro-list*; completely undocumented.
> > > (ii) cl--simple-expr-p: Talks about "side effects", but not what they
> > >   are side effects of.  Doesn't describe it's parameters or return
> > >   value.  It's unclear what it is that "executes quickly".
> > > (iii) cl--expr-contains: It's unclear what X and Y are, and what "refers
> > >   to" means.
> > > (iv) cl--expr-contains-any; completely undocumented.
> > > (v) cl--expr-depends-p: It's unclear what X and Y are, though Y appears
> > >   to be some sort of container of symbols.  It's unclear what sort of
> > >   "dependency" the function handles, or what "may" means in the context.
>
> > > There are many more.
>
> > These are all internal functions and implementation details.  They're
> > not necessary at all for users of cl-lib.el, only for its developers.
>
> So, you're not prepared to rewrite their doc strings as you promised
> yesterday, any more.

I thought the whole problem is that code using cl-lib.el is hard to use.
You said cl-labels was "obscure", so I was prepared to enhance its
docstring, yes.  That's a public macro.  Along with 4 other "obscure"
public ones.  Documentation cl-lib's implementation details is something
else entirely.  It doesn't hurt if there's suspicion that cl-lib.el is
misbehaving in function or efficiency.   But that's not the matter in
discussion here and it doesn't solve the presumed problem of not
understanding code that makes uses cl-lib.

> > What problem are you trying to solve by enhancing these docstrings?
> Being able to debug Emacs problems.

What problem specifically are you having trouble with?  What do you
ultimately want to achieve that you can't without docstrings for
cl-lib.el's implementation details.  This might be an XY problem
(https://en.wikipedia.org/wiki/XY_problem)

> cl-lib uses itself, and needs debugging too.  Debugging code that uses
> abstractions often requires penetrating these abstractions and checking
> their innards.

Well, no.  I don't go read OS source code whenever I have doubts
about how to use a system call or how to read a given piece of
code that uses that system call, I read the manual page for the system
call itself.  Just as I don't pull up a magnifying glass to check if
my CPU's transistors are doing what they should, I trust the data sheet
describing the instructions.  So, no, at least not desirably.  That's
the whole argument for good docstrings: that it relieves you from
having to penetrate innards  to understand a function's interface.
Feels a bit odd to be defending age-old ideas of encapsulation and
information-hiding here.

To summarize, I thought it would be absolutely clear that I volunteered
to write docstrings for public functions, public interfaces.  I'm
amazed it wasn't, but I apologize for the misunderstanding nonetheless.
Docstrings for things that the user sees, is confused by, finds
obscure, etc. Give me 5 of  those, and I'll rework their docstrings for
clarity.  I suggest cl-labels since you singled it out and some
sequence manipulating functions which just say: "Keywords supported:
:test :test-not :key :start :end :from-end" but don't explain what
each of those keyword arguments do.

Another useful thing would be to extend eldoc support in Elisp mode
to be able to highlight the currently active keyword argument.  I'll
also look into that, but no promises.

Thanks,
João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Fri, 10 Nov 2023 12:53:10 +0000
> Cc: Dmitry Gutov <dmitry@gutov.dev>,
>  Gerd Möllmann <gerd.moellmann@gmail.com>, 
>  Björn Bidar <bjorn.bidar@thaodan.de>, emacs-devel@gnu.org
> 
> > cl-lib uses itself, and needs debugging too.  Debugging code that uses
> > abstractions often requires penetrating these abstractions and checking
> > their innards.
> 
> Well, no.  I don't go read OS source code whenever I have doubts
> about how to use a system call or how to read a given piece of
> code that uses that system call, I read the manual page for the system
> call itself.

This misses the point which I think Alan wanted to make.  The point is
that when some cl-lib function seems to return an unexpected result,
one needs to debug that function.  And since Edebug has only
rudimentary and frequently completely inadequate support for debugging
cl-lib abstractions, one must then attempt at analyzing and
understanding those abstractions as part of debugging.

And if you never had such moments when some OS syscall produced an
unexpected result and the man pages failed to explain why, then I
guess you have yet a few TIL situations in your future ;-)

> Just as I don't pull up a magnifying glass to check if
> my CPU's transistors are doing what they should, I trust the data sheet
> describing the instructions.

Strawmen aside, this flies in the face of one of the great advantages
of working with Free Software: if you want and need to, you can find
out what happens on any level, including the lowest level of OS system
calls.  Trying to laugh it away is a non-starter around here.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Alan Mackenzie]

Hello, João.

On Fri, Nov 10, 2023 at 12:53:10 +0000, João Távora wrote:
> On Fri, Nov 10, 2023 at 12:13 PM Alan Mackenzie <acm@muc.de> wrote:

> > On Thu, Nov 09, 2023 at 13:41:48 +0000, João Távora wrote:
> > > On Thu, Nov 9, 2023 at 1:36 PM Alan Mackenzie <acm@muc.de> wrote:

> > > > On Thu, Nov 09, 2023 at 12:40:24 +0000, João Távora wrote:
> > > > > On Thu, Nov 9, 2023 at 11:49 AM Dmitry Gutov <dmitry@gutov.dev> wrote:

> > > > > > Improving cl-lib's documentation would be a welcome effort.

> > > > > For sure, and not a hard one as well, as all those functions and
> > > > > macros are pretty good, often flawless emulations of CL functions that
> > > > > are impeccably documented in

> > > > >   http://www.lispworks.com/documentation/HyperSpec/Front/

> > > > > Which is of free access (though not of a compatible license, I
> > > > > think).  But if people can point to the 5 most confusing functions
> > > > > they think are poorly documented, I volunteer to rewrite the
> > > > > docstrings for them.

> > > > How much are you prepared to do?  I don't have a list of the _most_
> > > > confusing doc strings, there are too many to chose from.  But starting
> > > > at the start of cl-macs.el, we have:

> > > > (i) cl--compiler-macro-list*; completely undocumented.
> > > > (ii) cl--simple-expr-p: Talks about "side effects", but not what they
> > > >   are side effects of.  Doesn't describe it's parameters or return
> > > >   value.  It's unclear what it is that "executes quickly".
> > > > (iii) cl--expr-contains: It's unclear what X and Y are, and what "refers
> > > >   to" means.
> > > > (iv) cl--expr-contains-any; completely undocumented.
> > > > (v) cl--expr-depends-p: It's unclear what X and Y are, though Y appears
> > > >   to be some sort of container of symbols.  It's unclear what sort of
> > > >   "dependency" the function handles, or what "may" means in the context.

> > > > There are many more.

> > > These are all internal functions and implementation details.  They're
> > > not necessary at all for users of cl-lib.el, only for its developers.

> > So, you're not prepared to rewrite their doc strings as you promised
> > yesterday, any more.

> I thought the whole problem is that code using cl-lib.el is hard to use.

More precisely, it's hard to understand and to debug, precisely because
it uses cl-* functions.

> You said cl-labels was "obscure", so I was prepared to enhance its
> docstring, yes.  That's a public macro.  Along with 4 other "obscure"
> public ones.

Ah, so your offer of fixing 5 doc strings is limited to obscure objects.
That wasn't clear from the outset, and wasn't what you said.

Documentation won't help cl-labels become any less obscure.  I've
debugged code using it, but I havn't any clue what it does.  There is
nothing about its functionality which sticks in memory.  Its name is
poor - "labels" isn't a verb, and what it does has nothing to do with
labels.

> Documentation cl-lib's implementation details is something else
> entirely.  It doesn't hurt if there's suspicion that cl-lib.el is
> misbehaving in function or efficiency.   But that's not the matter in
> discussion here and it doesn't solve the presumed problem of not
> understanding code that makes uses cl-lib.

It would solve part of the problem.

> > > What problem are you trying to solve by enhancing these docstrings?
> > Being able to debug Emacs problems.

> What problem specifically are you having trouble with?  What do you
> ultimately want to achieve that you can't without docstrings for
> cl-lib.el's implementation details.  This might be an XY problem
> (https://en.wikipedia.org/wiki/XY_problem)

I'm not having any specific problem at the moment.  The need for accurate
readable doc strings, particularly for code as difficult and obscure as
cl-*, should be self-evident.

I don't know what an "XY problem" is and can't be bothered to look up
your reference.  Thanks for not explaining clearly what you mean.

> > cl-lib uses itself, and needs debugging too.  Debugging code that uses
> > abstractions often requires penetrating these abstractions and checking
> > their innards.

> Well, no.  I don't go read OS source code whenever I have doubts
> about how to use a system call or how to read a given piece of
> code that uses that system call, I read the manual page for the system
> call itself.

One Emacs bug I solved involved hacking into the Linux kernel.  The
solution to the bug was to upgrade to a specific kernel version or later.

> Just as I don't pull up a magnifying glass to check if my CPU's
> transistors are doing what they should, I trust the data sheet
> describing the instructions.  So, no, at least not desirably.  That's
> the whole argument for good docstrings: that it relieves you from
> having to penetrate innards  to understand a function's interface.
> Feels a bit odd to be defending age-old ideas of encapsulation and
> information-hiding here.

Total strawman argument.

> To summarize, I thought it would be absolutely clear that I volunteered
> to write docstrings for public functions, public interfaces.

No, it wasn't clear.

> I'm amazed it wasn't, but I apologize for the misunderstanding
> nonetheless.  Docstrings for things that the user sees, is confused by,
> finds obscure, etc. Give me 5 of  those, and I'll rework their
> docstrings for clarity.

I did so in my last post, and you've just snipped them without comment.
That's twice I've given you five bad doc strings, and you aren't acting
on either set.  That took a significant amount of my time.

Thanks.

It seems to me you are going back on your undertaking to amend these five
doc strings.

> I suggest cl-labels since you singled it out and some sequence
> manipulating functions which just say: "Keywords supported: :test
> :test-not :key :start :end :from-end" but don't explain what each of
> those keyword arguments do.

That would certainly be useful, yes.

> Another useful thing would be to extend eldoc support in Elisp mode
> to be able to highlight the currently active keyword argument.  I'll
> also look into that, but no promises.

That is something else entirely, unrelated to the poor quality of the doc
strings that we're talking about.

> Thanks,
> João

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 10, 2023 at 2:25 PM Alan Mackenzie <acm@muc.de> wrote:

> I did so in my last post, and you've just snipped them without comment.
> That's twice I've given you five bad doc strings, and you aren't acting
> on either set.  That took a significant amount of my time.

Those are not public functions, Alan.  Even Dmitry pointed that out
immediately.

> It seems to me you are going back on your undertaking to amend these five
> doc strings.

Stop insinuating that.  I'm sorry you lost time looking at those
functions (by why???).  But it's not my fault you have no distinction of
public/private interface.  Or at least not one I share.  But I do, and
the Emacs documentation does.  The Emacs and Elisp manual doesn't
describe '--' functions, period. Even checkdoc.el has this concept.
Docstrings for internal functions don't hurt, but they're not the
essential parts to describe a library.  Surely not the first things
you should focus on, because internals are by definition prone to
one could re-implement all of cl-lib.el keeping the same interfaces
and their docstrings with wholly new internals.  I don't know why
am I'm having to clarify this.

Anyway, take care, Alan.
João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Alan Mackenzie]

Hello again, João.

On Fri, Nov 10, 2023 at 14:35:38 +0000, João Távora wrote:
> On Fri, Nov 10, 2023 at 2:25 PM Alan Mackenzie <acm@muc.de> wrote:

> > I did so in my last post, and you've just snipped them without comment.
> > That's twice I've given you five bad doc strings, and you aren't acting
> > on either set.  That took a significant amount of my time.

> Those are not public functions, Alan.  Even Dmitry pointed that out
> immediately.

The second set were public functions.  And as Eli pointed out,
non-public functions need documenting too.

> > It seems to me you are going back on your undertaking to amend these
> > five doc strings.

> Stop insinuating that.

You made an undertaking, the text of which was still in this thread
until you snipped it from your last post.

I now accept you won't be amending the doc strings I suggested, or
probably any others.  I think that's a shame, and I'm disappointed.

> I'm sorry you lost time looking at those functions (by why???).  But
> it's not my fault you have no distinction of public/private interface.

What sort of a smear is that meant to be?  It's just that I'm aware of
the need to document internal functions, you seem not to be.  As a
cl-lib enthusiast, you would have been best placed to fix this doc.

> Or at least not one I share.  But I do, and the Emacs documentation
> does.  The Emacs and Elisp manual doesn't describe '--' functions,
> period. Even checkdoc.el has this concept.  Docstrings for internal
> functions don't hurt, but they're not the essential parts to describe
> a library.  Surely not the first things you should focus on, because
> internals are by definition prone to one could re-implement all of
> cl-lib.el keeping the same interfaces and their docstrings with wholly
> new internals.  I don't know why am I'm having to clarify this.

I don't think there's much point continuing this bit of the conversation
any further.

> Anyway, take care, Alan.
> João

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 10, 2023 at 3:11 PM Alan Mackenzie <acm@muc.de> wrote:

> > Those are not public functions, Alan.  Even Dmitry pointed that out
> > immediately.
>
> The second set were public functions.  And as Eli pointed out,
> non-public functions need documenting too.

Oh then I missed that second set in all this jumble.  Can you please
post it again?

Thanks,
João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Alan Mackenzie]

On Fri, Nov 10, 2023 at 15:14:05 +0000, João Távora wrote:
> On Fri, Nov 10, 2023 at 3:11 PM Alan Mackenzie <acm@muc.de> wrote:

> > > Those are not public functions, Alan.  Even Dmitry pointed that out
> > > immediately.
> >
> > The second set were public functions.  And as Eli pointed out,
> > non-public functions need documenting too.

> Oh then I missed that second set in all this jumble.  Can you please
> post it again?

It's in my post in this thread posted at Date: Fri, 10 Nov 2023 12:13:25
+0000.

Or to be found at
https://lists.gnu.org/archive/html/emacs-devel/2023-11/msg00439.html.

> Thanks,
> João

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 10, 2023 at 3:47 PM Alan Mackenzie <acm@muc.de> wrote:
>
> On Fri, Nov 10, 2023 at 15:14:05 +0000, João Távora wrote:
> > On Fri, Nov 10, 2023 at 3:11 PM Alan Mackenzie <acm@muc.de> wrote:
>
> > > > Those are not public functions, Alan.  Even Dmitry pointed that out
> > > > immediately.
> > >
> > > The second set were public functions.  And as Eli pointed out,
> > > non-public functions need documenting too.
>
> > Oh then I missed that second set in all this jumble.  Can you please
> > post it again?
>
> It's in my post in this thread posted at Date: Fri, 10 Nov 2023 12:13:25
> +0000.
>
> Or to be found at
> https://lists.gnu.org/archive/html/emacs-devel/2023-11/msg00439.html.

Alright, here are two of the five functions documented in a patch.
Will look at the other 3 later.  By the way, if you have a web browser,
the so-called Common Lisp hyperspec is an excellent resource
to learn to work with CL functions.

Here is the doc for FILL and REPLACE:

  http://www.lispworks.com/documentation/HyperSpec/Body/f_fill.htm
  http://www.lispworks.com/documentation/HyperSpec/Body/f_replac.htm

Disclamer:  I'm open to tweaking these docstrings I wrote to
some reasonable degree, like grammar fixes, reordering sentences,
 etc.  Constructive criticism only. I will not indulge endless
hair-splitting, bikeshedding or deep personal considerations of
what proper docstrings should contain and look like, especially
if I suspect it is fueled by any deep disdain (or "generic aversion"
as someone put it) of the cl-lib.el library, such as has been patent
so far in some posts of this discussion so far.

IOW I agree with you that the current docstrings are subpar, so
this is why I volunteered to enhance them, but this is my honest
writing and my own considerations of what a good docstring should
look like, based on many years experience with Lisp and Emacs, and
how it may help a honest user of these functions put them to good
use or read code containing them.

And now the patch for 2 of the 5 functions you mentioned,
João

diff --git a/lisp/emacs-lisp/cl-seq.el b/lisp/emacs-lisp/cl-seq.el
index ec481121ae6..dd9f37b0a82 100644
--- a/lisp/emacs-lisp/cl-seq.el
+++ b/lisp/emacs-lisp/cl-seq.el
@@ -162,9 +162,19 @@ cl-reduce

 ;;;###autoload
 (defun cl-fill (cl-seq cl-item &rest cl-keys)
-  "Fill the elements of SEQ with ITEM.
-\nKeywords supported:  :start :end
-\n(fn SEQ ITEM [KEYWORD VALUE]...)"
+  "Replace elements of SEQ between START and END with ITEM.
+SEQ is a Lisp sequence.  It is destructively modified and
+returned.
+
+START and END are indexes like in `aref' or `elt'.  They
+designate the subsequence of SEQ to operate on.  If END is nil,
+process the sequence to the end. They default to 0 and nil,
+respectively, meaning process the whole SEQ.
+
+START and END are keyword arguments.  See info node `(cl) Program
+Structure > Argument Lists' for details.
+
+\n(fn SEQ ITEM &key START END...)"
   (cl--parsing-keywords ((:start 0) :end) ()
     (if (listp cl-seq)
  (let ((p (nthcdr cl-start cl-seq))
@@ -182,10 +192,30 @@ cl-fill

 ;;;###autoload
 (defun cl-replace (cl-seq1 cl-seq2 &rest cl-keys)
-  "Replace the elements of SEQ1 with the elements of SEQ2.
-SEQ1 is destructively modified, then returned.
-\nKeywords supported:  :start1 :end1 :start2 :end2
-\n(fn SEQ1 SEQ2 [KEYWORD VALUE]...)"
+  "Replace the elements of SEQ1 with elements of SEQ2.
+SEQ1 and SEQ2 are both Lisp sequences.  SEQ1 is destructively
+modified and returned.
+
+START1, END1, START2 and END2 can be indexes like in `aref' or
+`elt'.  Each pair designates two subsequences of SEQ1 and SEQ2,
+respectively, to operate on.  If END1 or END2 is nil, consider
+the respective sequences to the end.
+
+Consecutive elements of the subsequence of SEQ1 are replaced by
+consecutive elements of the subsequence of SEQ2.
+
+If the subsequences vary in length, the shorter one determines
+how many elements are replaced.  Extra elements in either
+subsequence are ignored.
+
+START1 and START2 default to 0, END1 and END2 default to nil,
+meaning replace as much of SEQ1 as possible with elements from
+SEQ2.
+
+START1, END1, START2 and END2 are keyword arguments.  See info
+node `(cl) Program Structure > Argument Lists' for details.
+
+\n(fn SEQ1 SEQ2 &key START1 END1 START2 END2...)"
   (cl--parsing-keywords ((:start1 0) :end1 (:start2 0) :end2) ()
     (if (and (eq cl-seq1 cl-seq2) (<= cl-start2 cl-start1))
  (or (= cl-start1 cl-start2)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Alan Mackenzie]

Hello, João.

On Fri, Nov 10, 2023 at 16:10:36 +0000, João Távora wrote:

[ .... ]

> Alright, here are two of the five functions documented in a patch.
> Will look at the other 3 later.

Thanks!  These two look like just the job!

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 10, 2023 at 8:24 PM Alan Mackenzie <acm@muc.de> wrote:
>
> Hello, João.
>
> On Fri, Nov 10, 2023 at 16:10:36 +0000, João Távora wrote:
>
> [ .... ]
>
> > Alright, here are two of the five functions documented in a patch.
> > Will look at the other 3 later.
>
> Thanks!  These two look like just the job!

Very happy you like them, will do the others ASAP, and
maybe even some more.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Michael Heerdegen]

Alan Mackenzie <acm@muc.de> writes:

> I don't think there's much point continuing this bit of the
> conversation any further.

There is much point.  Please, again, let's calm down a bit.  João is not
personally responsible for the things you criticize.  He will maybe not
be able to fix all of these points in a couple of days.  And he does not
have to prove that he can.  This doesn't mean it is not possible in a
longer term.

Alan, I totally understand that you are annoyed.  But impatience is not
helpful in this case, it will rather prevent that we get things done in
a good way.

Michael.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Alan Mackenzie]

Hello, Michael.

Thanks for the kind message!

On Sat, Nov 11, 2023 at 07:21:25 +0100, Michael Heerdegen wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > I don't think there's much point continuing this bit of the
> > conversation any further.

> There is much point.  Please, again, let's calm down a bit.  João is not
> personally responsible for the things you criticize.  He will maybe not
> be able to fix all of these points in a couple of days.  And he does not
> have to prove that he can.  This doesn't mean it is not possible in a
> longer term.

> Alan, I totally understand that you are annoyed.  But impatience is not
> helpful in this case, it will rather prevent that we get things done in
> a good way.

There was actually a specific misunderstanding between João and me,
which has now been resolved.  He has, so far, amended two doc strings
beautifully.

> Michael.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: What's missing in ELisp that makes people want to use cl-lib?

[Michael Heerdegen]

João Távora <joaotavora@gmail.com> writes:

> I suggest cl-labels since you singled it out and some
> sequence manipulating functions which just say: "Keywords supported:
> :test :test-not :key :start :end :from-end" but don't explain what
> each of those keyword arguments do.

I dunno if this would fit in our concept, but it would be nice if those
functions supporting keywords had a link (or an initially folded
subsection) where the keywords are explained in a quick way.  One needs
them too seldom to remember all the details until the next time.

Michael. 

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Sat, Nov 11, 2023 at 6:08 AM Michael Heerdegen
<michael_heerdegen@web.de> wrote:
>
> João Távora <joaotavora@gmail.com> writes:
>
> > I suggest cl-labels since you singled it out and some
> > sequence manipulating functions which just say: "Keywords supported:
> > :test :test-not :key :start :end :from-end" but don't explain what
> > each of those keyword arguments do.
>
> I dunno if this would fit in our concept, but it would be nice if those
> functions supporting keywords had a link (or an initially folded
> subsection) where the keywords are explained in a quick way.  One needs
> them too seldom to remember all the details until the next time.

I think these keyword arguments are just like normal elisp
optional arguments, they are just passed by name.  So the
same problems of writing/explaining them apply.

As to whether keyword arguments are intrinsically needed more
or less frequently than optional arguments is IMO, very
debatable, and entirely dependent on the function.  Again, just
like optional args. I don't think I've ever used read-from-minibuffer's
optional KEYMAP arg, for example.  Or INHERIT-INPUT-METHOD to
completing-read.  I use :KEY and :TEST quite often for cl-lib sequence
less frequently :FROM-END and even less frequently  :TEST-NOT.

Where I can agree with you is that is seems a bit suboptimal to
re-describe -- specifically for CL keyword arguments, _not_ keyword
arguments in general --  the semantics of the elements of the
standard set :KEY, :TEST, :FROM-END, etc.  I.e. there could
be some shortcuts to help facilitate this writing in cl-lib.el,
maybe some special docstring construct.  But copy-pasting it's
not that horrible either, and they can be summarily described
in single line with consistent style between the functions that
use them.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I think these keyword arguments are just like normal elisp
  > optional arguments, they are just passed by name.  So the
  > same problems of writing/explaining them apply.

  > As to whether keyword arguments are intrinsically needed more
  > or less frequently than optional arguments is IMO, very
  > debatable, and entirely dependent on the function.

I think both of these questions are minor issues that threaten to act
as red herrings.  Keyword arguments are better than optional
positional arguments in some circumstances, and the optional positional
arguments are better in some circumstances.

Keyword arguments are better -- sometimes much better -- when one
function accepts many different optional arguments.  There are
functions in Emacs that accept many positional optional arguments, and
some of them would be cleaner if they accepted keyword arguments.

Why don't they?  Because Emacs Lisp doesn't make it easy to accept
keyword arguments.  It is especially a pain in a primitive.  And many
of those functions were defined before Emacs Lisp had such a thing as
a keyword, let alone a keyword argument.

On the other hand, positional optional arguments are better when a
function accepts only one or two of them and they are used often.
They are easier to pass, and easier to remember,

Keyword arguments are easier to use when a function accepts many
different arguments pass.  But this can backfire when it encourages
people to give a function many keyword arguments, on the assumption
that "It doesn't cost us anything to give it more."

It may not cost much more at run time to decode those arguments,
but it costs a lot more to remember what they do, and to use them.

What Common Lisp did to sequence functions was the worst of
everything.  It gave a set of many keyword arguments to functions
which didn't have any, then changed the defaults for those functions
so that one usually needed to pass those keyword arguments,

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 09/11/2023 15:36, Alan Mackenzie wrote:
> (i) cl--compiler-macro-list*; completely undocumented.
> (ii) cl--simple-expr-p: Talks about "side effects", but not what they
>    are side effects of.  Doesn't describe it's parameters or return
>    value.  It's unclear what it is that "executes quickly".
> (iii) cl--expr-contains: It's unclear what X and Y are, and what "refers
>    to" means.
> (iv) cl--expr-contains-any; completely undocumented.
> (v) cl--expr-depends-p: It's unclear what X and Y are, though Y appears
>    to be some sort of container of symbols.  It's unclear what sort of
>    "dependency" the function handles, or what "may" means in the context.

These aren't public-facing functions, though.

Adding docs wouldn't hurt, but it's not like these could be removed due 
to the absence of them (that should depend on which public callers they 
have).

Re: What's missing in ELisp that makes people want to use cl-lib?

[Alan Mackenzie]

Hello, Dmitry.

On Thu, Nov 09, 2023 at 13:48:34 +0200, Dmitry Gutov wrote:

> On 09/11/2023 12:34, Alan Mackenzie wrote:
> > As a concrete plan, I would propose the following for discussion:

> > We should deprecate those functions/macros/variables in cl-lib that have
> > no doc string, or a substandard one.  This includes "internal" functions,
> > too.  Also to be deprecated are obscure functions/m/v (such as
> > cl-labels).

> I'm not sure we can remove cl-labels -- it's useful enough for the cases 
> where it does help, and there are callers inside and outside of Emacs.

> Whatever other removals, would probably need to be discussed case-by-case.

There are probably too many such functions/macros/variables to make that
practicable.  If we did do this, the discussion for each f/m/v would
likely balloon out into a thread as long as this one, and nothing much
would get done.

> > Having done this, we recode code currently using those deprecated f/m/v.

> > (Here a "substandard" doc string is contrasted with an adequate one,
> > which does all of the following:
> > (i) It says what the function/macro_does_, or what the variable_is_.
> > (ii) It describes the form and meaning of each parameter, and its
> >    relationship to (i).
> > (iii) If the return value is significant, it describes this.
> > (iv) It describes all effects on the global state, such as where it
> >    writes results to, and suchlike.)

> Improving cl-lib's documentation would be a welcome effort.

Yes, people have been saying this for years.  I wrote a chapter on
cl-print.el for cl.info a couple of months back.  Other than that nobody
seems prepared to do anything much on this front.

What is needed is a sustained effort from a cl-lib enthusiast to fix the
50 - 100 doc strings which are missing or substandard.  Well, João has
offered to start off with 5 doc strings, which is good.  Maybe something
will come of that.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Alan Mackenzie <acm@muc.de> writes:

> On Thu, Nov 09, 2023 at 08:24:14 +0100, Gerd Möllmann wrote:
>> Alan Mackenzie <acm@muc.de> writes:
>
>>>Just take one look at this "reality" you're so supportive of: the
>>>widespread use of cl-lib, not just in people's own projects, but
>>>throughout the core of Emacs, has multiplied the size of Lisp language
>>>part of Emacs by a factor of around 3.   This is a gross increase in
>>>complexity for maintainers that is not justified by the slight increase
>>>in facility that cl-lib (along with things like seq.el and oclosures)
>>>gives.
>
>>>Throughout this long discussion, this indiscriminate use of cl-lib has
>>>been supported only by occasional contributers.  Those who actually
>>>maintain other people's code, apart from (I think) Eli, Richard and
>>>myself, have been conspicuously silent.  None of us three have favoured
>>>such use of cl-lib.
>
>>>Occasional contributors may be fascinated by cl-lib, and learn enough
>>>of it to use random bits of it in their code.  The trouble is, each
>>>such contributor uses a different piece of cl-lib, with the result that
>>>those who end up maintaining it need to know a far greater part of it
>>>just to cope.
>
>>>This factor of 3 is, I believe, a significant barrier to new
>>>programmers coming into Emacs; Elisp is just that much more difficult
>>>than it was in the past.  And it isn't just for newcomers that it is
>>>more difficult.  I spend a significant amount of debugging time having
>>>to look up doc strings and manual pages for obscure cl-lib (etc.)
>>>functions.
>
>> > That is the current reality.
>
>> Maybe you could elaborate on what the plan then could look like?
>> Or is it not about a plan, but something else? 
>
> You stripped all the context.  I've put it back again.
>
> As a concrete plan, I would propose the following for discussion:

Thanks for that. I find it much easier to digest than this clean/unclean
thing.

And wow, that will not be popular :-).

>
> We should deprecate those functions/macros/variables in cl-lib that have
> no doc string, or a substandard one.  This includes "internal" functions,
> too.  Also to be deprecated are obscure functions/m/v (such as
> cl-labels).

Am I right in assuming that this is not about the documentation itself,
but just some selection criterium for reducing the size of cl-lib?

> Having done this, we recode code currently using those deprecated
> f/m/v.

What would recode mean? Using seq/map?

>
> (Here a "substandard" doc string is contrasted with an adequate one,
> which does all of the following:
> (i) It says what the function/macro _does_, or what the variable _is_.
> (ii) It describes the form and meaning of each parameter, and its
>   relationship to (i).
> (iii) If the return value is significant, it describes this.
> (iv) It describes all effects on the global state, such as where it
>   writes results to, and suchlike.)
>
> This would reduce the size of cl-lib to something more manageable than
> what we have at the moment.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Alan Mackenzie]

Hello, Gerd.

On Thu, Nov 09, 2023 at 13:19:28 +0100, Gerd Möllmann wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > On Thu, Nov 09, 2023 at 08:24:14 +0100, Gerd Möllmann wrote:
> >> Alan Mackenzie <acm@muc.de> writes:

> >>>Just take one look at this "reality" you're so supportive of: the
> >>>widespread use of cl-lib, not just in people's own projects, but
> >>>throughout the core of Emacs, has multiplied the size of Lisp language
> >>>part of Emacs by a factor of around 3.   This is a gross increase in
> >>>complexity for maintainers that is not justified by the slight increase
> >>>in facility that cl-lib (along with things like seq.el and oclosures)
> >>>gives.

> >>>Throughout this long discussion, this indiscriminate use of cl-lib has
> >>>been supported only by occasional contributers.  Those who actually
> >>>maintain other people's code, apart from (I think) Eli, Richard and
> >>>myself, have been conspicuously silent.  None of us three have favoured
> >>>such use of cl-lib.

> >>>Occasional contributors may be fascinated by cl-lib, and learn enough
> >>>of it to use random bits of it in their code.  The trouble is, each
> >>>such contributor uses a different piece of cl-lib, with the result that
> >>>those who end up maintaining it need to know a far greater part of it
> >>>just to cope.

> >>>This factor of 3 is, I believe, a significant barrier to new
> >>>programmers coming into Emacs; Elisp is just that much more difficult
> >>>than it was in the past.  And it isn't just for newcomers that it is
> >>>more difficult.  I spend a significant amount of debugging time having
> >>>to look up doc strings and manual pages for obscure cl-lib (etc.)
> >>>functions.

> >> > That is the current reality.

> >> Maybe you could elaborate on what the plan then could look like?
> >> Or is it not about a plan, but something else? 

> > You stripped all the context.  I've put it back again.

> > As a concrete plan, I would propose the following for discussion:

> Thanks for that. I find it much easier to digest than this clean/unclean
> thing.

> And wow, that will not be popular :-).

It will be a mixed popular/unpopular, with probably few people in the
middle.  Back when you were the Maintainer, I think there was a ban on
the use of cl.el and friends, except for at compilation time.  I don't
think that caused any problems.

> > We should deprecate those functions/macros/variables in cl-lib that have
> > no doc string, or a substandard one.  This includes "internal" functions,
> > too.  Also to be deprecated are obscure functions/m/v (such as
> > cl-labels).

> Am I right in assuming that this is not about the documentation itself,
> but just some selection criterium for reducing the size of cl-lib?

The most troublesome cl-lib functions are those without adequate
documentation.  They're the ones that waste unbounded amounts of time
when trying to debug something which uses them.  That's why I'd like to
do away with them.  There are few people willing to fix the doc strings
in cl-lib, though João has volunteered to make a start.

> > Having done this, we recode code currently using those deprecated
> > f/m/v.

> What would recode mean? Using seq/map?

I hadn't really thought of seq.el or map.el.  What do they do?  But it's
clear that working code can be written without these or cl-lib.

> > (Here a "substandard" doc string is contrasted with an adequate one,
> > which does all of the following:
> > (i) It says what the function/macro _does_, or what the variable _is_.
> > (ii) It describes the form and meaning of each parameter, and its
> >   relationship to (i).
> > (iii) If the return value is significant, it describes this.
> > (iv) It describes all effects on the global state, such as where it
> >   writes results to, and suchlike.)

> > This would reduce the size of cl-lib to something more manageable than
> > what we have at the moment.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Alan Mackenzie wrote:

> Just take one look at this "reality" you're so supportive
> of: the widespread use of cl-lib, not just in people's own
> projects, but throughout the core of Emacs, has multiplied
> the size of Lisp language part of Emacs by a factor of
> around 3.

So say I write a function like this, which has nothing to do
with cl-lib BTW.

(defun nand (&rest conditions)
  (when (member nil conditions)
    t) )

Some would say it is consistent, since we have `not' and
`and', so why not NAND as well?

Others would say it is cleaner in source than (not (and ...))
- especially if you find use of that recurring, since it is
a single function, not a combination of two.

Let's then say I put it in a package that makes its way into
Emacs, and some people start to use it since they also find it
consistent, clean, and useful.

What has then happened?

Have I introduced something that is clean, consistent,
and useful?

Or have I increased the size of the Lisp language by 1, making
it harder for the maintainers to do their work?

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Alan Mackenzie wrote:

> The most troublesome cl-lib functions are those without
> adequate documentation. They're the ones that waste
> unbounded amounts of time when trying to debug something
> which uses them. That's why I'd like to do away with them.

Why not instead provide them with docstrings, I think everyone
can agree they should have them?

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Bob Rogers]

   From: Emanuel Berg <incal@dataswamp.org>
   Date: Thu, 09 Nov 2023 16:13:29 +0100

   Alan Mackenzie wrote:

   > The most troublesome cl-lib functions are those without
   > adequate documentation. They're the ones that waste
   > unbounded amounts of time when trying to debug something
   > which uses them. That's why I'd like to do away with them.

   Why not instead provide them with docstrings, I think everyone
   can agree they should have them?

As a long-time CL/EL hacker, I will also undertake to document cl-lib
functions.  I think that is a much easier, and safer, alternative to
trying to recode cl-* usage.

   FWIW, I am also of the philosophy that simple functions should be
used where possible and cl-* functions where necessary, all in the name
of keeping my own code concise and readable.  But, alas, readability is
in the eye of the beholder.

					-- Bob Rogers
					   http://www.rgrjr.com/

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >    Why not instead provide them with docstrings, I think everyone
  >    can agree they should have them?

  > As a long-time CL/EL hacker, I will also undertake to document cl-lib
  > functions.  I think that is a much easier, and safer, alternative to
  > trying to recode cl-* usage.

Those functions certainly should have doc strings, and adding some is
a good thing to do.  But that will not cure the larger problem
caused by too much usage of those functions.  They were meant to be used
in a few obsure corners, not used so often that maintainers need to know
about them.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Bob Rogers]

   From: Richard Stallman <rms@gnu.org>
   Date: Fri, 10 Nov 2023 22:13:51 -0500

   Those functions certainly should have doc strings, and adding some is
   a good thing to do.

Naturally.

   But that will not cure the larger problem caused by too much usage of
   those functions.  They were meant to be used in a few obsure corners,
   not used so often that maintainers need to know about them.

   -- 
   Dr Richard Stallman (https://stallman.org)
   Chief GNUisance of the GNU Project (https://gnu.org)
   Founder, Free Software Foundation (https://fsf.org)
   Internet Hall-of-Famer (https://internethalloffame.org)

I know Emacs has many obscure corners.  But I didn't think that any were
beyond the reach of the maintainers.  ;-}

					-- Bob

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >    But that will not cure the larger problem caused by too much usage of
  >    those functions.  They were meant to be used in a few obsure corners,
  >    not used so often that maintainers need to know about them.

  > I know Emacs has many obscure corners.  But I didn't think that any were
  > beyond the reach of the maintainers.  ;-}

I have the impression that that is meant as a clever riposte to the
words you qupted, but I don't understand the point.  The words you qupted
are not clever wit.  They are meant to be serious and straightforward.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Bob Rogers]

   From: Richard Stallman <rms@gnu.org>
   Date: Sun, 12 Nov 2023 22:07:47 -0500

   [[[ To any NSA and FBI agents reading my email: please consider    ]]]
   [[[ whether defending the US Constitution against all enemies,     ]]]
   [[[ foreign or domestic, requires you to follow Snowden's example. ]]]

     >    But that will not cure the larger problem caused by too much usage of
     >    those functions.  They were meant to be used in a few obsure corners,
     >    not used so often that maintainers need to know about them.

     > I know Emacs has many obscure corners.  But I didn't think that any were
     > beyond the reach of the maintainers.  ;-}

   I have the impression that that is meant as a clever riposte to the
   words you qupted, but I don't understand the point.  The words you qupted
   are not clever wit.  They are meant to be serious and straightforward.

It seemed odd to me that anything could be beyond what the maintainers
would need to know and deal with.  But I guess that ignores the matter
of threshold.  So I withdraw my off-the-mark jibe.

					-- Bob

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > It seemed odd to me that anything could be beyond what the maintainers
  > would need to know and deal with.  But I guess that ignores the matter
  > of threshold.

You understand it now.

                   So I withdraw my off-the-mark jibe.

Thank you.  I appreciate that.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Alan Mackenzie <acm@muc.de> writes:

>> Thanks for that. I find it much easier to digest than this clean/unclean
>> thing.
>
>> And wow, that will not be popular :-).
>
> It will be a mixed popular/unpopular, with probably few people in the
> middle.  Back when you were the Maintainer, I think there was a ban on
> the use of cl.el and friends, except for at compilation time.  I don't
> think that caused any problems.

Not that I remember, but it's of course a long time ago.

But somehow, CL discussions and the clean/unclean thing from Richard
were always present. At least I seem to remember that.

>> > We should deprecate those functions/macros/variables in cl-lib that have
>> > no doc string, or a substandard one.  This includes "internal" functions,
>> > too.  Also to be deprecated are obscure functions/m/v (such as
>> > cl-labels).
>
>> Am I right in assuming that this is not about the documentation itself,
>> but just some selection criterium for reducing the size of cl-lib?
>
> The most troublesome cl-lib functions are those without adequate
> documentation.  They're the ones that waste unbounded amounts of time
> when trying to debug something which uses them.  That's why I'd like to
> do away with them.  There are few people willing to fix the doc strings
> in cl-lib, though João has volunteered to make a start.

Ok, thanks.

>> > Having done this, we recode code currently using those deprecated
>> > f/m/v.
>
>> What would recode mean? Using seq/map?
>
> I hadn't really thought of seq.el or map.el.  What do they do?

I would describe both as sets of generic functions for sequence handling
and mapping over sequences. I'm personally not using them, as I wrote here:

  https://lists.gnu.org/archive/html/emacs-devel/2023-11/msg00122.html

That spawned answers from Eli and a sub-thread from Richard, which I
personally found unconvincing.

But I'd recommend in any case that you build your own opinion.

> But it's
> clear that working code can be written without these or cl-lib.
>
>> > (Here a "substandard" doc string is contrasted with an adequate one,
>> > which does all of the following:
>> > (i) It says what the function/macro _does_, or what the variable _is_.
>> > (ii) It describes the form and meaning of each parameter, and its
>> >   relationship to (i).
>> > (iii) If the return value is significant, it describes this.
>> > (iv) It describes all effects on the global state, such as where it
>> >   writes results to, and suchlike.)
>
>> > This would reduce the size of cl-lib to something more manageable than
>> > what we have at the moment.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Philip Kaludercic wrote:

> Could you explain what you mean by "traditional" Emacs Lisp?

Traditional Elisp does not use concepts from ML and CL - it
merely implements them.

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: rms@gnu.org,  joaotavora@gmail.com,  adam@alphapapa.net,
>   emacs-devel@gnu.org,  stefankangas@gmail.com
> Date: Thu, 02 Nov 2023 21:26:00 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > AFAIU, it means that Emacs Lisp traditionally doesn't use keyword
> > arguments, except as relatively rare exceptions.
> 
> It is used rather prominently in `define-minor-mode' or
> `define-derived-mode', or do you specifically mean keyword arguments to
> functions?

I did mention exceptions, didn't I?

And define-derived-mode supports keyword arguments only since Emacs
22; the original implementation didn't.

> >> what constitutes "Emacs Lisp"?  It would
> >> seem peculiar if it were to be defined by the arbitrary decisions of the
> >> past, constrained by the contingent circumstances of the time.
> >
> > Those "arbitrary decisions" are what got us to where we are now, 40
> > years later.  So some respect for those "arbitrary decisions" is due,
> > I think.
> 
> No disrespect meant, but I am not sure we are thinking of the same
> things.  An "arbitrary decision" usually doesn't matter much, like
> calling a function rplacd or setcdr.  If a decision got us to where we
> are now, I would say it wasn't that arbitrary, but a good one?

Exactly my point.  So what did you mean by "It would seem peculiar if
[what constitutes Emacs Lisp] were to be defined by the arbitrary
decisions of the past"?

> > IMNSHO, extending Emacs Lisp as the language is not the main goal of
> > Emacs development.  Emacs Lisp is not a programming language on its
> > own, it is a language for implementing and extending features and
> > extensions in Emacs.  Thus, the main goal of Emacs development is to
> > develop applications and application-level features, and provide more
> > opportunities for extending the core where that is considered useful.
> > What we have in Emacs Lisp is IMO ample for that purpose.  Moreover,
> > most participants in Emacs development are not experts in programming
> > languages, their expertise is elsewhere (which is definitely a Good
> > Thing).
> 
> Of course not extending it for its own sake, but I would assume that
> making it easier for users to write practical and useful code should be
> something that is valued.

We should consider such additions carefully, weighing their advantages
against the disadvantages: introducing "alien" syntax, making the
language larger, etc.

> > Objectively, adding new syntax and semantics to Emacs Lisp does make
> > the source code harder to read and maintain, because it makes the
> > language larger and requires familiarization with those new language
> > features, which more often than not look and feel completely different
> > from the "traditional" Emacs Lisp.  So even if we conclude that these
> > additions are worthwhile, we should not pretend they come without a
> > price, and IMO we should think carefully whether their use is
> > justified before we use them in each and every case.
> 
> Could you explain what you mean by "traditional" Emacs Lisp?

Basically, the language as it is, without macros whose syntax is
different from Emacs Lisp.  For example, cl-loop has syntax that to my
eyes is starkly not Emacs Lisp, because it uses many keyword-like
parts that look like they were lifted from Fortran.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: rms@gnu.org,  joaotavora@gmail.com,  adam@alphapapa.net,
>>   emacs-devel@gnu.org,  stefankangas@gmail.com
>> Date: Thu, 02 Nov 2023 21:26:00 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > AFAIU, it means that Emacs Lisp traditionally doesn't use keyword
>> > arguments, except as relatively rare exceptions.
>> 
>> It is used rather prominently in `define-minor-mode' or
>> `define-derived-mode', or do you specifically mean keyword arguments to
>> functions?
>
> I did mention exceptions, didn't I?
>
> And define-derived-mode supports keyword arguments only since Emacs
> 22; the original implementation didn't.

Ok, I interpreted this as in "few usages", not in "few definitions".

>> >> what constitutes "Emacs Lisp"?  It would
>> >> seem peculiar if it were to be defined by the arbitrary decisions of the
>> >> past, constrained by the contingent circumstances of the time.
>> >
>> > Those "arbitrary decisions" are what got us to where we are now, 40
>> > years later.  So some respect for those "arbitrary decisions" is due,
>> > I think.
>> 
>> No disrespect meant, but I am not sure we are thinking of the same
>> things.  An "arbitrary decision" usually doesn't matter much, like
>> calling a function rplacd or setcdr.  If a decision got us to where we
>> are now, I would say it wasn't that arbitrary, but a good one?
>
> Exactly my point.  So what did you mean by "It would seem peculiar if
> [what constitutes Emacs Lisp] were to be defined by the arbitrary
> decisions of the past"?

That "setcar" is supposed to be more Elisp-ish than "rplacd", or things
like that. 

>> > IMNSHO, extending Emacs Lisp as the language is not the main goal of
>> > Emacs development.  Emacs Lisp is not a programming language on its
>> > own, it is a language for implementing and extending features and
>> > extensions in Emacs.  Thus, the main goal of Emacs development is to
>> > develop applications and application-level features, and provide more
>> > opportunities for extending the core where that is considered useful.
>> > What we have in Emacs Lisp is IMO ample for that purpose.  Moreover,
>> > most participants in Emacs development are not experts in programming
>> > languages, their expertise is elsewhere (which is definitely a Good
>> > Thing).
>> 
>> Of course not extending it for its own sake, but I would assume that
>> making it easier for users to write practical and useful code should be
>> something that is valued.
>
> We should consider such additions carefully, weighing their advantages
> against the disadvantages: introducing "alien" syntax, making the
> language larger, etc.

What I had in mind were extensions like the recent `with-restriction'.
Defining something like this came to my mine on more than one occasion
when combining `save-restriction' and `narrow-to-region' in exactly the
same pattern as I had done many times before, because it seems natural
to have it combined in a single form.  I don't think this is necessarily
alien, perhaps even natural in the long term.

>> > Objectively, adding new syntax and semantics to Emacs Lisp does make
>> > the source code harder to read and maintain, because it makes the
>> > language larger and requires familiarization with those new language
>> > features, which more often than not look and feel completely different
>> > from the "traditional" Emacs Lisp.  So even if we conclude that these
>> > additions are worthwhile, we should not pretend they come without a
>> > price, and IMO we should think carefully whether their use is
>> > justified before we use them in each and every case.
>> 
>> Could you explain what you mean by "traditional" Emacs Lisp?
>
> Basically, the language as it is, without macros whose syntax is
> different from Emacs Lisp.  For example, cl-loop has syntax that to my
> eyes is starkly not Emacs Lisp, because it uses many keyword-like
> parts that look like they were lifted from Fortran.

Then again (cl-)loop is a peculiar example; even among CL programmers
there are many that dislike using it on the same grounds.  As a macro,
it doesn't really even attempt to blend into the surrounding language,
but uses special keywords and syntax parsing, that has both its
advantages and disadvantages, but it is still used if only because CL
doesn't have a simple `while' macro.

This is a much more stark example than pushnew.  I could imagine that
pushnew could be implemented without a keyword argument, if we accepted
three arguments (newelt place compare-fn).

But there are a plenty of definitions in cl-lib that in some form or
another I think would be nice to have in Elisp proper as well: list*,
flet, defmacro/defgeneric, defstruct would be some examples that come to
mind.  This is not only my personal perspective, but the feeling I get
from reviewing Elisp packages that include cl-lib for only two or three
definitions.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: rms@gnu.org,  joaotavora@gmail.com,  adam@alphapapa.net,
>   emacs-devel@gnu.org,  stefankangas@gmail.com
> Date: Fri, 03 Nov 2023 07:48:21 +0000
> 
> >> >> what constitutes "Emacs Lisp"?  It would
> >> >> seem peculiar if it were to be defined by the arbitrary decisions of the
> >> >> past, constrained by the contingent circumstances of the time.
> >> >
> >> > Those "arbitrary decisions" are what got us to where we are now, 40
> >> > years later.  So some respect for those "arbitrary decisions" is due,
> >> > I think.
> >> 
> >> No disrespect meant, but I am not sure we are thinking of the same
> >> things.  An "arbitrary decision" usually doesn't matter much, like
> >> calling a function rplacd or setcdr.  If a decision got us to where we
> >> are now, I would say it wasn't that arbitrary, but a good one?
> >
> > Exactly my point.  So what did you mean by "It would seem peculiar if
> > [what constitutes Emacs Lisp] were to be defined by the arbitrary
> > decisions of the past"?
> 
> That "setcar" is supposed to be more Elisp-ish than "rplacd", or things
> like that. 

I didn't mean traditional names, I meant traditional syntax and
semantics of constructs.

> >> Of course not extending it for its own sake, but I would assume that
> >> making it easier for users to write practical and useful code should be
> >> something that is valued.
> >
> > We should consider such additions carefully, weighing their advantages
> > against the disadvantages: introducing "alien" syntax, making the
> > language larger, etc.
> 
> What I had in mind were extensions like the recent `with-restriction'.

I doubt that this is what bothers Richard (and me, to some extent).

> >> Could you explain what you mean by "traditional" Emacs Lisp?
> >
> > Basically, the language as it is, without macros whose syntax is
> > different from Emacs Lisp.  For example, cl-loop has syntax that to my
> > eyes is starkly not Emacs Lisp, because it uses many keyword-like
> > parts that look like they were lifted from Fortran.
> 
> Then again (cl-)loop is a peculiar example; even among CL programmers
> there are many that dislike using it on the same grounds.

Grepping our own sources for "(cl-loop" comes back with more than 760
hits.  That's more than one use for every 2 Lisp files in our tree.

And cl-loop is not the only example of additions whose syntax deviates
from Emacs Lisp traditions.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> >> Could you explain what you mean by "traditional" Emacs Lisp?
>> >
>> > Basically, the language as it is, without macros whose syntax is
>> > different from Emacs Lisp.  For example, cl-loop has syntax that to my
>> > eyes is starkly not Emacs Lisp, because it uses many keyword-like
>> > parts that look like they were lifted from Fortran.
>> 
>> Then again (cl-)loop is a peculiar example; even among CL programmers
>> there are many that dislike using it on the same grounds.
>
> Grepping our own sources for "(cl-loop" comes back with more than 760
> hits.  That's more than one use for every 2 Lisp files in our tree.
>
> And cl-loop is not the only example of additions whose syntax deviates
> from Emacs Lisp traditions.

It certainly has fans, though I believe this is more due to the utility
and flexibility than the syntax.  If there were a Elisp'ish alternative
(in the sense in which pcase is also an alternative to cl-case --
assuming that you see pcase as Elisp'ish?), then I honestly believe it
could be more popular than cl-loop.  I would be disappointed if being
Elisp'ish would mean that these kinds of language features per se
wouldn't be permissible.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > what constitutes "Emacs Lisp"?  It would
  > > seem peculiar if it were to be defined by the arbitrary decisions of the
  > > past, constrained by the contingent circumstances of the time.

  > Those "arbitrary decisions" are what got us to where we are now, 40
  > years later.  So some respect for those "arbitrary decisions" is due,
  > I think.

Not only that, but -- these decusions were not arbitrary in the first
place.  They were based on thought and embodied an idea of design.
That's why they add up to a coherent whole.

And yes, they add up to Emacs Lisp as it is -- "where we are now" is
the sum of them.

  > > 1. Not standardised; it is possible to extend the language without
  > >    having to worry about how many implementations will follow along

  > IMNSHO, extending Emacs Lisp as the language is not the main goal of
  > Emacs development.  Emacs Lisp is not a programming language on its
  > own, it is a language for implementing and extending features and
  > extensions in Emacs.

That is absolutely right!  But there is a second error in the point
(1) that you are responding to: the idea that extending a license is
good, that more complexity in the form of language constructs

Complexity of a language imposes a cost on all users of that language.
Sometimes the right choice is to refuse ti extend the language.

  > > Emacs Lisp can learn from interesting ideas that other
  > > languages provide, adapt and add them, making them available to
  > > everyone.

  > It certainly can.  The question is: should it?  Since we are not
  > driven by any standard, it is completely up to us to make those
  > decisions, and we should IMO make them judiciously and carefully,
  > taking the downsides into consideration.  In particular, I hope people
  > agree that making a language too large and complex is not a good
  > thing in the long run.

I think the same.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Petteri Hintsanen]

Eli Zaretskii <eliz@gnu.org> writes:

>> Emacs Lisp can learn from interesting ideas that other
>> languages provide, adapt and add them, making them available to
>> everyone.
>
> It certainly can.  The question is: should it?  Since we are not
> driven by any standard, it is completely up to us to make those
> decisions, and we should IMO make them judiciously and carefully,
> taking the downsides into consideration.  In particular, I hope people
> agree that making a language too large and complex is not a good
> thing in the long run.

I don't know anything about Common Lisp, and only fundamentals of Emacs
Lisp, but I think the above is really the crux of this discussion.

I believe that the complexity of our tools (including programming
languages) affects the complexity of our products.  Therefore, a kind of
methodological minimalism is valid, as it tends to lead to simpler
solutions, which are almost always preferable.

Additionally, since debugging is at least twice as hard as writing new
code--and I'd extrapolate that debugging code written by others is at
least four times as hard--any efforts to make debugging easier are
commendable.  Often this boils down to general principles: write
clearly, don't be too clever, do the same thing the same way, accumulate
idioms [1, 2].  Limiting the number of acceptable language features,
either by preference or deprecation, is another way towards this aim.

Would it be useful to document recommended and/or avoidable practices,
for example by extending Appendix D (Tips and Conventions) in Emacs Lisp
reference manual?  Or maybe a separate set of guidelines for code that
is, or is likely to be, included in Emacs itself?

For a recent example of a growing hodgepodge complexity, consider C++.
C++98 was already a Byzantine language.  C++11 made it significantly
more complex, and the recent editions are borderline crazy.  Even the
creator of the language seems to agree [3].  Nowadays I cannot recommend
learning C++ for any novice programmer in good faith.  It would be sad
to see Emacs Lisp take the same path.  (Even though I agree this is
comparing apples to oranges; C++ is truly a general programming language
in the widest sense, while Emacs Lisp is not.)

Thanks,
Petteri

[1] Kernighan and Plauger: The Elements of Programming Style.  (1978)
[2] A J Perlis: Epigrams in Programming.  (1982)
[3] B Stroustrup: Remember the Vasa!  (2018)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Petteri Hintsanen <petterih@iki.fi>
> Cc: Philip Kaludercic <philipk@posteo.net>,  rms@gnu.org,
>   joaotavora@gmail.com,  adam@alphapapa.net,  emacs-devel@gnu.org,
>   stefankangas@gmail.com
> Date: Sun, 12 Nov 2023 13:08:20 +0200
> 
> Would it be useful to document recommended and/or avoidable practices,
> for example by extending Appendix D (Tips and Conventions) in Emacs Lisp
> reference manual?

It would, but first we need to have a kind of consensus on what those
practices are.  Currently, as this discussion demonstrates, we don't
have such a consensus.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Petteri Hintsanen wrote:

> I believe that the complexity of our tools (including
> programming languages) affects the complexity of our
> products. Therefore, a kind of methodological minimalism is
> valid, as it tends to lead to simpler solutions, which are
> almost always preferable.

Minimalism is not a clear concept. If we imagine an ideal
hospital, the best in the world at everything, is this
"maximalist" because it does everything and has all the best
equipment, or is it "minimalist" because it ONLY has that and
nothing unnecessary?

Instead of thinking if one should have a lot or a little,
I think one should have the best stuff available, and have
that as well organized as possible. Because if it isn't good,
and not organized, having just a little of it won't
help, really.

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > IMNSHO, extending Emacs Lisp as the language is not the main goal of
  > Emacs development.  Emacs Lisp is not a programming language on its
  > own, it is a language for implementing and extending features and
  > extensions in Emacs.  Thus, the main goal of Emacs development is to
  > develop applications and application-level features, and provide more
  > opportunities for extending the core where that is considered useful.

I agree fully.

  > Objectively, adding new syntax and semantics to Emacs Lisp does make
  > the source code harder to read and maintain, because it makes the
  > language larger and requires familiarization with those new language
  > features, which more often than not look and feel completely different
  > from the "traditional" Emacs Lisp.  So even if we conclude that these
  > additions are worthwhile, we should not pretend they come without a
  > price, and IMO we should think carefully whether their use is
  > justified before we use them in each and every case.

I agree with that point too.

Extensions to Emacs Lisp can have very different effects
on the use of Emacs Lisp.  We should distinguish between

1. Narrow-purpose extensions, such as open-network-stream

2. General-purposes extensions, such as pcase.

The narrow-purpose exensions tend to be useful without increasing the
effective complexity of programming.  If you want to do what
open-network-stream does, you will find it very helpful.  Otherwise,
it probably won't be called from the code you work on, so its
existence won't affect you at all -- you will never need to know about
it.

General-purposes estensions are just the opposite.  They could show up
in any code whatsoever, so everyone will need to learn about all of
them.  Thus, they burden everyone.  But they are never _needed_ in any
code/


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

[ Richard, with this message, I am also replying to your message in the
other thread ]

On Wed, Nov 1, 2023 at 1:48 AM Richard Stallman <rms@gnu.org> wrote:

>   >   Anyway, the point is that to hack on long-lived files
>   > such as lisp/minibuffer.el, one can't really "ignore" the new
>   > dictionary of seq.el anymore.
>
> That's exactly the problem.

Then why this laser-focus on cl-lib.el?  Why not criticize the use of
seq.el and map.el and pcase.el all of which "add many functions, which
have many details"?  Why are these libraries seemingly exempt from this
discussion?

The above question is mostly rhetorical, I am _not_ criticizing these
libraries or advocating restricting seq.el and map.el and pcase.el.

Reading certain parts of Emacs core has become impossible without
getting well acquainted with Stefan's Monnier pcase.el which I had to do
some 7-8 years ago.  It was an entirely alien creature when I first
saw it.  It is directly inspired by pattern matching of the ML language.

When reading a pcase,  I occasionally think to myself that it's overkill
complexity, that "in the old days" it was all much simpler.  In many
other cases I've learned to appreciate, and now it's part of my Lisp
toolbox too.

I think it's important to allow the programmers and maintainers working
on specific sections of the code a certain freedom to recommend
or select certain abstractions.  But Emacs's Lisp code base is now
very extensive and any kind of core-wide ban or rewrites of certain
patterns is counter-productive and highly contentious.  Let's be
very careful when going down that path.

On Thu, Nov 2, 2023 at 2:28 AM Richard Stallman <rms@gnu.org> wrote:
>
> It might be ok to add some keyword arguments to `sort', which are more
> unusual and complex to use, but not to simple constructs like
> `pushnew'.  This is Emacs Lisp, not Common Lisp.

Adding keyword arguments to a given function doesn't transform
Emacs Lisp into Common Lisp in any meaningful way.  Just as
translating a work by Shakespeare into Portuguese doesn't transform
Portuguese into English.  It just allows Portuguese speakers
to enjoy some very decent, maybe even essential, poetry.

Has Emacs Lisp become ML through the use of pcase.el?  Of course not.
It's become easier to write many things in Emacs Lisp, if anything.

Frequently, the argument advanced for not wanting to import feature
A or B from this other language, is "Emacs Lisp is to be kept
simple and not have this complexity".  Fair enough, programmers' craft
is foremost about managing complexity.  So very often a decision to
simplify here and now breeds enormous complexity elsewhere later.
Simplicity that proved effective and beneficial 30 years ago, in
a different context, might not really be those things today, or
30 years from now.

For example, the early decision not to have namespaces in Emacs Lisp,
taken in the name of simplicity, may have facilitated development of
Emacs features in a time where there were reasonably few packages.
But as the ecosystem grew, it created -- and still creates  -- many
challenges.

For example, one of the challenges it creates is this discussion
right here:  to be able to easily delimit what that "core" or
"standard" Emacs Lisp is -- which is what Philip Addressed.  There
is in fact no such thing.

This is _not_ necessarily advocating for namespaces/packages in Emacs
Lisp, it's just my way of pointing out that the argument of "simplicity"
is much more nuanced.  Is a keyword-argument-less 'pushnew' simpler
than 'cl-pushnew'?  Yes it is, undoubtedly.  Will the former breed
less complexity than the latter?  Very unlikely IMO.

Instead we should look at how much more complex than the hypothetical
'eq'-comparing version of 'pushnew' is `cl-pushnew` in the end.  In
the argumentless version it's exactly the same.  When a keyword
argument _is_ needed (say we need to test with #'equal or
#'my-special-comp)  doesn't that increase in complexity pay off
three-fold almost immediately?  IOW versatility of a given utility
must obviously be taken in consideration.

By the way, if it hasn't yet become apparent to you or other readers,
the keywords usually accepted to cl-sort, cl-pushnew, etc all belong
to a fairly small dictionary with consistent meaning.  For example,
the ':key' accepted by 'cl-sort' tells the thing to consider when
comparing, and the ':key' accepted by  'cl-pushnew' also tells
the thing to consider when comparing.  It is not more useful to
'cl-sort' than it is to 'cl-pushnew'.

Keyword arguments are IMHO one of the most elegant mechanisms to
promote maintainability, and generally typing less code.
Of course, there might be other possible, unique and original ways of
achieving the same of course, just as pcase.el showed me there were
other ways of doing destructuring.   But choosing any one option
just for the sake of "not being like any other language" is not the
right path to take.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

João Távora wrote:

>>> Anyway, the point is that to hack on long-lived files such
>>> as lisp/minibuffer.el, one can't really "ignore" the new
>>> dictionary of seq.el anymore.
>>
>> That's exactly the problem.
>
> Then why this laser-focus on cl-lib.el? Why not criticize
> the use of seq.el and map.el and pcase.el all of which "add
> many functions, which have many details"? Why are these
> libraries seemingly exempt from this discussion?

Indeed, this has been mentioned several times now and the
answer we get is "There is no special treatment of cl-lib.el".
Hard to believe.

> The above question is mostly rhetorical, I am _not_
> criticizing these libraries or advocating restricting seq.el
> and map.el and pcase.el.
>
> Reading certain parts of Emacs core has become impossible
> without getting well acquainted with Stefan's Monnier
> pcase.el which I had to do some 7-8 years ago. It was an
> entirely alien creature when I first saw it. It is directly
> inspired by pattern matching of the ML language.
>
> When reading a pcase, I occasionally think to myself that
> it's overkill complexity, that "in the old days" it was all
> much simpler. In many other cases I've learned to
> appreciate, and now it's part of my Lisp toolbox too.
>
> I think it's important to allow the programmers and
> maintainers working on specific sections of the code
> a certain freedom to recommend or select certain
> abstractions. But Emacs's Lisp code base is now very
> extensive and any kind of core-wide ban or rewrites of
> certain patterns is counter-productive and highly
> contentious. Let's be very careful when going down
> that path.

Very much so.

> Has Emacs Lisp become ML through the use of pcase.el?
> Of course not. It's become easier to write many things in
> Emacs Lisp, if anything.

It has become the SME(cl), the Standard Meta Emacs
Common Lisp. Deal with it.

-- 
underground experts united
https://dataswamp.org/~incal

RE: [External] : Re: What's missing in ELisp that makes people want to use cl-lib?

[Drew Adams]

My 2c:

There's a difference between (1) adding &key to
Elisp generally and (2) letting more functions
take advantage of :key and :test, in particular.

(We have :test for `make-hash-table', but I
think that's about it.)

Needing multiple functions to allow for what
:key and :test provide isn't really "simpler".
Likewise for needing positional KEY and TEST
optional args.  :key and :test are convenient.

Heresy or not, I'd welcome more allowance of
them, regardless of any coincidence, or lack
of it, with existing Common Lisp functions.

It would be nice to be able to provide for
their use in user-defined functions, without
any need to load anything from `cl-*'.

Just one stodgy opinion.

Re: [External] : Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Thu, Nov 2, 2023 at 3:20 PM Drew Adams <drew.adams@oracle.com> wrote:

> There's a difference between (1) adding &key to
> Elisp generally and (2) letting more functions
> take advantage of :key and :test, in particular.

Good point.  Very good point.  It's just
that (1) happens to be a very good way to do (2).

But you're 100% right for sure it's not the only
way and (2) is ultimately what I'm after as well.

> (We have :test for `make-hash-table', but I
> think that's about it.)

We have keyword arguments for more functions,
make-process, json-serialize are two that
come to mind, pretty sure there are others.

> It would be nice to be able to provide for
> their use in user-defined functions, without
> any need to load anything from `cl-*'.

So why not introduce a 'definitelynotcl-defun'
macro with support for '&definitelynotkey' that
does that? ;-)

João

Re: [External] : Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

João Távora wrote:

> So why not introduce a 'definitelynotcl-defun' macro with
> support for '&definitelynotkey' that does that? ;-)

We don't want do stop anyone from introducing new stuff.
If they manage to get it into Emacs, it can't be that bad.
If they also get other people to use it, it must be
even better.

For the people who are maintaining it, I think to some extent
it would come with the job to handle code that isn't
necessarily written as the maintainer would like it to be.
Since it is originally written by another person, as
I understand the case to be in terms of this discussion.
A person more friendly to cl-lib?

But nothing prevents the maintainer from not using cl-lib
him/herself. And, to what extent can code that relies heavily
on cl-lib be left alone by the maintainer? As long as it
works? Do we have bugs that typically require big re-writes of
existing code?

If so, I agree the maintainer's job isn't easy.

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > There's a difference between (1) adding &key to
  > Elisp generally and (2) letting more functions
  > take advantage of :key and :test, in particular.

That is a valid distinction.

I would be happy to see a keyword argument facility provided for
function definitions in Emacs Lisp.  There are occasions where it is
good for a function to accept keyword arguments -- especially those
functions that need to accept lots of arguments.

What I object to are the keyword arguments used in Common Lisp to call
simple, everyday functions.  For instance, the sequence functions.
(For this reason, I prefer the seq- functions to the Common Lisp
sequence functions.)

But we can split that issue by making another distinction, between
(2a) sequence functions that _accept_ the Common Lisp sequence
function keywords, such as :key and :test, and
(2b) frequently needing to pass those arguments.

(2a) does not make most code any more clumsy.  It affects only the
_definitions_ of these sequence functions.  What makes code ugly is
(2b).

Logically, (2a) does not force (2b), but the two are directly related.
The designers of Common Lisp saw (2a) as providing an opportunity to
"simplify" by eliminating the predefined functions to compare using
`equal' and to compare using `eq'.  By default they use `eql',
which says "a pox on both `memq' and `member'."

In practice, that makes the Common Lisp functions inconvenient to use.

I would not be unhappy if Emacs Lisp functions started to support :key
and ;test where they make sense, provided they did this in an Emacs
Lisp way: default comparison is `equal'.

(We should take care to implement it without a slowdown in the case
where the keyword args are not specified.)

This means that `member' would accept ;test, but it would still
compare using `equal' by default (if :test was nil or not specified).
And there would still be `memq' to compare with `eq'.

When I say "new functions", that includes any Common Lisp functions
that are currently available in cl-seq.el with a cl- prefix but NOT
treated as part of Emacs Lisp.  To treat them as part of Emacs Lisp,
we should make them compatible, we should (1) make them default to
Emacs Lisp tradition and (2) accompany them with a corresponding q
function, if comparing using `eq' is useful for that function.

We would also rename the function, as part of the bargain, to avoid an
incompatibility.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

RE: [External] : Re: What's missing in ELisp that makes people want to use cl-lib?

[Drew Adams]

>   > There's a difference between (1) adding &key to
>   > Elisp generally and (2) letting more functions
>   > take advantage of :key and :test, in particular.
> 
> That is a valid distinction.
> 
> I would be happy to see a keyword argument facility provided for
> function definitions in Emacs Lisp.  There are occasions where it is
> good for a function to accept keyword arguments -- especially those
> functions that need to accept lots of arguments.
> 
> What I object to are the keyword arguments used in Common Lisp to call
> simple, everyday functions.  For instance, the sequence functions.
> (For this reason, I prefer the seq- functions to the Common Lisp
> sequence functions.)
> 
> But we can split that issue by making another distinction, between
> (2a) sequence functions that _accept_ the Common Lisp sequence
> function keywords, such as :key and :test, and
> (2b) frequently needing to pass those arguments.
> 
> (2a) does not make most code any more clumsy.  It affects only the
> _definitions_ of these sequence functions.  What makes code ugly is
> (2b).
> 
> Logically, (2a) does not force (2b), but the two are directly related.
> The designers of Common Lisp saw (2a) as providing an opportunity to
> "simplify" by eliminating the predefined functions to compare using
> `equal' and to compare using `eq'.  By default they use `eql',
> which says "a pox on both `memq' and `member'."
> 
> In practice, that makes the Common Lisp functions inconvenient to use.
> 
> I would not be unhappy if Emacs Lisp functions started to support :key
> and ;test where they make sense, provided they did this in an Emacs
> Lisp way: default comparison is `equal'.
> 
> (We should take care to implement it without a slowdown in the case
> where the keyword args are not specified.)
> 
> This means that `member' would accept ;test, but it would still
> compare using `equal' by default (if :test was nil or not specified).
> And there would still be `memq' to compare with `eq'.
> 
> When I say "new functions", that includes any Common Lisp functions
> that are currently available in cl-seq.el with a cl- prefix but NOT
> treated as part of Emacs Lisp.  To treat them as part of Emacs Lisp,
> we should make them compatible, we should (1) make them default to
> Emacs Lisp tradition and (2) accompany them with a corresponding q
> function, if comparing using `eq' is useful for that function.
> 
> We would also rename the function, as part of the bargain, to avoid an
> incompatibility.

FWIW, to me, all of what you say there is welcome news.

When incorporating whatever you describe as incorporating
from cl-* into Elisp (using `equal' as default), I hope
you mean giving it some other name than the CL name it
has (ignoring the `cl-' prefix or not).  I think that's
what you have in mind, when you say this:

  > We would also rename the function, as part of the
  > bargain, to avoid an incompatibility.

IOW, to me it would be fine to get such a function for
Elisp, but _not_ to have it keep the CL name but with
altered semantics (e.g. `equal' default, not `eql').

I've long argued that if we have a Common Lisp emulation
library then it should be a reasonable emulation, and
not gratuitously deviate from CL behavior/semantics.

It would not be good to keep the CL name for a function
but change its behavior from defaulting with `eql' to
defaulting with `equal'.  Better to give it a different
name.  At _least_ drop the `cl-' prefix, but better to
avoid any confusion by giving it an altogether different
name.

RE: [External] : Re: What's missing in ELisp that makes people want to use cl-lib?

[Bob Rogers]

   From: Drew Adams <drew.adams@oracle.com>
   Date: Tue, 7 Nov 2023 16:39:16 +0000

   >   > There's a difference between (1) adding &key to
   >   > Elisp generally and (2) letting more functions
   >   > take advantage of :key and :test, in particular.
   > 
   > That is a valid distinction.
   > 
   > I would be happy to see a keyword argument facility provided for
   > function definitions in Emacs Lisp.  There are occasions where it is
   > good for a function to accept keyword arguments -- especially those
   > functions that need to accept lots of arguments . . .

   FWIW, to me, all of what you say there is welcome news . . .

Likewise.  I have especially wished for a native &key; I think it will
greatly help Elisp code readability.

					-- Bob

Re: [External] : Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > When incorporating whatever you describe as incorporating
  > from cl-* into Elisp (using `equal' as default), I hope
  > you mean giving it some other name than the CL name it
  > has (ignoring the `cl-' prefix or not).

Yes, for sure.  (I thought I had made that explicit.)  The natural
name to use is the existing Emacs Lisp name -- for instance, `member'.

I'm not arguing for adding the :key and :test keyword args to every
function that could support them, or to any functions at all.  But IF
and WHEN we want to have functions supporting those keyword arguments
as parts of Emacs Lisp that programmers need to know about, this is
the clean way to do it.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [External] : Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Richard Stallman <rms@gnu.org> writes:

> We would also rename the function, as part of the bargain, to avoid an
> incompatibility.

I don't know what this bargain is about, but I somehow got the vibes of
a yet another new interface being on the horizon. Just saying.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > It might be more aesthetically or ideologically pleasing, but it would 
  > be much less useful.

If we add a function `pushnew' to Emacs Lisp, it will not have a :text
argument.  We are not going to start adding those arguments to
ordinary Emavs Lisp functions.

We could define `pushnew-equal' as well as `pushnew'.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > It might be more aesthetically or ideologically pleasing, but it would 
  > be much less useful.

If we add a function `pushnew' to Emacs Lisp, it will have a calling convention
like other Emacs Lisp functions -- not using CL-style keyword arguments.

We could define two functions, `pushnew' and `pushnq', if they are
both worth adding.  Or maybe the version that uses `equal' would DTRT
for most calls and we would only need that one.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Lisp files that load cl-lib in problematical ways

[João Távora]

On Sat, Oct 28, 2023 at 4:20 AM Richard Stallman <rms@gnu.org> wrote:

> I think you've put your finger on it.
>
> I see nothing wrong with having an optional extension which implements
> (more or less) Common Lisp in Emacs Lisp.  That's what I thought we
> had done.
>
> However, I object to incorporating (more or less) Common Lisp an essential part
> of Emacs Lisp.  And it looks like we've been drifting into that.
>
> There is no sharp line between the one and the other, and no simple
> test to determine which of those two our current situation actually
> is.  Rather, there is a spectrum that runs from "CL support is
> available but you can ignore it" to "the CL constructs are an
> essential and unavoidable part of Emacs Lisp."
>
> The specific practical questions I've asked are efforts to evaluate
> where we are now along that spectrum.  Of course, the answer to that
> isn't precise either.  But I was very surprised to learn how far
> Emacs has gone towards the latter end.

I'm going add my 2c here.  I think this whole discussion, which I haven't
followed very closely, might be slanted due to a misunderstanding.

Most, maybe all, programming languages have two different ways they can
evolve.  One is to add or modify the API offered by their standard library
by using the language itself, and another is to add or modify properties
of the fundamental mechanics of the language by changing any of the
runtime, interpreter or the compiler.

The latter can't be done using the language itself.  Now, Common Lisp
has a number of features that belong to the latter group that Emacs Lisp
doesn't have and there's no significant motion in that direction.
Examples are a namespacing system based on packages, a programmable
reader, a much more advanced condition handling system.  I'm sure
I'm forgetting some.

In fact the only way that Emacs Lisp, the core language, has approached
Common Lisp in recent years to any degree was through the addition of
lexical binding, and I think that most people here would agree that
was a very good thing.

Well, maybe native compilation also counts.  Then again, another very
good thing, I suppose.

cl-lib.el, like any other library, doesn't add -- because it really
can't -- any features of that calibre to Emacs Lisp.  It just adds
functions and macros much in the way that lots of other libraries
add functions and macros.  For example, the seq.el library is another
much later library (2014) that offers much of the same functionality of
cl-lib.el in wholly new abstractions that, say, the 2000's Emacs Lisp
coder won't be familiar with.  Another such library is pcase.
Does use of these libraries mutate Emacs Lisp programs to the point
that they are unrecognizable as Emacs Lisp programs?  Are the pcase
macros fundamentally easier to read than, say, cl-case macros?  Is
seq's sequence manipulating dictionary better than CL's?  Is
seq-some really better than cl-some? Isn't that entirely a question
of personal preference?

seq.el is preloaded and nobody seems to care (and I also don't, to
be honest).  Anyway, the point is that to hack on long-lived files
such as lisp/minibuffer.el, one can't really "ignore" the new
dictionary of seq.el anymore.

If one were to compare apples to apples one could argue that CL's
functions are _more_  accessible and quite well documented, since
both the thought put into the design of those interfaces and the use
they've seen by Lisp programmers of various breeds far exceed those
of seq.el and pcase.el.

Just my 2c,
João

RE: [External] : Re: Lisp files that load cl-lib in problematical ways

[Drew Adams]

> If one were to compare apples to apples one could
> argue that CL's functions are _more_  accessible
> and quite well documented,

Yes, if you mean their documentation in the
Common Lisp reference doc (esp. CLTL2, but even
the Hyperspec).

But no, not so well documented, if you mean the
Emacs doc for them.  (Just one opinion.)

> both the thought put into the design of those
> interfaces and the use they've seen by Lisp
> programmers of various breeds far exceed those
> of seq.el and pcase.el.

+1.  Far exceed.

Personally, I'd prefer that the Common Lisp
sequence functions just be added to Elisp, in
place of those in seq.el - keeping anything
truly Elisp-specific and found to be sorely
missing from C.L.  (Of course, this might mean
supporting their keyword args.)

I can't speak so definitively about `pcase.el'.
Some C.L. things could be used directly in place
of some of what `pcase' etc. do.  But patterned
binding, which pcase also does, is very useful.

(Patterned binding could usefully be separate
from any conditional behavior, but that's just
a personal opinion, and far down the river
that's passed under the bridge.)

By "just add to Elisp" I mean essentially (1)
get rid of the `cl-' prefix and (2) add the
construct, preferably "full-blown" but limited
in some ways if necessary, to Elisp.  Some
constructs could be preloaded, some autoloaded.
___

Every few months or so we hear input about
vastly multiplying the number of Emacs users
by changing this or that behavior or name to
something that potential newcomers might be
more familiar with.

There might be something to say, in terms of
leveraging existing familiarity with C.L. (and
perhaps in terms of reuse of some existing
C.L. code out there), for integrating more
C.L. stuff directly into Elisp.  The C.L.
"community" might not be on the order of, say,
the JavaScript or Python "community"..., but
still.  (And I'm pretty sure that many C.L.
programmers already use Emacs.)

Again, just one opinion.

(You might note that I'm both (1) for _not_
requiring `cl-lib.el' gratuitously in my
libraries and (2) integrating some more of
`cl-*.el' into Elisp.  For me, that's not a
contradiction.  If it seems to be one for
someone else, so be it.)

Re: [External] : Re: Lisp files that load cl-lib in problematical ways

[João Távora]

Drew Adams <drew.adams@oracle.com> writes:

> But no, not so well documented, if you mean the
> Emacs doc for them.  (Just one opinion.)

You're right, of course, but only because the Hyperspec and CLTL2 and
are top-class quality documentation.  The Emacs docs for this library
aren't up to that standard, but they're not terrible either.  Nothing
that can't be plugged, anyway.  Each function could use a link to the
Hyperspec maybe.

> I can't speak so definitively about `pcase.el'.
> Some C.L. things could be used directly in place
> of some of what `pcase' etc. do.  But patterned
> binding, which pcase also does, is very useful.

Sure, of course.  I use pcase sometimes, though cl-destructure-case is
also very useful.  The main advantage of pcase, as far as I udnerstand,
is that it is programmable, but I've not been able to get it.  So in
Eglot I rolled my own macros for fancier destructuring (see eglot.el's
eglot-dbind, for example)

I also agree many things could be brought into no-prefix Elisp
functions.

João

Re: [External] : Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

João Távora wrote:

> I also agree many things could be brought into no-prefix
> Elisp functions.

They can, but won't that just be a new name, and a new place?

Maybe some people will feel better about it that way but
I think to a lot of people it won't matter because they feel
that in terms of technology cl-lib is already Elisp and see no
harm using it whenever it is useful.

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: Lisp files that load cl-lib in problematical ways

[João Távora]

On Mon, Oct 30, 2023 at 2:44 PM Emanuel Berg <incal@dataswamp.org> wrote:
>
> João Távora wrote:
>
> > I also agree many things could be brought into no-prefix
> > Elisp functions.
>
> They can, but won't that just be a new name, and a new place?

Presumably the autoload embargo is lifted for those.

> Maybe some people will feel better about it that way but
> I think to a lot of people it won't matter because they feel
> that in terms of technology cl-lib is already Elisp and see no
> harm using it whenever it is useful.

As do I, but I can't use it in preloaded files.

João

Re: [External] : Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

João Távora wrote:

> As do I, but I can't use it in preloaded files.

Yes, so it will be a little win then.

But what will happen with the cl- versions, are we to keep
them? If not, maybe the people who wrote the library will
protest, and also, all the code that is currently cl- prefixed
will have to be changed.

Maybe reimplement them _and_ keep the cl- ones in their
current place?

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

>> As do I, but I can't use it in preloaded files.
>
> Yes, so it will be a little win then.
>
> But what will happen with the cl- versions, are we to keep
> them? If not, maybe the people who wrote the library will
> protest, and also, all the code that is currently cl-
> prefixed will have to be changed.
>
> Maybe reimplement them _and_ keep the cl- ones in their
> current place?

Here is the top 100 list.

A few hits don't make sense since the command is imperfect,
but other than that the list should be a pretty good
reflection of the situation.

Good luck with this :)

$ grep -E -h -o 'cl-[-a-z]*' **/*.el | sort | uniq -c | sort -r -b -n | head -n 100

   1416 cl-defmethod
    979 cl-loop
    720 cl-incf
    614 cl-lib
    486 cl-assert
    309 cl-letf
    303 cl-seq
    281 cl-defstruct
    257 cl-defgeneric
    227 cl-case
    218 cl-list
    209 cl-end
    189 cl-call-next-method
    172 cl-start
    171 cl-pushnew
    149 cl-defun
    143 cl-decf
    142 cl-p
    139 cl-keys
    135 cl-flet
    132 cl-return
    125 cl--loop-args
    114 cl--generic-
    106 cl-block
    104 cl-macs
    103 cl-destructuring-bind
     99 cl-check-type
     95 cl-some
     89 cl-
     89 cl-kwds
     88 cl-typep
     76 cl-prin
     72 cl-getf
     71 cl-extra
     69 cl-x
     69 cl-callf
     67 cl-pred
     67 cl--find-class
     66 cl-remove-if-not
     66 cl-defmacro
     65 cl-labels
     63 cl-second
     62 cl-return-from
     62 cl-mode
     62 cl-count
     61 cl-or-cast
     60 cl-set-difference
     53 cl-remove-if
     53 cl-generic
     51 cl-plusp
     51 cl-first
     51 cl-ecase
     50 cl-cvs
     49 cl-res
     49 cl-position
     48 cl-item
     47 cl-third
     47 cl-program
     47 cl-mapcan
     47 cl-every
     46 cl-tree
     45 cl-tempo-comma
     45 cl-substitute
     44 cl-current-ic
     43 cl-subseq
     42 cl-key
     40 cl-mapcar
     39 cl-tempo-right-paren
     39 cl-tempo-left-paren
     39 cl-rest
     39 cl-function
     39 cl-func
     39 cl-dolist
     38 cl-print-object
     38 cl-f
     37 cl-tempo-tags
     37 cl--slot-descriptor-name
     37 cl-macrolet
     37 cl-do
     36 cl-progv
     36 cl-find-if
     35 cl-embed-code
     34 cl-new
     34 cl-evenp
     33 cl-gensym
     33 cl-defsubst
     33 cl-check-register
     32 cl-print
     32 cl-pos
     32 cl-get-next-code
     31 cl-adjoin
     30 cl-remove-duplicates
     30 cl--loop-body
     30 cl-block-end-regexp
     30 cl-block-begin-regexp
     30 cl-basic-offset
     29 cl-remove
     29 cl-i
     28 cl-union
     28 cl-old-struct-compat-mode

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Here is the top 100 list.

  > A few hits don't make sense since the command is imperfect,
  > but other than that the list should be a pretty good
  > reflection of the situation.

Would someoe like to filter that list to remove everything not defined
in cl-lib.el itself?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [External] : Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Richard Stallman wrote:

>> Here is the top 100 list.
>>
>> A few hits don't make sense since the command is imperfect,
>> but other than that the list should be a pretty good
>> reflection of the situation.
>
> Would someoe like to filter that list to remove everything
> not defined in cl-lib.el itself?

One can do this in zsh by means of an extended glob pattern.

We see that `cl-pushnew' is on position #15, used 169 times.
So it is popular, but perhaps less so than expected?

`cl-loop' (#2, 979 times) and `cl-incf' (#3, 715 times) are as
expected very popular.

And there are more stuff that has been mentioned that appears
on the top 10 part of the list.

It is the list processing langauge :P

#! /bin/zsh
#
# this file:
#   https://dataswamp.org/~incal/conf/.zsh/emacs

setopt extendedglob

cle () {
    cd ~/src/emacs
    grep -E -h -o 'cl-[-a-z]+' **/*.el~lisp/emacs-lisp/cl-lib.el |
        sort          |
        uniq -c       |
        sort -r -b -n |
        head -n 100
}

$ cle
   1416 cl-defmethod
    979 cl-loop
    715 cl-incf
    611 cl-lib
    486 cl-assert
    309 cl-letf
    301 cl-seq
    279 cl-defstruct
    257 cl-defgeneric
    227 cl-case
    209 cl-end
    208 cl-list
    189 cl-call-next-method
    172 cl-start
    169 cl-pushnew
    149 cl-defun
    142 cl-p
    142 cl-decf
    135 cl-flet
    132 cl-return
    130 cl-keys
    125 cl--loop-args
    114 cl--generic-
    106 cl-block
    103 cl-destructuring-bind
    101 cl-macs
     99 cl-check-type
     95 cl-some
     89 cl-kwds
     88 cl-typep
     76 cl-prin
     72 cl-getf
     67 cl-pred
     67 cl--find-class
     66 cl-remove-if-not
     66 cl-defmacro
     66 cl-callf
     65 cl-labels
     65 cl-extra
     63 cl-x
     62 cl-second
     62 cl-return-from
     62 cl-mode
     62 cl-count
     61 cl-or-cast
     60 cl-set-difference
     53 cl-remove-if
     53 cl-generic
     51 cl-ecase
     50 cl-plusp
     50 cl-first
     50 cl-cvs
     49 cl-position
     47 cl-program
     47 cl-mapcan
     47 cl-every
     46 cl-third
     46 cl-res
     45 cl-tempo-comma
     45 cl-substitute
     44 cl-current-ic
     43 cl-subseq
     42 cl-key
     41 cl-item
     39 cl-tempo-right-paren
     39 cl-tempo-left-paren
     39 cl-function
     39 cl-dolist
     38 cl-print-object
     38 cl-mapcar
     37 cl-tempo-tags
     37 cl--slot-descriptor-name
     37 cl-macrolet
     37 cl-do
     36 cl-progv
     36 cl-find-if
     35 cl-f
     35 cl-embed-code
     34 cl-tree
     34 cl-func
     33 cl-gensym
     33 cl-evenp
     33 cl-defsubst
     33 cl-check-register
     32 cl-rest
     32 cl-print
     32 cl-pos
     32 cl-get-next-code
     30 cl-remove-duplicates
     30 cl--loop-body
     30 cl-block-end-regexp
     30 cl-block-begin-regexp
     30 cl-basic-offset
     29 cl-remove
     29 cl-i
     28 cl-union
     28 cl-member
     28 cl-adjoin
     27 cl-new
     27 cl-mismatch

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Wed, 01 Nov 2023 12:16:26 +0100
> 
> Richard Stallman wrote:
> 
> > Would someoe like to filter that list to remove everything
> > not defined in cl-lib.el itself?
> 
> One can do this in zsh by means of an extended glob pattern.
> 
> We see that `cl-pushnew' is on position #15, used 169 times.
> So it is popular, but perhaps less so than expected?
> 
> `cl-loop' (#2, 979 times) and `cl-incf' (#3, 715 times) are as
> expected very popular.
> 
> And there are more stuff that has been mentioned that appears
> on the top 10 part of the list.
> 
> It is the list processing langauge :P
> 
> #! /bin/zsh
> #
> # this file:
> #   https://dataswamp.org/~incal/conf/.zsh/emacs
> 
> setopt extendedglob
> 
> cle () {
>     cd ~/src/emacs
>     grep -E -h -o 'cl-[-a-z]+' **/*.el~lisp/emacs-lisp/cl-lib.el |
>         sort          |
>         uniq -c       |
>         sort -r -b -n |
>         head -n 100
> }
> 
> $ cle
>    1416 cl-defmethod
>     979 cl-loop
>     715 cl-incf
>     611 cl-lib
>     486 cl-assert
>     309 cl-letf
>     301 cl-seq
>     279 cl-defstruct
>     257 cl-defgeneric
>     227 cl-case
>     209 cl-end
>     208 cl-list
>     189 cl-call-next-method
>     172 cl-start
>     169 cl-pushnew
>     149 cl-defun
>     142 cl-p
>     142 cl-decf
>     135 cl-flet
>     132 cl-return
>     130 cl-keys
>     125 cl--loop-args
>     114 cl--generic-
>     106 cl-block
>     103 cl-destructuring-bind
>     101 cl-macs
>      99 cl-check-type
>      95 cl-some
>      89 cl-kwds
>      88 cl-typep
>      76 cl-prin
>      72 cl-getf
>      67 cl-pred
>      67 cl--find-class
>      66 cl-remove-if-not
>      66 cl-defmacro
>      66 cl-callf
>      65 cl-labels
>      65 cl-extra
>      63 cl-x
>      62 cl-second
>      62 cl-return-from
>      62 cl-mode
>      62 cl-count
>      61 cl-or-cast
>      60 cl-set-difference
>      53 cl-remove-if
>      53 cl-generic
>      51 cl-ecase
>      50 cl-plusp
>      50 cl-first
>      50 cl-cvs
>      49 cl-position
>      47 cl-program
>      47 cl-mapcan
>      47 cl-every
>      46 cl-third
>      46 cl-res
>      45 cl-tempo-comma
>      45 cl-substitute
>      44 cl-current-ic
>      43 cl-subseq
>      42 cl-key
>      41 cl-item
>      39 cl-tempo-right-paren
>      39 cl-tempo-left-paren
>      39 cl-function
>      39 cl-dolist
>      38 cl-print-object
>      38 cl-mapcar
>      37 cl-tempo-tags
>      37 cl--slot-descriptor-name
>      37 cl-macrolet
>      37 cl-do
>      36 cl-progv
>      36 cl-find-if
>      35 cl-f
>      35 cl-embed-code
>      34 cl-tree
>      34 cl-func
>      33 cl-gensym
>      33 cl-evenp
>      33 cl-defsubst
>      33 cl-check-register
>      32 cl-rest
>      32 cl-print
>      32 cl-pos
>      32 cl-get-next-code
>      30 cl-remove-duplicates
>      30 cl--loop-body
>      30 cl-block-end-regexp
>      30 cl-block-begin-regexp
>      30 cl-basic-offset
>      29 cl-remove
>      29 cl-i
>      28 cl-union
>      28 cl-member
>      28 cl-adjoin
>      27 cl-new
>      27 cl-mismatch

This list is not what Richard asked for, AFAIU.  Richard asked for a
list of functions defined in cl-lib.el, and only functions defined in
cl-lib.el.  The list above shows stuff that doesn't fit the request;
some examples of irrelevant symbols:

  cl-defmethod
  cl-defgeneric
  cl-loop
  cl-assert
  cl-defstruct
  cl-case
  cl-end
  cl-list

and many more.

I conclude that the above script is not a good way of producing a
useful list that Richard asked for.

Re: [External] : Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Eli Zaretskii wrote:

> This list is not what Richard asked for, AFAIU.
> Richard asked for a list of functions defined in cl-lib.el,

There are 20 `defun' in cl-lib.el, however one, "last*", is
commented out, so that leaves the following 19:

cl--defalias
cl--do-subst
cl--old-struct-type-of
cl--set-buffer-substring
cl--set-substring
cl-acons
cl-adjoin
cl-constantly
cl-copy-list
cl-digit-char-p
cl-evenp
cl-ldiff
cl-list*
cl-mapcar
cl-oddp
cl-pairlis
cl-proclaim
cl-subst
cl-values-list

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Wed, 01 Nov 2023 13:49:57 +0100
> 
> There are 20 `defun' in cl-lib.el, however one, "last*", is
> commented out, so that leaves the following 19:
> 
> cl--defalias
> cl--do-subst
> cl--old-struct-type-of
> cl--set-buffer-substring
> cl--set-substring
> cl-acons
> cl-adjoin
> cl-constantly
> cl-copy-list
> cl-digit-char-p
> cl-evenp
> cl-ldiff
> cl-list*
> cl-mapcar
> cl-oddp
> cl-pairlis
> cl-proclaim
> cl-subst
> cl-values-list

Thanks.  The "cl--*" ones should also be removed, I think (as they
are not supposed to be called from outsider cl-lib.el), but other than
that, I think this is the list.  And I cannot fail to point out that
the list is quite short, not nearly as long as the discussion seemed
to indicate.  Maybe we need to add to the list the functions in
cl-extra.el and cl-seq.el?

Re: [External] : Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Eli Zaretskii wrote:

> And I cannot fail to point out that the list is quite short,
> not nearly as long as the discussion seemed to indicate.
> Maybe we need to add to the list the functions in
> cl-extra.el and cl-seq.el?

`how-many' "defun" in cl-lib related files:

72 cl-macs.el
64 cl-seq.el
55 cl-extra.el
46 cl-generic.el
31 cl-indent.el
20 cl-lib.el
15 cl-loaddefs.el
14 cl-print.el
 9 cl-preloaded.el

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: Lisp files that load cl-lib in problematical ways

[Alan Mackenzie]

Hello, Eli.

On Wed, Nov 01, 2023 at 15:07:31 +0200, Eli Zaretskii wrote:
> > From: Emanuel Berg <incal@dataswamp.org>
> > Date: Wed, 01 Nov 2023 13:49:57 +0100

> > There are 20 `defun' in cl-lib.el, however one, "last*", is
> > commented out, so that leaves the following 19:

> > cl--defalias
> > cl--do-subst
> > cl--old-struct-type-of
> > cl--set-buffer-substring
> > cl--set-substring
> > cl-acons
> > cl-adjoin
> > cl-constantly
> > cl-copy-list
> > cl-digit-char-p
> > cl-evenp
> > cl-ldiff
> > cl-list*
> > cl-mapcar
> > cl-oddp
> > cl-pairlis
> > cl-proclaim
> > cl-subst
> > cl-values-list

> Thanks.  The "cl--*" ones should also be removed, I think (as they
> are not supposed to be called from outsider cl-lib.el), but other than
> that, I think this is the list.  And I cannot fail to point out that
> the list is quite short, not nearly as long as the discussion seemed
> to indicate.  Maybe we need to add to the list the functions in
> cl-extra.el and cl-seq.el?

I think so.  Richard's concern is the sheer bulk of cl-*, which adds
complexity to the language by virtue of the mass of detail which both
new and experienced Emacs hackers have to cope with.  That complexity
exists regardless of when those functions and macros get loaded.

So, from that point of view, cl-extra.el and cl-seq.el are certainly
relevant, as is cl-macs.el, and maybe one or two others.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [External] : Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> Date: Wed, 1 Nov 2023 14:09:15 +0000
> Cc: Emanuel Berg <incal@dataswamp.org>, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > Maybe we need to add to the list the functions in
> > cl-extra.el and cl-seq.el?
> 
> I think so.  Richard's concern is the sheer bulk of cl-*, which adds
> complexity to the language by virtue of the mass of detail which both
> new and experienced Emacs hackers have to cope with.  That complexity
> exists regardless of when those functions and macros get loaded.
> 
> So, from that point of view, cl-extra.el and cl-seq.el are certainly
> relevant, as is cl-macs.el, and maybe one or two others.

No, cl-macs is not relevant, since those are just macros, they leave
no trace after compiling.

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

You seem to be judging features in terms of how much fundamental
change it makes in Emacs Lisp.  What you say about these changes, at
that level, seems to be valid, but it's a different issue, thus a
change of subject.

The problem of widespread use of cl-lib is that it adds a lot of
details to what one needs to know to work on Elisp programs.

  > In fact the only way that Emacs Lisp, the core language, has approached
  > Common Lisp in recent years to any degree was through the addition of
  > lexical binding,

Lexical binding, as a language feature, adds little complexity of
details to Emacs Lisp.  Once you kow what it is, the only added detail
is how to enable it for a given source file, and that is very simple.

  > Well, maybe native compilation also counts.

That too adds little complexity, expecially if you don't turn it on
when you build Emacs.  I haven't had to learn anything to understand
Elisp programs because of the existence of native compilation.

  > cl-lib.el, like any other library, doesn't add -- because it really
  > can't -- any features of that calibre to Emacs Lisp.  It just adds
  > functions and macros much in the way that lots of other libraries
  > add functions and macros.

This is what maks cl-lib such a pain: it adds many functions, which
have many details.  When lots of Emacs Lisp programs use cl-lib, that
means all those details become more things everyone working on Emacs
Lisp programs needs to know.

In other words, you're judging various changes by how deeply they
change the Emacs Lisp language, but the problem I'm concerned with is
about how many superficial changes they make.

  >   Anyway, the point is that to hack on long-lived files
  > such as lisp/minibuffer.el, one can't really "ignore" the new
  > dictionary of seq.el anymore.

That's exactly the problem.

  > If one were to compare apples to apples one could argue that CL's
  > functions are _more_  accessible and quite well documented,

The documentation that counts, for Emacs Lisp, is documentation that
is suitable to integrate into Emacs.  It has to be technically
suitable (written in Texinfo) and legally suitable (under a free
license compatile with the GFDL).  If it isn't in Texinfo, integrating
it into Emacs would be a substantial job, If it isn't
license-compatible, we would have to rewrite it starting from zero.

If it isn't free, we cannot refer to it as documentation at all.

Is there clear and well-written documentation for cl-lib which fits
those criteria?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: rogers@rgrjr.com, incal@dataswamp.org, emacs-devel@gnu.org
> Date: Tue, 31 Oct 2023 21:48:57 -0400
> 
> Is there clear and well-written documentation for cl-lib which fits
> those criteria?

We have cl.texi.  Whether it's clear and well-written is a matter of
opinion.

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: acm@muc.de, emacs-devel@gnu.org
> Date: Tue, 24 Oct 2023 22:47:23 -0400
> 
>   > It would not have helped to define cl-pushnew on cl-macs, because
>   > cl-pushnew calls cl-adjoin, which is a function defined on cl-lib.  So
>   > basically cl-pushnew is a (rather thin) wrapper around cl-adjoin, and
>   > thus it is correctly defined on the same file where cl-adjoin is
>   > defined.
> 
> That is a good point -- for that particular solution I proposed.
> 
> But if we let ourselves be a little more flexible, how about this
> solution: install this
> 
>         (while (string-match "\\W" abbrev pos)
>           (or (memq (aref abbrev (match-beginning 0)) badchars)
>               (push (aref abbrev (match-beginning 0)) badchars))
>           (setq pos (1+ pos)))
> 
> instead of the current code:
> 
>         (while (string-match "\\W" abbrev pos)
>           (cl-pushnew (aref abbrev (match-beginning 0)) badchars)
>           (setq pos (1+ pos)))

If someone wants to work on such rewrites, I'm not sure I will object,
provided that the result is clean, tested, and is not horribly
complicated.  But personally, I see no need for investing any effort
in such rewrites, since there's nothing wrong with this code and
nothing wrong with loading cl-lib when its facilities are put to a
good use.

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

What's wrong with using cl-lib in many files, for example abbrev.el ad
debug.el, is that each one is a step towards making Emacs depend
pervasively on cl-lib.

With each step in that direction, it becomes gradually harder and
harder to work on Emacs at all without being familiar with the CL functions.
That general dependence is the problem I want to prevent.

So I've decided to start fixing some of those files not to use the
run-time CL features..

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: acm@muc.de, emacs-devel@gnu.org
> Date: Wed, 25 Oct 2023 22:27:32 -0400
> 
> What's wrong with using cl-lib in many files, for example abbrev.el ad
> debug.el, is that each one is a step towards making Emacs depend
> pervasively on cl-lib.

I don't understand how cl-lib here is different from other packages
that are used pervasively in Emacs.  We have quite a few of them that
aren't preloaded.

> With each step in that direction, it becomes gradually harder and
> harder to work on Emacs at all without being familiar with the CL functions.
> That general dependence is the problem I want to prevent.

This ship has sailed long ago.  cl-lib is used in many packages, both
those bundled with Emacs and those unbundled.  It is considered a
convenience library by many Lisp programmers who contribute to Emacs.
In this situation, rejecting code that uses cl-lib is
counter-productive and detrimental to attracting new contributors.

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > I investigated byte-run.el.  I don't think it uses any cl- facility.
  > > Rather, it declares a let-variable whose name starts with cl-.

  > byte-run.el indeed doesn't need cl-lib, and loading byte-run.elc
  > doesn't load cl-lib.  So I see no problem here, either.  IOW, this is
  > a false positive due to the naïve search pattern used to detect this.

That's right.  I suggest renaming the variable to eliminate the false positive.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [External] : Re: Shrinking the C core

[Arsen Arsenović]

[-- Attachment #1: Type: text/plain, Size: 774 bytes --]

Hi,

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > We have them in Elisp as well, `cl-remove-if' and
>   > `cl-remove-if-not', both in cl-seq.el.
>
> A partial emulation of some Common Lisp functions is present
> in the cl-lib library, for emulation purposes.  It is not supposed to
> be used a lot.

Apologies for my ignorance (and late reply), but why not?  It hosts some
quite useful functions.

For instance, I'm not aware of an ``flet'' equivalent in Elisp proper.

Thanks in advance, have a lovely day.
-- 
Arsen Arsenović

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 251 bytes --]

Re: [External] : Re: Shrinking the C core

[Alfred M. Szmidt]

   For instance, I'm not aware of an ``flet'' equivalent in Elisp proper.

There was a flet, but it had different semantics from cl-flet.  And
was thus removed / made obsolete.

Re: [External] : Re: Shrinking the C core

[Emanuel Berg]

Alfred M. Szmidt wrote:

>> For instance, I'm not aware of an ``flet'' equivalent in
>> Elisp proper.
>
> There was a flet, but it had different semantics from
> cl-flet. And was thus removed / made obsolete.

You used to be able to use "labels" for that. But it is now
removed, however there is `cl-labels'.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Shrinking the C core

[Arthur Miller]

>> Date: Wed, 9 Aug 2023 21:19:11 -0400
>> From: "Eric S. Raymond" <esr@thyrsus.com>
>> Cc: emacs-devel@gnu.org
>> 
>> Po Lu <luangruo@yahoo.com>:
>> > "Eric S. Raymond" <esr@thyrsus.com> writes:
>> > 
>> > > When I first worked on Emacs code in the 1980s Lisp was already fast
>> > > enough, and machine speeds have gone up by something like 10^3 since.
>> > > I plain don't believe the "slower" part can be an issue on modern
>> > > hardware, not even on tiny SBCs.
>> > 
>> > Can you promise the same, if your changes are not restricted to one or
>> > two functions in fileio.c, but instead pervade throughout C source?
>> 
>> Yes, in fact, I can. Because if by some miracle we were able to
>> instantly rewrite the entirety of Emacs in Python (which I'm not
>> advocating, I chose it because it's the slowest of the major modern
>> scripting languages) basic considerations of clocks per second would
>> predict it to run a *dead minimum* of two orders of magnitude faster
>> than the Emacs of, say, 1990.
>> 
>> And 1990 Emacs was already way fast enough for the human eye and
>> brain, which can't even register interface lag of less than 0.17
>> seconds (look up the story of Jef Raskin and how he exploited this
>> psychophysical fact in the design of the Canon Cat sometime; it's very
>> instructive). The human auditory system can perceive finer timeslices,
>> down to about 0.02s in skilled musicians, but we're not using elisp
>> for audio signal processing.
> 
> This kind of argument is inherently flawed: it's true that today's
> machines are much faster than those in, say, 1990, but Emacs nowadays
> demands much more horsepower from the CPU than it did back then.
> What's more, Emacs is still a single-threaded Lisp machine, although
> in the last 10 years CPU power develops more and more in the direction
> of multiple cores and execution units, with single execution units
> being basically as fast (or as slow) today as they were a decade ago.
> 
> And if these theoretical arguments don't convince you, then there are
> facts: the Emacs display engine, for example, was completely rewritten
> since the 1990s, and is significantly more expensive than the old one
> (because it lifts several of the gravest limitations of the old
> redisplay).  Similarly with some other core parts and internals.
> 
> We are trying to make Lisp programs faster all the time, precisely
> because users do complain about annoying delays and slowness.  Various
> optimizations in the byte-compiler and the whole native-compilation
> feature are parts of this effort, and are another evidence that the
> performance concerns are not illusory in Emacs.  And we are still not
> there yet: people still do complain from time to time, and not always
> because someone selected a sub-optimal algorithm where better ones
> exist.
> 
> The slowdown caused by moving one primitive to Lisp might be
> insignificant, but these slowdowns add up and eventually do show in
> user-experience reports.  Rewriting code in Lisp also increases the GC
> pressure, and GC cycles are known as one of the significant causes of
> slow performance in quite a few cases.  We are currently tracking the
> GC performance (see the emacs-gc-stats@gnu.org mailing list) for that
> reason, in the hope that we can modify GC and/or its thresholds to
> improve performance.
> 
>> If you take away nothing else from this conversation, at least get it
>> through your head that "more Lisp might make Emacs too slow" is a
>> deeply, *deeply* silly idea. It's 2023 and the only ways you can make
>> a user-facing program slow enough for response lag to be noticeable
>> are disk latency on spinning rust, network round-trips, or operations
>> with a superlinear big-O in critical paths.  Mere interpretive overhead
>> won't do it.
> 
> We found this conclusion to be false in practice, at least in Emacs
> practice.
> 
>> > Finally, you haven't addressed the remainder of the reasons I itemized.
>> 
>> They were too obvious, describing problems that competent software
>> engineers know how to prevent or hedge against, and you addressed me
>> as though I were a n00b that just fell off a cabbage truck. My
>> earliest contributions to Emacs were done so long ago that they
>> predated the systematic Changelog convention; have you heard the
>> expression "teaching your grandmother to suck eggs"?  My patience for
>> that sort of thing is limited.
> 
> Please be more patient, and please consider what others here say to be
> mostly in good-faith and based on non-trivial experience.  If
> something in what others here say sounds like an offense to your
> intelligence, it is most probably a misunderstanding: for most people
> here English is not their first language, so don't expect them to
> always be able to find the best words to express what they want to
> say.

Very interesting discussion going on for a long time.

I think you are all correct, and wrong to an extent, but I believe that
nobody has touched the fundamental issue: emacs design is flawed beyond
repair for todays machines. Not necessarily in pejorative meaning, but
to repair Emacs you would have to significantly rework internals, to the
point of entire design rewrite. Emacs is a child of its time (like
everything else). It was designed for the time of single-core slow
machine, and its design makes sense in that perspective. However, for
todays multicore machines, the fact that a lisp machine is slapped on
top of an existing text editor (Gosslings I guess), and everything is
shared via global state, can't be addressed in any other way but to
rewrite Emacs core from ground up. No amount of patch slapping onto the
current design can compensate for the lack of appropriate desing.

RMS has one abandoned TECO design to adapt Emacs to new times; perhaps
the time has come to do it again.

Sure, you can rework the C core completely, and implement better GC,
threads, and what not, but it seems signficant less work to re-implement
Emacs in a language that already has proper infrastructure in the place,
such as Common Lisp on ccl/sbcl, instead of duplicating all the work.

I understand machines of the time were to slow for a language as big as
Common Lisp when Emacs switched from TECO to C, but today the situation
is different. I am quite sure that sbcl would offer enough speed for a
text editor implementation. As you all have already noticed, speed is
limited by a human factor anyway.

We also see many projects that are duplicating Emacs idea, and with all
right; the ideas of Emacs are indeed sound and very good. But it is a
shame to abandon 40 years of development and the community that Emacs
has, something other projects can't compete with. Emacs Lisp is de
facto, *the* best documented Lisp out there, and Emacs is one of the
best documented open source projects available, kudos for that to
everyone involved. It would be a pittiful waste to just throw away all
that documentation and work done, as well as the vivid community that
Emacs has. In my personal opinon, those are *the* qualities of Emacs as
a project.

I personally believe that Emacs as a project would benefit of complete
core rewrite, as well as unifying the extension and the implementation
language for several reasons. Yes, it is a lot of work, but even
rewriting C core to adapt Emacs to modern machines is a lot of work in
itself, if it is even possible. You can either let it be, and slowly but
surely see Emacs usage decline, or you can rewrite Emacs core (again) to
better suit modern multicore machines.

Re: Shrinking the C core

[Po Lu]

Arthur Miller <arthur.miller@live.com> writes:

> Very interesting discussion going on for a long time.
>
> I think you are all correct, and wrong to an extent, but I believe that
> nobody has touched the fundamental issue: emacs design is flawed beyond
> repair for todays machines. Not necessarily in pejorative meaning, but
> to repair Emacs you would have to significantly rework internals, to the
> point of entire design rewrite. Emacs is a child of its time (like
> everything else). It was designed for the time of single-core slow
> machine, and its design makes sense in that perspective. However, for
> todays multicore machines, the fact that a lisp machine is slapped on
> top of an existing text editor (Gosslings I guess), and everything is
> shared via global state, can't be addressed in any other way but to
> rewrite Emacs core from ground up. No amount of patch slapping onto the
> current design can compensate for the lack of appropriate desing.

Unix was designed for 16-bit uniprocessor machines, where the only form
of ``interlocking'' was:

   int x = spltty ();
   splx (x);

But today, both free BSD Unix and proprietary Unix scale to SMPs with
hundereds of processors, exploiting intricate interlocking around
individual kernel data structures.  The perfect antithesis to your
standpoint...

Re: Shrinking the C core

[Arthur Miller]

Po Lu <luangruo@yahoo.com> writes:

> Arthur Miller <arthur.miller@live.com> writes:
>
>> Very interesting discussion going on for a long time.
>>
>> I think you are all correct, and wrong to an extent, but I believe that
>> nobody has touched the fundamental issue: emacs design is flawed beyond
>> repair for todays machines. Not necessarily in pejorative meaning, but
>> to repair Emacs you would have to significantly rework internals, to the
>> point of entire design rewrite. Emacs is a child of its time (like
>> everything else). It was designed for the time of single-core slow
>> machine, and its design makes sense in that perspective. However, for
>> todays multicore machines, the fact that a lisp machine is slapped on
>> top of an existing text editor (Gosslings I guess), and everything is
>> shared via global state, can't be addressed in any other way but to
>> rewrite Emacs core from ground up. No amount of patch slapping onto the
>> current design can compensate for the lack of appropriate desing.
>
> Unix was designed for 16-bit uniprocessor machines, where the only form
> of ``interlocking'' was:
>
>    int x = spltty ();
>    splx (x);
>
> But today, both free BSD Unix and proprietary Unix scale to SMPs with
> hundereds of processors, exploiting intricate interlocking around
> individual kernel data structures.  The perfect antithesis to your
> standpoint...

In which way is it "the perfect antithesis" to my standpoint, and what
is my standpoint?

If you wrote:

"where the only form of ``interlocking'' **is**",

than it would certainly be a "perfect anthithesis", but since you are
using the term *was* it means it *is no more*, a past tense, something
that has changed, and change is what I have suggested.

I think you are missunderstanding what I am saying: I am saying that
design needs to be changed; and the other thing is that I am suggesting
to rewrite the core API in CL instead of C, since you will get all those
things that constitute the Lisp Machine out for free; a better garbage
collector, a better threading, and better Lisp that are people are often
asking for. Of course you can implement all that stuff in C as well, it
just that it means redoing work that other people has done elsewhere.

Re: Shrinking the C core

[Po Lu]

Arthur Miller <arthur.miller@live.com> writes:

> In which way is it "the perfect antithesis" to my standpoint, and what
> is my standpoint?

That Emacs must be rewritten from the drawing board to leverage
multiprocessor operation.

> If you wrote:
>
> "where the only form of ``interlocking'' **is**",
>
> than it would certainly be a "perfect anthithesis", but since you are
> using the term *was* it means it *is no more*, a past tense, something
> that has changed, and change is what I have suggested.

Multiple organizations took Unix and independently transformed it into
what it is today: CMU, Novell, Sun, and of course the many volunteers
who orchestrate the development of the free BSD systems.

What fundamental issues (aside from manpower and time, of course) impede
Emacs from undergoing the same transformation?  The Unix experience
indicates that this is well-trodden ground.

> I think you are missunderstanding what I am saying: I am saying that
> design needs to be changed; and the other thing is that I am suggesting
> to rewrite the core API in CL instead of C, since you will get all those
> things that constitute the Lisp Machine out for free; a better garbage
> collector, a better threading, and better Lisp that are people are often
> asking for. Of course you can implement all that stuff in C as well, it
> just that it means redoing work that other people has done elsewhere.

Last I checked, none of the popular Common Lisp systems supported MS-DOS
or ran satisfactorily on Android, nor was Common Lisp an especially well
known language.  There is a reason Emacs is written in C, and why its
present design has endured for so long: it is portable, it is reasonably
fast, and it is a subject familiar to all programmers.

Other Lisp systems aren't panaceas either: rewritten in Common Lisp or
not, the present Emacs design will still require interlocking to operate
safely in a multiprocessor environment.  Ergo, such a rewrite will only
compound the effort needed to produce a hypothetical ``Lockmacs'' (so to
speak) with the additional pains required to rewrite Emacs in a new
language, and one very few of the present maintainers are well versed in
at that.

Re: Shrinking the C core

[Arthur Miller]

Po Lu <luangruo@yahoo.com> writes:

> Arthur Miller <arthur.miller@live.com> writes:
>
>> In which way is it "the perfect antithesis" to my standpoint, and what
>> is my standpoint?
>
> That Emacs must be rewritten from the drawing board to leverage
> multiprocessor operation.

I think you have missunderstand what I mean; I can try make it clear
that I mean parts that considers Emacs core, shared state and so on.

>> If you wrote:
>>
>> "where the only form of ``interlocking'' **is**",
>>
>> than it would certainly be a "perfect anthithesis", but since you are
>> using the term *was* it means it *is no more*, a past tense, something
>> that has changed, and change is what I have suggested.
>
> Multiple organizations took Unix and independently transformed it into
> what it is today: CMU, Novell, Sun, and of course the many volunteers
> who orchestrate the development of the free BSD systems.

You are missing the point: I wasn't talking about *who* changed the
Unix; the point here is that it got *changed*.

> What fundamental issues (aside from manpower and time, of course) impede
> Emacs from undergoing the same transformation?  The Unix experience
> indicates that this is well-trodden ground.

None. And nobody has suggested that such issues exists either, it is your
construction. On contrary, I am saying that Emacs should undergo such
transformation :).

>> I think you are missunderstanding what I am saying: I am saying that
>> design needs to be changed; and the other thing is that I am suggesting
>> to rewrite the core API in CL instead of C, since you will get all those
>> things that constitute the Lisp Machine out for free; a better garbage
>> collector, a better threading, and better Lisp that are people are often
>> asking for. Of course you can implement all that stuff in C as well, it
>> just that it means redoing work that other people has done elsewhere.
>
> Last I checked, none of the popular Common Lisp systems supported MS-DOS

True. MS-DOS and Windows 95 would have to go away. Perhaps some other
popular OS from 1980s-1990s? 

> or ran satisfactorily on Android, nor was Common Lisp an especially well
> known language.  There is a reason Emacs is written in C, and why its
> present design has endured for so long: it is portable, it is reasonably
> fast, and it is a subject familiar to all programmers.

Well yes, C is portable and fast, nobody denies that. So are some CL
compilers. 

> Other Lisp systems aren't panaceas either: rewritten in Common Lisp or

Nobody said it is "panaceas". But sbcl would provide a better lisp
machine, in terms of garbagecollector, threading etc; something you will
have to develop for Emacs eventually. You heard that one about a guy
with name Isaac Newton and standing on shoulders of others? :)

> not, the present Emacs design will still require interlocking to operate
> safely in a multiprocessor environment.  Ergo, such a rewrite will only
> compound the effort needed to produce a hypothetical ``Lockmacs'' (so to

Just reimplementing the C core would not, because you wouldn't use
multiple threads. But if you would like to thread Emacs command loop or
internals itself, than you would have to reimplement it that way of
course. Think of Java Swing; at least 20 years ago when I war
programming Java, it was not thread safe, but worker threads were OK.

> speak) with the additional pains required to rewrite Emacs in a new
> language, and one very few of the present maintainers are well versed in
> at that.

I am sure present maintainers are well versed with CL, just not very
interested to rewrite Emacs in CL; and I wouldn't expected them to be
either :-). 

Observe that I am presenting some food for thought; I don't expect Emacs
devs to jump and say "Yes! What a great idea!". But I wouldn't be
suprised if at least some of them have had the idea in their mind at
some point in time. 

Re: Shrinking the C core

[Po Lu]

Arthur Miller <arthur.miller@live.com> writes:

> I think you have missunderstand what I mean; I can try make it clear
> that I mean parts that considers Emacs core, shared state and so on.

That's what I was referring to, yes.

> You are missing the point: I wasn't talking about *who* changed the
> Unix; the point here is that it got *changed*.

My point is, Unix has demonstrated many times over that programs written
in C can be interlocked, and doing so is more productive than a rewrite.

> None. And nobody has suggested that such issues exists either, it is your
> construction. On contrary, I am saying that Emacs should undergo such
> transformation :).

That contradicts your assertions below, where you propose returning to
the drawing board to produce an Emacs written in Common Lisp.  Curiously
enough, I can't locate any precedent for large and complex Common Lisp
programs being modified for multi-processor operation.

> True. MS-DOS and Windows 95 would have to go away. Perhaps some other
> popular OS from 1980s-1990s? 

They wouldn't have to go away if Emacs remains written in C.

> Well yes, C is portable and fast, nobody denies that. So are some CL
> compilers. 

Which Common Lisp compiler is capable of linking with Java code through
the JNI?  (ABCL doesn't work, as the Android JVM can't interpret the
bytecode it generates.)

> Nobody said it is "panaceas". But sbcl would provide a better lisp
> machine, in terms of garbagecollector, threading etc; something you will
> have to develop for Emacs eventually. You heard that one about a guy
> with name Isaac Newton and standing on shoulders of others? :)

SBCL only supports kernel-based threads on a handful of systems, and its
garbage collection strategies vary between machine types.  I'm sure we
would be capable of supporting more while remaining portable, and Common
Lisp leaves much to be desired as a pair of shoulders to stand on.

> Just reimplementing the C core would not, because you wouldn't use
> multiple threads. But if you would like to thread Emacs command loop or
> internals itself, than you would have to reimplement it that way of
> course. Think of Java Swing; at least 20 years ago when I war
> programming Java, it was not thread safe, but worker threads were OK.

You're okay with Emacs crashing if threads, wielded incorrectly, trample
over internal state?  If so, the global lock could be lifted from Emacs
Lisp in under a week.

Instead of denigrating the C language, Emacs's C core, and so many other
components critical to Emacs that have been subject to unjustified
invective in recent times, why not direct your attention to a more
productive task, such as identifying the complex interdependencies
between different pieces of Emacs's global state to establish a detailed
plan for their interlocking?  For this task, the most important tool is
a control-flow visualizer such as cflow, and an ample supply of paper.

> I am sure present maintainers are well versed with CL, just not very
> interested to rewrite Emacs in CL; and I wouldn't expected them to be
> either :-). 

Two data points: I'm not an adept Common Lisp programmer, nor is Eli.
I'm not even an Emacs Lisp programmer by vocation.

Re: Shrinking the C core

[Arthur Miller]

Po Lu <luangruo@yahoo.com> writes:

> Arthur Miller <arthur.miller@live.com> writes:
>
>> I think you have missunderstand what I mean; I can try make it clear
>> that I mean parts that considers Emacs core, shared state and so on.
>
> That's what I was referring to, yes.
>
>> You are missing the point: I wasn't talking about *who* changed the
>> Unix; the point here is that it got *changed*.
>
> My point is, Unix has demonstrated many times over that programs written
> in C can be interlocked, and doing so is more productive than a rewrite.

Unix which? Unix is not a single program; it is a family of operating
systems, and I am quite sure nobody runs the same kernel they did back in
1970. I don't know what you are talking about you have
"demonestrated". You said yourself *was* and not *is* which suggest that
those kernels are rewritten in order to support those C programs that
can be interlocked. Again you are constructing yourself things here: I
haven't said it is impossible to write mulththreaded C application that
can use locking.

There are many applicaitons that have been redesigned. Emacs has been
redesigned since its first incarnations as TECO editor if I have
understand the history. For the very last time: I haven't
said Emacs *can not be redesigned*, on the contrary, I said Emacs
*should be* redesigned. Emacs core so to say. You are fighting against
things I haven't said; I don't know why, if it is lack of English skill,
or something else, but we are talking constantly beside each other.

> in C can be interlocked, and doing so is more productive than a rewrite.

Are you saying that a non-multihtreaded program written in C can
suddenly become multithreaded without being re-written to support
threading? Nevermind, you are totally, on purpose or lack of
understanding, missing the point: you have to rewrite stuff to make it
multithreaded regardless of the language if you intention is to have
Emacs core multithreaded. Supporting multithreading is not same as
having entire Emacs internals be multithreaded.

>> None. And nobody has suggested that such issues exists either, it is your
>> construction. On contrary, I am saying that Emacs should undergo such
>> transformation :).
>
> That contradicts your assertions below, where you propose returning to

No, it does not because you have missunderstood that I mean something else.

> the drawing board to produce an Emacs written in Common Lisp.  Curiously

I propose porting core API to Common Lisp because that would save you
implementing all those things that you need to make Emacs
multithreaded, better GC etc, because it is not a trivial task.

Of course you can re-write all that stuff in C too, nobody is denying
that it is possible. I am saying there is no need to reduplicate the
work, and there are other benefits of having Emacs in Common Lisp.

> enough, I can't locate any precedent for large and complex Common Lisp
> programs being modified for multi-processor operation.

Look harder.

>> True. MS-DOS and Windows 95 would have to go away. Perhaps some other
>> popular OS from 1980s-1990s? 
>
> They wouldn't have to go away if Emacs remains written in C.

Tough argument indeed.

It is certainly very importnat to keep the development back to
support two ancient non-free OS:s.

>> Well yes, C is portable and fast, nobody denies that. So are some CL
>> compilers. 
>
> Which Common Lisp compiler is capable of linking with Java code through
> the JNI?  (ABCL doesn't work, as the Android JVM can't interpret the
> bytecode it generates.)

I don't know what you are trying to say; that sound like very uninformed
statement based on missunderstanding. Calling Java via native API and
generating bytecode that runs Lisp on top of Java as ABCL does are two
completely different and separate things. I am quite sure Emacs does not
generate Java byte code. But I don't see how is it even relevant, I
haven't suggested to run Emacs on JVM. Perhaps it is possible who know,
but I have suggested to use sbcl which is a native compiler specialized
for compiling Lisp (and runs on Android as well).

>> Nobody said it is "panaceas". But sbcl would provide a better lisp
>> machine, in terms of garbagecollector, threading etc; something you will
>> have to develop for Emacs eventually. You heard that one about a guy
>> with name Isaac Newton and standing on shoulders of others? :)
>
> SBCL only supports kernel-based threads on a handful of systems, and its

"only" ? :-) I can asure you that Lispers have access to both kernel and
user space threads. Search on Bordeaux threads.

> garbage collection strategies vary between machine types.  I'm sure we

Does it?

> would be capable of supporting more while remaining portable, and Common
> Lisp leaves much to be desired as a pair of shoulders to stand on.

I am also sure everything is possible, it is software, as a wise old
grey head once said. It is just how much effort and resource you are
pouring into it. SBCL runs on basically all important platforms on which
Emacs runs, minus as mentioned DOS and Windows95; I don't know if there is
 some other. But it is not the point. The point is: there is another
community working on the same problems you have or wish to solve. And
they have come far away than you. You could re-use their work, and if
you miss somethign add those missing parts so everyone would benefit, or
you can live in an isolated island and do everything yourself.

>> Just reimplementing the C core would not, because you wouldn't use
>> multiple threads. But if you would like to thread Emacs command loop or
>> internals itself, than you would have to reimplement it that way of
>> course. Think of Java Swing; at least 20 years ago when I war
>> programming Java, it was not thread safe, but worker threads were OK.
>
> You're okay with Emacs crashing if threads, wielded incorrectly, trample
> over internal state?  If so, the global lock could be lifted from Emacs
> Lisp in under a week.

I am telling you that sbcl/ccl has already solved those problems you
have to solve in order to have multithreading without having to lift
your lock.

> Instead of denigrating the C language, Emacs's C core, and so many other
> components critical to Emacs that have been subject to unjustified
> invective in recent times, why not direct your attention to a more

I don't think I understand how I "denigrate the C language" and what
"unjustify invective" in this case is, but I don't think it
matters. Once you also said I should be forbidden to talk about Emacs.

>                                    direct your attention to a more
> productive task, such as identifying the complex interdependencies
> between different pieces of Emacs's global state to establish a detailed
> plan for their interlocking?  For this task, the most important tool is
> a control-flow visualizer such as cflow, and an ample supply of paper.

Can we save nature and our selves?

>> I am sure present maintainers are well versed with CL, just not very
>> interested to rewrite Emacs in CL; and I wouldn't expected them to be
>> either :-). 
>
> Two data points: I'm not an adept Common Lisp programmer, nor is Eli.
> I'm not even an Emacs Lisp programmer by vocation.

Nor are you sole Emacs devs, aren't you? I don't think calling names is
appropriate, so I want, but I can come up with at least four people who
work on Emacs and who are experienced Lispers. How active they are in
development recently I don't know, I don't follow the list every day as
I am used to, but I certainly haven't targeted Eli nor you with that one.

Re: Shrinking the C core

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Would you please stop arguing for rewriting Emacs in Common Lisp?  It
is a non-starter.

It would be an enormouse job -- including rewriting the Emacs Lisp
Referance Manual.  We can't be sure how much the benefit would be,
because Common Lisp has some drawbacks too.  But we know the cost is
prohibitive.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Shrinking the C core

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Would you please stop arguing for rewriting Emacs in Common Lisp?  It
  > is a non-starter.

  > It would be an enormouse job -- including rewriting the Emacs Lisp
  > Referance Manual.

Also, there are aspects of Common Lisp which i rejected as clumsy and
best avoided.  All those keyword arguments!  I intentionally excluded
tham from Emacs and I am not going to let them in.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Shrinking the C core

[Arthur Miller]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > Would you please stop arguing for rewriting Emacs in Common Lisp?  It
>   > is a non-starter.
>
>   > It would be an enormouse job -- including rewriting the Emacs Lisp
>   > Referance Manual.
>
> Also, there are aspects of Common Lisp which i rejected as clumsy and

With all respect to you, but sound to me like an emotional argument, not
a rational one.

I don't know why you reject them as clumsy, but why does it matter to us
if a tool includes features we don't use? I don't know if I understand
it correctly, but CL was made so other Lisps can be abstracted on top of
it; so we use those features to abstract stuff on top of those lengthy
verbose functions? In other words we can hide those keyword params
behind elispy abstractions if we don't like them?

> best avoided.

Why are they best avoided? Is there some technical reason or is it
psychological?

> best avoided.  All those keyword arguments!  I intentionally excluded
> tham from Emacs and I am not going to let them in.

I don't want to be impolite, but they are already in; via cl-lib.el.

Re: Shrinking the C core

[Alan Mackenzie]

Hello, Arthur.

On Wed, Sep 06, 2023 at 07:04:43 +0200, Arthur Miller wrote:
> Richard Stallman <rms@gnu.org> writes:

> > [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> > [[[ whether defending the US Constitution against all enemies,     ]]]
> > [[[ foreign or domestic, requires you to follow Snowden's example. ]]]

> >   > Would you please stop arguing for rewriting Emacs in Common Lisp?  It
> >   > is a non-starter.

> >   > It would be an enormouse job -- including rewriting the Emacs Lisp
> >   > Referance Manual.

> > Also, there are aspects of Common Lisp which i rejected as clumsy and

> With all respect to you, but sound to me like an emotional argument, not
> a rational one.

It's a rational argument expressed in emotional terms for simplicity.

> I don't know why you reject them as clumsy, but why does it matter to us
> if a tool includes features we don't use?

It matters a very great deal.  In practice you cannot avoid "using" these
features if you have to understand or debug somebody else's code.

> I don't know if I understand it correctly, but CL was made so other
> Lisps can be abstracted on top of it; so we use those features to
> abstract stuff on top of those lengthy verbose functions? In other
> words we can hide those keyword params behind elispy abstractions if we
> don't like them?

That's complexity and bloat.  Far better not to have them in the first
place.

> > best avoided.

> Why are they best avoided? Is there some technical reason or is it
> psychological?

It's because they are not needed.  Bear in mind, Common Lisp is a massive
language, much like C++ or PL/1 or Algol-68.  With such a language,
nobody uses all of it (it's just too big), but everybody has her own
personal subset.  This creates difficulty when somebody else has to
understand that first hacker's code.

CL is a niche language; it has not captured the hacker mindset.  I think
it is just too big and too unwieldy.

> > best avoided.  All those keyword arguments!  I intentionally excluded
> > tham from Emacs and I am not going to let them in.

> I don't want to be impolite, but they are already in; via cl-lib.el.

Yes.  There was a time not so long ago when cl.el was banned from use in
our Lisp code, except for at compile time.  Our Emacs Lisp was small,
simple to understand, and easy to learn.  Now things in cl-lib.el get
used as if they are just a normal part of Emacs Lisp.  Our language is
thus MUCH more difficult to understand, perhaps by a factor of somewhere
between 3 and 10.  When perusing even established parts of Emacs I groan
inwardly every time I encounter one of these needless cl-lib features.
It stops me dead, forcing me to consult doc strings (which are often
missing and often inadequate even when they are present) or even manuals.

Were we to rewrite Emacs in Common Lisp, these things would get worse
fairly quickly.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Shrinking the C core

[Arthur Miller]

Alan Mackenzie <acm@muc.de> writes:

> Hello, Arthur.
>
> On Wed, Sep 06, 2023 at 07:04:43 +0200, Arthur Miller wrote:
>> Richard Stallman <rms@gnu.org> writes:
>
>> > [[[ To any NSA and FBI agents reading my email: please consider    ]]]
>> > [[[ whether defending the US Constitution against all enemies,     ]]]
>> > [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>> >   > Would you please stop arguing for rewriting Emacs in Common Lisp?  It
>> >   > is a non-starter.
>
>> >   > It would be an enormouse job -- including rewriting the Emacs Lisp
>> >   > Referance Manual.
>
>> > Also, there are aspects of Common Lisp which i rejected as clumsy and
>
>> With all respect to you, but sound to me like an emotional argument, not
>> a rational one.
>
> It's a rational argument expressed in emotional terms for simplicity.

What you basicaly say: it somehow is a summarization of some more
advanced/complicated reasons which can't be explained, but have to be
expressed in the term "I don't like it" for the simplicity?

I don't think it is a rational argument, but I explicitly didn't want to
speculate why RMS feels so, but asked him to clarify for pure respect
for him as a person. You can try to teach me why is it a rational
argument and why it has to be expressed in emotional terms for
simplicity, and I promise I'll try to understand it as best as I can,
but I have hard time to see such an explanation, because the argument is
truly just an opinion.

CL for sure has things that in retrospect can be debated about, but I
don't think keyword arguments are one of those.

Also note that keyword arguments, or at least similar principle is used
in other parts of Emacs, not just cl-lib; for example look at
define-minor-mode. While define-minor-mode macro does not use "&key" as
an explicit keyword, the arguments to it are a plist which basicaly
gives you the very same effect. The "new" defvar-keymap is another
example.

Keyword arguments are undeniably useful because they let us omit
arguments we are not interested in, and type only the ones we are
interested in. I don't see how they complicate a Lisp by much; we have
&rest and &optional so what is problem having &key?

In my personal opinion it is by far more clumsy to have two defun
macros, and to prefix one with cl- for no good reason at all; and than
have to explain for users why they are two different versions, how they
differ, when they should use each, why would they like to use one or
another etc. If ordinary "defun" was simply upgraded to act as cl-defun,
potential user dilemas and explanations would be avoided.

>> I don't know why you reject them as clumsy, but why does it matter to us
>> if a tool includes features we don't use?
>
> It matters a very great deal.  In practice you cannot avoid "using" these
> features if you have to understand or debug somebody else's code.

In practice most of code will use some common part of the language, and
relatively little code will use some specialized parts. In a big
language that targets many different kind of developers, there will
always be features that are needed by some developers but not by
all. Falacy in your, by now very tired and wornout argument, is the
assumption that everyone will use everything in the language all the
time, which certainly isn't the case.

Interestingly, yesterday why commuting home after the work, I have heard
a talk by Stroustrup, quite recent just a few week old:

https://www.youtube.com/watch?v=eo-4ZSLn3jc

The talk is more or less about that very argument: why is C++ so big and
why are there so many features in c++. (If the link gets removed search
on Stroustrup C++ 2023). I think that was one of better Stroustrup's
talks, definitely worth watching.

There is also an incredibly inspirational talk by Steel (search on
"Growing the Language"); I have seen it just a few weeks ago myself:

https://www.youtube.com/watch?v=lw6TaiXzHAE

Observe that Steel gave us the "most minimal" and "elegant" Lisp as
Schemer's like to think of Scheme, which is probably important to
remember in the context of this discussion. If you don't watch it, the
essence is in this summary, and in my opinion is really a clever
insight:

"Over the last quarter-century Guy Steele has been convinced that trying
to design a complete and perfect programming language is the worst thing
you can do. A programming language (including its associated libraries)
must grow over time as its user community and its development community
grow. This is a different situation from 25 years ago, when all such
communities were relatively small. The difference is a problem of
scale. As a result, programming language design now and in the future is
necessarily as much a matter of social engineering as technical
engineering and must rely more on a set of general principles than on a
set of specific technical decisions."

>> I don't know if I understand it correctly, but CL was made so other
>> Lisps can be abstracted on top of it; so we use those features to
>> abstract stuff on top of those lengthy verbose functions? In other
>> words we can hide those keyword params behind elispy abstractions if we
>> don't like them?
>
> That's complexity and bloat.  Far better not to have them in the first
> place.

Complexity and bloat by which metric?

Emacs has long time ago exceeded CL in numbers of "core functions". By
the time I have extracted all Lisp defuns from the C core, a couple of
months ago or so, it was ~1800 functions in C core. That is by far many
more than what CL standard defines. I know that some of Emacs devs are
lurking on Reddit behind pseudonims, I don't know if you are one of
them, but there are also quite many users who consider Emacs to be quite
bloated and would like many of some older parts to be expunded out of
the core.

Anyway, by the nature of Lisp, there are very few actual language
concepts, and there are quite few additional concepts that CL offers
that need support of the system that are not found in Emacs; but most of
additions to the language are actually looking exactly like the language
itself. The nice thing is that, when you write a defun in Lisp, it
becomes part of the language itself, unlike languages like Java, C, C++
etc where users can not extend the language itself since the syntax has
to be encoded in the compiler. In other words, it means, that even if
you don't want it, people can extend it the way they like it, and cl.el
was a proof.

>> > best avoided.
>
>> Why are they best avoided? Is there some technical reason or is it
>> psychological?
>
> It's because they are not needed.  Bear in mind, Common Lisp is a massive

I don't know if you agree with me or not, but in order to drive from
Nuremberg to Berlin you certainly don't need AC in a car; but on a hot
summer day, with 40 degrees (celsius) it is very nice to have AC in the
car, isn't it? Sure, keyword arguments are not needed; but they are
handy from time to time. Optional or rest arguments are not needed
either, but they too are nice to have sometimes, aren't they? What
makes keyword arguments more "clumsy" then? 

To be honest to you, I really didn't want to answer this mail, because I
see just very same arguments I have seen many, many times before. I had
very much, very same sentiment about C++ as you, and I didn't even know
much of CL at all; I just thought it was big and bloated and didn't want
to use it, untill not so long time ago. Untill perhaps a year or two
ago, when I invested some time in learning more about CL and tried to
understand why things are as they are, not just in CL, but in general;
C++ very much included.

> It's because they are not needed.  Bear in mind, Common Lisp is a massiveI
> language, much like C++ or PL/1 or Algol-68.  With such a language,
> nobody uses all of it (it's just too big), but everybody has her own
> personal subset.  This creates difficulty when somebody else has to
> understand that first hacker's code.

I totally understand you; but I believe this is a bit misguided. For the
first, for the same argument as above, nobody uses the entire language
because they don't need it; not because it is too big. Some parts target
certain specialized domains which are needed by some groups and not by
everyone. I think that is more true for C++ and less for CL, because CL
is not that massive as you are trying to make it. CL is also a child of its
time, just like Emacs, and some parts are probably in need of a
revision, but it certainly stands the proof of time when it comes to
core language principles.

If we look at this:

(defun directory-files (directory &optional full match nosort count)
...)

When you call this function in the code; how easy is to remember the
order and number of arguments? Was it 5 or 6 arguments? I had fortune to
contribute the idea for the 5th argument, and yet I typed by a misstake
the wrong number:

CL-USER> (directory-files "./" nil nil nil nil 5)
; Debugger entered on #<SB-INT:SIMPLE-PROGRAM-ERROR "invalid number of arguments: ~S" {1001E92C63}>
[1] CL-USER> 
; Evaluation aborted on #<SB-INT:SIMPLE-PROGRAM-ERROR "invalid number of arguments: ~S" {1001E92C63}>
CL-USER> (directory-files "./" nil nil nil 5)
("alloc.lisp" "buffer.lisp" "callint.lisp" "callproc.lisp" "casefiddle.lisp")
CL-USER>

If I only change &optional to &key:

(defun directory-files (directory &key full match nosort count) ... )

I can now type:

CL-USER> (directory-files "./" :count 5)
("#dired.lisp#" ".#dired.lisp" "alloc.lisp" "buffer.lisp" "callint.lisp")
CL-USER> 

How "clumsy" is that? Honestly, I am not trying to be PITA or devil's
advocate or anything like that; but I think that everyone will agree
that the second one is both nicer, less error prone to type and puts
less cognitive load on users to remember or lookup the information.

Just as a remark: by just having it in CL I was able to just go to the
code, and change "optional" for "key", C-x C-e in Sly and I changed the
implementation of "directory-files", nice? For the record if you want to
try it (just a fast hack for the demonstration purpose, needs cl-ppcre lib):

(defun directory-files (directory &key full match nosort count)
  (let ((files
          (if full
              (mapcar #'namestring (uiop:directory-files directory))
              (mapcar #'file-namestring (uiop:directory-files directory)))))
    (when match
      (let ((scanner (ppcre:create-scanner match))
            matches)
        (dolist (filename files)
          (when (ppcre:scan scanner filename)
            (push filename matches)))
        (setf files matches)))
    (unless nosort
      (setq files (sort files #'string-lessp)))
    (when (and (numberp count) (> count 0))
      (when (> count (length files))
        (setf count (length files)))
         (setf files (subseq files 0 count)))
    files))

Observe also that I agree with you; keyword arguments are not needed,
but are very nice to have.

> CL is a niche language; it has not captured the hacker mindset.  I think
> it is just too big and too unwieldy.
>
>> > best avoided.  All those keyword arguments!  I intentionally excluded
>> > tham from Emacs and I am not going to let them in.
>
>> I don't want to be impolite, but they are already in; via cl-lib.el.
>
> Yes.  There was a time not so long ago when cl.el was banned from use in
> our Lisp code, except for at compile time.  Our Emacs Lisp was small,
> simple to understand, and easy to learn.  Now things in cl-lib.el get
> used as if they are just a normal part of Emacs Lisp.  Our language is
> thus MUCH more difficult to understand, perhaps by a factor of somewhere
> between 3 and 10.  When perusing even established parts of Emacs I groan
> inwardly every time I encounter one of these needless cl-lib features.
> It stops me dead, forcing me to consult doc strings (which are often
> missing and often inadequate even when they are present) or even manuals.

"MUCH more difficult" for whom in which sense?

What you are telling is that "good old times were so much better" which
is just an emotional argument. You are actually just reinforcing the
same sentiment that RMS expressed. I shared that sentiment myself, but I
think it is misguided.

We can certainly speak about "old ways", let us take the or-idiom or how
should I call it: initialization of the default value for optional arguments:

(defun foo (&optional who-am-I)
  (let ((who-am-I (or who-am-I "foo")))
    (message "I am %s." who-am-I)))

Is that really better than typing:

(cl-defun foo (&optional (who-am-I "foo"))
  (message "I am %s." who-am-I))

The user has to learn the idiom, which uses operator "or" to perform
something that visually has nothing to do with the intention of the
code, and also has to type the additional let-form each and every
time. Than the users who are not familiar with the idiom will perhaps
come up with their own version, using setq or some other thing, and you
will really have to think what the user wanted to say with their code if
you had to debug it. Is it better than seing an initialiser and knowing
directly what is the default value and everyone using uniform syntax?

There was a discussion, perhaps a couple of years ago, I think on Emacs
help list, I don't remember the detials, but I think something about
car, cdr, cdar, cddr & co, and using "nth" instead of them. Monnier said a
clever thing there: using those old functions is like writing in
assembler, but without the benefit of additional speed (something in
that sense). I think he was more than correct there. It is much more
clear to say (nth N list), than to chain cars and cdrs together.

A bit further, that really is about abstractions. We can for sure do
lots of stuff manually, without higher-level abstractions; but higher
level abstractions, like using initializer in a defun arguments, help us
convey the meaning more clearly and concize, we can rationalize about
abstractions, and they are probably easier to remember than teaching
people how to use some common pattern or idiom.

I also happened to read the CMUCL manual just yesterday. They have a
nice writing about using structures instead of lists in CL:

"Even if structures weren't more efficient than other representations,
structure use would still be attractive because programs that use
structures in appropriate ways are much more maintainable and robust
than programs written using only lists. For example:
     

    (rplaca (caddr (cadddr x)) (caddr y))

could have been written using structures in this way:
     

    (setf (beverage-flavor (astronaut-beverage x)) (beverage-flavor y))

The second version is more maintainable because it is easier to
understand what it is doing."

https://cmucl.org/downloads/doc/cmu-user-2010-05-03/compiler-hint.html#toc189

> Were we to rewrite Emacs in Common Lisp, these things would get worse
> fairly quickly.

I think you are very, very wrong about that one; on the contrary we
would get some tools that Emacs is lacking to manage the code
complexity, but I don't think they are actually applicable on the
existing code, so they don't matter much. But things certainly wouldn't
get worse. That is just my opinion of course, you can think I am an
idiot and I can be wrong, but at least give me a reason to change my
mind that isn't based on subjective opinion.

RE: [External] : Re: Shrinking the C core

[Drew Adams]

FWIW, +1 for your mail, Arthur -
pretty much each of the points you made.
___

As one who's used CL (long ago) but is
no expert about it or Elisp or Lisp
generally, I happen to agree about the
usefulness of keyword args _for users_.

I can't really speak to implementation,
compilation, maintenance, etc., all of
which are of course also important.

AFAIK there has never been a real, open,
serious discussion about keyword args
for Elisp.  And (I think) I've always
respected the decision to not bring up
the question.  But I do appreciate it
being at least _presented_, if not put
on the table for outright discussion.

In general, I like that Richard speaks
up and decides, and I generally agree
with his judgments as helmsman.  But on
this one my own experience tells me
something different.

Not that I have any interesting opposing
arguments.  Nor do I want to argue about
this.  But my experience with CL has led
me to appreciate the ease and handiness
of using keyword args.

Argument order, and the need to provide
all args up through the last optional
one you really want to provide, are an
unnecessary burden on human readers of
code.  Writers too, of course, but it's
more important that code be clear for
readers (including writers as readers).

Unnecessary burden.  So Occam asks "Why
then?".  In his role as _implementer_ of
a language Occam might ask "Why bother
with keywords args?"  But in his role as
user I think his question would be "Why
always need to know & respect arg order?"

I also haven't noticed that having named
arguments is detrimental to code that
analyses or generates code.  I have some,
but less, experience with that - maybe
it's something to consider; dunno.
___

That's likely all I'll say about this.
Just one opinion.  Keyword args can be
incredibly useful _for users_.

Re: [External] : Re: Shrinking the C core

[Arthur Miller]

Drew Adams <drew.adams@oracle.com> writes:

> FWIW, +1 for your mail, Arthur -
> pretty much each of the points you made.
> ___
>
> As one who's used CL (long ago) but is
> no expert about it or Elisp or Lisp
> generally, I happen to agree about the
> usefulness of keyword args _for users_.

Interestingly, how things can be percieved wrongly over the wire.
I always percieved you as an CL veteran and expert :).

> I can't really speak to implementation,
> compilation, maintenance, etc., all of
> which are of course also important.
>
> AFAIK there has never been a real, open,
> serious discussion about keyword args
> for Elisp.  And (I think) I've always
> respected the decision to not bring up
> the question.  But I do appreciate it
> being at least _presented_, if not put
> on the table for outright discussion.
>
> In general, I like that Richard speaks
> up and decides, and I generally agree
> with his judgments as helmsman.  But on
> this one my own experience tells me
> something different.
>
> Not that I have any interesting opposing
> arguments.  Nor do I want to argue about
> this.  But my experience with CL has led
> me to appreciate the ease and handiness
> of using keyword args.

I really didn't want to argue about keyword arguments either, but
Richard brought them up himself. Like you, and probably everyone else
who is familiar with Richards writing through the years, through his
articles and this very mailing list, I am aware of the elephant in the
room, but honestly, I think his friends, Eli and others, are too kind and
would do Richard more favor if they told him when he is dead wrong and
made him to challenge his own view on some questions from time to
time. This reminded me of John Cleese and Faulty Towers, no idea if you
had that in States, but I think every European here knows which episode
I am thinking of at the moment.

> Argument order, and the need to provide
> all args up through the last optional
> one you really want to provide, are an
> unnecessary burden on human readers of
> code.  Writers too, of course, but it's
> more important that code be clear for
> readers (including writers as readers).
>
> Unnecessary burden.  So Occam asks "Why
> then?".  In his role as _implementer_ of
> a language Occam might ask "Why bother
> with keywords args?"  But in his role as
> user I think his question would be "Why
> always need to know & respect arg order?"
>
> I also haven't noticed that having named
> arguments is detrimental to code that
> analyses or generates code.  I have some,
> but less, experience with that - maybe
> it's something to consider; dunno.
> ___
>
> That's likely all I'll say about this.
> Just one opinion.  Keyword args can be
> incredibly useful _for users_.

Thank you, for wording this so concisely.  You are much better with
words than me. I am not a good writer, not even in my own language, so
mine explanations and arguments are always long and round-about for some
reason.

Best regards

Re: [External] : Re: Shrinking the C core

[Eli Zaretskii]

> From: Arthur Miller <arthur.miller@live.com>
> Cc: Alan Mackenzie <acm@muc.de>,  Richard Stallman <rms@gnu.org>,
>  "luangruo@yahoo.com" <luangruo@yahoo.com>,  "emacs-devel@gnu.org"
>  <emacs-devel@gnu.org>
> Date: Sat, 09 Sep 2023 13:55:46 +0200
> 
> I really didn't want to argue about keyword arguments either, but
> Richard brought them up himself. Like you, and probably everyone else
> who is familiar with Richards writing through the years, through his
> articles and this very mailing list, I am aware of the elephant in the
> room, but honestly, I think his friends, Eli and others, are too kind and
> would do Richard more favor if they told him when he is dead wrong and
> made him to challenge his own view on some questions from time to
> time.

This comes out quite condescending, FYI.  You probably didn't mean
that, but the result is not nice anyway.  One more reason to stop this
dispute about personal stylistic preferences, which was futile from
the get-go.

Re: [External] : Re: Shrinking the C core

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Arthur Miller <arthur.miller@live.com>
>> Cc: Alan Mackenzie <acm@muc.de>,  Richard Stallman <rms@gnu.org>,
>>  "luangruo@yahoo.com" <luangruo@yahoo.com>,  "emacs-devel@gnu.org"
>>  <emacs-devel@gnu.org>
>> Date: Sat, 09 Sep 2023 13:55:46 +0200
>> 
>> I really didn't want to argue about keyword arguments either, but
>> Richard brought them up himself. Like you, and probably everyone else
>> who is familiar with Richards writing through the years, through his
>> articles and this very mailing list, I am aware of the elephant in the
>> room, but honestly, I think his friends, Eli and others, are too kind and
>> would do Richard more favor if they told him when he is dead wrong and
>> made him to challenge his own view on some questions from time to
>> time.
>
> This comes out quite condescending, FYI.  You probably didn't mean
> that, but the result is not nice anyway.

No, I certainly didn't mean as patronizing if that is what "condecending".

> that, but the result is not nice anyway.  One more reason to stop this
> dispute about personal stylistic preferences, which was futile from
> the get-go.

I agree it is futile.

No problems; thanks for telling me.

RE: [External] : Re: Shrinking the C core

[Drew Adams]

> > FWIW, +1 for your mail, Arthur -
> > pretty much each of the points you made.
> > ___
> >
> > As one who's used CL (long ago) but is
> > no expert about it or Elisp or Lisp
> > generally, I happen to agree about the
> > usefulness of keyword args _for users_.
> 
> Interestingly, how things can be percieved wrongly over the wire.
> I always percieved you as an CL veteran and expert :).

Nope.  Just a long-ago user.  Being around a
long time can sometimes give the impression
of being a veteran.

> I think his friends, Eli and others, are too kind and
> would do Richard more favor if they told him when he
> is dead wrong 

I have no reason to think Richard is dead
wrong about whether Elisp should have
keyword args.  I just related my (good)
experience with them in CL, as one user,
long ago.

Elisp could of course go in many directions.
That it hasn't taken certain directions I'm
generally thankful for, FWIW.  It's not a
bad language, and not every proposal or
imagined change represents real progress. ;-)

Re: [External] : Re: Shrinking the C core

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I implemented Common Lisp for the Lisp Machine in 1983.
That included the generic sequence functions with their
many keyword arguments.

In 1984 I wrote Emacs Lisp and decided to omit all that.
I have no regrets.  Their absence makes Emacs Lisp easier to use.

Keyword arguments are fine for an unusual, heavyweight operation such
as creating a frame.  The sort of function whose doc you need to look
up again each time you use it.  When something is as cumbersome as
that, the question of how to pass the arguments is a pile of minor
details that you'll see when you look it up.

But it's a pain in the neck to make simple everyday functions such as
`append' and `length' that cumbersome just for the sake of following a
general system.  Part of this system is that the default argument
values follow the system, rather than following the traditional Lisp
functions -- so getting the traditional behavior requires actually
specifying the keyword arguments to override defaults, almost every
time.

Switching to Common Lisp would cause those painful incompatibilities.
It is unacceptable.

Defining the CL functions with a name prefix is ok, because the
traditional Lisp functions are still available compatibly and we don't
have to use the CL functions.  This is what we have now.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [External] : Re: Shrinking the C core

[João Távora]

On Mon, Sep 11, 2023 at 1:40 AM Richard Stallman <rms@gnu.org> wrote:

> Keyword arguments are fine for an unusual, heavyweight operation such
> as creating a frame.  The sort of function whose doc you need to look
> up again each time you use it.  When something is as cumbersome as
> that, the question of how to pass the arguments is a pile of minor
> details that you'll see when you look it up.

It's possible that in 1984 there weren't any good at-point-documentation
systems that immediately show the available keyword arguments of a given
call being typed.  I was very young at the time, so I can't attest to it.

But nowadays such systems exist, including built-in, in Emacs.

But even in the absence of those conveniences, each family of functions
in CL, (say, the sequence manipulating functions) take standard sets
of keyword arguments (like :START, :END, :FROM-END, :KEY, :TEST).

This makes them very easy to remember and makes using e.g. CL's SORT
much more practical than Emacs Lisp 'sort'.

> But it's a pain in the neck to make simple everyday functions such as
> `append' and `length' that cumbersome just for the sake of following a
> general system.

APPEND and LENGTH in Common Lisp do not take keyword arguments,
(nor have they ever?) so I believe your argument is weakened by
criticizing a suggestion that no-one is offering.

Let's look at a traditional Elisp macro define-minor-mode.

According to the CVS history, you created this function in 1997, with
three positional optional arguments.  Then more and more arguments
came along.  Possibly due to the realization by Emacs developers
around 2000, already more than 20 years ago,  that positional arguments
are maintenance hazards, the macro now accepts keyword arguments
(and the positional arguments are actively discouraged).

Unfortunately, because cl-lib.el wasn't used for this particular change,
the aforementioned at-point documentation facilities aren't available,
which, to some of us, would appear just another manifestation of
Greenspun's 10th rule (which I assume everyone in this niche topic
is familar with).

Many other functions with a large amount of optional arguments
(completing-read comes to mind) would be much, much easier
to use with keyword arguments.  Without them, we find ourselves
wondering about how many nils to sprinkle before the argument
we want to pass.

João

RE: [External] : Re: Shrinking the C core

[Drew Adams]

I thought my previous post in this thread would
likely be my only such, but I'd like now to add a
point, FWIW.

Richard, you spoke about implementing support for
keyword args in ~CL (Lisp-machine Lisp code that
you wrote), and how you found their presence to be
a bother for implementers and (I think) users.

I mentioned that I appreciated _using_ keyword args
with CL, long ago.  I'll add that it was especially
in code that I wrote that I found them useful (as
opposed to their presence in standard CL code).
That is, for _users_ of my code (but including me,
while developing and testing).  And especially for
functions that allowed for more than a few args.

In a nutshell, with keyword args a single function
can replace multiple related/similar functions.

And when you have multiple things to juggle, it
helps to name them, and it helps to not need to
address them in a particular order.

Here's an analogy:

Imagine Lisp without &optional args.  You'd end by
defining more functions of the same "family".

IME, the same difference applies to the absence of
keyword args.  You end up defining more functions,
for the convenience of not having to specifying a
zillion nil args.  Plus you have to pay attention
to arg order.

Of course sometimes it makes sense to define a
separate function or two, to cover particular
common use cases.  But in general optional args
are handy, and so are keyword args, for the same
reason: many functions in one - a single name to
rule them all. ;-)

Now, if you always _had to_ use keyword args, so
you could _never_ just provide args without names
(in the proper order), then that would definitely
be an unnecessary bother.  We probably all agree
about that.

In Lisp all args to functions are evaluated, and
there really is no user-level dependence on the
order of their evaluation.  Given that, Occam says
that arg order doesn't matter, and there's no real
_need_ to specify args in an order, filling in nil
args as needed to get to the last optional arg you
need.

From a user point of view, being able to have the
option of specifying an arg by its position OR by
its name is just a plus - I don't see any downside.

From the point of view of a language implementer
things might be different, of course.

Re: [External] : Re: Shrinking the C core

[Tomas Hlavaty]

On Mon 11 Sep 2023 at 16:10, João Távora <joaotavora@gmail.com> wrote:
> This makes them very easy to remember and makes using e.g. CL's SORT
> much more practical than Emacs Lisp 'sort'.

not really, it seems that CL:SORT is a bad example

the CL spec defines sort like this:

   sort sequence predicate &key key => sorted-sequence
   stable-sort sequence predicate &key key => sorted-sequence

(btw why two functions and not extra stablep keyword argument?)

it could have been defined as:

   sort sequence predicate &optional key => sorted-sequence
   stable-sort sequence predicate &optional key => sorted-sequence

and the same could be done in elisp which would be backwards compatible

> Let's look at a traditional Elisp macro define-minor-mode.

the usual CL argument list does not seem to be able to express arguments
of such shape

it looks like whoever extended the original argument list did it
"weirdly" using custom ad-hoc single-use argument list parser.

> are maintenance hazards, the macro now accepts keyword arguments

in CL, the arguments would normally be in a list before body, something
like

   (define-minor-mode MODE ([KEYWORD VAL ... ]) [DOC] &rest BODY)

Re: [External] : Re: Shrinking the C core

[João Távora]

On Mon, Sep 11, 2023 at 9:37 PM Tomas Hlavaty <tom@logand.com> wrote:

> not really, it seems that CL:SORT is a bad example

The example was for comparing to Emacs's lisp 'sort', where
IMO it's much easier to do

  (cl-sort things-having-foo #'< :key #'foo)

than

  (sort things-having-foo (lambda (a b) (< (foo a) (foo b))))

As to the advantage of @key key vs &optional key it is -- in my
eyes,  at least -- that you already know its meaning from many other
functions.

> (btw why two functions and not extra stablep keyword argument?)

Why not? Maybe that makes passing sort and stable-sort to higher
order functions more practical?

> it could have been defined as:
>
>    sort sequence predicate &optional key => sorted-sequence
>    stable-sort sequence predicate &optional key => sorted-sequence
>
> and the same could be done in elisp which would be backwards compatible

Sure, but if according to your conjecture Elisp's grew a &optional
arg, why couldn't it grow a &key arg and be backwards compatible?

> > Let's look at a traditional Elisp macro define-minor-mode.
>
> the usual CL argument list does not seem to be able to express arguments
> of such shape
>
> it looks like whoever extended the original argument list did it
> "weirdly" using custom ad-hoc single-use argument list parser.

Quite likely the cl machinery wasn't available at the time...  But yes,
Greenspun's 10th.

> > are maintenance hazards, the macro now accepts keyword arguments
>
> in CL, the arguments would normally be in a list before body, something
> like
>
>    (define-minor-mode MODE ([KEYWORD VAL ... ]) [DOC] &rest BODY)

OK, but this is irrelevant.  This is a macro, not a function.
For all practical purposes it was extended with keyword arguments.
In fact you could have used cl-lib's cl-destructuring-bind.

João

Re: [External] : Re: Shrinking the C core

[Tomas Hlavaty]

On Mon 11 Sep 2023 at 22:10, João Távora <joaotavora@gmail.com> wrote:
> On Mon, Sep 11, 2023 at 9:37 PM Tomas Hlavaty <tom@logand.com> wrote:
>> not really, it seems that CL:SORT is a bad example
>
> The example was for comparing to Emacs's lisp 'sort', where
> IMO it's much easier to do
>
>   (cl-sort things-having-foo #'< :key #'foo)
>
> than
>
>   (sort things-having-foo (lambda (a b) (< (foo a) (foo b))))

That has nothing to do with &key.
It has everything to do with the lack of the KEY argument
and the KEY argument could as well be &optional instead of &key.


If you find lack of the key argument "much not easier", you can define
your own sort, something like:

   (defun sort3 (seq pred key)
     (sort seq (lambda (a b)
                 (funcall pred
                   (funcall key a)
                   (funcall key b)))))
where

   (sort3 things-having-foo #'< #'foo)

is "much easier to do" than

   (cl-sort things-having-foo #'< :key #'foo)

or

   (sort things-having-foo (lambda (a b) (< (foo a) (foo b))))

Additionally, you'll get much nicer tool support automatically,
e.g. autodoc tells me:

   sort3: (SEQ PRED KEY)
   cl-sort: (SEQ PREDICATE [KEYWORD VALUE]...)

One can see that autodoc for cl-sort is severely crippled.

> As to the advantage of @key key vs &optional key it is -- in my
> eyes,  at least -- that you already know its meaning from many other
> functions.

The point Richard raised was that your "many other functions" with &key
are bad way of doing it.

Simply naming the argument KEY would achieve the same goal of already
knowing its meaning from many other functions.  Making it &key does not
change that.

>> (btw why two functions and not extra stablep keyword argument?)
>
> Why not? Maybe that makes passing sort and stable-sort to higher
> order functions more practical?

That speculation does not sound plausible.

By that logic, CL would probably define 4 functions:

   sort sequence predicate => sorted-sequence
   sort3 sequence predicate key => sorted-sequence
   stable-sort sequence predicate => sorted-sequence
   stable-sort3 sequence predicate key => sorted-sequence

>> > Let's look at a traditional Elisp macro define-minor-mode.
>>
>> the usual CL argument list does not seem to be able to express arguments
>> of such shape
>>
>> it looks like whoever extended the original argument list did it
>> "weirdly" using custom ad-hoc single-use argument list parser.
>
> Quite likely the cl machinery wasn't available at the time...  But yes,
> Greenspun's 10th.
>
>> > are maintenance hazards, the macro now accepts keyword arguments
>>
>> in CL, the arguments would normally be in a list before body, something
>> like
>>
>>    (define-minor-mode MODE ([KEYWORD VAL ... ]) [DOC] &rest BODY)
>
> OK, but this is irrelevant.  This is a macro, not a function.
> For all practical purposes it was extended with keyword arguments.
> In fact you could have used cl-lib's cl-destructuring-bind.

No, that's missing the point.  CL itself is not able to express the
argument list structure in the shape it is implemented in
define-minor-mode.  You need custom argument parser anyway.

autodoc for define-minor-mode is also useless:

   define-minor-mode: (MODE DOC [KEYWORD VAL ... &rest BODY])

The way define-minor-mode sneaks in keyword arguments is pretty bad.  It
implements custom argument list parser and makes tools that understand
and work with argument lists useless, which causes extra manual work in
other areas.  If there was a unified argument list parser implemented in
one place and used consistently, it would avoid lots of manual work.

Back to the [KEYWORD VAL ... ] example, putting the argument list of the
macro into an extra list would allow one to use unified argument list
parser, but the way define-minor-mode arguments are specified at the
moment, CL:DESTRUCTURING-BIND cannot destructure such structure (&key
before &rest/&body is malformed).

Another example, take &body as opposed to &rest.  It automatically
declares the intent and how the indentation is meant to be computed.
With &rest indentation needs to be specified individually and manually
per case.  Lack of &body in elisp is annoying inconvenience.  Example:
(with-help-window BUFFER-OR-NAME &rest BODY) has (declare (indent 1))
which would not be needed if it was (with-help-window BUFFER-OR-NAME
&body BODY).

I think that rather than arguing about keyword arguments only, it would
be better to push for a unified argument list format which would be
followed, understood and supported by relevant tools.  So far, argument
list structure in elisp grew into ad-hoc mess which one needs to decode
manually from docstring on case by case basis; and no amount of CL would
help.

Re: [External] : Re: Shrinking the C core

[João Távora]

On Tue, Sep 12, 2023 at 8:52 PM Tomas Hlavaty <tom@logand.com> wrote:

> It has everything to do with the lack of the KEY argument
> and the KEY argument could as well be &optional instead of &key.

Yes, it could, but should it?  I think not, because consistency.
You think yes, because whatever, so let's just disagree.

> If you find lack of the key argument "much not easier", you can define
> your own sort, something like:

The point with having good library functions is that the user doesn't
need to define her ad-hoc versions.

> One can see that autodoc for cl-sort is severely crippled.

That's a problem with autodoc and cl-sort's definition, not its protocol.
Functions defined with cl-func are much better and autodoc functionality
for SLIME/SLY is spot on and detects the correct argument.  This is
no technical hurdle.

João

Re: [External] : Re: Shrinking the C core

[Emanuel Berg]

João Távora wrote:

> Many other functions with a large amount of optional
> arguments (completing-read comes to mind) would be much,
> much easier to use with keyword arguments. Without them, we
> find ourselves wondering about how many nils to sprinkle
> before the argument we want to pass.

Agreed, it is better with keyword arguments than a long list
of nils because of optional arguments.

But it is better yet to not have the functions take so many
arguments in the first place, but to split them up and have
the function name be more specific what is going to happen.

Sometimes this just doesn't happen, it's life and I have
a hard time seeing any situation (long list of nils for
optional arguments vs. keywords) being a real problem
to anyone?

-- 
underground experts united
https://dataswamp.org/~incal

RE: [External] : Re: Shrinking the C core

[Drew Adams]

(Again, moving to emacs-tangents@gnu.org.)

> Agreed, it is better with keyword arguments than 
> a long list of nils because of optional arguments.
> 
> But it is better yet to not have the functions take so many
> arguments in the first place, but to split them up and have
> the function name be more specific what is going to happen.

Not necessarily.  It depends - on context, how
often each combination is used, and even user
preferences.

With 10 different functions, instead of one,
to represent a "family" of functions, users
can wonder "Which should I use?".  The problem
introduced then is finding good function names
and specifying, in the doc of each (or in some
"family" doc) just what the differences are,
and how they can (or can't) be used together.

IOW, it can make sense to use a "family" name
and keywords, with a single explanation of the
keywords, together.  A single explanation lets
you address what combinations make sense etc.

Happily, if keywords are at least available
(possible) in the language, you have a choice
of which approach to use.  Nothing _requires_
you to define a function with keyword args,
instead of defining separate functions (up to
N-factorial of them!).

Re: [External] : Re: Shrinking the C core

[Eric S. Raymond]

Richard Stallman <rms@gnu.org>:
> Switching to Common Lisp would cause those painful incompatibilities.
> It is unacceptable.

I did some checking into this a few weeks ago and even wrote a bit of Lisp to do
an audit.  There are other reasons I decided to get up to speed on Common Lisp
recently, and because I have Emacs Lisp engraved on my forebrain this interested
me in getting a good grasp on how incompatible with SBCL it actally is.

Count of elisp core functions: 1476 
Total SBCL/elisp function name collisions: 145 
Primitives identical in elisp and SBCL: 116
Primitives not known identical: 28

The first two lines are crunched from querying SBCL's symbol list and
groveling through the Emacs sourcefiles; I can supply the SBCL code I
write to do this if anybody cares.

The second two lines are from me reading the SBCL documentation a lot;
I needed immediate motivation to do that and this was a good one.

I was able to write trivial implementations for 15 of those 28 collisions.
The 14 I didn't implement are these:

(require read-char read random process-status process-plist print
princ prin1-to-string prin1 load documentation defvar aref)

What this means is that it would, in principle, be possible to write
an SBCL core that implements all of the Emacs primitives, with only a rather
small amount of shim code required to make existing elisp code execute
in an environment where it thinks it sees *all* elisp primitives.

The code would look something like this, where I have deleted all but one emulation
to showcase setting up the emulation environment.

	;; Implement functions shared between CL and elisp in an elisp-like way

	(defun elisp-append (&rest sequences)
	  "(append &rest SEQUENCES)

	  Probably introduced at or before Emacs version 18.
	  This function does not change global state, including the match data.

	Concatenate all the arguments and make the result a list.
	The result is a list whose elements are the elements of all the arguments.
	Each argument may be a list, vector or string.
	The last argument is not copied, just used as the tail of the new list."
	  (apply 'append (mapcar
			  (lambda (it)
			    (if (listp it) it (listp it)))
			  sequences))
	  )

	(defmacro within-elisp (form)
	  "Evaluate FORM in an environment with elisp behavior."
	  `(flet ((append #'emacs-append)
		 (delete-file-internal #'elisp-delete-file-internal)
		 (eval #'elisp-eval)
		 (format #'elisp-format)
		 (intern #'elisp-intern)
		 (make-hash-table #'elisp-make-hash-table)
		 (make-string #'elisp-make-string)
		 (rename-file #'elisp-rename-file)
		 (sort #'elisp-sort)
		 (string #'elisp-string)
		 (string-equal #'elisp-string-equal)
		 (string-lessp #'elisp-string-lessp)
		 (symbol-value #'elisp-symbol-value)
		 (type-of #'elisp-type-of)
		 (unintern #'elisp-unintern)
		 (yes-or-no-p #'elisp-yes-or-no-p)
		 )
	    ,form)
	  )

With this encapsulation, "painful incompatiblities" between elisp and
a core written in SBCL would never need be visible to anyone who put
an .el extension on their code to tell the core it should be executed
using within-elisp.

I'm not minimizing the amount of work an SBCL core would take,
there's a lot of devil in the details of 1476 + 14 primitive
functions.  Nor am I interested in joining a political argument
about whether it should be done; I have too much else on my
plate for that.

But it could be done. There is a technical path forward to it.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: [External] : Re: Shrinking the C core

[Arthur Miller]

"Eric S. Raymond" <esr@thyrsus.com> writes:

> Richard Stallman <rms@gnu.org>:
>> Switching to Common Lisp would cause those painful incompatibilities.
>> It is unacceptable.
>
> I did some checking into this a few weeks ago and even wrote a bit of Lisp to do
> an audit.  There are other reasons I decided to get up to speed on Common Lisp
> recently, and because I have Emacs Lisp engraved on my forebrain this interested
> me in getting a good grasp on how incompatible with SBCL it actally is.
>
> Count of elisp core functions: 1476 
> Total SBCL/elisp function name collisions: 145 
> Primitives identical in elisp and SBCL: 116
> Primitives not known identical: 28
>
> The first two lines are crunched from querying SBCL's symbol list and
> groveling through the Emacs sourcefiles; I can supply the SBCL code I
> write to do this if anybody cares.
>
> The second two lines are from me reading the SBCL documentation a lot;
> I needed immediate motivation to do that and this was a good one.
>
> I was able to write trivial implementations for 15 of those 28 collisions.
> The 14 I didn't implement are these:
>
> (require read-char read random process-status process-plist print
> princ prin1-to-string prin1 load documentation defvar aref)
>
> What this means is that it would, in principle, be possible to write
> an SBCL core that implements all of the Emacs primitives, with only a rather
> small amount of shim code required to make existing elisp code execute
> in an environment where it thinks it sees *all* elisp primitives.
>
> The code would look something like this, where I have deleted all but one emulation
> to showcase setting up the emulation environment.
>
> 	;; Implement functions shared between CL and elisp in an elisp-like way
>
> 	(defun elisp-append (&rest sequences)
> 	  "(append &rest SEQUENCES)
>
> 	  Probably introduced at or before Emacs version 18.
> 	  This function does not change global state, including the match data.
>
> 	Concatenate all the arguments and make the result a list.
> 	The result is a list whose elements are the elements of all the arguments.
> 	Each argument may be a list, vector or string.
> 	The last argument is not copied, just used as the tail of the new list."
> 	  (apply 'append (mapcar
> 			  (lambda (it)
> 			    (if (listp it) it (listp it)))
> 			  sequences))
> 	  )
>
> 	(defmacro within-elisp (form)
> 	  "Evaluate FORM in an environment with elisp behavior."
> 	  `(flet ((append #'emacs-append)
> 		 (delete-file-internal #'elisp-delete-file-internal)
> 		 (eval #'elisp-eval)
> 		 (format #'elisp-format)
> 		 (intern #'elisp-intern)
> 		 (make-hash-table #'elisp-make-hash-table)
> 		 (make-string #'elisp-make-string)
> 		 (rename-file #'elisp-rename-file)
> 		 (sort #'elisp-sort)
> 		 (string #'elisp-string)
> 		 (string-equal #'elisp-string-equal)
> 		 (string-lessp #'elisp-string-lessp)
> 		 (symbol-value #'elisp-symbol-value)
> 		 (type-of #'elisp-type-of)
> 		 (unintern #'elisp-unintern)
> 		 (yes-or-no-p #'elisp-yes-or-no-p)
> 		 )
> 	    ,form)
> 	  )
>
> With this encapsulation, "painful incompatiblities" between elisp and
> a core written in SBCL would never need be visible to anyone who put
> an .el extension on their code to tell the core it should be executed
> using within-elisp.

There is a little bit more than just this; but that one was very clever
to smooth out basic differences. It all depends on the ambition of
course. I do wrap cl funcitons and re-implement some of those that are
missing in CL just to be able to attach the doc strings to symbols, and
counting on sbcl inlining to remove the function call. I am not sure yet
if it is a good strategy, but I can do it relatively automated. Looks
like this:

(defun take (n list)
  "Return the first N elements of LIST.
If N is zero or negative, return nil.
If N is greater or equal to the length of LIST, return LIST (or a copy)."
  ;; todo - this can probably be done without taking the length of the list
  (cl:when (cl:> n 0)
    (cl:let ((l (cl:length list)))
      (when (cl:> n l) (setf n l))
      (cl:subseq list 0 n))))

(defun ntake (n list)
  "Modify LIST to keep only the first N elements.
If N is zero or negative, return nil.
If N is greater or equal to the length of LIST, return LIST unmodified.
Otherwise, return LIST after truncating it."
  (cl:when (cl:> n 0)
    (cl:let ((l list))
      (eli:while (cl:and (cl:> n 1) (cl:cdr l))
             (cl:setf l (cl:cdr l))
             (cl:decf n))
      (cl:rplacd l nil)
      list)))

(declaim (inline nthcdr))
(defun nthcdr (n list)
  "Take cdr N times on LIST, return the result."
  (cl:nthcdr n list))

(declaim (inline nth))
(defun nth (n list)
  "Return the Nth element of LIST.
N counts from zero.  If LIST is not that long, nil is returned."
  (cl:nth n list))

But tere is more than just functions and macros.

The reader needs some help; Emacs uses different syntax for characters
and escapes for example; ? vs #\, but the escape syntax is were a real
hear pulling is. Also Emacs does not have character type but basically
uses whatever int comes out of C and thus Emacs API can do arithmetic
operaions and comparisons on characters whereas in CL they have to use
character specific operators. I have implemented the reader to
understand ? and escapes and to spit out integer codes instead of cl
characters so I don't need to dispatch on each mathematical operation.

There is still more, "markers are numbers" in Emacs, so mathematical
operations are applicable to those as well. I haven't decided if I will
do same as they do in C sources or I'll try to implement static dispatch
for generic functions and have mathematical operations as generic
functions, I have to test how it works, but there are other things like
those that have to be taken care of. And there is more, that is
just the tip of the iceberg :).

I am also talking only about lisp implemented in C core. There weould be
need to do some work on the elisp side too; eieo, cl-lib, defun/defmacro,
stuff like that, but that is probably by far lesser and easier to fix.

> I'm not minimizing the amount of work an SBCL core would take,
> there's a lot of devil in the details of 1476 + 14 primitive
> functions.  Nor am I interested in joining a political argument
> about whether it should be done; I have too much else on my
> plate for that.

Yes, something like that; it is volumous and lots of work, for one
person probably impossible, but it is not impossible to do. 

Re: [External] : Re: Shrinking the C core

[Gerd Möllmann]

"Eric S. Raymond" <esr@thyrsus.com> writes:

 > But it could be done. There is a technical path forward to it.

Which would have to cope with buffer-local bindings.
Just saying :-).

Re: [External] : Re: Shrinking the C core

[Arthur Miller]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> "Eric S. Raymond" <esr@thyrsus.com> writes:
>
>  > But it could be done. There is a technical path forward to it.
>
> Which would have to cope with buffer-local bindings.
> Just saying :-).

I think they can be dealt with in at least two different ways; but I haven't got
so far yet, so I might be wrong.

Re: [External] : Re: Shrinking the C core

[Eli Zaretskii]

[Redirected to emacs-tangents]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: Richard Stallman <rms@gnu.org>,  Drew Adams <drew.adams@oracle.com>,
>  arthur.miller@live.com,  acm@muc.de,  luangruo@yahoo.com,
>  emacs-devel@gnu.org
> Date: Tue, 12 Sep 2023 06:38:00 +0200
> 
> "Eric S. Raymond" <esr@thyrsus.com> writes:
> 
>  > But it could be done. There is a technical path forward to it.
> 
> Which would have to cope with buffer-local bindings.

Right.  And the display code.  And text en/decoding.  And input queue.
And faces.  Etc. etc. -- these are all written in C, but are full of
Lisp data structures and calls into Lisp, so separating them is
anything but easy or even straightforward.

People who have enough talents, knowledge, and energy to do this kind
of job will be better off, and will serve the community better, if
they design an Emacs differently, taking into consideration the main
lessons we've learned.  That "neo-Emacs" could have a mode whereby it
worked in a way compatible with the current Emacs, so that people who
must run the old applications could run them without changes.  But it
should be based on different, more modern architectural decisions.
Handling of buffer text, GC, the display engine, threads -- all these
and more needs to be rethought and preferably replaced with more
modern solutions, to avoid bumping into the same limitations right
from the get-go.

Re: [External] : Re: Shrinking the C core

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> [Redirected to emacs-tangents]
>
>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>> Cc: Richard Stallman <rms@gnu.org>,  Drew Adams <drew.adams@oracle.com>,
>>  arthur.miller@live.com,  acm@muc.de,  luangruo@yahoo.com,
>>  emacs-devel@gnu.org
>> Date: Tue, 12 Sep 2023 06:38:00 +0200
>> 
>> "Eric S. Raymond" <esr@thyrsus.com> writes:
>> 
>>  > But it could be done. There is a technical path forward to it.
>> 
>> Which would have to cope with buffer-local bindings.
>
> Right.  And the display code.  And text en/decoding.  And input queue.
> And faces.  Etc. etc. -- these are all written in C, but are full of
> Lisp data structures and calls into Lisp, so separating them is
> anything but easy or even straightforward.

Which are all components that have grown immensely in both volume and
complexity over the past 20 years or so, which is to say, since you
(Eric) last expended significant amounts of time developing Emacs.  Not
a single individual understands everything, particularly not to the
extent necessary to orchestrate a rewrite or re-implementation.

> People who have enough talents, knowledge, and energy to do this kind
> of job will be better off

So it is already a stretch to assume that such people will materialize
at all.

Re: [External] : Re: Shrinking the C core

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

> [Redirected to emacs-tangents]
>
>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>> Cc: Richard Stallman <rms@gnu.org>,  Drew Adams <drew.adams@oracle.com>,
>>  arthur.miller@live.com,  acm@muc.de,  luangruo@yahoo.com,
>>  emacs-devel@gnu.org
>> Date: Tue, 12 Sep 2023 06:38:00 +0200
>> 
>> "Eric S. Raymond" <esr@thyrsus.com> writes:
>> 
>>  > But it could be done. There is a technical path forward to it.
>> 
>> Which would have to cope with buffer-local bindings.
>
> Right.  And the display code.  And text en/decoding.  And input queue.
> And faces.  Etc. etc. -- these are all written in C, but are full of
> Lisp data structures and calls into Lisp, so separating them is
> anything but easy or even straightforward.
>
> People who have enough talents, knowledge, and energy to do this kind
> of job will be better off, and will serve the community better, if
> they design an Emacs differently, taking into consideration the main
> lessons we've learned.

I don't know what you have learned, but I have learned that Guy Steel
was correct when he told the world back in 1998 already: don't really on
a few people for the development, instead make things extensible by
the users. In an applicaiton that is almost a life style for many users
and where most of users are developers as well it makes sense to use the
same implementation and extension language, because it removes a barrier
for design and experimentation. For web developers it might make sense
to write their tools in JS, for Lisp developers it make sense to use
Lisp for their tools. That would make for more sustainable development.

> lessons we've learned.  That "neo-Emacs" could have a mode whereby it
> worked in a way compatible with the current Emacs, so that people who
> must run the old applications could run them without changes.  But it

Such a "mode" would still require you to implement al that stuff you
have now, there is no way around, and I am quite sure you know it.

Also; there is nothing that says that you can't have different
implementation under the hood. There is so much narrow-mindedness and
assumptions from you. Instead of assuming bunch of what I think or don't
think, as you always do, why don't you just ask me? I didn't answered
further in our private correspondence because of your constant assuming
what I think or don't think and so on. Ask me instead; if I haven't think
of something, or thought wrong, I'll be glad to learn about it.

> should be based on different, more modern architectural decisions.
> Handling of buffer text, GC, the display engine, threads -- all these
> and more needs to be rethought and preferably replaced with more
> modern solutions, to avoid bumping into the same limitations right
> from the get-go.

Yes, and what in your opinion *is* a suggestion to completely remove the
C code, which actually was a very relevant in a thread about shrinking
the C core? Also, to answer all those who can't put 1 + 1 togther by
themselves: I have suggested to remove completely C core in a thread
about shrinking the C core. I think a "maximal shrink" of C core is
quite on topic :-).

About your "neo-design", just implementing the editor in a Lisp machine
instead of having a Lisp machine in the editor is itself already a
radical design change. Not to mention that the Lisp machine suggested
already has threading and some other tools that would make life much
easier so you could concentrate on actually improving the editor instead
of the Lisp machine. Look at other similar applications already doing it
in that "clumsy" CL; Lem already has several rendering backends. How
many do you have?

Nobody says that you have to implement stuff under the hood the same
way; I have said we need C core and elisp semantics implemented in
CL. It is laborous but possible. Under the hood it could be implemented
with any changes needed, and in the future design could be exchanged for
something else, complemented etc.

Anyway, your rhetorics give allusion that Emacs is dead, users should go
to greener pastures you who want something more modern dead suites their
needs. I don't know, but those are vibes I get from your arguments.

Anyone can *already* use other "more modern applications". Reason users
don't use Hemlock or Climax or don't rush to Lem (perhaps they should)
is, as we have already discussed in private and Reddit, because they
can't run Emacs packages in them. People don't want Emacs-like editor,
they want GNU Emacs. Which is good, and which you are aware of from both
private discussion and Reddit. Sure, loosing *some* applications is OK,
there is a natural regression too, some things are not maintained or get
irrelevant for other reasons, but not all.

Another thing is your way to talk to people and keep this community; I
was told I am a lazy idiot who came here to beg someone to write
something for me, and I am told to watch my mouth when I answered. Great
community you are maintaining, thank you for that.

Re: [External] : Re: Shrinking the C core

[Eli Zaretskii]

> From: Arthur Miller <arthur.miller@live.com>
> Cc: Gerd Möllmann <gerd.moellmann@gmail.com>,
>   esr@thyrsus.com,
>   rms@gnu.org,  drew.adams@oracle.com,  acm@muc.de,  luangruo@yahoo.com,
>   emacs-tangents@gnu.org
> Date: Tue, 12 Sep 2023 21:58:43 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > [Redirected to emacs-tangents]
> >
> >> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> >> Cc: Richard Stallman <rms@gnu.org>,  Drew Adams <drew.adams@oracle.com>,
> >>  arthur.miller@live.com,  acm@muc.de,  luangruo@yahoo.com,
> >>  emacs-devel@gnu.org
> >> Date: Tue, 12 Sep 2023 06:38:00 +0200
> >> 
> >> "Eric S. Raymond" <esr@thyrsus.com> writes:
> >> 
> >>  > But it could be done. There is a technical path forward to it.
> >> 
> >> Which would have to cope with buffer-local bindings.
> >
> > Right.  And the display code.  And text en/decoding.  And input queue.
> > And faces.  Etc. etc. -- these are all written in C, but are full of
> > Lisp data structures and calls into Lisp, so separating them is
> > anything but easy or even straightforward.
> >
> > People who have enough talents, knowledge, and energy to do this kind
> > of job will be better off, and will serve the community better, if
> > they design an Emacs differently, taking into consideration the main
> > lessons we've learned.
> 
> I don't know what you have learned, but I have learned that Guy Steel
> was correct when he told the world back in 1998 already: don't really on
> a few people for the development, instead make things extensible by
> the users. In an applicaiton that is almost a life style for many users
> and where most of users are developers as well it makes sense to use the
> same implementation and extension language, because it removes a barrier
> for design and experimentation. For web developers it might make sense
> to write their tools in JS, for Lisp developers it make sense to use
> Lisp for their tools. That would make for more sustainable development.
> 
> > lessons we've learned.  That "neo-Emacs" could have a mode whereby it
> > worked in a way compatible with the current Emacs, so that people who
> > must run the old applications could run them without changes.  But it
> 
> Such a "mode" would still require you to implement al that stuff you
> have now, there is no way around, and I am quite sure you know it.
> 
> Also; there is nothing that says that you can't have different
> implementation under the hood. There is so much narrow-mindedness and
> assumptions from you. Instead of assuming bunch of what I think or don't
> think, as you always do, why don't you just ask me? I didn't answered
> further in our private correspondence because of your constant assuming
> what I think or don't think and so on. Ask me instead; if I haven't think
> of something, or thought wrong, I'll be glad to learn about it.
> 
> > should be based on different, more modern architectural decisions.
> > Handling of buffer text, GC, the display engine, threads -- all these
> > and more needs to be rethought and preferably replaced with more
> > modern solutions, to avoid bumping into the same limitations right
> > from the get-go.
> 
> Yes, and what in your opinion *is* a suggestion to completely remove the
> C code, which actually was a very relevant in a thread about shrinking
> the C core? Also, to answer all those who can't put 1 + 1 togther by
> themselves: I have suggested to remove completely C core in a thread
> about shrinking the C core. I think a "maximal shrink" of C core is
> quite on topic :-).
> 
> About your "neo-design", just implementing the editor in a Lisp machine
> instead of having a Lisp machine in the editor is itself already a
> radical design change. Not to mention that the Lisp machine suggested
> already has threading and some other tools that would make life much
> easier so you could concentrate on actually improving the editor instead
> of the Lisp machine. Look at other similar applications already doing it
> in that "clumsy" CL; Lem already has several rendering backends. How
> many do you have?
> 
> Nobody says that you have to implement stuff under the hood the same
> way; I have said we need C core and elisp semantics implemented in
> CL. It is laborous but possible. Under the hood it could be implemented
> with any changes needed, and in the future design could be exchanged for
> something else, complemented etc.
> 
> Anyway, your rhetorics give allusion that Emacs is dead, users should go
> to greener pastures you who want something more modern dead suites their
> needs. I don't know, but those are vibes I get from your arguments.
> 
> Anyone can *already* use other "more modern applications". Reason users
> don't use Hemlock or Climax or don't rush to Lem (perhaps they should)
> is, as we have already discussed in private and Reddit, because they
> can't run Emacs packages in them. People don't want Emacs-like editor,
> they want GNU Emacs. Which is good, and which you are aware of from both
> private discussion and Reddit. Sure, loosing *some* applications is OK,
> there is a natural regression too, some things are not maintained or get
> irrelevant for other reasons, but not all.

I don't know how to respond to this tirade, nor even where did it came
from and what did I do to deserve such rudeness.

I expressed my opinions on what would be a worthwhile "rewrite" of
Emacs.  Feel free to disagree with what I said, but why do you have to
mention my alleged "narrow-mindedness", or accuse me in making some
assumptions about what you think, or claim that I think Emacs is dead?
What I wrote was about Emacs and its future, not about you.  And no, I
don't think Emacs is dead, or I wouldn't be wasting my free time on
this job.

> Another thing is your way to talk to people and keep this community; I
> was told I am a lazy idiot who came here to beg someone to write
> something for me, and I am told to watch my mouth when I answered. Great
> community you are maintaining, thank you for that.

You are mistaken: I don't maintain this community.  I can barely tell
people to use kinder words, and even then they don't always listen.

Re: [External] : Re: Shrinking the C core

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Arthur Miller <arthur.miller@live.com>
>> Cc: Gerd Möllmann <gerd.moellmann@gmail.com>,
>>   esr@thyrsus.com,
>>   rms@gnu.org,  drew.adams@oracle.com,  acm@muc.de,  luangruo@yahoo.com,
>>   emacs-tangents@gnu.org
>> Date: Tue, 12 Sep 2023 21:58:43 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > [Redirected to emacs-tangents]
>> >
>> >> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>> >> Cc: Richard Stallman <rms@gnu.org>,  Drew Adams <drew.adams@oracle.com>,
>> >>  arthur.miller@live.com,  acm@muc.de,  luangruo@yahoo.com,
>> >>  emacs-devel@gnu.org
>> >> Date: Tue, 12 Sep 2023 06:38:00 +0200
>> >> 
>> >> "Eric S. Raymond" <esr@thyrsus.com> writes:
>> >> 
>> >>  > But it could be done. There is a technical path forward to it.
>> >> 
>> >> Which would have to cope with buffer-local bindings.
>> >
>> > Right.  And the display code.  And text en/decoding.  And input queue.
>> > And faces.  Etc. etc. -- these are all written in C, but are full of
>> > Lisp data structures and calls into Lisp, so separating them is
>> > anything but easy or even straightforward.
>> >
>> > People who have enough talents, knowledge, and energy to do this kind
>> > of job will be better off, and will serve the community better, if
>> > they design an Emacs differently, taking into consideration the main
>> > lessons we've learned.
>> 
>> I don't know what you have learned, but I have learned that Guy Steel
>> was correct when he told the world back in 1998 already: don't really on
>> a few people for the development, instead make things extensible by
>> the users. In an applicaiton that is almost a life style for many users
>> and where most of users are developers as well it makes sense to use the
>> same implementation and extension language, because it removes a barrier
>> for design and experimentation. For web developers it might make sense
>> to write their tools in JS, for Lisp developers it make sense to use
>> Lisp for their tools. That would make for more sustainable development.
>> 
>> > lessons we've learned.  That "neo-Emacs" could have a mode whereby it
>> > worked in a way compatible with the current Emacs, so that people who
>> > must run the old applications could run them without changes.  But it
>> 
>> Such a "mode" would still require you to implement al that stuff you
>> have now, there is no way around, and I am quite sure you know it.
>> 
>> Also; there is nothing that says that you can't have different
>> implementation under the hood. There is so much narrow-mindedness and
>> assumptions from you. Instead of assuming bunch of what I think or don't
>> think, as you always do, why don't you just ask me? I didn't answered
>> further in our private correspondence because of your constant assuming
>> what I think or don't think and so on. Ask me instead; if I haven't think
>> of something, or thought wrong, I'll be glad to learn about it.
>> 
>> > should be based on different, more modern architectural decisions.
>> > Handling of buffer text, GC, the display engine, threads -- all these
>> > and more needs to be rethought and preferably replaced with more
>> > modern solutions, to avoid bumping into the same limitations right
>> > from the get-go.
>> 
>> Yes, and what in your opinion *is* a suggestion to completely remove the
>> C code, which actually was a very relevant in a thread about shrinking
>> the C core? Also, to answer all those who can't put 1 + 1 togther by
>> themselves: I have suggested to remove completely C core in a thread
>> about shrinking the C core. I think a "maximal shrink" of C core is
>> quite on topic :-).
>> 
>> About your "neo-design", just implementing the editor in a Lisp machine
>> instead of having a Lisp machine in the editor is itself already a
>> radical design change. Not to mention that the Lisp machine suggested
>> already has threading and some other tools that would make life much
>> easier so you could concentrate on actually improving the editor instead
>> of the Lisp machine. Look at other similar applications already doing it
>> in that "clumsy" CL; Lem already has several rendering backends. How
>> many do you have?
>> 
>> Nobody says that you have to implement stuff under the hood the same
>> way; I have said we need C core and elisp semantics implemented in
>> CL. It is laborous but possible. Under the hood it could be implemented
>> with any changes needed, and in the future design could be exchanged for
>> something else, complemented etc.
>> 
>> Anyway, your rhetorics give allusion that Emacs is dead, users should go
>> to greener pastures you who want something more modern dead suites their
>> needs. I don't know, but those are vibes I get from your arguments.
>> 
>> Anyone can *already* use other "more modern applications". Reason users
>> don't use Hemlock or Climax or don't rush to Lem (perhaps they should)
>> is, as we have already discussed in private and Reddit, because they
>> can't run Emacs packages in them. People don't want Emacs-like editor,
>> they want GNU Emacs. Which is good, and which you are aware of from both
>> private discussion and Reddit. Sure, loosing *some* applications is OK,
>> there is a natural regression too, some things are not maintained or get
>> irrelevant for other reasons, but not all.
>
> I don't know how to respond to this tirade, nor even where did it came
> from and what did I do to deserve such rudeness.

Sory if it came out as a tirade, it answer on the mail before, your
thoughts in that mail, and also me complaining about some persons taking
very much freedom in how they talk to or about me, and when answered
back I am threatened to be blocked from here. It perhaps should have
been three mails instead of one, sorry if it is too diffused or
confused, it was the intention to be rude to you.

> I expressed my opinions on what would be a worthwhile "rewrite" of
> Emacs.  Feel free to disagree with what I said, but why do you have to
> mention my alleged "narrow-mindedness", or accuse me in making some
> assumptions about what you think, or claim that I think Emacs is dead?
> What I wrote was about Emacs and its future, not about you.  And no, I
> don't think Emacs is dead, or I wouldn't be wasting my free time on
> this job.
>
>> Another thing is your way to talk to people and keep this community; I
>> was told I am a lazy idiot who came here to beg someone to write
>> something for me, and I am told to watch my mouth when I answered. Great
>> community you are maintaining, thank you for that.
>
> You are mistaken: I don't maintain this community.  I can barely tell
> people to use kinder words, and even then they don't always listen.

Well you can; but you didn't.

I am forced to change the mailing list while other people are not, and I
am told to "dropp of" and "watch my mouth or risk to be blocked (I
believe what he meant)" while called "for lazy idiot" becaue of bringing
up what someone found uncomfortable to talk about, and I don't see that
person being threaten the way I am; and I am not able to bring that up,
than I guess I am not welcome here.

Re: [External] : Re: Shrinking the C core

[Po Lu]

Arthur Miller <arthur.miller@live.com> writes:

> Sory if it came out as a tirade, it answer on the mail before, your
> thoughts in that mail, and also me complaining about some persons taking
> very much freedom in how they talk to or about me, and when answered
> back I am threatened to be blocked from here. It perhaps should have
> been three mails instead of one, sorry if it is too diffused or
> confused,

No one ever threatened to block you from the list.

> it was the intention to be rude to you.

[...]

> I am forced to change the mailing list while other people are not, and I
> am told to "dropp of" and "watch my mouth or risk to be blocked (I
> believe what he meant)" while called "for lazy idiot" becaue of bringing
> up what someone found uncomfortable to talk about, and I don't see that
> person being threaten the way I am; and I am not able to bring that up,
> than I guess I am not welcome here.

Au contraire, everyone has been asked to transfer this thread to
emacs-tangents, not only you.  The important detail being that others
have long since grown weary of said thread, electing to direct their
attention to things more worthwhile instead.

Additionally, you were never labeled a ``lazy idiot'', and furthermore
never threatened with ``blocking''.

Would you please dispense with the self-victimizing?  If anything, the
only persistently aggressive participant has been you, perpetually
vituperating against your interlocutors with accusations of censorship
and sectarianism whilst demonstrating precisely those traits that you
claim to reject.

With that said, feel free to continue posting long diatribes advocating
rewrites of Emacs in Common Lisp, now that the thread has been moved off
emacs-devel.  But please remove me from the carbon copy list, and mind
that talking never does anything in Emacs: it never did, and it never
will.

TIA.

Re: [External] : Re: Shrinking the C core

[Arthur Miller]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> I implemented Common Lisp for the Lisp Machine in 1983.
> That included the generic sequence functions with their
> many keyword arguments.
>
> In 1984 I wrote Emacs Lisp and decided to omit all that.
> I have no regrets.  Their absence makes Emacs Lisp easier to use.

Thank you for explanation and clarficiation. Now at least I understand
why you don't like them.

> Keyword arguments are fine for an unusual, heavyweight operation such
> as creating a frame.  The sort of function whose doc you need to look
> up again each time you use it.  When something is as cumbersome as
> that, the question of how to pass the arguments is a pile of minor
> details that you'll see when you look it up.
>
> But it's a pain in the neck to make simple everyday functions such as
> `append' and `length' that cumbersome just for the sake of following a
> general system.  Part of this system is that the default argument
> values follow the system, rather than following the traditional Lisp
> functions -- so getting the traditional behavior requires actually
> specifying the keyword arguments to override defaults, almost every
> time.

Of course, but nobody suggests it is all-in. `length' is not a keyworded
in CL either. It would be madness to do something like naming all
arguments in all functions. But it is convenient to have keywords in
some places, like for example in define-minor-mode or make-process, or
even "new" define-keymap.

> Switching to Common Lisp would cause those painful incompatibilities.
> It is unacceptable.
>
> Defining the CL functions with a name prefix is ok, because the
> traditional Lisp functions are still available compatibly and we don't
> have to use the CL functions.  This is what we have now.

If cl-lib was loaded into the lisp image, we could even rename cl-defun
to defun, after we have initialized all the other stuff that goes into
the dump, and have "upgraded" vesion of defun and would be fully
compatible with traditional Lisp functions. We could even go further and
remove cl- prefix from bunch of symbols to make them less cumbersome to
use, and everything would stil be compatible with traditional Lisp
functions.

I think there is a thing that makes Lisp a bit special compared to other
programming languages, and Steel mentions it briefly in his legendary
talk: anyone can add a new feature to the language and it becomes part
of the vocabulary without distinction from the built-in stuff. Keyword
arguments does not require anything special from the C runtime, and we
see them pop-up in different forms in elisp. Perhaps elisp would be
better off to accept them and make cl-defun the norm, instead of seing
macros like define-minor-mode re-implement the technique and some other
macro possibly do something similar which leads to more less maintanable
code.

But that is just loud thinking, since my point wasn't to suggest any
changes, I just wanted to understand why you didn't like keyword
arguments per se, so thank you very much for the answer.

Re: [External] : Re: Shrinking the C core

[Emanuel Berg]

We should extend the current design so that Emacs is
multi-threaded, for this to happen we should agree on a model
how this is supposed to be done, and after that it is just
a matter what needs to be done and what parts need to be
adopted to realize that model.

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: Shrinking the C core

[Emanuel Berg]

Arthur Miller wrote:

> Of course, but nobody suggests it is all-in. `length' is not
> a keyworded in CL either. It would be madness to do
> something like naming all arguments in all functions. But it
> is convenient to have keywords in some places [...]

Keywords typically make for shorter interfaces and function
calls that communicate more, however the shorter interface
advantage and the need to have function calls to communicate
more should be put against the typical downside with this
style, which is very long and complicated functions that do
a lot more than they should, often.

And, because very long functions isn't good style anyway,
those should probably be split up - and if they are split up,
the interfaces will be shorter and the style clear that way
instead, so the need for keywords to do that will also be
reduced or disappear, really.

>> Defining the CL functions with a name prefix is ok, because
>> the traditional Lisp functions are still available
>> compatibly and we don't have to use the CL functions.
>> This is what we have now.
>
> If cl-lib was loaded into the lisp image, we could even
> rename cl-defun to defun, after we have initialized all the
> other stuff that goes into the dump, and have "upgraded"
> vesion of defun and would be fully compatible with
> traditional Lisp functions. We could even go further and
> remove cl- prefix from bunch of symbols to make them less
> cumbersome to use, and everything would stil be compatible
> with traditional Lisp functions.

Indeed, that could be done for every cl-X case where X is
a non-cl-lib Elisp function and cl-X already does everything
X does with the same interface. Those could be cancelled out.

For the cases where there is no such X outside of cl-lib, e.g.
`cl-incf' and `cl-decf' (no "incf" or "decf" anymore?) the
cl-lib prefix could also be dropped. Another example is
`cl-loop'. One could solve this with aliases, possibly.

Because yes, it looks a bit strange that such rudimentary
stuff has to be prefixed to note their belonging to some
particular implementation or library, and even more so when
there isn't any competition or alternative, even?

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: Shrinking the C core

[Tomas Hlavaty]

On Tue 12 Sep 2023 at 04:30, Arthur Miller <arthur.miller@live.com> wrote:
> Perhaps elisp would be better off to accept them and make cl-defun the
> norm, instead of seing macros like define-minor-mode re-implement the
> technique and some other macro possibly do something similar which
> leads to more less maintanable code.

cl-defmacro would not help with define-minor-mode because it cannot
describe the special argument list structure in this case.

Re: [External] : Re: Shrinking the C core

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Of course, but nobody suggests it is all-in. `length' is not a keyworded
  > in CL either.

Oops, I thought it was -- but the time when I implemented and used
Common Lisp was 40 years ago.

I am pretty sure `member' used keyword arguments, and I think that
getting behavior equivalent to traditional Lisp `member' required
specifying a keyword argument.  I never forgot that, during that
period, because I got reminded of it almost every day.

  >  But it is convenient to have keywords in
  > some places, like for example in define-minor-mode or make-process, or
  > even "new" define-keymap.

I would not object to using keyword arguments for functions like that
-- complex and cumbersome to use, and not used often.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Shrinking the C core

[Gerd Möllmann]

 >> AFAIK, all the Emacs-like implementations mentioned on the
 >> web page run on unmodified CL implementations, so you get
 >> the speed of that CL implementation.
 >
 > Uhm, what's an "unmodified CL implementation"?

The implemention as released by the project or vendor.

 >
 > And what implementations are those, you mean they are not as
 > fast as SBCL?

What the Emacs-alikes support varies.  I haven't checked recently, but I 
know that Lem, for instance, started by only supporting SBCL.  Hemlock 
is part of CMUCL and wasn't portable, hence Portable Hemlock.  There's 
also an Emacs-alike that is part of Lispworks' CL implementation 
(commencial).  And so on.  Please consult the project pages.

For some benchmarks on 10 CL implementations, please see

https://www.cliki.net/performance%20benchmarks

BTW, I get DNS errors when sending mail to you via Gmail. Don't know 
what's happening.

Reporting-MTA: dns; dataswamp.org

Final-Recipient: rfc822; incal@dataswamp.org
Action: failed

Re: Shrinking the C core

[Ihor Radchenko]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> BTW, I get DNS errors when sending mail to you via Gmail. Don't know 
> what's happening.

Same for me.

RE: [External] : Re: Shrinking the C core

[Drew Adams]

> > BTW, I get DNS errors when sending mail to you via Gmail. Don't know
> > what's happening.
> 
> Same for me.

Ditto.  And it's been pointed out before...

Re: [External] : Re: Shrinking the C core

[Emanuel Berg]

Drew Adams wrote:

>>> BTW, I get DNS errors when sending mail to you via Gmail.
>>> Don't know what's happening.
>> 
>> Same for me.
>
> Ditto. And it's been pointed out before ...

Seems reasonable as, in this thread alone, three people have
pointed it out.

No, I get your replies here and mails as well for that matter,
but, admittedly, I was fiddling with DNSSEC the other day, and
maybe I br0ke something. Investigating, but have no idea at
the moment what could have caused it.

-- 
underground experts united
https://dataswamp.org/~incal

Shrinking the C core

[Eric S. Raymond]

Recently I have been refamiliarizing myself with the Emacs C core.

Some days ago, as a test that I understand the C core API and the
current build recipe, I made and pushed a small commit that moved
the policy code in delete-file out to Lisp, basing it on a smaller
and simpler new entry point named delete-file-internal (this is
parallel to the way delete-directory is already partitioned).

I've since been poking around the C core code and am now wondering why
there is so much C-core code that seems like it could be pushed out to
Lisp.  For example, in src/fileio.c:

DEFUN ("unhandled-file-name-directory", Funhandled_file_name_directory,
       Sunhandled_file_name_directory, 1, 1, 0,
       doc: /* Return a directly usable directory name somehow associated with FILENAME.
A `directly usable' directory name is one that may be used without the
intervention of any file name handler.
If FILENAME is a directly usable file itself, return
\(file-name-as-directory FILENAME).
If FILENAME refers to a file which is not accessible from a local process,
then this should return nil.
The `call-process' and `start-process' functions use this function to
get a current directory to run processes in.  */)
  (Lisp_Object filename)
{
  Lisp_Object handler;

  /* If the file name has special constructs in it,
     call the corresponding file name handler.  */
  handler = Ffind_file_name_handler (filename, Qunhandled_file_name_directory);
  if (!NILP (handler))
    {
      Lisp_Object handled_name = call2 (handler, Qunhandled_file_name_directory,
					filename);
      return STRINGP (handled_name) ? handled_name : Qnil;
    }

  return Ffile_name_as_directory (filename);
}

Why is this in C? Is there any reason not to push it out to Lisp and
reduce the core complexity?  More generally, if I do this kind of
refactor, will I be stepping on anyone's toes?
--
						>>esr>>

Re: Shrinking the C core

[Po Lu]

"Eric S. Raymond" <esr@thyrsus.com> writes:

> Recently I have been refamiliarizing myself with the Emacs C core.
>
> Some days ago, as a test that I understand the C core API and the
> current build recipe, I made and pushed a small commit that moved
> the policy code in delete-file out to Lisp, basing it on a smaller
> and simpler new entry point named delete-file-internal (this is
> parallel to the way delete-directory is already partitioned).
>
> I've since been poking around the C core code and am now wondering why
> there is so much C-core code that seems like it could be pushed out to
> Lisp.  For example, in src/fileio.c:
>
> DEFUN ("unhandled-file-name-directory", Funhandled_file_name_directory,
>        Sunhandled_file_name_directory, 1, 1, 0,
>        doc: /* Return a directly usable directory name somehow associated with FILENAME.
> A `directly usable' directory name is one that may be used without the
> intervention of any file name handler.
> If FILENAME is a directly usable file itself, return
> \(file-name-as-directory FILENAME).
> If FILENAME refers to a file which is not accessible from a local process,
> then this should return nil.
> The `call-process' and `start-process' functions use this function to
> get a current directory to run processes in.  */)
>   (Lisp_Object filename)
> {
>   Lisp_Object handler;
>
>   /* If the file name has special constructs in it,
>      call the corresponding file name handler.  */
>   handler = Ffind_file_name_handler (filename, Qunhandled_file_name_directory);
>   if (!NILP (handler))
>     {
>       Lisp_Object handled_name = call2 (handler, Qunhandled_file_name_directory,
> 					filename);
>       return STRINGP (handled_name) ? handled_name : Qnil;
>     }
>
>   return Ffile_name_as_directory (filename);
> }
>
> Why is this in C? Is there any reason not to push it out to Lisp and
> reduce the core complexity?

There is a plenitude of such reasons.  Whenever some code is moved to
Lisp, its structure and history is lost.  Often, comments within the
extracted C code remain, but the code itself is left ajar.  Bootstrap
problems are frequently introduced, as well as latent bugs.  And Emacs
becomes ever so much slower.

These are not simply theoretical concerns, but problems I've encountered
many times in practice.  Compounding that, fileio.c is abundant with
complex logic amended and iteratively refined over many years, which is
dangerously prone to loss or mistranscription during a refactor or a
rewrite.

Finally, this specific case is because we don't want to provide Lisp
with an easy means to bypass file name handlers.  All primitives
operating on file names should thus consult file name handlers, enabling
packages like TRAMP to continue operating correctly.

> More generally, if I do this kind of refactor, will I be stepping on
> anyone's toes?

Probably.  I think a better idea for a first project is this item in
etc/TODO:

** A better display of the bar cursor
Distribute a bar cursor of width > 1 evenly between the two glyphs on
each side of the bar (what to do at the edges?).

which has been on my plate for a while.  I would be grateful to anyone
who decides to preempt me.

Thanks in advance!

Re: Shrinking the C core

[Eric S. Raymond]

Po Lu <luangruo@yahoo.com>:
> There is a plenitude of such reasons.  Whenever some code is moved to
> Lisp, its structure and history is lost.  Often, comments within the
> extracted C code remain, but the code itself is left ajar.  Bootstrap
> problems are frequently introduced, as well as latent bugs.  And Emacs
> becomes ever so much slower.

When I first worked on Emacs code in the 1980s Lisp was already fast
enough, and machine speeds have gone up by something like 10^3 since.
I plain don't believe the "slower" part can be an issue on modern
hardware, not even on tiny SBCs. 

> Finally, this specific case is because we don't want to provide Lisp
> with an easy means to bypass file name handlers.  All primitives
> operating on file names should thus consult file name handlers, enabling
> packages like TRAMP to continue operating correctly.

If calling the file-name handlers through Lisp is a crash landing,
you were already out of luck.  Go have a look at delete-directory.

> Probably.  I think a better idea for a first project is this item in
> etc/TODO:

This would ... not be my first project. :-)
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Shrinking the C core

[Po Lu]

"Eric S. Raymond" <esr@thyrsus.com> writes:

> When I first worked on Emacs code in the 1980s Lisp was already fast
> enough, and machine speeds have gone up by something like 10^3 since.
> I plain don't believe the "slower" part can be an issue on modern
> hardware, not even on tiny SBCs.

Can you promise the same, if your changes are not restricted to one or
two functions in fileio.c, but instead pervade throughout C source?

Finally, you haven't addressed the remainder of the reasons I itemized.

Re: Shrinking the C core

[Eric S. Raymond]

Po Lu <luangruo@yahoo.com>:
> "Eric S. Raymond" <esr@thyrsus.com> writes:
> 
> > When I first worked on Emacs code in the 1980s Lisp was already fast
> > enough, and machine speeds have gone up by something like 10^3 since.
> > I plain don't believe the "slower" part can be an issue on modern
> > hardware, not even on tiny SBCs.
> 
> Can you promise the same, if your changes are not restricted to one or
> two functions in fileio.c, but instead pervade throughout C source?

Yes, in fact, I can. Because if by some miracle we were able to
instantly rewrite the entirety of Emacs in Python (which I'm not
advocating, I chose it because it's the slowest of the major modern
scripting languages) basic considerations of clocks per second would
predict it to run a *dead minimum* of two orders of magnitude faster
than the Emacs of, say, 1990.

And 1990 Emacs was already way fast enough for the human eye and
brain, which can't even register interface lag of less than 0.17
seconds (look up the story of Jef Raskin and how he exploited this
psychophysical fact in the design of the Canon Cat sometime; it's very
instructive). The human auditory system can perceive finer timeslices,
down to about 0.02s in skilled musicians, but we're not using elisp
for audio signal processing.

If you take away nothing else from this conversation, at least get it
through your head that "more Lisp might make Emacs too slow" is a
deeply, *deeply* silly idea. It's 2023 and the only ways you can make
a user-facing program slow enough for response lag to be noticeable
are disk latency on spinning rust, network round-trips, or operations
with a superlinear big-O in critical paths.  Mere interpretive overhead
won't do it.

> Finally, you haven't addressed the remainder of the reasons I itemized.

They were too obvious, describing problems that competent software
engineers know how to prevent or hedge against, and you addressed me
as though I were a n00b that just fell off a cabbage truck. My
earliest contributions to Emacs were done so long ago that they
predated the systematic Changelog convention; have you heard the
expression "teaching your grandmother to suck eggs"?  My patience for
that sort of thing is limited.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Shrinking the C core

[Dmitry Gutov]

On 10/08/2023 04:19, Eric S. Raymond wrote:
> basic considerations of clocks per second would
> predict it to run a*dead minimum*  of two orders of magnitude faster
> than the Emacs of, say, 1990.

In addition to the examples made by others, I'll say that the sizes of 
software projects have increased from 1990 as well. So if you have a 
Lisp routine that simply enumerates the files in one project, it has to 
do proportionally more work.

 > And 1990 Emacs was already way fast enough for the human eye and
 > brain, which can't even register interface lag of less than 0.17
 > seconds (look up the story of Jef Raskin and how he exploited this
 > psychophysical fact in the design of the Canon Cat sometime; it's very
 > instructive). The human auditory system can perceive finer timeslices,
 > down to about 0.02s in skilled musicians, but we're not using elisp
 > for audio signal processing.

I've had to expend significant effort on many occasions to keep various 
command execution times below 0.17, or 0.02, or etc.

Which is to say, while I'm very much in favor of the "lean core" concept 
myself, we should accompany far-reaching changes like that with 
appropriate benchmarking.

Re: Shrinking the C core

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

There are occasiona when it is useful, for added flexibility,
to move some function from C to Lisp.  However, stability
is an important goal for Emacs, so we should not even try
to move large amounts of code to Lisp just for the sake
of moving code to Lisp.

The rate at whic we have added features already causes significant
instability.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Shrinking the C core

[Emanuel Berg]

Richard Stallman wrote:

> There are occasiona when it is useful, for added
> flexibility, to move some function from C to Lisp. However,
> stability is an important goal for Emacs, so we should not
> even try to move large amounts of code to Lisp just for the
> sake of moving code to Lisp.

It is just cool that Emacs is written in two languages, it is
like to continents or something connected by bridges
overlapping certain areas.

Maybe we can use SBCL to compile Elisp, and after that
integrate it into Emacs so you would be able to run all old
Elisp without changing the code, only now it would in fact be
Common Lisp implementing Elisp, i.e. the opposite of our
cl-lib (`cl-loop' etc).

Maybe some normalization efforts would have to be done to the
syntax here and there as preparatory steps ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: Shrinking the C core

[tomas]

[-- Attachment #1: Type: text/plain, Size: 235 bytes --]

On Sat, Aug 12, 2023 at 05:22:45AM +0200, Emanuel Berg wrote:

> Maybe we can use SBCL to compile Elisp, and after that
> integrate it into Emacs [...]

- https://www.cliki.net/CL-Emacs
- https://xkcd.com/927/

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Shrinking the C core

[Emanuel Berg]

tomas wrote:

> https://xkcd.com/927/

  HOW STANDARDS PROLIFERATE (See: A C chargers, character
  encodings, instant messaging, etc.) SITUATION: There are 14
  competing standards. Geek: 14?! Ridiculous! We need to
  develop one universal standard that covers everyone's use
  cases. Fellow Geek: Yeah! Soon: SITUATION: There are 15
  competing standards. {{Title text: Fortunately, the charging
  one has been solved now that we've all standardized on
  mini-USB. Or is it micro-USB? Shit.}}

Okay, but here it isn't about joining the CL standard, it is
the situation that we have "the Lisp editor" yet our Lisp is
much slower than other people's Lisp, and for no good reason
what I can understand as Emacs is C, and SBCL is C. What's the
difference, why is one so much faster than the other?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Shrinking the C core

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1060 bytes --]

On Sun, Aug 13, 2023 at 01:09:38AM +0200, Emanuel Berg wrote:
> tomas wrote:
> 
> > https://xkcd.com/927/

Try a bit of lateral thinking: what if, before embarking into
a "let's do everything new, the old sucks" kind of thing those
proposing it would, at least, have a look at the attempts made
in this direction, and, you know, try to learn about why no
one of them took over the house?

This might yield a more interesting attempt.

As I see it, the main challenge for an Emacs maintainer isn't
that it is software, nor that it is a big, complex piece of
software. Rather, that its community is huge and diverse. Folks
are using it in extremely different ways (in part due to the
project's age), and moving something will break one usage dating
back to 1997 or something.

Still moving forward is a little wonder, and I'm genuinely in
awe of Eli's job (although I'm not happy about each and every
of his decisions, but I think that'll happen to everyone, due
to the situation sketched above and is thus part of it).

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

RE: [External] : Re: Shrinking the C core

[Drew Adams]

> Still moving forward is a little wonder,
> and I'm genuinely in awe of Eli's job
>  (although I'm not happy about each and
> every [one] of his decisions, but I think
> that'll happen to everyone...

+1.

Re: Shrinking the C core

[Andreas Schwab]

On Aug 13 2023, Emanuel Berg wrote:

> Okay, but here it isn't about joining the CL standard, it is
> the situation that we have "the Lisp editor" yet our Lisp is
> much slower than other people's Lisp, and for no good reason
> what I can understand as Emacs is C, and SBCL is C.

But SBCL is not portable.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Shrinking the C core

[Emanuel Berg]

Andreas Schwab wrote:

>> Okay, but here it isn't about joining the CL standard, it
>> is the situation that we have "the Lisp editor" yet our
>> Lisp is much slower than other people's Lisp, and for no
>> good reason what I can understand as Emacs is C, and SBCL
>> is C.
>
> But SBCL is not portable.

Step one would be to identify why SBCL is so much faster.
Say it is faster for reasons A, B, C and D. Surely A, B, C and
D are not all unportable features, so one would first try add
the same thing to the Elisp model.

If that fails maybe one would consider ECL or some other
faster, yet portable solution ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: Shrinking the C core

[Alfred M. Szmidt]

   Step one would be to identify why SBCL is so much faster.

The short reason why SBCL is faster is that SBCL uses lots of type
checking and interference, which allows it to figure out optimizations
better.  And then using assembly to implement such optimizations.  CL
code compiled without DECLARE/DECLAIM is generally slow, but can be
made fast if you are very careful, and lucky that the things you need
are implemented as VOPs -- which are entierly unportable.

It also has the luxury of using a language that is set more or less in
stone, that doesn't change constantly.  Which means the developers can
spend quite a bit more time on optimizing what they have.

   If that fails maybe one would consider ECL or some other
   faster, yet portable solution ...

"Porting" Emacs to CL is not just slap ECL or some CL implementation
on top.  Adding type checking of the sort that SBCL does, would be
possible but ALOT of work.  It would most definitly require a entierly
new VM, and with that comes dragons.

Re: Shrinking the C core

[Ihor Radchenko]

"Alfred M. Szmidt" <ams@gnu.org> writes:

>    Step one would be to identify why SBCL is so much faster.
>
> The short reason why SBCL is faster is that SBCL uses lots of type
> checking and interference, which allows it to figure out optimizations
> better.  And then using assembly to implement such optimizations.  CL
> code compiled without DECLARE/DECLAIM is generally slow, but can be
> made fast if you are very careful, and lucky that the things you need
> are implemented as VOPs -- which are entierly unportable.

AFAIK, Elisp is full of type checks (all these BUFFERP/CHECK_STRING in
C code). Also, AFAIR, native-comp is making use of some of the type
checks as well, allowing some extra optimizations.

So, there might be some room for improvement after all.

Do you have some references detailing what SBCL does?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Shrinking the C core

[Alfred M. Szmidt]

   AFAIK, Elisp is full of type checks (all these BUFFERP/CHECK_STRING in
   C code). Also, AFAIR, native-comp is making use of some of the type
   checks as well, allowing some extra optimizations.

SBCL does far more detailed checks than that.  Which is why it makes
it very unportable, since the base case is trivial, but the optimized
one is not.  Check for example FLOOR in sbcl/src/code/numbers.lisp ,
and compare it to FLOOR in Emacs Lisp.

SBCL has many luxuries that Emacs does not have.

   So, there might be some room for improvement after all.

There is always a door for that ... but someone needs to open it.

   Do you have some references detailing what SBCL does?

http://www.sbcl.org/sbcl-internals/  and the source code.

Re: Shrinking the C core

[Emanuel Berg]

Alfred M. Szmidt wrote:

> one is not. Check for example FLOOR in
> sbcl/src/code/numbers.lisp , and compare it to FLOOR in
> Emacs Lisp.

Are we talking `floor' as in (floor 3.1415) ; 3 ?

How do we make a benchmark for that? Run it 1000+ times for
random floats? But maybe SBCL has faster random as well!

Actually, even I have a better random than Elisp:

  https://dataswamp.org/~incal/emacs-init/random-urandom/

It is more random than `random', using the Linux random.
But I didn't compare them in terms of speed. It is a so called
dynamic module, that is, not built-in but still written in C.

Anyway, I'd be happy to add something `floor' to the
benchmarks, for sure!

-- 
underground experts united
https://dataswamp.org/~incal

Re: Shrinking the C core

[Ihor Radchenko]

Emanuel Berg <incal@dataswamp.org> writes:

> Are we talking `floor' as in (floor 3.1415) ; 3 ?
>
> How do we make a benchmark for that? Run it 1000+ times for
> random floats? But maybe SBCL has faster random as well!

You can generate random number sequence first, excluding it from the
benchmark, and then benchmark mapping #'floor on the already generated
sequence. Ideally, use the same sequence for both Elisp and SBCL.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Shrinking the C core

[Emanuel Berg]

Ihor Radchenko wrote:

>> Are we talking `floor' as in (floor 3.1415) ; 3 ?
>>
>> How do we make a benchmark for that? Run it 1000+ times for
>> random floats? But maybe SBCL has faster random as well!
>
> You can generate random number sequence first, excluding it
> from the benchmark, and then benchmark mapping #'floor on
> the already generated sequence. Ideally, use the same
> sequence for both Elisp and SBCL.

You are right, good idea. But maybe it is already known why
floor is slower in Elisp than SBCL?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Shrinking the C core

[Ihor Radchenko]

Emanuel Berg <incal@dataswamp.org> writes:

>> You can generate random number sequence first, excluding it
>> from the benchmark, and then benchmark mapping #'floor on
>> the already generated sequence. Ideally, use the same
>> sequence for both Elisp and SBCL.
>
> You are right, good idea. But maybe it is already known why
> floor is slower in Elisp than SBCL?

The discussion about floor started from Alfred using `floor' as an
example where CL uses system-dependent optimizations and thus being much
faster. https://yhetil.org/emacs-devel/E1qVSLD-00079S-Gg@fencepost.gnu.org/ 

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Shrinking the C core

[Emanuel Berg]

Ihor Radchenko wrote:

>>> You can generate random number sequence first, excluding
>>> it from the benchmark, and then benchmark mapping #'floor
>>> on the already generated sequence. Ideally, use the same
>>> sequence for both Elisp and SBCL.
>>
>> You are right, good idea. But maybe it is already known why
>> floor is slower in Elisp than SBCL?
>
> The discussion about floor started from Alfred using `floor'
> as an example where CL uses system-dependent optimizations
> and thus being much faster.

So the answer to the question, Why is SBCL faster?
is "optimizations". And the answer to the question, Why don't
we have those optimizations? is "they are not portable"?

But isn't that what we do already with compilation and in
particular native compilation, why can't that add
optimizations for the native system?

Some commands/Elisp on that (compilation and native
compilation) as a side note, maybe someone finds them
entertaining:

  https://dataswamp.org/~incal/conf/.zsh/install-emacs
  https://dataswamp.org/~incal/emacs-init/native.el

-- 
underground experts united
https://dataswamp.org/~incal

Re: Shrinking the C core

[Ihor Radchenko]

Emanuel Berg <incal@dataswamp.org> writes:

>> The discussion about floor started from Alfred using `floor'
>> as an example where CL uses system-dependent optimizations
>> and thus being much faster.
>
> So the answer to the question, Why is SBCL faster?
> is "optimizations". And the answer to the question, Why don't
> we have those optimizations? is "they are not portable"?

Looking at
https://github.com/sbcl/sbcl/blob/master/src/code/numbers.lisp#L390,
they employ certain x86-64-, x86-, ppc64-specific optimizations.

Althrough, Elisp's rounding_driver is not too bad actually. It also does
shortcuts depending on the argument type.

AFAIU, the main difference in SBCL vs. Elisp is that Elisp type checks
are often called repetitively on the same values. Even though the checks
are rather fast (typecheck is usually just a single xor + equal
operation), repetitive calls do add up.

And even this is not a definitive answer. I do not think that we can
point out a single reason why SBCL is faster. I am not even sure if SBCL
is _always_ faster.

> But isn't that what we do already with compilation and in
> particular native compilation, why can't that add
> optimizations for the native system?

If we talk about type checking, Elisp uses dynamic typing and
compilation cannot do much about it. Native compilation also does not
touch C subroutines - the place where typechecks are performed.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Shrinking the C core

[Alfred M. Szmidt]

   And even this is not a definitive answer. I do not think that we can
   point out a single reason why SBCL is faster. I am not even sure if SBCL
   is _always_ faster.

Always faster than what?  What are you comparing?  SBCL is a compiler,
Emacs is more than that.  

It should be quite obvious why SBCL is faster than the Emacs Lisp VM
(or even native).  Just look at this call to (car "foo"), and compare
what happens in Emacs.

* (disassemble 'foo)
; disassembly for FOO
; Size: 166 bytes. Origin: #x225D873F                         ; FOO
; 3F:       488B042590060020 MOV RAX, [#x20000690]
; 47:       488945F8         MOV [RBP-8], RAX
; 4B:       48892C2560060020 MOV [#x20000660], RBP
; 53:       488B142518000020 MOV RDX, [#x20000018]
; 5B:       488D4210         LEA RAX, [RDX+16]
; 5F:       483B042520000020 CMP RAX, [#x20000020]
; 67:       7770             JA L2
; 69:       4889042518000020 MOV [#x20000018], RAX
; 71: L0:   488B0570FFFFFF   MOV RAX, [RIP-144]               ; "foo"
; 78:       488902           MOV [RDX], RAX
; 7B:       48C7420817010020 MOV QWORD PTR [RDX+8], #x20000117  ; NIL
; 83:       80CA07           OR DL, 7
; 86:       48312C2560060020 XOR [#x20000660], RBP
; 8E:       7402             JEQ L1
; 90:       CC09             INT3 9                           ; pending interrupt trap
; 92: L1:   4C8D4424F0       LEA R8, [RSP-16]
; 97:       4883EC30         SUB RSP, 48
; 9B:       BFAF0B1520       MOV EDI, #x20150BAF              ; 'LIST
; A0:       488B3551FFFFFF   MOV RSI, [RIP-175]               ; '(VALUES
                                                              ;   (SIMPLE-ARRAY ..))
; A7:       488B0552FFFFFF   MOV RAX, [RIP-174]               ; '("foo")
; AE:       498940F0         MOV [R8-16], RAX
; B2:       488B054FFFFFFF   MOV RAX, [RIP-177]               ; "(CAR \"foo\")"
; B9:       498940E8         MOV [R8-24], RAX
; BD:       49C740E017010020 MOV QWORD PTR [R8-32], #x20000117  ; NIL
; C5:       B90C000000       MOV ECX, 12
; CA:       498928           MOV [R8], RBP
; CD:       498BE8           MOV RBP, R8
; D0:       B882B12620       MOV EAX, #x2026B182              ; #<FDEFN SB-C::%COMPILE-TIME-TYPE-ERROR>
; D5:       FFD0             CALL RAX
; D7:       CC10             INT3 16                          ; Invalid argument count trap
; D9: L2:   6A10             PUSH 16
; DB:       FF1425B0080020   CALL [#x200008B0]                ; #x21A00540: LIST-ALLOC-TRAMP
; E2:       5A               POP RDX
; E3:       EB8C             JMP L0
NIL
* 

   > But isn't that what we do already with compilation and in
   > particular native compilation, why can't that add
   > optimizations for the native system?

   If we talk about type checking, Elisp uses dynamic typing and
   compilation cannot do much about it. Native compilation also does not
   touch C subroutines - the place where typechecks are performed.

SBCL implements a Lisp, Lisp by definition is dynamically typed. 

Re: Shrinking the C core

[Emanuel Berg]

Alfred M. Szmidt wrote:

>    And even this is not a definitive answer. I do not think
>    that we can point out a single reason why SBCL is faster.
>    I am not even sure if SBCL is _always_ faster.
>
> Always faster than what? What are you comparing?

We are working on it.

> SBCL is a compiler, Emacs is more than that.

Including SBCL with SLIME, but that would still be CL with
SBCL and not Elisp which is what we are (not) comparing with.

> It should be quite obvious why SBCL is faster than the Emacs
> Lisp VM (or even native). Just look at this call to (car
> "foo"), and compare what happens in Emacs.
>
> * (disassemble 'foo)
> ; disassembly for FOO
> ; Size: 166 bytes. Origin: #x225D873F                         ; FOO
> ; 3F:       488B042590060020 MOV RAX, [#x20000690]
> ; 47:       488945F8         MOV [RBP-8], RAX
> ; 4B:       48892C2560060020 MOV [#x20000660], RBP
> ; 53:       488B142518000020 MOV RDX, [#x20000018]
> ; 5B:       488D4210         LEA RAX, [RDX+16]
> ; 5F:       483B042520000020 CMP RAX, [#x20000020]
> ; 67:       7770             JA L2
> ; 69:       4889042518000020 MOV [#x20000018], RAX
> ; 71: L0:   488B0570FFFFFF   MOV RAX, [RIP-144]               ; "foo"
> ; 78:       488902           MOV [RDX], RAX
> ; 7B:       48C7420817010020 MOV QWORD PTR [RDX+8], #x20000117  ; NIL
> ; 83:       80CA07           OR DL, 7
> ; 86:       48312C2560060020 XOR [#x20000660], RBP
> ; 8E:       7402             JEQ L1
> ; 90:       CC09             INT3 9                           ; pending interrupt trap
> ; 92: L1:   4C8D4424F0       LEA R8, [RSP-16]
> ; 97:       4883EC30         SUB RSP, 48
> ; 9B:       BFAF0B1520       MOV EDI, #x20150BAF              ; 'LIST
> ; A0:       488B3551FFFFFF   MOV RSI, [RIP-175]               ; '(VALUES
>                                                               ;   (SIMPLE-ARRAY ..))
> ; A7:       488B0552FFFFFF   MOV RAX, [RIP-174]               ; '("foo")
> ; AE:       498940F0         MOV [R8-16], RAX
> ; B2:       488B054FFFFFFF   MOV RAX, [RIP-177]               ; "(CAR \"foo\")"
> ; B9:       498940E8         MOV [R8-24], RAX
> ; BD:       49C740E017010020 MOV QWORD PTR [R8-32], #x20000117  ; NIL
> ; C5:       B90C000000       MOV ECX, 12
> ; CA:       498928           MOV [R8], RBP
> ; CD:       498BE8           MOV RBP, R8
> ; D0:       B882B12620       MOV EAX, #x2026B182              ; #<FDEFN SB-C::%COMPILE-TIME-TYPE-ERROR>
> ; D5:       FFD0             CALL RAX
> ; D7:       CC10             INT3 16                          ; Invalid argument count trap
> ; D9: L2:   6A10             PUSH 16
> ; DB:       FF1425B0080020   CALL [#x200008B0]                ; #x21A00540: LIST-ALLOC-TRAMP
> ; E2:       5A               POP RDX
> ; E3:       EB8C             JMP L0
> NIL
> * 

Okay?

>> If we talk about type checking, Elisp uses dynamic typing
>> and compilation cannot do much about it. Native compilation
>> also does not touch C subroutines - the place where
>> typechecks are performed.
>
> SBCL implements a Lisp, Lisp by definition is
> dynamically typed.

Only for the kind of use (code) that we are used to. See this:

  https://medium.com/@MartinCracauer/static-type-checking-in-the-programmable-programming-language-lisp-79bb79eb068a

For example

(defunt meh5c ((int p1) (int p2))
    (+ p1 p2))
(meh5c 1 2) ; ==> 3

with defunt being a macro that uses declare.

A simple example is given earlier in the text,

(defun meh (p1)
  (declare (fixnum p1))
  (+ p1 3))

-- 
underground experts united
https://dataswamp.org/~incal

Re: Shrinking the C core

[Alfred M. Szmidt]

Please keep the CC intact, not everyone subscribed.

   > It should be quite obvious why SBCL is faster than the Emacs
   > Lisp VM (or even native). Just look at this call to (car
   > "foo"), and compare what happens in Emacs.
   >
   > * (disassemble 'foo)
   > ; disassembly for FOO
   > ; Size: 166 bytes. Origin: #x225D873F                         ; FOO
   > ; 3F:       488B042590060020 MOV RAX, [#x20000690]
   > ; 47:       488945F8         MOV [RBP-8], RAX
   > ; 4B:       48892C2560060020 MOV [#x20000660], RBP
   > ; 53:       488B142518000020 MOV RDX, [#x20000018]
   > ; 5B:       488D4210         LEA RAX, [RDX+16]
   > ; 5F:       483B042520000020 CMP RAX, [#x20000020]
   > ; 67:       7770             JA L2
   > ; 69:       4889042518000020 MOV [#x20000018], RAX
   > ; 71: L0:   488B0570FFFFFF   MOV RAX, [RIP-144]               ; "foo"
   > ; 78:       488902           MOV [RDX], RAX
   > ; 7B:       48C7420817010020 MOV QWORD PTR [RDX+8], #x20000117  ; NIL
   > ; 83:       80CA07           OR DL, 7
   > ; 86:       48312C2560060020 XOR [#x20000660], RBP
   > ; 8E:       7402             JEQ L1
   > ; 90:       CC09             INT3 9                           ; pending interrupt trap
   > ; 92: L1:   4C8D4424F0       LEA R8, [RSP-16]
   > ; 97:       4883EC30         SUB RSP, 48
   > ; 9B:       BFAF0B1520       MOV EDI, #x20150BAF              ; 'LIST
   > ; A0:       488B3551FFFFFF   MOV RSI, [RIP-175]               ; '(VALUES
   >                                                               ;   (SIMPLE-ARRAY ..))
   > ; A7:       488B0552FFFFFF   MOV RAX, [RIP-174]               ; '("foo")
   > ; AE:       498940F0         MOV [R8-16], RAX
   > ; B2:       488B054FFFFFFF   MOV RAX, [RIP-177]               ; "(CAR \"foo\")"
   > ; B9:       498940E8         MOV [R8-24], RAX
   > ; BD:       49C740E017010020 MOV QWORD PTR [R8-32], #x20000117  ; NIL
   > ; C5:       B90C000000       MOV ECX, 12
   > ; CA:       498928           MOV [R8], RBP
   > ; CD:       498BE8           MOV RBP, R8
   > ; D0:       B882B12620       MOV EAX, #x2026B182              ; #<FDEFN SB-C::%COMPILE-TIME-TYPE-ERROR>
   > ; D5:       FFD0             CALL RAX
   > ; D7:       CC10             INT3 16                          ; Invalid argument count trap
   > ; D9: L2:   6A10             PUSH 16
   > ; DB:       FF1425B0080020   CALL [#x200008B0]                ; #x21A00540: LIST-ALLOC-TRAMP
   > ; E2:       5A               POP RDX
   > ; E3:       EB8C             JMP L0
   > NIL
   > * 

   Okay?

I guess that you do not understand the above?  Or what?  Do you know
and understand what happens in Emacs when a similar call is done?  It
is far more than "166 bytes".

   >> If we talk about type checking, Elisp uses dynamic typing
   >> and compilation cannot do much about it. Native compilation
   >> also does not touch C subroutines - the place where
   >> typechecks are performed.
   >
   > SBCL implements a Lisp, Lisp by definition is
   > dynamically typed.

   Only for the kind of use (code) that we are used to. See this:

     https://medium.com/@MartinCracauer/static-type-checking-in-the-programmable-programming-language-lisp-79bb79eb068a

This has literally nothing to do with the difference between static
typing, and dynamic typing.  The author, and you, have it completeley
backwards to the point where I need to suggest that you take sometime
to read up on basic Lisp compilers, and then look into very good Lisp
compilers (CMUCL and SBCL come to mind).  Since it is already showing
that it is very hard to even explain basic Lisp compiler behaviour
without going to fundamentals.

Re: Shrinking the C core

[Ihor Radchenko]

"Alfred M. Szmidt" <ams@gnu.org> writes:

> Please keep the CC intact, not everyone subscribed.
>
>    > It should be quite obvious why SBCL is faster than the Emacs
>    > Lisp VM (or even native). Just look at this call to (car
>    > "foo"), and compare what happens in Emacs.
>    >
>    > * (disassemble 'foo)
>    > ; disassembly for FOO
>    > ; Size: 166 bytes. Origin: #x225D873F                         ; FOO
>> ...
>    Okay?
>
> I guess that you do not understand the above?  Or what?  Do you know
> and understand what happens in Emacs when a similar call is done?  It
> is far more than "166 bytes".

It would be helpful if you show us what happens in Elisp with a similar
call. Especially after native compilation.

I am asking genuinely because `car' (1) has dedicated opt code and thus
should be one of the best-optimized function calls on Elisp side; (2)
Fcar is nothing but

/* Take the car or cdr of something whose type is not known.  */
INLINE Lisp_Object
CAR (Lisp_Object c)
{
  if (CONSP (c))
    return XCAR (c); // <- XCONS (c)->u.s.car
  if (!NILP (c))
    wrong_type_argument (Qlistp, c);
  return Qnil;
}

So, it is a very simple example that can actually explain the basic
differences between Elisp and CL. It would be nice if you (considering
your low-level understanding) can provide us with an analysis of what is
different between Elisp and CL implementations of such a simple
function.

> This has literally nothing to do with the difference between static
> typing, and dynamic typing.  The author, and you, have it completeley
> backwards ...

I am sorry, because it was my message that started the confusion.
I was mostly referring to separation between Elisp
interpreted/byte/native code and C subrs. AFAIU, static analysis info
cannot be passed between these two parts of Emacs runtime: subr cannot
know in advance what Lisp_Object type it is working on, even if static
analysis of the caller Elisp code has such information (e.g. from GCC
JIT compiler).

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Shrinking the C core

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: Emanuel Berg <incal@dataswamp.org>, emacs-devel@gnu.org
> Date: Sun, 20 Aug 2023 15:36:34 +0000
> 
> "Alfred M. Szmidt" <ams@gnu.org> writes:
> 
> > I guess that you do not understand the above?  Or what?  Do you know
> > and understand what happens in Emacs when a similar call is done?  It
> > is far more than "166 bytes".
> 
> It would be helpful if you show us what happens in Elisp with a similar
> call.

See below.

> Especially after native compilation.

Native compilation doesn't affect 'car', because it's a primitive.

> I am asking genuinely because `car' (1) has dedicated opt code and thus
> should be one of the best-optimized function calls on Elisp side; (2)
> Fcar is nothing but
> 
> /* Take the car or cdr of something whose type is not known.  */
> INLINE Lisp_Object
> CAR (Lisp_Object c)
> {
>   if (CONSP (c))
>     return XCAR (c); // <- XCONS (c)->u.s.car
>   if (!NILP (c))
>     wrong_type_argument (Qlistp, c);
>   return Qnil;
> }

It's very easy to see the code of 'car' in Emacs.  All you need is run
GDB:

  $ gdb ./emacs
  ...
  (gdb) disassemble /m Fcar

Re: Shrinking the C core

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> It would be helpful if you show us what happens in Elisp with a similar
>> call.
>
> See below.

Sorry, I was not clear. I was asking to help comparing between
disassembly of Elisp and CL versions. I myself is not familiar with
assembly code.

> It's very easy to see the code of 'car' in Emacs.  All you need is run
> GDB:
>
>   $ gdb ./emacs
>   ...
>   (gdb) disassemble /m Fcar

So, while I can do this mechanically, it will not understand it.
Not to the level to draw conclusions about what is different in Elisp
compared to CL.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Shrinking the C core

[Alfred M. Szmidt]

   > It's very easy to see the code of 'car' in Emacs.  All you need is run
   > GDB:
   >
   >   $ gdb ./emacs
   >   ...
   >   (gdb) disassemble /m Fcar

   So, while I can do this mechanically, it will not understand it.
   Not to the level to draw conclusions about what is different in Elisp
   compared to CL.

The issue is not Emacs Lisp vs. Common Lisp.  What you mean is what
the difference is between SBCL and GNU Emacs.'

The question can be rephrased as what is the difference between GNU
Emacs and GCC?  Why is GCC so much faster?  And if you phrase it like
that you will see that it really doesn't make much sense anymore,
since you are comparing different things.

Emacs could implement optimizaations that GCC does for C, or ADA
.. but it breaks down very quickly.

Re: Shrinking the C core

[Ihor Radchenko]

"Alfred M. Szmidt" <ams@gnu.org> writes:

> The issue is not Emacs Lisp vs. Common Lisp.  What you mean is what
> the difference is between SBCL and GNU Emacs.'
>
> The question can be rephrased as what is the difference between GNU
> Emacs and GCC?  Why is GCC so much faster?  And if you phrase it like
> that you will see that it really doesn't make much sense anymore,
> since you are comparing different things.

> Emacs could implement optimizaations that GCC does for C, or ADA
> .. but it breaks down very quickly.

Not really. Native compilation already uses GCC. At least on the byte
code instructions and, separately, in subr code.
There is more than just GCC vs byte code VM in it.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Shrinking the C core

[Alfred M. Szmidt]

   > The issue is not Emacs Lisp vs. Common Lisp.  What you mean is what
   > the difference is between SBCL and GNU Emacs.'
   >
   > The question can be rephrased as what is the difference between GNU
   > Emacs and GCC?  Why is GCC so much faster?  And if you phrase it like
   > that you will see that it really doesn't make much sense anymore,
   > since you are comparing different things.

   > Emacs could implement optimizaations that GCC does for C, or ADA
   > .. but it breaks down very quickly.

   Not really. Native compilation already uses GCC. At least on the byte
   code instructions and, separately, in subr code.
   There is more than just GCC vs byte code VM in it.

It is not about native compilation!  It is about what OPTIMIZATIONS
can be done to the actual code flow.  Just using GCC doesn't do ANY
optimizations to how the Lisp code is optimized or how its flow is
changed due to optimizations!  

Re: Shrinking the C core

[Ihor Radchenko]

"Alfred M. Szmidt" <ams@gnu.org> writes:

>    Not really. Native compilation already uses GCC. At least on the byte
>    code instructions and, separately, in subr code.
>    There is more than just GCC vs byte code VM in it.
>
> It is not about native compilation!  It is about what OPTIMIZATIONS
> can be done to the actual code flow.  Just using GCC doesn't do ANY
> optimizations to how the Lisp code is optimized or how its flow is
> changed due to optimizations!  

Then, what does GCC do? AFAIK, GCC JIT takes the Elisp byte code,
transforms it into JIT pseudocode, and optimizes the actual code flow.

For example, when I write

(when (> x y) (when (> x y) x))

I expect GCC JIT to throw away the duplicate comparison.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Shrinking the C core

[Alfred M. Szmidt]

   "Alfred M. Szmidt" <ams@gnu.org> writes:

   >    Not really. Native compilation already uses GCC. At least on the byte
   >    code instructions and, separately, in subr code.
   >    There is more than just GCC vs byte code VM in it.
   >
   > It is not about native compilation!  It is about what OPTIMIZATIONS
   > can be done to the actual code flow.  Just using GCC doesn't do ANY
   > optimizations to how the Lisp code is optimized or how its flow is
   > changed due to optimizations!  

   Then, what does GCC do? AFAIK, GCC JIT takes the Elisp byte code,
   transforms it into JIT pseudocode, and optimizes the actual code flow.

What does GCC do _WHERE_?  What backend? What language? You're
speaking in such broad terms that it makes it impossible to continue
this discussion.  I don't know how the native compilation works, but
no matter what you feed to GCC it cannot do magic and any optimization
should be done on what the Emacs compiler does.

   For example, when I write

   (when (> x y) (when (> x y) x))

   I expect GCC JIT to throw away the duplicate comparison.

Why do you expect that?  Why do you think it is duplicate?  Where are
the guarantees that > or WHEN don't have side-effects?  Do you know
the exact type of X and Y so you can skip a cascade of type checks to
pick the right comparison operator?  Can you use fixnum comparison of
a specific bit width? Do you need to use bignum comparison?  

That is the type of information SBCL knows about, or allows the user
to specify.  Emacs does not have that today, and that incures one set
of overhead.  There are plenty more...

Re: Shrinking the C core

[Ihor Radchenko]

"Alfred M. Szmidt" <ams@gnu.org> writes:

>    Then, what does GCC do? AFAIK, GCC JIT takes the Elisp byte code,
>    transforms it into JIT pseudocode, and optimizes the actual code flow.
>
> What does GCC do _WHERE_?  What backend? What language? You're
> speaking in such broad terms that it makes it impossible to continue
> this discussion.  I don't know how the native compilation works, but
> no matter what you feed to GCC it cannot do magic and any optimization
> should be done on what the Emacs compiler does.

Native compilation provides the necessary information about Elisp to GCC.
Otherwise, native compilation would be useless.
You may check out the details in
https://zenodo.org/record/3736363 and
https://toobnix.org/w/1f997b3c-00dc-4f7d-b2ce-74538c194fa7

>    For example, when I write
>
>    (when (> x y) (when (> x y) x))
>
>    I expect GCC JIT to throw away the duplicate comparison.
>
> Why do you expect that?  Why do you think it is duplicate?  Where are
> the guarantees that > or WHEN don't have side-effects?  Do you know
> the exact type of X and Y so you can skip a cascade of type checks to
> pick the right comparison operator?  Can you use fixnum comparison of
> a specific bit width? Do you need to use bignum comparison?  

At least some of these questions are answered by the code on Emacs side.
Native compiler transforms the Elisp byte code, using its knowledge
about function purity, types, and maybe other things, into LIMP that can
be fed to GCC JIT. Then, GCC JIT uses the provided info to do actual
optimization.

> That is the type of information SBCL knows about, or allows the user
> to specify.  Emacs does not have that today, and that incures one set
> of overhead.  There are plenty more...

AFAIK, users cannot specify type info manually, but types are tracked
when transforming Elisp byte code into LIMP representation.

The only problem (AFAIU) is that GCC JIT cannot reach inside subr level,
so all these information does not benefit Emacs functions implemented in
C.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Shrinking the C core

[Ihor Radchenko]

Ihor Radchenko <yantar92@posteo.net> writes:

> The only problem (AFAIU) is that GCC JIT cannot reach inside subr level,
> so all these information does not benefit Emacs functions implemented in
> C.

If I am right here, it might actually be worth it to rewrite some of the
subroutines into Elisp. For example rounding_driver (called by
`floor') code is full of runtime type checks:

  CHECK_NUMBER (n);
  if (NILP (d))
...
  CHECK_NUMBER (d);
...
  if (FIXNUMP (d))
      if (XFIXNUM (d) == 0)
...
      if (FIXNUMP (n))
...
  else if (FLOATP (d))
      if (XFLOAT_DATA (d) == 0)
  int nscale = FLOATP (n) ? double_integer_scale (XFLOAT_DATA (n)) : 0;
..

During native compilation, if type information and n and d is available,
GCC might have a chance to cut a number of branches away from the above
code.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Shrinking the C core

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: eliz@gnu.org, incal@dataswamp.org, emacs-devel@gnu.org
> Date: Sun, 20 Aug 2023 19:24:36 +0000
> 
> If I am right here, it might actually be worth it to rewrite some of the
> subroutines into Elisp. For example rounding_driver (called by
> `floor') code is full of runtime type checks:
> 
>   CHECK_NUMBER (n);
>   if (NILP (d))
> ...
>   CHECK_NUMBER (d);
> ...
>   if (FIXNUMP (d))
>       if (XFIXNUM (d) == 0)
> ...
>       if (FIXNUMP (n))
> ...
>   else if (FLOATP (d))
>       if (XFLOAT_DATA (d) == 0)
>   int nscale = FLOATP (n) ? double_integer_scale (XFLOAT_DATA (n)) : 0;
> ..
> 
> During native compilation, if type information and n and d is available,
> GCC might have a chance to cut a number of branches away from the above
> code.

Cut them how?  AFAICT, none of the tests above are redundant.

Re: Shrinking the C core

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>>   CHECK_NUMBER (n);
>>   if (NILP (d))
>>     return FLOATP (n) ? double_to_integer (double_round (XFLOAT_DATA (n))) : n;
>> ...
>> During native compilation, if type information and n and d is available,
>> GCC might have a chance to cut a number of branches away from the above
>> code.
>
> Cut them how?  AFAICT, none of the tests above are redundant.

Consider the following:

(let ((a 10))
  (setq a (+ a 100))
  (floor a nil))

During compilation of the above code, the compiler will know that a is a
positive integer. Therefore, CHECK_NUMBER, NILP, and FLOATP are not
necessary and can be omitted in the call to `floor':

(let ((a 10))
 (setq a (+ a 100))
 a)

However, GCC JIT has no information about the internal structure of the
`floor' subr. Hence, it is currently unable to perform such
optimization.

It could, if it were somehow given an information about `floor'
implementation.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Shrinking the C core

[Po Lu]

Ihor Radchenko <yantar92@posteo.net> writes:

> (let ((a 10))
>   (setq a (+ a 100))
>   (floor a nil))
>
> During compilation of the above code, the compiler will know that a is a
> positive integer. Therefore, CHECK_NUMBER, NILP, and FLOATP are not
> necessary and can be omitted in the call to `floor':
>
> (let ((a 10))
>  (setq a (+ a 100))
>  a)
>
> However, GCC JIT has no information about the internal structure of the
> `floor' subr. Hence, it is currently unable to perform such
> optimization.
>
> It could, if it were somehow given an information about `floor'
> implementation.

This should thus be implemented in the native compiler, without
affecting the code of Ffloor itself.

Re: Shrinking the C core

[Ihor Radchenko]

Po Lu <luangruo@yahoo.com> writes:

>> However, GCC JIT has no information about the internal structure of the
>> `floor' subr. Hence, it is currently unable to perform such
>> optimization.
>>
>> It could, if it were somehow given an information about `floor'
>> implementation.
>
> This should thus be implemented in the native compiler, without
> affecting the code of Ffloor itself.

I do understand that the approach you propose is indeed used, for
example, for `car' in emit_lval_XCAR. However, is it practical for
functions like `floor'?

`car' implementation is very unlikely to change in future. But `floor'
and other functions (we should not be limited to `floor') may change
their implementations. The extra "native comp" copy of the
implementation will need to be always synchronized with the original
implementation. I doubt that it is practical maintenance-wise.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Shrinking the C core

[Po Lu]

Ihor Radchenko <yantar92@posteo.net> writes:

> I do understand that the approach you propose is indeed used, for
> example, for `car' in emit_lval_XCAR. However, is it practical for
> functions like `floor'?
>
> `car' implementation is very unlikely to change in future.

Why not?

> But `floor' and other functions (we should not be limited to `floor')
> may change their implementations. The extra "native comp" copy of the
> implementation will need to be always synchronized with the original
> implementation. I doubt that it is practical maintenance-wise.

How and why so?  How are Fcar and Ffloor divergent in this department?

Re: Shrinking the C core

[Ihor Radchenko]

Po Lu <luangruo@yahoo.com> writes:

>> `car' implementation is very unlikely to change in future.
>
> Why not?

Mostly because such basic functions are rarely changed.
Of course, it is not impossible that `car' is changed in future.

>> But `floor' and other functions (we should not be limited to `floor')
>> may change their implementations. The extra "native comp" copy of the
>> implementation will need to be always synchronized with the original
>> implementation. I doubt that it is practical maintenance-wise.
>
> How and why so?  How are Fcar and Ffloor divergent in this department?

`floor' might also be rather stable. I was mostly referring to "we
should not be limited to `floor'" - it may be a problem for other
functions.

But let me rephrase it in other terms: what you propose will require
maintaining two separate implementations of subroutines - one in C, and
one specially tailored to GCC JIT psudocode. This may be doable for a
small set of core primitives, but not scalable if we want to make more
subroutines benefit from GGC JIT optimizations.

Another idea, if rewriting in Elisp is not feasible, could be somehow
structuring the internal C code in such a way that we can derive GCC
JIT pseudocode right from the C function bodies.

For example, Ffloor could be (1) split into smaller functions dedicated
to certain argument type combinations; (2) record a metadata readable by
native comp code about which small function correspond to different
argument types. Then, native comp can emit direct calls to these smaller
(and faster) functions when the type is known.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

RE: [External] : Re: Shrinking the C core

[Drew Adams]

> Mostly because such basic functions are rarely changed.
> Of course, it is not impossible that `car' is changed in future.

Perhaps it will become "electric"...

RE: [External] : Re: Shrinking the C core

[Drew Adams]

> > The discussion about floor started from Alfred using `floor'
> > as an example where CL uses system-dependent optimizations
> > and thus being much faster.
> 
> So the answer to the question, Why is SBCL faster?
> is "optimizations". And the answer to the question, Why don't
> we have those optimizations? is "they are not portable"?

https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node103.html#SECTION001300000000000000000

Common Lisp is a standard.  Different implementations
of it should respect the standard, but the standard
allows for different behaviors to some extent, esp.
wrt performance.  CL has multiple ways of declaring
different levels of optimization, which a given
implementation can support or not.

Particular optimizations are not expected to be
portable.