Re: [External] : Re: Shrinking the C core

Re: [External] : Re: Shrinking the C core

[Arthur Miller]

>[[[ To any NSA and FBI agents reading my email: please consider    ]]]
>[[[ whether defending the US Constitution against all enemies,     ]]]
>[[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>  > Of course, but nobody suggests it is all-in. `length' is not a keyworded
>  > in CL either.
>
>Oops, I thought it was -- but the time when I implemented and used
>Common Lisp was 40 years ago.
>
>I am pretty sure `member' used keyword arguments, and I think that
>getting behavior equivalent to traditional Lisp `member' required
>specifying a keyword argument.  I never forgot that, during that
>period, because I got reminded of it almost every day.
>
>  >  But it is convenient to have keywords in
>  > some places, like for example in define-minor-mode or make-process, or
>  > even "new" define-keymap.
>
>I would not object to using keyword arguments for functions like that
>-- complex and cumbersome to use, and not used often.

Well, than we pretty much agree 100% about keyword args. 40 years ago I
was playing in the woods, but as I have learned Lisp some barely 2 or 3
years ago, thanks to Emacs Lisp, that is pretty much how I understand
keyword arguments and why they are used, so that is why it felt a bit
strange when you said you don't want them, but I understand now that you
might have felt aversion if they were used everywhere; it certainly is a
PITA in functions with small number of arguments.

Question is also, if Maclisp was developed further, to current day, if
and how it would evolve itself. As time goes buy we discover new idioms,
patterns, have different needs etc. These things changes and new ideas
emerges; it is very much probable that Maclisp itself would look
differently than what it was when CL was conceived. I am sure they
wouldn't have those paires of complementatry functions
(remove-if/remove-if-not) which make for a big part of CL verbosity too.

RE: [External] : Re: Shrinking the C core

[Drew Adams]

[Moving reply to e-tangents.]

> > I would not object to using keyword arguments for functions
> > like that -- complex and cumbersome to use, and not used often.
> 
> Well, than we pretty much agree 100% about keyword args.

+1.

Except maybe for the "not used often".  Depends on the
"often" use.  If used often without any keyword args (or
optional args, for that matter), what difference does it
make that the function _has_ keyword/optional args?  By
definition and convention, such args are generally not
for the most common (the default) use case.

But the general point, I think, is that keyword (and
optional) args turn one function into a family of several,
and that can be useful.  But the "base" function, called without such args, can be just as convenient as if there
were no such args possible.

Re: [External] : Re: Shrinking the C core

[Arthur Miller]

Drew Adams <drew.adams@oracle.com> writes:

> [Moving reply to e-tangents.]
>
>> > I would not object to using keyword arguments for functions
>> > like that -- complex and cumbersome to use, and not used often.
>> 
>> Well, than we pretty much agree 100% about keyword args.
>
> +1.
>
> Except maybe for the "not used often".  Depends on the
> "often" use.  If used often without any keyword args (or
> optional args, for that matter), what difference does it
> make that the function _has_ keyword/optional args?  By
> definition and convention, such args are generally not
> for the most common (the default) use case.

Well yes, of course, it is corect. I think it depends on what "not often
used" referred to. I interepretted it as not often used argument, not
the function itself. Those more often used but possible to omit are of
course usually optional and in front of the optional list, but those
that are used in some more specialized circumstance or by very few are
perhaps good candidates for keyword arguments? I don't know, that is how
I perecieve it.

> But the general point, I think, is that keyword (and
> optional) args turn one function into a family of several,
> and that can be useful.  But the "base" function, called without such args, can be just as convenient as if there
> were no such args possible.

Yes indeed; and I agree with what you wrote in your longer second mail
about that. They do lessen the need for more functions which means less
API surface to document and potentially lookup and learn. Isn't a good
example that window functions that I wanted to make into same once in
the context of help/info patch? I don't remember which it were tbh
:). Something with selecting window on just current frame or all frames
or something like that.

Re: [External] : Re: Shrinking the C core

[Emanuel Berg]

Arthur Miller wrote:

> I am sure they wouldn't have those paires of complementatry
> functions (remove-if/remove-if-not) which make for a big
> part of CL verbosity too.

What is wrong with them?

We have them in Elisp as well, `cl-remove-if' and
`cl-remove-if-not', both in cl-seq.el.

Those functions are pretty useful IMO including use which
includes the keywords, all tho that is optional.

The only thing negative is the documentation, see for example
the docstring of `cl-remove-if-not':

  Remove all items not satisfying PREDICATE in SEQ.
  This is a non-destructive function; it makes a copy of SEQ
  if necessary to avoid corrupting the original SEQ.

  Keywords supported:  :key :count :start :end :from-end

As you see, it only says what keywords are "supported", not
what they express or how they effect the execution of
the function.

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: Shrinking the C core

[Emanuel Berg]

Arthur Miller wrote:

> Yes indeed; and I agree with what you wrote in your longer
> second mail about that. They do lessen the need for more
> functions which means less API surface to document and
> potentially lookup and learn. Isn't a good example that
> window functions that I wanted to make into same once in the
> context of help/info patch? I don't remember which it were
> tbh :). Something with selecting window on just current
> frame or all frames or something like that.

I think, in general and ranked from best to worse,

1. Many functions, all with few arguments
2. One function with not-that-many optional arguments
3. One function with keyword arguments
4. One function with too many optional arguments

Documentation is not a problem for case 1 since small
functions with clear names and few arguments typically
requires just 1~3 lines of documentation.

This documentation is also easy to browse, you won't have to
navigate the help buffer or search it for what an argument or
keyword does, written far below somewhere in the text.

-- 
underground experts united
https://dataswamp.org/~incal

RE: [External] : Re: Shrinking the C core

[Drew Adams]

> I think, in general and ranked from best to worse,
> 
> 1. Many functions, all with few arguments
> 2. One function with not-that-many optional arguments
> 3. One function with keyword arguments
> 4. One function with too many optional arguments

I don't think any such generalization is
very helpful - regardless of how one might
choose to order those.

> Documentation is not a problem for case 1 since small
> functions with clear names and few arguments typically
> requires just 1~3 lines of documentation.
> 
> This documentation is also easy to browse, you won't have to
> navigate the help buffer or search it for what an argument or
> keyword does, written far below somewhere in the text.

I think you're missing the point I made about
a "family" of functions.

Just having separate "many functions, all with
few arguments" can make it more difficult to
understand which is which, what's what, and
when to use this one or that one.

More generally, it's harder to see what the
familial relations are.  You can't as easily
see the forest because of all the trees.

Re: [External] : Re: Shrinking the C core

[Emanuel Berg]

Drew Adams wrote:

> I think you're missing the point I made about a "family"
> of functions.
>
> Just having separate "many functions, all with few
> arguments" can make it more difficult to understand which is
> which, what's what, and when to use this one or that one.
>
> More generally, it's harder to see what the familial
> relations are. You can't as easily see the forest because of
> all the trees.

Indeed, you have to be familiar with the functions in order to
use them. Just as you have to be familiar with the optional
arguments and keywords to use them. So there is no difference
in that regard.

But I think the functions are easier to see than arguments and
keywords which are inside functions, because when I use
functions, I'm myself outside, not inside of them.

Or put it like this: If I have a computer that can do 100
things, I prefer 100 functions to 2 functions, one with
50 optional arguments and the other with 50 keywords.

-- 
underground experts united
https://dataswamp.org/~incal

RE: [External] : Re: Shrinking the C core

[Drew Adams]

> > I think you're missing the point I made about a "family"
> > of functions.
> >
> > Just having separate "many functions, all with few
> > arguments" can make it more difficult to understand which is
> > which, what's what, and when to use this one or that one.
> >
> > More generally, it's harder to see what the familial
> > relations are. You can't as easily see the forest because of
> > all the trees.
> 
> Indeed, you have to be familiar with the functions in order to
> use them. Just as you have to be familiar with the optional
> arguments and keywords to use them. So there is no difference
> in that regard.

There is a difference.  The doc of for the family,
i.e., all the variants of the function, is in one
place.  And that doc can, and typically does,
describe them together, i.e., it describes their
relationships.

And another, obvious, difference, for things like
apropos search and completion of commands.

> But I think the functions are easier to see than arguments and
> keywords which are inside functions, because when I use
> functions, I'm myself outside, not inside of them.

"The functions" can't be "easier to see" than the
single function, which encompasses the family.

Of course, you haven't made clear what you mean
by "to see" or your being "outside"/"inside"
functions.  So maybe you have some unsaid meanings
that fit your "easier to see" characterization.
 
> Or put it like this: If I have a computer that can do 100
> things, I prefer 100 functions to 2 functions, one with
> 50 optional arguments and the other with 50 keywords.

There's no need to exaggerate or deal with such
a level of abstraction.  The ability to have
&optional, &rest, and keyword arguments doesn't
prevent anyone from not making use of any of those
and instead defining 8000 separate functions (to
further the exaggeration).

Do you use &optional?  &rest?  Why, since you
apparently have a blanket rule that using
multiple separate functions is always better.

Re: [External] : Re: Shrinking the C core

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > We have them in Elisp as well, `cl-remove-if' and
  > `cl-remove-if-not', both in cl-seq.el.

A partial emulation of some Common Lisp functions is present
in the cl-lib library, for emulation purposes.  It is not supposed to
be used a lot.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [External] : Re: Shrinking the C core

[Alfred M. Szmidt]

   [[[ To any NSA and FBI agents reading my email: please consider    ]]]
   [[[ whether defending the US Constitution against all enemies,     ]]]
   [[[ foreign or domestic, requires you to follow Snowden's example. ]]]

     > We have them in Elisp as well, `cl-remove-if' and
     > `cl-remove-if-not', both in cl-seq.el.

   A partial emulation of some Common Lisp functions is present
   in the cl-lib library, for emulation purposes.  It is not supposed to
   be used a lot.

And if there are functions or features that make sense from Common
Lisp, they can always be added piecemeal.  There is no need to make
Emacs Lisp complicated for the sake of compatibility with Common Lisp.

Someone mentioned &optional, and with default values -- that would be
a nice addition to Emacs Lisp.

Re: [External] : Re: Shrinking the C core

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Sat, 16 Sep 2023 20:46:24 -0400
> 
> A partial emulation of some Common Lisp functions is present
> in the cl-lib library, for emulation purposes.  It is not supposed to
> be used a lot.

466 out of 1637 Lisp files in Emacs require cl-lib (some of them only
during compilation, i.e. they use only the macros).

Re: [External] : Re: Shrinking the C core

[Emanuel Berg]

Eli Zaretskii wrote:

>> A partial emulation of some Common Lisp functions is
>> present in the cl-lib library, for emulation purposes.
>> It is not supposed to be used a lot.
>
> 466 out of 1637 Lisp files in Emacs require cl-lib (some of
> them only during compilation, i.e. they use only the
> macros).

466 out of 1637 files is 28%. It seems that, regardless of any
anti cl-lib sentiments present, the issue has been decided on
the field already.

(format "%d%%" (round (* 100 (/ 466 1637.0)))) ; 28%

The only thing that would have made it more striking is if
Emacs had 1337 files instead.

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: Shrinking the C core

[Emanuel Berg]

Alfred M. Szmidt wrote:

>>> We have them in Elisp as well, `cl-remove-if' and
>>> `cl-remove-if-not', both in cl-seq.el.
>>
>> A partial emulation of some Common Lisp functions is
>> present in the cl-lib library, for emulation purposes.
>> It is not supposed to be used a lot.
>
> And if there are functions or features that make sense from
> Common Lisp, they can always be added piecemeal. There is no
> need to make Emacs Lisp complicated for the sake of
> compatibility with Common Lisp.

I don't know why cl-lib was added to GNU Emacs and Emacs Lisp,
maybe it was, as you say, for emulation purposes and for the
sake of compatibility with Common Lisp. But the way it is used
today, as we just heard in 28% of vanilla Emacs files, isn't
because of CL emulation or compatibility purposes, but because
it adds useful features and covers aspects that non-cl-lib
Elisp leaves blank.

For functions like `cl-incf', where there is no corresponding
"incf" [I don't know whatever happened to it or why it was
dropped, or maybe it wasn't ever there before cl-lib?] one
could setup aliases or simply drop the "cl-" prefix (and set
an "cl-incf" alias to the new "incf" as not to break existing
code).

Then, for functions like `cl-defun' and `defun', one would
examine if those could be merged into a new "defun", with the
CL features brought over, e.g. the default &optional argument
value syntax. Here one would again use aliases to cover
the back.

For functions that cannot be merged because they are
inherently different, if such cases exist, one would keep the
"cl-" prefix and the incompatible non-cl-lib Elisp function.

But one could also just leave it the way it is! Since those
prefixes are hardly a big problem. And especially not compared
to all the good features cl-lib brings to our game.

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: Shrinking the C core

[Alfred M. Szmidt]

One should think carefully before thinking of what one should do, or
would do.  What one can do is anything... and that is not always the
best thing, these things should be done very carefully and on a case
by case basis.  So lets not go crazy.

Re: [External] : Re: Shrinking the C core

[Emanuel Berg]

Drew Adams wrote:

> "The functions" can't be "easier to see" than the single
> function

Many functions that each do little are easier to see than few
functions that each do a lot, because with many functions that
each do little, the function names are often enough to
understand what they do and how they work - and they require
none or very little documentation.

But with few functions that each do a lot, one has to check
the documentation for each function, and the documentation
will be huge, because few functions that each do a lot either
have to have an almost endless list of arguments or rely on an
equally long list of possible keywords to control
their behavior.

For example, (+ ...), (- ...) etc are preferable to
(arithmetic :operation 'addition ...),
(arithmetic :operation 'subtraction ...) etc.

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: Shrinking the C core

[Yuri Khan]

On Sun, 17 Sept 2023 at 17:46, Emanuel Berg <incal@dataswamp.org> wrote:

> For example, (+ ...), (- ...) etc are preferable to
> (arithmetic :operation 'addition ...),
> (arithmetic :operation 'subtraction ...) etc.

Let’s reframe that in more real-life examples. What is preferable and why:

1. a couple of functions
   a. ‘(my-sort-by SEQ LESSP)’ where LESSP is a function accepting two
elements X and Y and returning ‘t’ when X should be sorted before Y,
and
   b. ‘(my-sort-on SEQ KEY)’ where KEY is a function accepting an
element X and returning a number such that sorting by that number
yields the desired order; or

2. a single function that could be called as ‘(my-sort SEQ :by LESSP)’
or ‘(my-sort SEQ :key KEY)’, with the same semantics as above?

Does your answer change if we want to extend the API to also support
sorting in descending order? If it does, how?

RE: [External] : Re: Shrinking the C core

[Drew Adams]

> Many functions that each do little are easier to see than few
> functions that each do a lot, because with many functions that
> each do little, the function names are often enough to
> understand what they do and how they work - and they require
> none or very little documentation.
> 
> But with few functions that each do a lot, one has to check
> the documentation for each function, and the documentation
> will be huge, because few functions that each do a lot either
> have to have an almost endless list of arguments or rely on an
> equally long list of possible keywords to control
> their behavior.
> 
> For example, (+ ...), (- ...) etc are preferable to
> (arithmetic :operation 'addition ...),
> (arithmetic :operation 'subtraction ...) etc.

You continue to ignore this in your posts:

  The ability to have &optional, &rest, and keyword
  arguments doesn't prevent anyone from _not_ making
  use of any of those and instead defining 8000
  separate functions (to further the exaggeration).

  Do _you_ ever use &optional or &rest when defining
  functions?  If so, why, since you apparently have
  a blanket rule that using multiple separate
  functions is always better?
 
Everything you've said against keyword args (even
just allowing them?) applies equally to &optional
and &rest.  Should they be outlawed or deprecated?

Please answer the question: do you define functions
that use those?  If so, how do you reconcile that
practice with your claim that it's always better to
define separate functions instead?  Why do you use

 (defun foo (arg1 &optional arg2 arg3 arg4)...)

instead of these, since you argue that these must
be better?

 (defun foo1 (arg1) ...)
 (defun foo2 (arg1 arg2) ...)
 (defun foo3 (arg1 arg2 arg3) ...)
 (defun foo4 (arg1 arg2 arg3 arg4) ...)

It's a no-brainer that:

1. Providing the _possibility_ of using &optional,
   &rest - and, yes, keyword args - doesn't _oblige_
   anyone to define functions that use those.

   Such arguments just _allow_ you to define, and
   users to use, a function that takes the place
   of multiple related functions.  Or that serves
   as a helper/workhorse, to _define_ multiple
   related functions.

2. That provides a place and a name for the family
   as a whole, and a place for their _relations_
   to be set forth explicitly in doc.  You get not
   only individual views of particular trees but
   views of tree communities: particular forests.

3. Keyword args let users use fewer actual args
   than when &optional args are used - nils that
   are only positional placeholders disappear.

4. Keyword args make clear what the intention of
   each arg is (it's named!).  Contrast that with
   having to consult the doc each time to figure
   out what each positional arg means.

When calling a function, both (1) the need to stick
in positional-placeholder nils and (2) the need to
identify a keyword arg by its name make function
calls more verbose (an argument you could have made,
but didn't, in favor of not using &optional, &rest,
and keyword args).

It's a tradeoff, for not needing to define and use
more functions.  When defining functions _you_ get
to make that tradeoff decision.  That's the point -
the language doesn't decide for you, by offering
only one possibility.  Lisp says, "You're welcome!"

If every function had no relatives, you might have
an argument.  But if that were the case then no one
would ever define or use &optional or &rest or
keyword args, and those never would have been added
to Lisp in the first place.

(BTW, &rest is essentially just a shortcut for an
explicit list argument.  Some languages make you
pass a vector/sequence/list each time, even when
that's empty: [], "", ().)

Finally, perhaps the most common uses of &optional,
&rest, and keyword args are for defining a family
workhorse function, which is then used to define
other functions with fewer or no such args, which
serve particularly common use cases.

And often the most common use case, even for end
users, is just to call the workhorse function with
no such actual args.  IOW, the "family" function
is often, maybe even typically, defined so that it
can be used as is, with no such args, to provide a
useful default behavior.

Thus it has been, for decades.  And thus it will
continue to be...

RE: [External] : Re: Shrinking the C core

[Drew Adams]

> (format "%d%%" (round (* 100 (/ 466 1637.0)))) ; 28%


OT, but wrt `format':

Recently I've seen some questions on SE.Emacs that
resulted from users trying to use Common Lisp's
`format' in Elisp, that is, trying to use `format'
with DESTINATION first arg and with Common Lisp's
kind of CONTROL-STRING arg.

RE: [External] : Re: Shrinking the C core

[Drew Adams]

> For functions like `cl-incf', where there is no corresponding
> "incf" [I don't know whatever happened to it or why it was
> dropped, or maybe it wasn't ever there before cl-lib?] one
> could setup aliases or simply drop the "cl-" prefix (and set
> an "cl-incf" alias to the new "incf" as not to break existing
> code).

The CL emulation was originally in file `cl.el' and
its helper files, such as `cl-macs.el'.  At that time
functions/macros such as `cl-incf' and `cl-case' had
no `cl-' prefix.  E.g., Emacs 20:
___

 incf is a Lisp macro in `cl'.
 (incf PLACE &optional X)

 (incf PLACE [X]): increment PLACE by X (1 by default).
 PLACE may be a symbol, or any generalized variable allowed by `setf'.
 The return value is the incremented value of PLACE.
___

 case is a Lisp macro in `cl-macs'.
 (case EXPR &rest CLAUSES)

 (case EXPR CLAUSES...): evals EXPR, chooses from CLAUSES on that value.
 Each clause looks like (KEYLIST BODY...).  EXPR is evaluated and compared
 against each key in each KEYLIST; the corresponding BODY is evaluated.
 If no clause succeeds, case returns nil.  A single atom may be used in
 place of a KEYLIST of one atom.  A KEYLIST of `t' or `otherwise' is
 allowed only in the final clause, and matches if no other keys match.
 Key values are compared by `eql'.
___

There's no `incf' or `case' in Elisp that would
conflict with the CL-emulation definitions, but
when Someone decided to prefix CL thingies with
`cl-' the decision was to do it wholesale, not
just where there was a collision/conflict with
existing Elisp thingies.

Re: [External] : Re: Shrinking the C core

[Emanuel Berg]

Yuri Khan wrote:

> 1. a couple of functions
>    a. ‘(my-sort-by SEQ LESSP)’ where LESSP is a function accepting two
>        elements X and Y and returning ‘t’ when X should be
>        sorted before Y, and
>    b. ‘(my-sort-on SEQ KEY)’ where KEY is a function accepting an
>        element X and returning a number such that sorting by
>        that number yields the desired order; or
>
> 2. a single function that could be called as ‘(my-sort SEQ
>    :by LESSP)’ or ‘(my-sort SEQ :key KEY)’, with the same
>    semantics as above?
>
> Does your answer change if we want to extend the API to also
> support sorting in descending order? If it does, how?

If there are several tendencies and in particular if those can
be combined one can absolutely use keywords (and extend by
adding more), but then we are at a level of complexity which
would translate to a huge amount of envisioned simple, one or
two argument functions, so instead of doing that, if one were
to stay at the simple function end of the spectrum, one would
use combinations of such simple functions, e.g.
(reverse (sort ... ))

At the opposite end where functions are insanely complex,
keywords are a way to make the interface more clear and
facilitate the submission of arguments from code.

So it is good keywords exist when it is complicated, in
general I prefer it when it is simple but sometimes the
complexity is such it is actually easier to not try to keep it
simple - since that is impossible - but to stay at the complex
side and take it from there.

Optional arguments don't really belong on that simple/complex
scale, rather they should be used when there is an intuitive
default, for example (forward-line &optional N) where
N defaults to 1.

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: Shrinking the C core

[Emanuel Berg]

Drew Adams wrote:

> your claim that it's always better to define separate
> functions instead?

You are claiming that - but I disagree.

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: Shrinking the C core

[Emanuel Berg]

Drew Adams wrote:

>> For functions like `cl-incf', where there is no
>> corresponding "incf" [I don't know whatever happened to it
>> or why it was dropped, or maybe it wasn't ever there before
>> cl-lib?] [...]
>
> The CL emulation was originally in file `cl.el' and its
> helper files, such as `cl-macs.el'. At that time
> functions/macros such as `cl-incf' and `cl-case' had no
> `cl-' prefix. [...]
>
> There's no `incf' or `case' in Elisp that would conflict
> with the CL-emulation definitions, but when Someone decided
> to prefix CL thingies with `cl-' the decision was to do it
> wholesale, not just where there was a collision/conflict
> with existing Elisp thingies.

Ah, I see!

Thanks for explaining.

So I remembered correctly then, that I did use "incf" before
`cl-incf'!

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: Shrinking the C core

[Emanuel Berg]

Drew Adams wrote:

>> (format "%d%%" (round (* 100 (/ 466 1637.0)))) ; 28%
>
> OT, but wrt `format':
>
> Recently I've seen some questions on SE.Emacs that resulted
> from users trying to use Common Lisp's `format' in Elisp,
> that is, trying to use `format' with DESTINATION first arg
> and with Common Lisp's kind of CONTROL-STRING arg.

Maybe we should have a "cl-format" then for the CL format of
`format'?

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: Shrinking the C core

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I think the reason we added `cl-' to all the CL emulations is to
inform people that they are not parts of what Emacs Lisp supports.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [External] : Re: Shrinking the C core

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > But the way it is used
  > today, as we just heard in 28% of vanilla Emacs files, isn't
  > because of CL emulation or compatibility purposes, but because
  > it adds useful features and covers aspects that non-cl-lib
  > Elisp leaves blank.

I consider this a problem.  Such frequent use of the cl facilities --
even though it is just the macros -- adds those cl macros to what
people need to know to understand those files.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [External] : Re: Shrinking the C core

[Emanuel Berg]

Richard Stallman wrote:

>> But the way it is used today, as we just heard in 28% of
>> vanilla Emacs files, isn't because of CL emulation or
>> compatibility purposes, but because it adds useful features
>> and covers aspects that non-cl-lib Elisp leaves blank.
>
> I consider this a problem. Such frequent use of the cl
> facilities -- even though it is just the macros -- adds
> those cl macros to what people need to know to understand
> those files.

Okay, I understand. I must admit I don't really care about
other people at such a fine grained level, but let's think
then ...

I think that non-programmers are a lost cause in this case
anyway and for programmers the complexity increase from
non-cl-lib Elisp to cl-lib is pretty much negligible?

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: Shrinking the C core

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Tue, 19 Sep 2023 13:21:59 +0200
> 
> for programmers the complexity increase from non-cl-lib Elisp to
> cl-lib is pretty much negligible?

No, it isn't.  There's quite a lot of different syntax and semantics
that need to be learned and mastered.

Re: [External] : Re: Shrinking the C core

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I think that non-programmers are a lost cause in this case
  > anyway and for programmers the complexity increase from
  > non-cl-lib Elisp to cl-lib is pretty much negligible?

This dichotomy oversimplifies the various levels of skill and
knowledge that people have for Emacs Lisp programming.  People's skill
at programming varies across a wide range.  Programmers' knowledge of
Emacs Lisp likewise -- there is so much you could learn if you want to
but few have time to learn it all.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [External] : Re: Shrinking the C core

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > 466 out of 1637 Lisp files in Emacs require cl-lib (some of them only
  > during compilation, i.e. they use only the macros).

If a file uses cl only during compilation (for macros) it is
not much of a problem.  How many use it at run time?

I reported one file a few weeks ago that is always (or nearly always)
loaded and uses cl-lib, forcin it to be nearly always loaded too.
That should be fixed.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [External] : Re: Shrinking the C core

[Emanuel Berg]

Richard Stallman wrote:

>> 466 out of 1637 Lisp files in Emacs require cl-lib (some of
>> them only during compilation, i.e. they use only the
>> macros).
>
> If a file uses cl only during compilation (for macros) it is
> not much of a problem. How many use it at run time?
>
> I reported one file a few weeks ago that is always (or
> nearly always) loaded and uses cl-lib, forcin it to be
> nearly always loaded too. That should be fixed.

But if it is shipped and included with vanilla Emacs, which it
is, how can it be either any more or any less (dis)encouraged
for use than anything else included on the same premises?

If something isn't part of vanilla Emacs one can maybe say
"see if you can do without it, because if you use it, it has
to be brought in externally adding complexity" - perhaps.

But it _is_ included so in terms of technology it is on the
same level as everything else included. Are we gonna have one
big toolbox containing a bunch of tools, but on certain tools
put on little stickers saying "don't use this"?

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: Shrinking the C core

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: incal@dataswamp.org, emacs-devel@gnu.org
> Date: Sat, 14 Oct 2023 21:53:56 -0400
> 
>   > 466 out of 1637 Lisp files in Emacs require cl-lib (some of them only
>   > during compilation, i.e. they use only the macros).
> 
> If a file uses cl only during compilation (for macros) it is
> not much of a problem.  How many use it at run time?

226.

> I reported one file a few weeks ago that is always (or nearly always)
> loaded and uses cl-lib, forcin it to be nearly always loaded too.
> That should be fixed.

Which file was that?  I cannot reason about its causes to load cl-lib
without knowing the details, which include looking into the file and
considering what it does.

Re: [External] : Re: Shrinking the C core

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Sun, 15 Oct 2023 04:46:45 +0200
> 
> Richard Stallman wrote:
> 
> >> 466 out of 1637 Lisp files in Emacs require cl-lib (some of
> >> them only during compilation, i.e. they use only the
> >> macros).
> >
> > If a file uses cl only during compilation (for macros) it is
> > not much of a problem. How many use it at run time?
> >
> > I reported one file a few weeks ago that is always (or
> > nearly always) loaded and uses cl-lib, forcin it to be
> > nearly always loaded too. That should be fixed.
> 
> But if it is shipped and included with vanilla Emacs, which it
> is, how can it be either any more or any less (dis)encouraged
> for use than anything else included on the same premises?
> 
> If something isn't part of vanilla Emacs one can maybe say
> "see if you can do without it, because if you use it, it has
> to be brought in externally adding complexity" - perhaps.
> 
> But it _is_ included so in terms of technology it is on the
> same level as everything else included. Are we gonna have one
> big toolbox containing a bunch of tools, but on certain tools
> put on little stickers saying "don't use this"?

This thread is not for casual users of Emacs, it is for active Emacs
developers and maintainers.  So please just read it and don't try to
chime in, as that doesn't help.

Re: [External] : Re: Shrinking the C core

[Emanuel Berg]

Eli Zaretskii wrote:

>>> 466 out of 1637 Lisp files in Emacs require cl-lib (some
>>> of them only during compilation, i.e. they use only the
>>> macros).
>> 
>> If a file uses cl only during compilation (for macros) it
>> is not much of a problem. How many use it at run time?
>
> 226.

That means that, out of a total number of 1637 Lisp files in
vanilla Emacs,

466 files, or 28%, require cl-lib overall;

240 files, or 15%, use cl-lib at compile time; and

226 files, or 14%, use cl-lib at run time.

-- 
underground experts united
https://dataswamp.org/~incal

Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I've changed the subject line to match what we are talking about.

  > 226 files, or 14%, use cl-lib at run time.

These cases may be more or less problematical, but not necessarily
equally so.

Which of those files are usually loaded when you start Emacs with
no arguments?  Those we should certainly fix.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Richard Stallman wrote:

> I've changed the subject line to match what we are
> talking about.
>
>> 226 files, or 14%, use cl-lib at run time.
>
> These cases may be more or less problematical, but not
> necessarily equally so.
>
> Which of those files are usually loaded when you start Emacs
> with no arguments? Those we should certainly fix.

I don't know, but at least one since you get the cl-lib stuff
even at 'emacs -Q'.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Thu, 19 Oct 2023 06:14:52 +0200
> 
> Richard Stallman wrote:
> 
> >> 226 files, or 14%, use cl-lib at run time.
> >
> > These cases may be more or less problematical, but not
> > necessarily equally so.
> >
> > Which of those files are usually loaded when you start Emacs
> > with no arguments? Those we should certainly fix.
> 
> I don't know, but at least one since you get the cl-lib stuff
> even at 'emacs -Q'.

That is not true, since in "emacs -Q" I get this:

  (featurep 'cl-lib) => nil

both in GUI and TTY (i.e. -nw) sessions.

So if cl-lib is loaded at startup, Emacs does that because of some
user customizations, not by default.

Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Eli Zaretskii wrote:

>>>> 226 files, or 14%, use cl-lib at run time.
>>>
>>> These cases may be more or less problematical, but not
>>> necessarily equally so.
>>>
>>> Which of those files are usually loaded when you start
>>> Emacs with no arguments? Those we should certainly fix.
>> 
>> I don't know, but at least one since you get the cl-lib
>> stuff even at 'emacs -Q'.
>
> That is not true, since in "emacs -Q" I get this:
>
>   (featurep 'cl-lib) => nil
>
> both in GUI and TTY (i.e. -nw) sessions.
>
> So if cl-lib is loaded at startup, Emacs does that because
> of some user customizations, not by default.

It is the debugger. Try this two times:

  (cl-loop for i from 0 to 10 do (+ 1 1))

The first time `cl-loop' isn't loaded, the second time it is.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Alan Mackenzie]

Hello, Richard.

On Wed, Oct 18, 2023 at 21:28:41 -0400, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]

> I've changed the subject line to match what we are talking about.

>   > 226 files, or 14%, use cl-lib at run time.

> These cases may be more or less problematical, but not necessarily
> equally so.

> Which of those files are usually loaded when you start Emacs with
> no arguments?  Those we should certainly fix.

With the following command, started from the lisp directory:

$ sed 's/^[^;]*(load "\(.*\)".*$/\1.el/; s/\.el\.el/.el/; t; d' loadup.el | \
sort | xargs grep -c "^[^;]*(cl-" | sed '/:0/d'

, I get the following files, with counts of '(cl-' usages:

abbrev.el:2
dnd.el:4
emacs-lisp/byte-run.el:1
emacs-lisp/cconv.el:15
emacs-lisp/cl-generic.el:178
emacs-lisp/cl-preloaded.el:45
emacs-lisp/eldoc.el:12
emacs-lisp/lisp-mode.el:14
emacs-lisp/nadvice.el:7
emacs-lisp/oclosure.el:31
emacs-lisp/seq.el:51
emacs-lisp/shorthands.el:1
emacs-lisp/syntax.el:6
emacs-lisp/tabulated-list.el:1
emacs-lisp/timer.el:1
font-lock.el:1
frame.el:6
international/mule-cmds.el:4
international/ucs-normalize.el:3
isearch.el:3
ldefs-boot.el:3
loaddefs.el:3
minibuffer.el:12
progmodes/elisp-mode.el:23
progmodes/prog-mode.el:1
register.el:23
replace.el:1
select.el:6
simple.el:9
startup.el:3
subr.el:1
tab-bar.el:4
term/android-win.el:7
term/haiku-win.el:11
term/ns-win.el:9
term/pc-win.el:7
term/pgtk-win.el:9
term/w32-win.el:12
term/x-win.el:8
uniquify.el:2
vc/vc-hooks.el:1

..  That's a total of 541 occurrences, just in the files that get dumped
with the Emacs binary.

I fear that that's the way things will stay, because it would be too
much work to fix.  But more because there are enough contributors who
think that cl-lib is just an ordinary part of Emacs to be used freely
without restraint.  I am not among this group of contributors.

> -- 
> Dr Richard Stallman (https://stallman.org)
> Chief GNUisance of the GNU Project (https://gnu.org)
> Founder, Free Software Foundation (https://fsf.org)
> Internet Hall-of-Famer (https://internethalloffame.org)

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Lisp files that load cl-lib in problematical ways

[Stephen Berman]

On Thu, 19 Oct 2023 07:54:12 +0300 Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Emanuel Berg <incal@dataswamp.org>
>> Date: Thu, 19 Oct 2023 06:14:52 +0200
>>
>> Richard Stallman wrote:
>>
>> >> 226 files, or 14%, use cl-lib at run time.
>> >
>> > These cases may be more or less problematical, but not
>> > necessarily equally so.
>> >
>> > Which of those files are usually loaded when you start Emacs
>> > with no arguments? Those we should certainly fix.
>>
>> I don't know, but at least one since you get the cl-lib stuff
>> even at 'emacs -Q'.
>
> That is not true, since in "emacs -Q" I get this:
>
>   (featurep 'cl-lib) => nil
>
> both in GUI and TTY (i.e. -nw) sessions.

I also get this in a freshly updated build from master both with -Q and
with -Q -nw.  In a freshly updated build from emacs-29 I also get it
with -Q -nw, but with just -Q (i.e. the GUI) it returns `t', and
evaluating `features' returns (time-date subr-x cl-loaddefs cl-lib
rmc...).  And in a GUI build from master from October 8, (featurep
'cl-lib) returns `t' and features returns (time-date subr-x cl-loaddefs
cl-lib rmc...).  In the GUI build from current master, features returns
(rmc iso-transl tooltip...), so it looks like a recent change in master
stopped loading subr-x at startup.

Steve Berman

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> Date: Thu, 19 Oct 2023 05:27:03 +0000
> Cc: Emanuel Berg <incal@dataswamp.org>, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> With the following command, started from the lisp directory:
> 
> $ sed 's/^[^;]*(load "\(.*\)".*$/\1.el/; s/\.el\.el/.el/; t; d' loadup.el | \
> sort | xargs grep -c "^[^;]*(cl-" | sed '/:0/d'
> 
> , I get the following files, with counts of '(cl-' usages:
> 
> abbrev.el:2
> dnd.el:4
> emacs-lisp/byte-run.el:1
> emacs-lisp/cconv.el:15
> emacs-lisp/cl-generic.el:178
> emacs-lisp/cl-preloaded.el:45
> emacs-lisp/eldoc.el:12
> emacs-lisp/lisp-mode.el:14
> emacs-lisp/nadvice.el:7
> emacs-lisp/oclosure.el:31
> emacs-lisp/seq.el:51
> emacs-lisp/shorthands.el:1
> emacs-lisp/syntax.el:6
> emacs-lisp/tabulated-list.el:1
> emacs-lisp/timer.el:1
> font-lock.el:1
> frame.el:6
> international/mule-cmds.el:4
> international/ucs-normalize.el:3
> isearch.el:3
> ldefs-boot.el:3
> loaddefs.el:3
> minibuffer.el:12
> progmodes/elisp-mode.el:23
> progmodes/prog-mode.el:1
> register.el:23
> replace.el:1
> select.el:6
> simple.el:9
> startup.el:3
> subr.el:1
> tab-bar.el:4
> term/android-win.el:7
> term/haiku-win.el:11
> term/ns-win.el:9
> term/pc-win.el:7
> term/pgtk-win.el:9
> term/w32-win.el:12
> term/x-win.el:8
> uniquify.el:2
> vc/vc-hooks.el:1
> 
> ..  That's a total of 541 occurrences, just in the files that get dumped
> with the Emacs binary.

Those use cl-lib during byte-compilation, they don't load cl-lib at
run time.  So this is not what Richard asked about, AFAIU.

> I fear that that's the way things will stay, because it would be too
> much work to fix.  But more because there are enough contributors who
> think that cl-lib is just an ordinary part of Emacs to be used freely
> without restraint.  I am not among this group of contributors.

We cannot possibly expect people to contribute code if we force them
not to use the macros they are used to.  If cl-lib is not loaded as
result, that is good enough for us, I think.

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Stephen Berman <stephen.berman@gmx.net>
> Cc: Emanuel Berg <incal@dataswamp.org>,  emacs-devel@gnu.org
> Date: Thu, 19 Oct 2023 09:34:39 +0200
> 
> On Thu, 19 Oct 2023 07:54:12 +0300 Eli Zaretskii <eliz@gnu.org> wrote:
> 
> >   (featurep 'cl-lib) => nil
> >
> > both in GUI and TTY (i.e. -nw) sessions.
> 
> I also get this in a freshly updated build from master both with -Q and
> with -Q -nw.  In a freshly updated build from emacs-29 I also get it
> with -Q -nw, but with just -Q (i.e. the GUI) it returns `t', and
> evaluating `features' returns (time-date subr-x cl-loaddefs cl-lib
> rmc...).  And in a GUI build from master from October 8, (featurep
> 'cl-lib) returns `t' and features returns (time-date subr-x cl-loaddefs
> cl-lib rmc...).

Please try figuring out what loads cl-lib in your case, as I cannot
reproduce this with the latest emacs-29 branch.

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Thu, 19 Oct 2023 07:18:47 +0200
> 
> Eli Zaretskii wrote:
> 
> >   (featurep 'cl-lib) => nil
> >
> > both in GUI and TTY (i.e. -nw) sessions.
> >
> > So if cl-lib is loaded at startup, Emacs does that because
> > of some user customizations, not by default.
> 
> It is the debugger.

Then that's fine, as the debugger is not normally loaded in a
production session.

Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

> It is the debugger. Try this two times:
>
>   (cl-loop for i from 0 to 10 do (+ 1 1))
>
> The first time `cl-loop' isn't loaded, the second time
> it is.

debug.el `require' cl-lib as its first line of non-comment
source, and then goes on to use it 6 times: `cl-prin1' (2),
`cl-defstruct' (1), and `cl-getf' (3).

cl-prin1 is found in cl-print.el, cl-defstruct in cl-macs.el
and cl-getf in cl-extra.el.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Stephen Berman]

On Thu, 19 Oct 2023 10:55:24 +0300 Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Stephen Berman <stephen.berman@gmx.net>
>> Cc: Emanuel Berg <incal@dataswamp.org>,  emacs-devel@gnu.org
>> Date: Thu, 19 Oct 2023 09:34:39 +0200
>>
>> On Thu, 19 Oct 2023 07:54:12 +0300 Eli Zaretskii <eliz@gnu.org> wrote:
>>
>> >   (featurep 'cl-lib) => nil
>> >
>> > both in GUI and TTY (i.e. -nw) sessions.
>>
>> I also get this in a freshly updated build from master both with -Q and
>> with -Q -nw.  In a freshly updated build from emacs-29 I also get it
>> with -Q -nw, but with just -Q (i.e. the GUI) it returns `t', and
>> evaluating `features' returns (time-date subr-x cl-loaddefs cl-lib
>> rmc...).  And in a GUI build from master from October 8, (featurep
>> 'cl-lib) returns `t' and features returns (time-date subr-x cl-loaddefs
>> cl-lib rmc...).
>
> Please try figuring out what loads cl-lib in your case, as I cannot
> reproduce this with the latest emacs-29 branch.

Neither can I now; I must have mistakenly used the previous build of
emacs-29 (also from October 8), which was still open.  Sorry for the
false alarm.

Steve Berman

Re: Lisp files that load cl-lib in problematical ways

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stephen Berman <stephen.berman@gmx.net>
>> Cc: Emanuel Berg <incal@dataswamp.org>,  emacs-devel@gnu.org
>> Date: Thu, 19 Oct 2023 09:34:39 +0200
>> 
>> On Thu, 19 Oct 2023 07:54:12 +0300 Eli Zaretskii <eliz@gnu.org> wrote:
>> 
>> >   (featurep 'cl-lib) => nil
>> >
>> > both in GUI and TTY (i.e. -nw) sessions.
>> 
>> I also get this in a freshly updated build from master both with -Q and
>> with -Q -nw.  In a freshly updated build from emacs-29 I also get it
>> with -Q -nw, but with just -Q (i.e. the GUI) it returns `t', and
>> evaluating `features' returns (time-date subr-x cl-loaddefs cl-lib
>> rmc...).  And in a GUI build from master from October 8, (featurep
>> 'cl-lib) returns `t' and features returns (time-date subr-x cl-loaddefs
>> cl-lib rmc...).
>
> Please try figuring out what loads cl-lib in your case, as I cannot
> reproduce this with the latest emacs-29 branch.

Maybe the native compiler as it was triggered for some native
compilation?

  Andrea

Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Eli Zaretskii wrote:

>> That's a total of 541 occurrences, just in the files that
>> get dumped with the Emacs binary.
>
> Those use cl-lib during byte-compilation, they don't load
> cl-lib at run time. So this is not what Richard asked
> about, AFAIU.

The "levels" introduced in this discussion are, it would seem

  - files that are used by vanilla Emacs that use cl-lib
  
  - such files that use it at compile time

  - such files that use it a run time

  - such files that are loaded when Emacs is executed with no
    arguments

but - what is the difference? I think it is pretty clear that
cl-lib is used quite broadly, probably because people find
it useful.

So what does it matter if it is "purged" from one or several
of these levels? What gain is there if that is achieved?

>> I fear that that's the way things will stay, because it
>> would be too much work to fix. But more because there are
>> enough contributors who think that cl-lib is just an
>> ordinary part of Emacs to be used freely without restraint.
>> I am not among this group of contributors.
>
> We cannot possibly expect people to contribute code if we
> force them not to use the macros they are used to. If cl-lib
> is not loaded as result, that is good enough for us,
> I think.

Again, what is the harm loading cl-lib?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Thu, 19 Oct 2023 13:19:06 +0200
> 
> The "levels" introduced in this discussion are, it would seem
> 
>   - files that are used by vanilla Emacs that use cl-lib
>   
>   - such files that use it at compile time
> 
>   - such files that use it a run time
> 
>   - such files that are loaded when Emacs is executed with no
>     arguments
> 
> but - what is the difference? I think it is pretty clear that
> cl-lib is used quite broadly, probably because people find
> it useful.
> 
> So what does it matter if it is "purged" from one or several
> of these levels? What gain is there if that is achieved?

It matters to us because we decided long ago to avoid loading cl-lib
at runtime in vanilla Emacs.  There are several good reasons: bloat,
unnecessary namespace pollution, etc.  But all this is not really
relevant for the code that you (or any one of us) write for their own
personal use, it is only relevant to code contributed to Emacs.

Re: Lisp files that load cl-lib in problematical ways

[Alan Mackenzie]

Hello, Eli.

On Thu, Oct 19, 2023 at 10:47:37 +0300, Eli Zaretskii wrote:
> > Date: Thu, 19 Oct 2023 05:27:03 +0000
> > Cc: Emanuel Berg <incal@dataswamp.org>, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > With the following command, started from the lisp directory:

> > $ sed 's/^[^;]*(load "\(.*\)".*$/\1.el/; s/\.el\.el/.el/; t; d' loadup.el | \
> > sort | xargs grep -c "^[^;]*(cl-" | sed '/:0/d'

> > , I get the following files, with counts of '(cl-' usages:

> > abbrev.el:2
[ .... ]
> > vc/vc-hooks.el:1

> > ..  That's a total of 541 occurrences, just in the files that get dumped
> > with the Emacs binary.

> Those use cl-lib during byte-compilation, they don't load cl-lib at
> run time.

They do, I think (see below).

> So this is not what Richard asked about, AFAIU.

Yes, you're right.

So, I counted all the occurrences of "cl-" not inside an
eval-when-compile, by piping all the dumped .el files through the
following script:

find-run-time-cl.el:
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defun run-time-cl ()
  (save-excursion
    (goto-char (point-min))
    (let ((count 0))
      (while (and (< (point) (point-max))
		  (re-search-forward "^[^;]*(\\(eval-when-compile\\|cl-\\)"
				     nil 'stop))
	(if (string= (match-string-no-properties 1) "cl-")
	    (progn
	      (setq count (1+ count)))
	      ;; (message "%s:%s %s" (buffer-file-name) (line-number-at-pos)
	      ;; 	       (buffer-substring-no-properties (line-beginning-position)
	      ;; 					       (line-end-position)))
	  (goto-char (1- (match-beginning 1)))
	  (forward-list)))
      (prin1 (format "%s:%s\n" (buffer-file-name) count) t))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

$ for f in `sed 's/^[^;]*(load "\(.*\)".*$/\1.el/; s/\.el\.el/.el/; t; d' \
  loadup.el | sort`; \
  do ../src/emacs -Q -batch -l ~/find-run-time-cl.el \
  --eval "(progn (find-file \"$f\") (run-time-cl))"; \
  done | sed '/:0$/d' | sed 's%^.*/lisp/%%'

and got the following results:

abbrev.el:2
dnd.el:4
emacs-lisp/byte-run.el:1
emacs-lisp/cconv.el:13
emacs-lisp/cl-generic.el:56
emacs-lisp/cl-preloaded.el:16
emacs-lisp/eldoc.el:6
emacs-lisp/lisp-mode.el:6
emacs-lisp/nadvice.el:2
emacs-lisp/oclosure.el:8
emacs-lisp/seq.el:25
emacs-lisp/shorthands.el:1
emacs-lisp/syntax.el:5
emacs-lisp/tabulated-list.el:1
emacs-lisp/timer.el:1
font-lock.el:1
frame.el:5
international/mule-cmds.el:3
international/ucs-normalize.el:1
isearch.el:3
ldefs-boot.el:3
loaddefs.el:3
minibuffer.el:12
progmodes/elisp-mode.el:11
progmodes/prog-mode.el:1
register.el:3
replace.el:1
select.el:3
simple.el:9
startup.el:1
subr.el:1
tab-bar.el:4
term/android-win.el:4
term/haiku-win.el:3
term/ns-win.el:2
term/pc-win.el:6
term/pgtk-win.el:3
term/w32-win.el:4
term/x-win.el:2
uniquify.el:2
vc/vc-hooks.el:1

That may not be 541 occurrences, but it's still 239.  Incidentally, when
I start emacs -Q, cl-lib isn't yet loaded for me, either.

> > I fear that that's the way things will stay, because it would be too
> > much work to fix.  But more because there are enough contributors who
> > think that cl-lib is just an ordinary part of Emacs to be used freely
> > without restraint.  I am not among this group of contributors.

> We cannot possibly expect people to contribute code if we force them
> not to use the macros they are used to.  If cl-lib is not loaded as
> result, that is good enough for us, I think.

We "force" them to use Emacs Lisp, but they still contribute.  The
problem is that use of cl- makes maintenance of other people's code much
harder, at least for me.  I'm sure I'm not alone.

cl- probably multiplies the size of the Lisp language parts of Elisp
(i.e. things like defun, defmacro, if, while, cond, ...) by around three.
I'm not able to learn all these fancy macros which are used rarely enough
not to be learnable, but frequently enough to cause maintenance headaches
- every time I encounter `if-let' and the like, it just stops me in my
tracks.  I either have to guess what it means, which makes me
uncomfortable, or laboriously consult a doc string, if any.

Yes, I know if-let isn't a cl- function, and its doc string is of
reasonable quality.  Many cl- functions don't have doc strings, or have
inadequate ones, however.

There was a time in the medium past when the use of run time cl- was not
allowed.  Emacs was then, from my point of view, much easier to maintain.
At some stage this rule was done away with, but I don't recall there
being any discussion on emacs-devel about this change.  There should have
been such discussion.

But we are where we are, as you said.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> Date: Thu, 19 Oct 2023 12:34:42 +0000
> Cc: rms@gnu.org, incal@dataswamp.org, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> So, I counted all the occurrences of "cl-" not inside an
> eval-when-compile, by piping all the dumped .el files through the
> following script:
> 
> find-run-time-cl.el:
> ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
> (defun run-time-cl ()
>   (save-excursion
>     (goto-char (point-min))
>     (let ((count 0))
>       (while (and (< (point) (point-max))
> 		  (re-search-forward "^[^;]*(\\(eval-when-compile\\|cl-\\)"
> 				     nil 'stop))
> 	(if (string= (match-string-no-properties 1) "cl-")
> 	    (progn
> 	      (setq count (1+ count)))
> 	      ;; (message "%s:%s %s" (buffer-file-name) (line-number-at-pos)
> 	      ;; 	       (buffer-substring-no-properties (line-beginning-position)
> 	      ;; 					       (line-end-position)))
> 	  (goto-char (1- (match-beginning 1)))
> 	  (forward-list)))
>       (prin1 (format "%s:%s\n" (buffer-file-name) count) t))))
> ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
> 
> $ for f in `sed 's/^[^;]*(load "\(.*\)".*$/\1.el/; s/\.el\.el/.el/; t; d' \
>   loadup.el | sort`; \
>   do ../src/emacs -Q -batch -l ~/find-run-time-cl.el \
>   --eval "(progn (find-file \"$f\") (run-time-cl))"; \
>   done | sed '/:0$/d' | sed 's%^.*/lisp/%%'
> 
> and got the following results:
> 
> abbrev.el:2
> dnd.el:4
> emacs-lisp/byte-run.el:1
> emacs-lisp/cconv.el:13
> emacs-lisp/cl-generic.el:56
> emacs-lisp/cl-preloaded.el:16
> emacs-lisp/eldoc.el:6
> emacs-lisp/lisp-mode.el:6
> emacs-lisp/nadvice.el:2
> emacs-lisp/oclosure.el:8
> emacs-lisp/seq.el:25
> emacs-lisp/shorthands.el:1
> emacs-lisp/syntax.el:5
> emacs-lisp/tabulated-list.el:1
> emacs-lisp/timer.el:1
> font-lock.el:1
> frame.el:5
> international/mule-cmds.el:3
> international/ucs-normalize.el:1
> isearch.el:3
> ldefs-boot.el:3
> loaddefs.el:3
> minibuffer.el:12
> progmodes/elisp-mode.el:11
> progmodes/prog-mode.el:1
> register.el:3
> replace.el:1
> select.el:3
> simple.el:9
> startup.el:1
> subr.el:1
> tab-bar.el:4
> term/android-win.el:4
> term/haiku-win.el:3
> term/ns-win.el:2
> term/pc-win.el:6
> term/pgtk-win.el:3
> term/w32-win.el:4
> term/x-win.el:2
> uniquify.el:2
> vc/vc-hooks.el:1

If the above is correct, then how do you explain that in "emacs -Q" I
don't see the cl-lib feature present?

My conclusion is that your program has a bug.

> That may not be 541 occurrences, but it's still 239.  Incidentally, when
> I start emacs -Q, cl-lib isn't yet loaded for me, either.

Exactly.  How come, if all of the above-mentioned files load it?

> > We cannot possibly expect people to contribute code if we force them
> > not to use the macros they are used to.  If cl-lib is not loaded as
> > result, that is good enough for us, I think.
> 
> We "force" them to use Emacs Lisp, but they still contribute.  The
> problem is that use of cl- makes maintenance of other people's code much
> harder, at least for me.  I'm sure I'm not alone.

I'm sorry, but the above is how I feel we should treat our
contributors, if we want to keep the existing ones and attract new
ones.

Re: Lisp files that load cl-lib in problematical ways

[Alan Mackenzie]

Hello, Eli.

On Thu, Oct 19, 2023 at 15:55:34 +0300, Eli Zaretskii wrote:
> > Date: Thu, 19 Oct 2023 12:34:42 +0000
> > Cc: rms@gnu.org, incal@dataswamp.org, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > So, I counted all the occurrences of "cl-" not inside an
> > eval-when-compile, by piping all the dumped .el files through the
> > following script:

[ .... ]

> > and got the following results:

> > abbrev.el:2
[ .... ]
> > vc/vc-hooks.el:1

> If the above is correct, then how do you explain that in "emacs -Q" I
> don't see the cl-lib feature present?

Because the files that use cl- don't have a (require 'cl-lib) in them.
Quite a lot of the function calls are things like cl-assert, cl-incf, or
cl-defgeneric, which won't cause the loading of cl- until they are
actually run.

> My conclusion is that your program has a bug.

Maybe, but there are definitely run time uses of cl- in our dumped .el
files, such as a cl-incf in font-lock.el which creates the gradually
increasing row of dots which are displayed for large files (or, at least,
used to be).

> > That may not be 541 occurrences, but it's still 239.  Incidentally, when
> > I start emacs -Q, cl-lib isn't yet loaded for me, either.

> Exactly.  How come, if all of the above-mentioned files load it?

See above.  

> > > We cannot possibly expect people to contribute code if we force them
> > > not to use the macros they are used to.  If cl-lib is not loaded as
> > > result, that is good enough for us, I think.

> > We "force" them to use Emacs Lisp, but they still contribute.  The
> > problem is that use of cl- makes maintenance of other people's code much
> > harder, at least for me.  I'm sure I'm not alone.

> I'm sorry, but the above is how I feel we should treat our
> contributors, if we want to keep the existing ones and attract new
> ones.

I'm not urging you to change anything; merely expressing regret at the
existing state of affairs, and possibly inviting contributors in general
to use cl-lib a little less.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Lisp files that load cl-lib in problematical ways

[Andrea Corallo]

Andrea Corallo <acorallo@gnu.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Stephen Berman <stephen.berman@gmx.net>
>>> Cc: Emanuel Berg <incal@dataswamp.org>,  emacs-devel@gnu.org
>>> Date: Thu, 19 Oct 2023 09:34:39 +0200
>>> 
>>> On Thu, 19 Oct 2023 07:54:12 +0300 Eli Zaretskii <eliz@gnu.org> wrote:
>>> 
>>> >   (featurep 'cl-lib) => nil
>>> >
>>> > both in GUI and TTY (i.e. -nw) sessions.
>>> 
>>> I also get this in a freshly updated build from master both with -Q and
>>> with -Q -nw.  In a freshly updated build from emacs-29 I also get it
>>> with -Q -nw, but with just -Q (i.e. the GUI) it returns `t', and
>>> evaluating `features' returns (time-date subr-x cl-loaddefs cl-lib
>>> rmc...).  And in a GUI build from master from October 8, (featurep
>>> 'cl-lib) returns `t' and features returns (time-date subr-x cl-loaddefs
>>> cl-lib rmc...).
>>
>> Please try figuring out what loads cl-lib in your case, as I cannot
>> reproduce this with the latest emacs-29 branch.
>
> Maybe the native compiler as it was triggered for some native
> compilation?

Okay, I confirm that comp.el loads cl-lib, so any jit compilation
triggered loads that.

emacs -Q on my system at the first run (not in the followings) loads
cl-lib because of that.

The compiler really needs cl-lib while running as needs to understand
user defined types (cl structs).

OTOH one thing we could do, if that's important, is to split the code
that only drives the async compilation (that actually happens in a
subprocess) so we don't load cl-lib in the Emacs the user is actually
using.  This should not be that hard (optimism mode on).

  Andrea

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> Date: Thu, 19 Oct 2023 13:28:28 +0000
> Cc: rms@gnu.org, incal@dataswamp.org, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > If the above is correct, then how do you explain that in "emacs -Q" I
> > don't see the cl-lib feature present?
> 
> Because the files that use cl- don't have a (require 'cl-lib) in them.
> Quite a lot of the function calls are things like cl-assert, cl-incf, or
> cl-defgeneric, which won't cause the loading of cl- until they are
> actually run.

Then there's no problem, from where I stand.

> Maybe, but there are definitely run time uses of cl- in our dumped .el
> files, such as a cl-incf in font-lock.el which creates the gradually
> increasing row of dots which are displayed for large files (or, at least,
> used to be).

Again, not a problem.

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Andrea Corallo <acorallo@gnu.org>
> Cc: Stephen Berman <stephen.berman@gmx.net>,  incal@dataswamp.org,
>   emacs-devel@gnu.org
> Date: Thu, 19 Oct 2023 09:44:51 -0400
> 
> Okay, I confirm that comp.el loads cl-lib, so any jit compilation
> triggered loads that.

This is OK, not a problem.

> OTOH one thing we could do, if that's important, is to split the code
> that only drives the async compilation (that actually happens in a
> subprocess) so we don't load cl-lib in the Emacs the user is actually
> using.  This should not be that hard (optimism mode on).

I don't see the need, but I will not object if you want to make such
changes.

Re: Lisp files that load cl-lib in problematical ways

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <acorallo@gnu.org>
>> Cc: Stephen Berman <stephen.berman@gmx.net>,  incal@dataswamp.org,
>>   emacs-devel@gnu.org
>> Date: Thu, 19 Oct 2023 09:44:51 -0400
>> 
>> Okay, I confirm that comp.el loads cl-lib, so any jit compilation
>> triggered loads that.
>
> This is OK, not a problem.
>
>> OTOH one thing we could do, if that's important, is to split the code
>> that only drives the async compilation (that actually happens in a
>> subprocess) so we don't load cl-lib in the Emacs the user is actually
>> using.  This should not be that hard (optimism mode on).
>
> I don't see the need, but I will not object if you want to make such
> changes.

Okay I'll think about it, might be a positive change even leaving aside
the cl-lib discussion.

Thanks

  Andrea

Re: Lisp files that load cl-lib in problematical ways

[Björn Bidar]

Alan Mackenzie <acm@muc.de> writes:

> Hello, Eli.
>> > I fear that that's the way things will stay, because it would be too
>> > much work to fix.  But more because there are enough contributors who
>> > think that cl-lib is just an ordinary part of Emacs to be used freely
>> > without restraint.  I am not among this group of contributors.
>
>> We cannot possibly expect people to contribute code if we force them
>> not to use the macros they are used to.  If cl-lib is not loaded as
>> result, that is good enough for us, I think.
>
> We "force" them to use Emacs Lisp, but they still contribute.  The
> problem is that use of cl- makes maintenance of other people's code much
> harder, at least for me.  I'm sure I'm not alone.
>
> cl- probably multiplies the size of the Lisp language parts of Elisp
> (i.e. things like defun, defmacro, if, while, cond, ...) by around three.
> I'm not able to learn all these fancy macros which are used rarely enough
> not to be learnable, but frequently enough to cause maintenance headaches
> - every time I encounter `if-let' and the like, it just stops me in my
> tracks.  I either have to guess what it means, which makes me
> uncomfortable, or laboriously consult a doc string, if any.
>
> Yes, I know if-let isn't a cl- function, and its doc string is of
> reasonable quality.  Many cl- functions don't have doc strings, or have
> inadequate ones, however.
>
> There was a time in the medium past when the use of run time cl- was not
> allowed.  Emacs was then, from my point of view, much easier to maintain.
> At some stage this rule was done away with, but I don't recall there
> being any discussion on emacs-devel about this change.  There should have
> been such discussion.
>

I think if people find cl- useful there are reasons for that.
That code changes, things get harder to read if one doesn't know the new
code is normal I think.

Resisting against change, doesn't seem like the best option to me.
But the only change that is necessary from my point of view is that the
documentation is on par.

As someone that is new in lisp programming I found that these shortcuts
that macros like these allow make things easier but in sometimes harder
to understand what's going on.

Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Eli Zaretskii wrote:

>> The "levels" introduced in this discussion are, it would
>> seem
>> 
>>   - files that are used by vanilla Emacs that use cl-lib
>>   
>>   - such files that use it at compile time
>> 
>>   - such files that use it a run time
>> 
>>   - such files that are loaded when Emacs is executed with
>>     no arguments
>> 
>> but - what is the difference? I think it is pretty clear
>> that cl-lib is used quite broadly, probably because people
>> find it useful.
>> 
>> So what does it matter if it is "purged" from one or
>> several of these levels? What gain is there if that
>> is achieved?
>
> It matters to us because we decided long ago to avoid
> loading cl-lib at runtime in vanilla Emacs.

But as we have seen in practice it _is_ loaded, be it by the
debugger, by native compilation, or any other package or piece
of Elisp that uses it, or uses something else that is using
it. It is all interconnected, and because it is used so much
already, using Emacs in practice implies having cl-lib loaded
at run time.

> There are several good reasons: bloat, unnecessary namespace
> pollution, etc.

People use it because it is useful: there is `cl-loop',
`cl-incf', `cl-decf', `cl-pushnew' and many others examples of
that. Those are not complicated to use, they are all prefixed
with cl- (as we see) and have good docstrings.

> But all this is not really relevant for the code that you
> (or any one of us) write for their own personal use, it is
> only relevant to code contributed to Emacs.

But we don't know what people who are starting to write Elisp
today for their personal use go on to contribute to Emacs
packages that make it to the ELPAs, or core Emacs for that
matter. That whole distinction does not make any sense to me.
If it is harmful, which it isn't, no one should use it and it
doesn't matter in what context, be it personal use, ELPA
packages, core Emacs, compile time, run time - what does
it matter?

On the other hand if it _is_ useful - and the number of people
using it and the number of files where it is used, and the
number of occurrences indicate that it is - there is no reason
to disencourage people from using it, anywhere.

You, and everyone else writing Elisp for that matter, are of
coures equally permitted not to use it, if they consider it
harmful, it just shouldn't be labeled detrimental as a policy
with really no good arguments supporting that claim that we
have seen thus far at least.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 1074 bytes --]


Emanuel Berg <incal@dataswamp.org> writes:

>> But all this is not really relevant for the code that you
>> (or any one of us) write for their own personal use, it is
>> only relevant to code contributed to Emacs.
>
> But we don't know what people who are starting to write Elisp
> today for their personal use go on to contribute to Emacs
> packages that make it to the ELPAs, or core Emacs for that
> matter. That whole distinction does not make any sense to me.

I have quite a few things in my .emmacs.d/init.el that are right for me
but would not be suitable for core.

That starts with (setq gc-cons-threshold (* 100 1024 1024))
;; Make gc pauses faster by decreasing the threshold again (from the increased initial).
(setq gc-cons-threshold (* 20 1024 1024))
;; speed up reading from external processes
(setq read-process-output-max (* 1024 1024)) ;; 1mb

and continues through my exwm setup, reaching key-chords and interactive
ido preview.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Dr. Arne Babenhauserheide wrote:

>> But we don't know what people who are starting to write
>> Elisp today for their personal use go on to contribute to
>> Emacs packages that make it to the ELPAs, or core Emacs for
>> that matter. That whole distinction does not make any sense
>> to me.
>
> I have quite a few things in my .emmacs.d/init.el that are
> right for me but would not be suitable for core.
>
> That starts with (setq gc-cons-threshold (* 100 1024 1024))
> ;; Make gc pauses faster by decreasing the threshold again
> (from the increased initial).
> (setq gc-cons-threshold (* 20 1024 1024))
> ;; speed up reading from external processes
> (setq read-process-output-max (* 1024 1024)) ;; 1mb
>
> and continues through my exwm setup, reaching key-chords and
> interactive ido preview.

Case closed :)

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Fri, 20 Oct 2023 05:02:14 +0200
> 
> Eli Zaretskii wrote:
> 
> >> So what does it matter if it is "purged" from one or
> >> several of these levels? What gain is there if that
> >> is achieved?
> >
> > It matters to us because we decided long ago to avoid
> > loading cl-lib at runtime in vanilla Emacs.
> 
> But as we have seen in practice it _is_ loaded, be it by the
> debugger, by native compilation, or any other package or piece
> of Elisp that uses it, or uses something else that is using
> it. It is all interconnected, and because it is used so much
> already, using Emacs in practice implies having cl-lib loaded
> at run time.

It is okay for cl-lib to be loaded when packages that use it are
loaded.  The same happens with gazillion other packages in Emacs; for
example, try loading Gnus or Org or Eshell, and see how much stuff
unrelated to these packages per se gets loaded as result, because
those packages need some non-default infrastructure.

What matters to us is that this unnecessary stuff is not loaded in
"emacs -Q", i.e. is not dumped and preloaded into the bare
uncustomized Emacs session.  This lets users start from the leanest
possible Emacs, then load what they need, without being "punished" by
loads of stuff they never need or want.

For Emacs to preload unnecessary stuff is unclean, and we try to avoid
that uncleanliness.

> > There are several good reasons: bloat, unnecessary namespace
> > pollution, etc.
> 
> People use it because it is useful: there is `cl-loop',
> `cl-incf', `cl-decf', `cl-pushnew' and many others examples of
> that. Those are not complicated to use, they are all prefixed
> with cl- (as we see) and have good docstrings.

This is not relevant to the issue at hand.

> > But all this is not really relevant for the code that you
> > (or any one of us) write for their own personal use, it is
> > only relevant to code contributed to Emacs.
> 
> But we don't know what people who are starting to write Elisp
> today for their personal use go on to contribute to Emacs
> packages that make it to the ELPAs, or core Emacs for that
> matter.

That's okay, they will learn it on-the-fly, from review comments to
their contributions, like with the rest of our coding conventions,
which are usually not clear to new contributors when they first
contribute.

> You, and everyone else writing Elisp for that matter, are of
> coures equally permitted not to use it, if they consider it
> harmful, it just shouldn't be labeled detrimental as a policy
> with really no good arguments supporting that claim that we
> have seen thus far at least.

See above: this is not relevant to the issue at hand.

Re: [External] : Re: Shrinking the C core

[Arsen Arsenović]

[-- Attachment #1: Type: text/plain, Size: 774 bytes --]

Hi,

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > We have them in Elisp as well, `cl-remove-if' and
>   > `cl-remove-if-not', both in cl-seq.el.
>
> A partial emulation of some Common Lisp functions is present
> in the cl-lib library, for emulation purposes.  It is not supposed to
> be used a lot.

Apologies for my ignorance (and late reply), but why not?  It hosts some
quite useful functions.

For instance, I'm not aware of an ``flet'' equivalent in Elisp proper.

Thanks in advance, have a lovely day.
-- 
Arsen Arsenović

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 251 bytes --]

Re: [External] : Re: Shrinking the C core

[Alfred M. Szmidt]

   For instance, I'm not aware of an ``flet'' equivalent in Elisp proper.

There was a flet, but it had different semantics from cl-flet.  And
was thus removed / made obsolete.

Re: Lisp files that load cl-lib in problematical ways

[Michael Heerdegen]

Alan Mackenzie <acm@muc.de> writes:

> > If the above is correct, then how do you explain that in "emacs -Q" I
> > don't see the cl-lib feature present?
>
> Because the files that use cl- don't have a (require 'cl-lib) in them.
> Quite a lot of the function calls are things like cl-assert, cl-incf, or
> cl-defgeneric, which won't cause the loading of cl- until they are
> actually run.

These are all macros.  Unless you explicitly load the .el source files,
nothing of these will cause loading cl-lib when the according code is
run.

Michael.

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Which of those files are usually loaded when you start Emacs
  > > with no arguments? Those we should certainly fix.

  > I don't know, but at least one since you get the cl-lib stuff
  > even at 'emacs -Q'.

That reply gives no useful information.  It only makes
a specious argument.

I'm starting to think that your presence on the list
msy be motivated by the goak of being a troll.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > That is not true, since in "emacs -Q" I get this:

  >   (featurep 'cl-lib) => nil

  > both in GUI and TTY (i.e. -nw) sessions.

Interesting.  I am glad I was mistaken about that file.
But I wonder how I made the mistake.

I recall sending a first email some weeks ago in which I said which
file caused loading of cl-lib.  But I can't find that outgoing message now.
Did anyone here receive that message?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Quite a lot of the function calls are things like cl-assert, cl-incf, or
  > cl-defgeneric, which won't cause the loading of cl- until they are
  > actually run.

Those are macros, right?  These macros are not supposed to need cl-lib
at run time; they are supposed to be handled (when the code is
compiled) at compile time.

Maybe people changed these macros and caused them to load cl-lib.

But maybe not all of them.

I looked into cl-assert and found that it generates a call to
cl--assertion-failed, which is defined in xo cl-preloaded.el.
So I think that macro is ok -- the problem in question doesn't
seem to happen in cl-assert.

Which of the cl- macros really generate calls that require cl-lib
to be loaded?  We should fix those.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Sat, 21 Oct 2023 01:21:16 -0400
> 
> Which of the cl- macros really generate calls that require cl-lib
> to be loaded?

I don't think there are any.

> We should fix those.

If they exist, yes.

Re: [External] : Re: Shrinking the C core

[Emanuel Berg]

Alfred M. Szmidt wrote:

>> For instance, I'm not aware of an ``flet'' equivalent in
>> Elisp proper.
>
> There was a flet, but it had different semantics from
> cl-flet. And was thus removed / made obsolete.

You used to be able to use "labels" for that. But it is now
removed, however there is `cl-labels'.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Eli Zaretskii wrote:

> What matters to us is that this unnecessary stuff is not
> loaded in "emacs -Q", i.e. is not dumped and preloaded into
> the bare uncustomized Emacs session. This lets users start
> from the leanest possible Emacs, then load what they need,
> without being "punished" by loads of stuff they never need
> or want.

In practice cl-lib will be loaded pretty much instantly even
from minor use, including from the starting point that is the
minimal 'emacs -Q'.

But if we focus on this level in particular, defined by
'emacs -Q' and no use whatsoever on top of that, then cl-lib
isn't loaded as you say.

Because 'emacs -Q' should load as little as possible, it makes
sense not to load cl-lib and, as has been showed, that doesn't
happen either.

So I suppose everyone is happy about that then. It will also
be easy for you to keep it that way by not introducing it to
anything that is loaded by 'emacs -Q'.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Alan Mackenzie]

Hello, Richard.

On Sat, Oct 21, 2023 at 01:21:16 -0400, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]

>   > Quite a lot of the function calls are things like cl-assert, cl-incf, or
>   > cl-defgeneric, which won't cause the loading of cl- until they are
>   > actually run.

> Those are macros, right?  These macros are not supposed to need cl-lib
> at run time; they are supposed to be handled (when the code is
> compiled) at compile time.

A lot of them are macros, yes.  When I filter out the macros from my
list, and also functions of the form cl-generic-* and cl--generic-*
(cl-generic is loaded at dump time), I'm left with these functions:

* cl-call-next-method
* cl--class-parents
cl-member
* cl-next-method-p
cl-old-struct-compat-mode
cl-plusp
cl-print-object
* cl--slot-descriptor-name
* cl--slot-descriptor-props
* cl--struct-class-children-sym
* cl--struct-class-parents
* cl--struct-register-child

..  I've marked with an asterisk those that are in cl-preloaded.el or
cl-generic.el.

Of the remaining functions:
(i) cl-member is used only in a compiler macro for add-to-list in
  subr.el.  This is OK.
(ii) cl-old-struct-compat-mode is called from cl-struct-define in
  cl-preloaded.el.  It is an auto-load function which only gets loaded
  when cl-struct-define is called with an obsolete calling convention.
(iii) cl-plusp is a defsubst, effectively a macro.
(iv) cl-print-object is a defgeneric function in cl-print.el.  There is
  a cl-defmethod of cl-print-object in nadvice.el.  This contains calls
  to cl-print-object methods defined in cl-print.el  Since
  cl-print-object is an auto-load, this should only cause cl-lib to get
  loaded at run time, not dump time.

So it would appear there are no cl- functions used in the dumped .el
files which would cause cl-lib to be loaded.

There was only one cl- special variable which got bound to a non-nil
value in the dumped .el files.  That was cl--generic-edebug-name,
defined and bound in cl-generic.el.

It would take a more rigorous analysis to find any cl- variables bound
to nil in these files.

> Maybe people changed these macros and caused them to load cl-lib.

> But maybe not all of them.

I'm convinced this is not the case, unless it is by binding cl- special
variables to nil.

> I looked into cl-assert and found that it generates a call to
> cl--assertion-failed, which is defined in xo cl-preloaded.el.
> So I think that macro is ok -- the problem in question doesn't
> seem to happen in cl-assert.

> Which of the cl- macros really generate calls that require cl-lib
> to be loaded?  We should fix those.

After scanning the source code, I'm convinced there aren't any, with the
above proviso.

> -- 
> Dr Richard Stallman (https://stallman.org)
> Chief GNUisance of the GNU Project (https://gnu.org)
> Founder, Free Software Foundation (https://fsf.org)
> Internet Hall-of-Famer (https://internethalloffame.org)

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Lisp files that load cl-lib in problematical ways

[Jens Schmidt]

On 2023-10-20  08:49, Eli Zaretskii wrote:

> For Emacs to preload unnecessary stuff is unclean, and we try to avoid
> that uncleanliness.

I have to admit that I don't fully understand the dump/bootstrap logic
of Emacs (yet), but doesn't inhibit the following test in Frequire already
any/most uncleanliness:

      /* This is to make sure that loadup.el gives a clear picture
         of what files are preloaded and when.  */
      if (will_dump_p () && !will_bootstrap_p ())
        {
          /* Avoid landing here recursively while outputting the
             backtrace from the error.  */
          gflags.will_dump_ = false;
          error ("(require %s) while preparing to dump",
                 SDATA (SYMBOL_NAME (feature)));
        }

At least that error fires during dump ("Dump mode: pdump") when I try to
require, for example, cl-lib from a file loaded from loadup.el.

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> Date: Sat, 21 Oct 2023 17:17:24 +0200
> Cc: emacs-devel@gnu.org
> From: Jens Schmidt <jschmidt4gnu@vodafonemail.de>
> 
> On 2023-10-20  08:49, Eli Zaretskii wrote:
> 
> > For Emacs to preload unnecessary stuff is unclean, and we try to avoid
> > that uncleanliness.
> 
> I have to admit that I don't fully understand the dump/bootstrap logic
> of Emacs (yet), but doesn't inhibit the following test in Frequire already
> any/most uncleanliness:
> 
>       /* This is to make sure that loadup.el gives a clear picture
>          of what files are preloaded and when.  */
>       if (will_dump_p () && !will_bootstrap_p ())
>         {
>           /* Avoid landing here recursively while outputting the
>              backtrace from the error.  */
>           gflags.will_dump_ = false;
>           error ("(require %s) while preparing to dump",
>                  SDATA (SYMBOL_NAME (feature)));
>         }

No, because there's more than one way of loading a Lisp package.
Moreover, we could have a situation where cl-lib is not preloaded, but
just starting "emacs -Q" loads it (e.g., via some autoloaded feature),
and that also should be avoided.

Re: Lisp files that load cl-lib in problematical ways

[Alan Mackenzie]

Hello, Eli.

On Sat, Oct 21, 2023 at 10:42:05 +0300, Eli Zaretskii wrote:
> > From: Richard Stallman <rms@gnu.org>
> > Cc: emacs-devel@gnu.org
> > Date: Sat, 21 Oct 2023 01:21:16 -0400

> > Which of the cl- macros really generate calls that require cl-lib
> > to be loaded?

> I don't think there are any.

> > We should fix those.

> If they exist, yes.

Just as a matter of interest, what loads cl-lib in my .emacs is
desktop.el.  It does this for a single call to cl-list* which could be
seen as rather contrived.

So any Emacs session loading desktop will also be loading cl-lib; that's
virtually any Emacs session in which real work will be getting done, I
think.

It would be fairly easy to remove cl-lib from desktop.el.  Or, on the
other hand, we could just as well preload cl-lib, and save an
unmeasurable amount of loading time at each Emacs startup.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Lisp files that load cl-lib in problematical ways

[Gerd Möllmann]

Alan Mackenzie <acm@muc.de> writes:

> It would be fairly easy to remove cl-lib from desktop.el.  Or, on the
> other hand, we could just as well preload cl-lib, and save an
> unmeasurable amount of loading time at each Emacs startup.

+1

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> Date: Sat, 21 Oct 2023 17:53:31 +0000
> Cc: rms@gnu.org, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> Just as a matter of interest, what loads cl-lib in my .emacs is
> desktop.el.  It does this for a single call to cl-list* which could be
> seen as rather contrived.
> 
> So any Emacs session loading desktop will also be loading cl-lib; that's
> virtually any Emacs session in which real work will be getting done, I
> think.
> 
> It would be fairly easy to remove cl-lib from desktop.el.  Or, on the
> other hand, we could just as well preload cl-lib, and save an
> unmeasurable amount of loading time at each Emacs startup.

No, we will not preload cl-lib.  But if some package loads cl-lib for
a small number of functions that can be easily replaced with
non-cl-lib functions, patches to make such packages leaner and meaner
will be welcome.

RE: [External] : Re: Lisp files that load cl-lib in problematical ways

[Drew Adams]

> > It would be fairly easy to remove cl-lib from desktop.el.

+1

Re: Lisp files that load cl-lib in problematical ways

[Alan Mackenzie]

Hello, Eli.

On Sat, Oct 21, 2023 at 22:00:16 +0300, Eli Zaretskii wrote:
> > Date: Sat, 21 Oct 2023 17:53:31 +0000
> > Cc: rms@gnu.org, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > Just as a matter of interest, what loads cl-lib in my .emacs is
> > desktop.el.  It does this for a single call to cl-list* which could be
> > seen as rather contrived.

> > So any Emacs session loading desktop will also be loading cl-lib; that's
> > virtually any Emacs session in which real work will be getting done, I
> > think.

> > It would be fairly easy to remove cl-lib from desktop.el.  Or, on the
> > other hand, we could just as well preload cl-lib, and save an
> > unmeasurable amount of loading time at each Emacs startup.

> No, we will not preload cl-lib.  But if some package loads cl-lib for
> a small number of functions that can be easily replaced with
> non-cl-lib functions, patches to make such packages leaner and meaner
> will be welcome.

Apologies, I was mistaken, it's not quite so simple.  desktop.el itself
could have its cl-lib dependency easily removed, but it loads
frameset.el, the bit of desktop that saves frame layouts.

frameset.el makes moderately heavy use of cl-lib, including at least two
externally visible cl-defun's where keyword parameters are used.  It
would be a lot of work to replace cl-lib in this file, and doing this
might be viewed as ungenerous towards its author, Juanma Barranquero.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Lisp files that load cl-lib in problematical ways

[Björn Bidar]

Emanuel Berg <incal@dataswamp.org> writes:

>> There are several good reasons: bloat, unnecessary namespace
>> pollution, etc.
>
> People use it because it is useful: there is `cl-loop',
> `cl-incf', `cl-decf', `cl-pushnew' and many others examples of
> that. Those are not complicated to use, they are all prefixed
> with cl- (as we see) and have good docstrings.
>
>> But all this is not really relevant for the code that you
>> (or any one of us) write for their own personal use, it is
>> only relevant to code contributed to Emacs.
>
> But we don't know what people who are starting to write Elisp
> today for their personal use go on to contribute to Emacs
> packages that make it to the ELPAs, or core Emacs for that
> matter. That whole distinction does not make any sense to me.
> If it is harmful, which it isn't, no one should use it and it
> doesn't matter in what context, be it personal use, ELPA
> packages, core Emacs, compile time, run time - what does
> it matter?

From my point of view in some way it does sound like that some are
against it for ideological reasons rather than technical.
E.g. those that makes code unreadable or lets rewrite something to drop
the dependency to not load it.
It does sound like a generational conflict, newer developers write code
sometimes different than older ones.
Why not use cl-lib when it makes it easier to achieve what I want easier
and easier to understand than "vanilla" elisp.
But as other said, this wasn't the point of the thread even when some
messages did sound very ideological.

>
> On the other hand if it _is_ useful - and the number of people
> using it and the number of files where it is used, and the
> number of occurrences indicate that it is - there is no reason
> to disencourage people from using it, anywhere.

I think these things are also often a long the lines we don't want you
to use thing XX for a reason, so discourage them from even mentioning
it.
Ideological vs practicism. 
Similar as when people mention other repositories for package.el that
are not from Gnu..

Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Björn Bidar wrote:

>>> But all this is not really relevant for the code that you
>>> (or any one of us) write for their own personal use, it is
>>> only relevant to code contributed to Emacs.
>>
>> But we don't know what people who are starting to write
>> Elisp today for their personal use go on to contribute to
>> Emacs packages that make it to the ELPAs, or core Emacs for
>> that matter. That whole distinction does not make any sense
>> to me. If it is harmful, which it isn't, no one should use
>> it and it doesn't matter in what context, be it personal
>> use, ELPA packages, core Emacs, compile time, run time -
>> what does it matter?
>
> From my point of view in some way it does sound like that
> some are against it for ideological reasons rather than
> technical. E.g. those that makes code unreadable or lets
> rewrite something to drop the dependency to not load it.
> It does sound like a generational conflict, newer developers
> write code sometimes different than older ones. Why not use
> cl-lib when it makes it easier to achieve what I want easier
> and easier to understand than "vanilla" elisp. But as other
> said, this wasn't the point of the thread even when some
> messages did sound very ideological.

Exactly, that's the way it sounds.

>> On the other hand if it _is_ useful - and the number of
>> people using it and the number of files where it is used,
>> and the number of occurrences indicate that it is - there
>> is no reason to disencourage people from using
>> it, anywhere.
>
> I think these things are also often a long the lines we
> don't want you to use thing XX for a reason, so discourage
> them from even mentioning it. Ideological vs practicism.

Again, exactly. Only it is difficult to pinpoint what is the
ideology or set of thoughts that drives the anti-cl-lib
tendency.

As for practicality, I think that has been showed many times
over by now, not the least in the amount of usage already
present in Emacs. There is no going around that, contributors
to Emacs have already voted with their fingers in favor of
cl-lib.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Gerd Möllmann]

Alan Mackenzie <acm@muc.de> writes:

> frameset.el makes moderately heavy use of cl-lib, including at least two
> externally visible cl-defun's where keyword parameters are used.  It
> would be a lot of work to replace cl-lib in this file, and doing this
> might be viewed as ungenerous towards its author, Juanma Barranquero.

Yes, that, and it would make an unmeasurable difference.

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Björn Bidar <bjorn.bidar@thaodan.de>
> Date: Sun, 22 Oct 2023 03:52:29 +0300
> 
> >From my point of view in some way it does sound like that some are
> against it for ideological reasons rather than technical.

I gave specific technical reasons for why we are trying to avoid
preloading cl-lib.

> Ideological vs practicism. 

It isn't.  Again, I explained the policy.  That Emanuel disregards
what I said and keeps claiming it's ideology or that people prefer
cl-lib etc. is not relevant to the actual issue at hand, nor to
specific questions Richard asked (and I answered).  This is purely a
technical, practical issue of interest to Emacs maintainers, it has no
bearing on how people write their own code or Emacs extensions they
submit to Emacs.

Of course, if cl-lib is used without any justification, patch review
comments will usually point that out, but in that case cl-lib is not
different from any other package, because Lisp code should not
gratuitously load stuff it can do without.  IOW, this is a matter of
writing clean code, not of preferring or not preferring cl-lib.

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > and got the following results:

  > abbrev.el:2

A generally useful minor mode as abbrev.el should not load cl-lib.
Why does it?  I investigated.

abbrev--check-chars uses the macro cl-pushnew.  Generally we
arrange for cl- macros not to load cl-lib.  But cl-pushnew
is _defined_ in cl-lib.  It ought to be defined with and like
other cl- macros that are meant for users to use.

That's a bug, and probably not a deep one.  Would someone please fix
it?

I investigated byte-run.el.  I don't think it uses any cl- facility.
Rather, it declares a let-variable whose name starts with cl-.

I think that if we investigate the other general-purpose packages
in the list you posted (I copied it below), we wlll find many such bugs that
got in because developers were not on guard against them.

Once we fix them all, how about if we add a regression test
to verify that various packages anyone might load at any time
do not use cl-lib.  With that, bugs like this would get caught
right away.

Here's the list, with some files crossed off that I've just handled.

;; abbrev.el:2
dnd.el:4
;; emacs-lisp/byte-run.el:1
emacs-lisp/cconv.el:13
emacs-lisp/cl-generic.el:56
emacs-lisp/cl-preloaded.el:16
emacs-lisp/eldoc.el:6
emacs-lisp/lisp-mode.el:6
emacs-lisp/nadvice.el:2
emacs-lisp/oclosure.el:8
emacs-lisp/seq.el:25
emacs-lisp/shorthands.el:1
emacs-lisp/syntax.el:5
emacs-lisp/tabulated-list.el:1
emacs-lisp/timer.el:1
font-lock.el:1
frame.el:5
international/mule-cmds.el:3
international/ucs-normalize.el:1
isearch.el:3
ldefs-boot.el:3
loaddefs.el:3
minibuffer.el:12
progmodes/elisp-mode.el:11
progmodes/prog-mode.el:1
register.el:3
replace.el:1
select.el:3
simple.el:9
startup.el:1
subr.el:1
tab-bar.el:4
term/android-win.el:4
term/haiku-win.el:3
term/ns-win.el:2
term/pc-win.el:6
term/pgtk-win.el:3
term/w32-win.el:4
term/x-win.el:2
uniquify.el:2
vc/vc-hooks.el:1


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > But as we have seen in practice it _is_ loaded, be it by the
  > debugger, by native compilation, or any other package or piece
  > of Elisp that uses it,

Let's fix the debugger, and native compilation, not to load cl-lib.

It's not so bad if you load a specialized package that serves your code
and it uses cl-lib.  But the debugger can be loaded to debug anything,
even programs you have nothing to do with.  It's rude for the debugger
to load cl-lib.

Likewise for native compilation.  You might recompile anything.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > So it would appear there are no cl- functions used in the dumped .el
  > files which would cause cl-lib to be loaded.

That is good news.  Thanks for studying this.

Nonetheless, we do have some problems.  For instance, the debugger
should not load cl-lib.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Which of the cl- macros really generate calls that require cl-lib
  > > to be loaded?

  > I don't think there are any.

If there aren't any, that's good -- but I think we have found a few.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Sun, 22 Oct 2023 22:08:55 -0400
> 
>   > But as we have seen in practice it _is_ loaded, be it by the
>   > debugger, by native compilation, or any other package or piece
>   > of Elisp that uses it,
> 
> Let's fix the debugger, and native compilation, not to load cl-lib.

That's not practical.  I don't see any reason to waste our resources
on doing that.

> It's not so bad if you load a specialized package that serves your code
> and it uses cl-lib.  But the debugger can be loaded to debug anything,
> even programs you have nothing to do with.  It's rude for the debugger
> to load cl-lib.

The debugger uses cl-lib for good reasons.

> Likewise for native compilation.  You might recompile anything.

Likewise here.

I don't see a problem in these using cl-lib, and don't think we should
be bothered about that.

Re: Lisp files that load cl-lib in problematical ways

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Richard Stallman <rms@gnu.org>
>> Cc: emacs-devel@gnu.org
>> Date: Sun, 22 Oct 2023 22:08:55 -0400
>>
>> Let's fix the debugger, and native compilation, not to load cl-lib.
>
> That's not practical.  I don't see any reason to waste our resources
> on doing that.

+1

> I don't see a problem in these using cl-lib, and don't think we should
> be bothered about that.

Yes, let's focus our efforts on fixing real problems.

Re: Lisp files that load cl-lib in problematical ways

[Alan Mackenzie]

Hello, Björn.

On Sun, Oct 22, 2023 at 03:52:29 +0300, Björn Bidar wrote:

[ .... ]

> >From my point of view in some way it does sound like that some are
> against it [the use of cl-lib] for ideological reasons rather than
> technical.  E.g. those that makes code unreadable or lets rewrite
> something to drop the dependency to not load it.

You seem to be saying that avoiding making code unreadable (i.e.
difficult to maintain) is an "ideological" thing.  Here "ideological"
means "can be dismissed without consideration", I think.

> It does sound like a generational conflict, newer developers write code
> sometimes different than older ones.

I think it's more likely to be a "conflict" between those who have to
maintain other people's code and those who merely write it.  But seeing
as how I'm the only member of the first group to express a view on this,
it's difficult to be sure.

> Why not use cl-lib when it makes it easier to achieve what I want easier
> and easier to understand than "vanilla" elisp.

That's a big "when" you use.  If the use of cl-lib actually did make
things "easier to understand", I would agree with you.  But it doesn't.
I'm speaking from bitter experience here, like trying to track down a bug
at one o'clock in the morning, and encountering some obscure cl-lib
function or macro.

A great number of them are obscure.  They have inadequate doc-strings, if
any at all.  That is to say, doc-strings which don't say what the thing
does, and don't say what the parameters mean or give their form.

As an example, take the last macro in cl-macs.el, `cl-deftype'.  Its doc
string is just

    "Define NAME as a new data type.
    The type name can then be used in `cl-typecase', `cl-check-type', etc."

..  At 01:00, I need to start asking what is NAME?  Is it a symbol?  Is it
a string?  Is it, perhaps, some type of list structure?  What does it
mean to "define" it?  And what exactly is meant by a "data type"?  And
there are the other two parameters ARGLIST and BODY which are not
mentioned at all, and would appear to have nothing to do with defining
data types.  In practice, this means having to study the source code of
several functions/macros in detail, something which would be not be
needed had it not been used in the first place.  This is unrewarding,
tedious, and time consuming.

This poor standard of doc strings is typical throughout cl-lib, even if
not universal.

> But as other said, this wasn't the point of the thread even when some
> messages did sound very ideological.

Some, perhaps, but not all.

> > On the other hand if it _is_ useful - and the number of people
> > using it and the number of files where it is used, and the
> > number of occurrences indicate that it is - there is no reason
> > to disencourage people from using it, anywhere.

I have outlined a reason above.  Fifty years ago, you could have been
saying the same about the goto statement.

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Sun, 22 Oct 2023 22:08:55 -0400
> 
>   > abbrev.el:2
> 
> A generally useful minor mode as abbrev.el should not load cl-lib.
> Why does it?  I investigated.
> 
> abbrev--check-chars uses the macro cl-pushnew.  Generally we
> arrange for cl- macros not to load cl-lib.  But cl-pushnew
> is _defined_ in cl-lib.  It ought to be defined with and like
> other cl- macros that are meant for users to use.

It would not have helped to define cl-pushnew on cl-macs, because
cl-pushnew calls cl-adjoin, which is a function defined on cl-lib.  So
basically cl-pushnew is a (rather thin) wrapper around cl-adjoin, and
thus it is correctly defined on the same file where cl-adjoin is
defined.  I see no problem here.

> I investigated byte-run.el.  I don't think it uses any cl- facility.
> Rather, it declares a let-variable whose name starts with cl-.

byte-run.el indeed doesn't need cl-lib, and loading byte-run.elc
doesn't load cl-lib.  So I see no problem here, either.  IOW, this is
a false positive due to the naïve search pattern used to detect this.

> I think that if we investigate the other general-purpose packages
> in the list you posted (I copied it below), we wlll find many such bugs that
> got in because developers were not on guard against them.

I very much doubt that we will find many such bugs, because we
actually pay close attention to these aspects as part of our patch
review process.  But, of course, there could be places we missed, so
if someone finds them, please do report them and/or propose patches to
fix them.

> Once we fix them all, how about if we add a regression test
> to verify that various packages anyone might load at any time
> do not use cl-lib.  With that, bugs like this would get caught
> right away.

It isn't easy to write such a test, since our test suite
infrastructure, ert.el, itself uses cl-lib extensively.  But if
someone can come up with a test, please do submit it, and TIA.

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> Date: Mon, 23 Oct 2023 11:07:27 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> This poor standard of doc strings is typical throughout cl-lib, even if
> not universal.

Patches to improve the documentation of CL macros and functions will
be most welcome.

Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Alan Mackenzie wrote:

>> From my point of view in some way it does sound like that
>> some are against it [the use of cl-lib] for ideological
>> reasons rather than technical. E.g. those that makes code
>> unreadable or lets rewrite something to drop the dependency
>> to not load it.
>
> You seem to be saying that avoiding making code unreadable
> (i.e. difficult to maintain) is an "ideological" thing.

From Eli we heard cl-lib adds bloat and fills up the global
namespace. From RMS and you we hear it makes code more
difficult to maintain.

At the same time Eli says it is just another Elisp package and
library which should be treated as any other, i.e. it should
only be used and brought in when needed. This, of course,
I think everyone will agree on.

But the other arguments are more difficult to see from here.
Rather than bloat and filling up the namespace unnecessarily,
I think it makes Elisp more powerful. So to me, cl-lib is
a good example och packages we do want, and the more
the merrier.

When there is no clear way to do something using non-cl-lib
Elisp, I see no harm in using cl-lib (on the contrary) and
I still don't understand why it appears to be considered OK in
some places and some stages of Emacs execution and not others.
Wherever it is useful and provides expressive and
computational power otherwise unavailable, it should be OK to
use. With the possible exception of 'emacs -Q', since there
the minimalist policy makes sense and one can make
that argument.

One can also think that cl-lib's presence can bring guys from
the Common Lisp world here, which is of course a positive
thing. And it doesn't mean now all Emacs will be Common Lisp.

> That's a big "when" you use. If the use of cl-lib actually
> did make things "easier to understand", I would agree with
> you. But it doesn't.

E.g. (cl-incf some-var) isn't better and more clean than (setq
some-var (1+ some-var)) ? I think it is! Other examples are,
as has been mentioned, `cl-decf' and `cl-pushnew'. I can see
that `cl-loop' has some higher threshold to understand and to
get used to but (1) it adds so much expressive power increase
in complexity is unavoidable and expected; and, (2) it still
should be no match for programmers maintaining Emacs?

> As an example, take the last macro in cl-macs.el,
> `cl-deftype'. Its doc string is just [...]

Well, we should improve the docstrings whenever they are
insufficient, this is the same as any other library.

>> On the other hand if it _is_ useful - and the number of
>> people using it and the number of files where it is used,
>> and the number of occurrences indicate that it is - there
>> is no reason to disencourage people from using
>> it, anywhere.
>
> I have outlined a reason above. Fifty years ago, you could
> have been saying the same about the goto statement.

50 years ago, some programming languages were so limited the
goto was needed. Today, some are so powerful, one can add
cl-lib to make them even better. And indeed, we now have
`cl-return', even ;)

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Eli Zaretskii wrote:

>> This poor standard of doc strings is typical throughout
>> cl-lib, even if not universal.
>
> Patches to improve the documentation of CL macros and
> functions will be most welcome.

`elint-current-buffer' does not find anything to correct in
cl-lib.el.

(checkdoc-current-buffer t) says the following:

*** cl-lib.el: checkdoc-current-buffer
cl-lib.el:133: Argument ‘keys’ should appear (as KEYS) in the doc string
cl-lib.el:160: All variables and subroutines might as well have a documentation string
cl-lib.el:166: All variables and subroutines might as well have a documentation string
cl-lib.el:197: All variables and subroutines might as well have a documentation string
cl-lib.el:197: All variables and subroutines might as well have a documentation string
cl-lib.el:226: Arguments occur in the doc string out of order
cl-lib.el:258: Argument ‘specs’ should appear (as SPECS) in the doc string
cl-lib.el:357: Argument ‘cl-func’ should appear (as CL-FUNC) in the doc string
cl-lib.el:450: Argument ‘rest’ should appear (as REST) in the doc string
cl-lib.el:488: Argument ‘cl-item’ should appear (as CL-ITEM) in the doc string
cl-lib.el:501: Argument ‘cl-new’ should appear (as CL-NEW) in the doc string
cl-lib.el:510: All variables and subroutines might as well have a documentation string
cl-lib.el:539: All variables and subroutines might as well have a documentation string
cl-lib.el:577: Probably "returns" should be imperative "return"

I CC this to the maintainer, but like to add that I personally
have not experienced the documentation of cl-lib bad in
any way. That said, for a person who knows the code well,
those warnings should be a pretty quick fix.

There are other files to cl-lib, of course.

Emacs version:
  GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, cairo
  version 1.16.0) of 2023-10-10 [commit
  239db5d5162f6f9a4a6735e176c8d306f18e50e7]

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Mon, 23 Oct 2023 14:18:51 +0200
> 
> >From Eli we heard cl-lib adds bloat and fills up the global
> namespace. From RMS and you we hear it makes code more
> difficult to maintain.
> 
> At the same time Eli says it is just another Elisp package and
> library which should be treated as any other, i.e. it should
> only be used and brought in when needed. This, of course,
> I think everyone will agree on.

So you get to decide who do you listen to.  Choose wisely.

Re: Lisp files that load cl-lib in problematical ways

[Alan Mackenzie]

Hello, Richard.

On Sun, Oct 22, 2023 at 22:11:31 -0400, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]

>   > So it would appear there are no cl- functions used in the dumped .el
>   > files which would cause cl-lib to be loaded.

> That is good news.  Thanks for studying this.

> Nonetheless, we do have some problems.  For instance, the debugger
> should not load cl-lib.

I would agree, but edebug _does_ load cl-lib.  So does desktop, as I've
already pointed out.  Even calling M-: (backtrace) causes cl-lib to get
loaded.  All of these, and likely many others, would take a lot of work
to fix.

It seems that while emacs -Q doesn't itself load cl-lib, there is now
very little, if anything, one can do in Emacs which doesn't load this
library.  I find this regrettable, but can't see there's much to be done
about it.

> -- 
> Dr Richard Stallman (https://stallman.org)
> Chief GNUisance of the GNU Project (https://gnu.org)
> Founder, Free Software Foundation (https://fsf.org)
> Internet Hall-of-Famer (https://internethalloffame.org)

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Lisp files that load cl-lib in problematical ways

[Gregory Heytings]

>
> For instance, the debugger should not load cl-lib.
>

Pretty much everything loads cl-lib nowadays, for example:

- C-s / C-r / M-% / C-M-%
- C-h C-h
- C-h t
- C-h w
- C-h k C-g
- M-x apropos
- clicking in the echo area
- clicking on the modeline
- hovering over the modeline or over the toolbar
- browsing the menus
- clicking on a completion in *Completions*
- CC Mode
- Shell-script mode
- VC (and opening any file in a version-controlled repository)
- Dired
- M-g g (or c or i)
- M-x shell
- M-x eshell
- M-x M-n
- M-. and M-? and M-, (Xref)
- C-x (
- M-&
- C-x m and Gnus and Rmail
- EWW
- Org

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> Date: Mon, 23 Oct 2023 12:54:33 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> Even calling M-: (backtrace) causes cl-lib to get loaded.

That's because we use cl-print to display the backtrace.  That change
was done intentionally and for good reasons.  So that is not a
"problem" that needs to be "fixed", from where I stand.

> It seems that while emacs -Q doesn't itself load cl-lib, there is now
> very little, if anything, one can do in Emacs which doesn't load this
> library.  I find this regrettable, but can't see there's much to be done
> about it.

Neither should we do anything about it (except fixing places where
this is done unintentionally, if there are such places).

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> Date: Mon, 23 Oct 2023 14:06:33 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: emacs-devel@gnu.org
> 
> Pretty much everything loads cl-lib nowadays, for example:

That's not the issue, from my POV.  the issue is whether all these
places load cl-lib for good reasons.  If not, they shouldn't.

Are any of these commands loading cl-lib without a good reason?

Re: Lisp files that load cl-lib in problematical ways

[Gregory Heytings]

>> Pretty much everything loads cl-lib nowadays, for example:
>
> That's not the issue, from my POV.  the issue is whether all these 
> places load cl-lib for good reasons.  If not, they shouldn't.
>
> Are any of these commands loading cl-lib without a good reason?
>

I don't know.  I was only pointing out that the debugger is just one of 
the many cases, and not the most likely one, in which cl-lib is loaded. 
I was also hinting that it would perhaps make sense to preload cl-lib, 
given that it's practically impossible to use Emacs without loading 
cl-lib.

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> Date: Mon, 23 Oct 2023 14:48:29 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: rms@gnu.org, emacs-devel@gnu.org
> 
> 
> >> Pretty much everything loads cl-lib nowadays, for example:
> >
> > That's not the issue, from my POV.  the issue is whether all these 
> > places load cl-lib for good reasons.  If not, they shouldn't.
> >
> > Are any of these commands loading cl-lib without a good reason?
> >
> 
> I don't know.  I was only pointing out that the debugger is just one of 
> the many cases, and not the most likely one, in which cl-lib is loaded. 
> I was also hinting that it would perhaps make sense to preload cl-lib, 
> given that it's practically impossible to use Emacs without loading 
> cl-lib.

It is also "practically impossible" to use Emacs without loading some
other packages, like help-fns, for example.  But we still don't (and
shouldn't) preload them.

Preloading unnecessary stuff is a slippery slope that once we start
there's no way of knowing where we end.  So let's not.

Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Eli Zaretskii wrote:

>> Pretty much everything loads cl-lib nowadays [...]
>
> That's not the issue, from my POV. the issue is whether all
> these places load cl-lib for good reasons. If not,
> they shouldn't.

But as you say, this is nothing specific to cl-lib.
Because without good reason, nothing should load anything.

> Are any of these commands loading cl-lib without
> a good reason?

There is no way to tell without examining each case. And case
here isn't a file but individual usage of whatever
cl-lib provides.

And, because cl-lib is just another Elisp library at the same
level and status as any other, we might generalize the
question from both ends, "Is any file loading anything without
good reason?" If so - yes, we should fix that.

And voilà, everyone agrees.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

> Preloading unnecessary stuff is a slippery slope that once we start
> there's no way of knowing where we end.  So let's not.

AFAIK, the main drawback is increased memory consumption.  The main
benefit is that Emacs starts faster.

Are there any other important considerations?

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > debug.el `require' cl-lib as its first line of non-comment
  > source, and then goes on to use it 6 times: `cl-prin1' (2),
  > `cl-defstruct' (1), and `cl-getf' (3).

  > cl-prin1 is found in cl-print.el, cl-defstruct in cl-macs.el
  > and cl-getf in cl-extra.el.

It sounds like this should be easy to fix.

I would expect that there is an important reason to call cl-prin1.
But I'd expect it is easy to avoid cl-getf -- that using
it struct some developer as "more elegant".

As for cl-defstruct, that is a macro, so it should not be an issue really.

Does this mean it would be simple to fix debug.el by writing out cl-getf,
loading cl-print.el, and not requirng cl-lib?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Mon, 23 Oct 2023 15:11:06 -0700
> Cc: rms@gnu.org, emacs-devel@gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Preloading unnecessary stuff is a slippery slope that once we start
> > there's no way of knowing where we end.  So let's not.
> 
> AFAIK, the main drawback is increased memory consumption.  The main
> benefit is that Emacs starts faster.
> 
> Are there any other important considerations?

Introduction of symbols that are not needed.  Also, some packages
modify the run-time environment when they load.

Re: Lisp files that load cl-lib in problematical ways

[Björn Bidar]

Stefan Kangas <stefankangas@gmail.com> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> Preloading unnecessary stuff is a slippery slope that once we start
>> there's no way of knowing where we end.  So let's not.
>
> AFAIK, the main drawback is increased memory consumption.  The main
> benefit is that Emacs starts faster.

About how much memory are we talking here? All this sounds like talk
 about micro optimizations.
Emacs speed is more limited about it being single threaded most of the
 time than anything else.
 

Re: Lisp files that load cl-lib in problematical ways

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stefan Kangas <stefankangas@gmail.com>
>> Date: Mon, 23 Oct 2023 15:11:06 -0700
>> Cc: rms@gnu.org, emacs-devel@gnu.org
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > Preloading unnecessary stuff is a slippery slope that once we start
>> > there's no way of knowing where we end.  So let's not.
>> 
>> AFAIK, the main drawback is increased memory consumption.  The main
>> benefit is that Emacs starts faster.
>> 
>> Are there any other important considerations?
>
> Introduction of symbols that are not needed.  Also, some packages
> modify the run-time environment when they load.

And the pure space or W32 dumped_heap occupied increases as more files
are preloaded.  Until such time as they are removed, this remains an
important factor in deciding what to load.

Re: Lisp files that load cl-lib in problematical ways

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

> Introduction of symbols that are not needed.

Sure, but that's rather subjective.  One could also argue that it is a
good thing to have certain symbols available immediately, for example
because they are typically needed.

> Also, some packages modify the run-time environment when they load.

Yeah, we do have some bugs in that department.  :-(

Re: Lisp files that load cl-lib in problematical ways

[Ihor Radchenko]

Stefan Kangas <stefankangas@gmail.com> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> Introduction of symbols that are not needed.
>
> Sure, but that's rather subjective.  One could also argue that it is a
> good thing to have certain symbols available immediately, for example
> because they are typically needed.

Could it be customizeable somehow?

I have seen articles using portable dumper to pre-load more libraries,
but it was rather cumbersome and fragile exercise. It might be of
interest for people interested in reducing the startup time to have
a command like M-x dump-emacs-using-already-loaded-built-in-libraries.
That way, users can pre-load the libraries that will be loaded anyway.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Björn Bidar <bjorn.bidar@thaodan.de>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Gregory Heytings <gregory@heytings.org>,
>   rms@gnu.org,  emacs-devel@gnu.org
> Date: Tue, 24 Oct 2023 11:10:06 +0300
> 
> Stefan Kangas <stefankangas@gmail.com> writes:
> 
> > Eli Zaretskii <eliz@gnu.org> writes:
> >
> >> Preloading unnecessary stuff is a slippery slope that once we start
> >> there's no way of knowing where we end.  So let's not.
> >
> > AFAIK, the main drawback is increased memory consumption.  The main
> > benefit is that Emacs starts faster.
> 
> About how much memory are we talking here? All this sounds like talk
>  about micro optimizations.

If we ignore such "micro optimizations", we will quickly bloat Emacs,
since those small amounts add up.  We have a lot of relatively small
potential additions that we make a point of not adding, because if we
decide adding one is okay, then why not add all the rest?  We don't
add such stuff to subr.el or to simple.el or to files.el, and keep
them separate, precisely because adding them all will not be
insignificant.

The only way of keeping this kind of bloat at bay is to resist adding
anything, no matter how small, because, like I said, it's a slippery
slope.

> Emacs speed is more limited about it being single threaded most of the
>  time than anything else.

I don't see the relevance, sorry.  We can work on making Emacs faster
without bloating it.

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Tue, 24 Oct 2023 01:35:21 -0700
> Cc: gregory@heytings.org, rms@gnu.org, emacs-devel@gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Introduction of symbols that are not needed.
> 
> Sure, but that's rather subjective.

No, it's completely objective: we add symbols to obarray that don't
need to be there.

> One could also argue that it is a good thing to have certain symbols
> available immediately, for example because they are typically
> needed.

We have autoloading to take care of that.

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: Eli Zaretskii <eliz@gnu.org>, gregory@heytings.org, rms@gnu.org,
>  emacs-devel@gnu.org
> Date: Tue, 24 Oct 2023 09:04:54 +0000
> 
> Stefan Kangas <stefankangas@gmail.com> writes:
> 
> > Eli Zaretskii <eliz@gnu.org> writes:
> >
> >> Introduction of symbols that are not needed.
> >
> > Sure, but that's rather subjective.  One could also argue that it is a
> > good thing to have certain symbols available immediately, for example
> > because they are typically needed.
> 
> Could it be customizeable somehow?

What do you want to be customizable, and how?

> I have seen articles using portable dumper to pre-load more libraries,
> but it was rather cumbersome and fragile exercise. It might be of
> interest for people interested in reducing the startup time to have
> a command like M-x dump-emacs-using-already-loaded-built-in-libraries.
> That way, users can pre-load the libraries that will be loaded anyway.

Isn't that already available using dump-emacs-portable?  Or am I
missing something?

Re: Lisp files that load cl-lib in problematical ways

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> > Sure, but that's rather subjective.  One could also argue that it is a
>> > good thing to have certain symbols available immediately, for example
>> > because they are typically needed.
>> 
>> Could it be customizeable somehow?
>
> What do you want to be customizable, and how?

What I have in mind is a custom list of packages to be dumped.

>> I have seen articles using portable dumper to pre-load more libraries,
>> but it was rather cumbersome and fragile exercise. It might be of
>> interest for people interested in reducing the startup time to have
>> a command like M-x dump-emacs-using-already-loaded-built-in-libraries.
>> That way, users can pre-load the libraries that will be loaded anyway.
>
> Isn't that already available using dump-emacs-portable?  Or am I
> missing something?

(1) It does not work interactively.
(2) Even making it work non-interactively is not necessarily trivial.
    See https://archive.casouri.cat/note/2020/painless-transition-to-portable-dumper/index.html

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Eli Zaretskii wrote:

>> About how much memory are we talking here? All this sounds
>> like talk about micro optimizations.
>
> If we ignore such "micro optimizations", we will quickly
> bloat Emacs, since those small amounts add up. We have a lot
> of relatively small potential additions that we make a point
> of not adding, because if we decide adding one is okay, then
> why not add all the rest? We don't add such stuff to subr.el
> or to simple.el or to files.el, and keep them separate,
> precisely because adding them all will not be insignificant.
>
> The only way of keeping this kind of bloat at bay is to
> resist adding anything, no matter how small, because, like
> I said, it's a slippery slope.

While I agree one shouldn't ignore micro optimizations in
general, you keep refering to useful Elisp as bloat. That is
disrespectful and also not true, it is useful code, and where
it is or isn't added or loaded doesn't change it into
something it isn't.

Also, instead of the general "slippery slope" model of thought
and offered explanation, it is better to either have a clear
policy "we add stuff that [...]". Then one can say "we don't
intend to include some-package.el because it doesn't fall
under our policy what stuff to add, because it is [...] and
the policy is [...]".

If one cannot formulate a general policy it will instead have
to be dealt with on a case-by-case basis. But then one would
like to hear the arguments for each such case suggested, and
in terms of what that package brings (or lacks) and where it
is suggested to be added, not that you want to keep "this kind
of bloat at bay".

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: stefankangas@gmail.com, gregory@heytings.org, rms@gnu.org,
>  emacs-devel@gnu.org
> Date: Tue, 24 Oct 2023 12:41:13 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> > Sure, but that's rather subjective.  One could also argue that it is a
> >> > good thing to have certain symbols available immediately, for example
> >> > because they are typically needed.
> >> 
> >> Could it be customizeable somehow?
> >
> > What do you want to be customizable, and how?
> 
> What I have in mind is a custom list of packages to be dumped.

You can easily edit lisp/loadup.el and add whatever you want.  So this
possibility already exists.

> >> I have seen articles using portable dumper to pre-load more libraries,
> >> but it was rather cumbersome and fragile exercise. It might be of
> >> interest for people interested in reducing the startup time to have
> >> a command like M-x dump-emacs-using-already-loaded-built-in-libraries.
> >> That way, users can pre-load the libraries that will be loaded anyway.
> >
> > Isn't that already available using dump-emacs-portable?  Or am I
> > missing something?
> 
> (1) It does not work interactively.

Doesn't M-: work?

> (2) Even making it work non-interactively is not necessarily trivial.
>     See https://archive.casouri.cat/note/2020/painless-transition-to-portable-dumper/index.html

These are bugs we still need to find and fix.  But adding arbitrary
Lisp packages to loadup.el is also likely to cause bugs, since
dumping Emacs is a tricky business in any case.

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Tue, 24 Oct 2023 14:43:48 +0200
> 
> Eli Zaretskii wrote:
> 
> > If we ignore such "micro optimizations", we will quickly
> > bloat Emacs, since those small amounts add up. We have a lot
> > of relatively small potential additions that we make a point
> > of not adding, because if we decide adding one is okay, then
> > why not add all the rest? We don't add such stuff to subr.el
> > or to simple.el or to files.el, and keep them separate,
> > precisely because adding them all will not be insignificant.
> >
> > The only way of keeping this kind of bloat at bay is to
> > resist adding anything, no matter how small, because, like
> > I said, it's a slippery slope.
> 
> While I agree one shouldn't ignore micro optimizations in
> general, you keep refering to useful Elisp as bloat.

No, I said that loading stuff that _is_not_necessary_ will _cause_
bloat.  Please don't misrepresent what I wrote.

> That is disrespectful and also not true, it is useful code, and
> where it is or isn't added or loaded doesn't change it into
> something it isn't.

See above.

> Also, instead of the general "slippery slope" model of thought
> and offered explanation, it is better to either have a clear
> policy "we add stuff that [...]".

The policy exists and is clear: we don' preload anything that isn't
needed by Emacs either during the dumping process or when it starts
after dumping.

Re: Lisp files that load cl-lib in problematical ways

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> > What do you want to be customizable, and how?
>> 
>> What I have in mind is a custom list of packages to be dumped.
>
> You can easily edit lisp/loadup.el and add whatever you want.  So this
> possibility already exists.

Indeed, custom dump is already possible, as described in the article I
linked to.

I meant something more interactive that takes care about arranging all
the internal details like running batch Emacs, setting up load-path, and
loading the packages.

>> > Isn't that already available using dump-emacs-portable?  Or am I
>> > missing something?
>> 
>> (1) It does not work interactively.
>
> Doesn't M-: work?

Nope.

Debugger entered--Lisp error: (error "Dumping Emacs currently works only in batch mode.  If you’d like it to work interactively, please consider contributing a patch to Emacs.")
  (dump-emacs-portable "~/tmp/emacs-pdumped")
  (eval (dump-emacs-portable "~/tmp/emacs-pdumped") t)

>> (2) Even making it work non-interactively is not necessarily trivial.
>>     See https://archive.casouri.cat/note/2020/painless-transition-to-portable-dumper/index.html
>
> These are bugs we still need to find and fix.  But adding arbitrary
> Lisp packages to loadup.el is also likely to cause bugs, since
> dumping Emacs is a tricky business in any case.

That's my point. If we can carefully select built-in packages that can
be safely dumped, we can generate Emacs executable tailored for user
configs.

For example, imaginary M-x dump-emacs-portable-interactively might check
the loaded features, filter them according to built-in libraries known
to be safe for dumping, and generate Emacs image that will pre-load
these libraries. This way, Emacs startup time will be reduced while not
adding unnecessary bloat, with "unnecessary" defined by the user config
rather than general considerations that must apply to every possible
Emacs user.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: stefankangas@gmail.com, gregory@heytings.org, rms@gnu.org,
>  emacs-devel@gnu.org
> Date: Tue, 24 Oct 2023 13:01:45 +0000
> 
> > These are bugs we still need to find and fix.  But adding arbitrary
> > Lisp packages to loadup.el is also likely to cause bugs, since
> > dumping Emacs is a tricky business in any case.
> 
> That's my point. If we can carefully select built-in packages that can
> be safely dumped, we can generate Emacs executable tailored for user
> configs.
> 
> For example, imaginary M-x dump-emacs-portable-interactively might check
> the loaded features, filter them according to built-in libraries known
> to be safe for dumping, and generate Emacs image that will pre-load
> these libraries. This way, Emacs startup time will be reduced while not
> adding unnecessary bloat, with "unnecessary" defined by the user config
> rather than general considerations that must apply to every possible
> Emacs user.

Feel free to work on such a feature, I'm sure some will like it,
especially people who for some reason start Emacs frequently.  It's a
lot of non-trivial work, though.

Re: Lisp files that load cl-lib in problematical ways

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> Feel free to work on such a feature, I'm sure some will like it,
> especially people who for some reason start Emacs frequently.  It's a
> lot of non-trivial work, though.

Not sure if I can allocate enough time soon enough, but as a quick
experimentation I tried

----- /tmp/dump.el -----
(require 'cl-lib)
(dump-emacs-portable "/tmp/emacs-dumped")
-------- end -----------

and got surprised by

> ./src/emacs -Q --batch -l /tmp/dump.el 
Dumping fingerprint: 83db7cb9b32874a4c4a0b923b2ca94f0d08a9a22ff41e18a2f92290276ef7d07

Error: error ("Trying to dump non fixed-up eln file")
  mapbacktrace(#f(compiled-function (evald func args flags) #<bytecode -0x154b0bb3d851781b>))
  debug-early-backtrace()
  debug-early(error (error "Trying to dump non fixed-up eln file"))
  dump-emacs-portable("/tmp/emacs-dumped")
  eval-buffer(#<buffer  *load*> nil "/tmp/dump.el" nil t)
  load-with-code-conversion("/tmp/dump.el" "/tmp/dump.el" nil t)
  load("/tmp/dump.el" nil t)
  command-line-1(("-l" "/tmp/dump.el"))
  command-line()
  normal-top-level()
Trying to dump non fixed-up eln file

Is it a bug? Or do I miss something important?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: stefankangas@gmail.com, gregory@heytings.org, rms@gnu.org,
>  emacs-devel@gnu.org
> Date: Tue, 24 Oct 2023 13:33:01 +0000
> 
> ----- /tmp/dump.el -----
> (require 'cl-lib)
> (dump-emacs-portable "/tmp/emacs-dumped")
> -------- end -----------
> 
> and got surprised by
> 
> > ./src/emacs -Q --batch -l /tmp/dump.el 
> Dumping fingerprint: 83db7cb9b32874a4c4a0b923b2ca94f0d08a9a22ff41e18a2f92290276ef7d07
> 
> Error: error ("Trying to dump non fixed-up eln file")
>   mapbacktrace(#f(compiled-function (evald func args flags) #<bytecode -0x154b0bb3d851781b>))
>   debug-early-backtrace()
>   debug-early(error (error "Trying to dump non fixed-up eln file"))
>   dump-emacs-portable("/tmp/emacs-dumped")
>   eval-buffer(#<buffer  *load*> nil "/tmp/dump.el" nil t)
>   load-with-code-conversion("/tmp/dump.el" "/tmp/dump.el" nil t)
>   load("/tmp/dump.el" nil t)
>   command-line-1(("-l" "/tmp/dump.el"))
>   command-line()
>   normal-top-level()
> Trying to dump non fixed-up eln file
> 
> Is it a bug? Or do I miss something important?

I think you are trying to do something that isn't supported.  Andrea,
am I right?

Your original idea was to add stuff to loadup, not to dump an already
dumped Emacs, and with AOT *.eln stuff on top of that.  Preloaded
*.eln files are handled specially by loadup.el during the dumping
process, and you tried doing that without observing the protocol of
*.eln dumping.

Re: Lisp files that load cl-lib in problematical ways

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Ihor Radchenko <yantar92@posteo.net>
>> Cc: stefankangas@gmail.com, gregory@heytings.org, rms@gnu.org,
>>  emacs-devel@gnu.org
>> Date: Tue, 24 Oct 2023 13:33:01 +0000
>> 
>> ----- /tmp/dump.el -----
>> (require 'cl-lib)
>> (dump-emacs-portable "/tmp/emacs-dumped")
>> -------- end -----------
>> 
>> and got surprised by
>> 
>> > ./src/emacs -Q --batch -l /tmp/dump.el 
>> Dumping fingerprint: 83db7cb9b32874a4c4a0b923b2ca94f0d08a9a22ff41e18a2f92290276ef7d07
>> 
>> Error: error ("Trying to dump non fixed-up eln file")
>>   mapbacktrace(#f(compiled-function (evald func args flags) #<bytecode -0x154b0bb3d851781b>))
>>   debug-early-backtrace()
>>   debug-early(error (error "Trying to dump non fixed-up eln file"))
>>   dump-emacs-portable("/tmp/emacs-dumped")
>>   eval-buffer(#<buffer  *load*> nil "/tmp/dump.el" nil t)
>>   load-with-code-conversion("/tmp/dump.el" "/tmp/dump.el" nil t)
>>   load("/tmp/dump.el" nil t)
>>   command-line-1(("-l" "/tmp/dump.el"))
>>   command-line()
>>   normal-top-level()
>> Trying to dump non fixed-up eln file
>> 
>> Is it a bug? Or do I miss something important?
>
> I think you are trying to do something that isn't supported.  Andrea,
> am I right?
>
> Your original idea was to add stuff to loadup, not to dump an already
> dumped Emacs, and with AOT *.eln stuff on top of that.  Preloaded
> *.eln files are handled specially by loadup.el during the dumping
> process, and you tried doing that without observing the protocol of
> *.eln dumping.


Yes that's correct, so far we never supported re-dumping Emacs with
elns.

I think it should be rather easy tho, I'll try to investigate a bit.

Bests

  Andrea

Re: Lisp files that load cl-lib in problematical ways

[Corwin Brust]

[-- Attachment #1: Type: text/plain, Size: 2062 bytes --]

On Tue, Oct 24, 2023 at 9:31 AM Andrea Corallo <acorallo@gnu.org> wrote:

> Eli Zaretskii <eliz@gnu.org> writes:
>
> >> From: Ihor Radchenko <yantar92@posteo.net>
> >> Cc: stefankangas@gmail.com, gregory@heytings.org, rms@gnu.org,
> >>  emacs-devel@gnu.org
> >> Date: Tue, 24 Oct 2023 13:33:01 +0000
> >>
> >> ----- /tmp/dump.el -----
> >> (require 'cl-lib)
> >> (dump-emacs-portable "/tmp/emacs-dumped")
> >> -------- end -----------
> >>
> >> and got surprised by
> >>
> >> Error: error ("Trying to dump non fixed-up eln file")
>

FTR, I tried the same experiment and got the same result.  Glad to know I'm
not the only person who thought to try this approach :)

I then also tried creating a lisp/site-load.el  with just (load
"emacs-lisp/cl-lib") and got this seemingly circular error, suggesting it
might not be so easy to include cl-lib in the dumper.  (Or maybe just that
I have no idea what I'm doing, which is true.)

Loading emacs-lisp/cl-lib (native compiled elisp)...

Error: error ("(require cl-macs) while preparing to dump")
  mapbacktrace(#[1028
"\1\4\203\24\0\301\302!\210\300\4!\210\301\303!\210\202\35\0\301\304!\210\3\3B\262\1\211\2035\0\300\1@!\210\211A\211\262\2\2035\0\301\305!\210\202!\0\301\306!\207"
[prin1 princ "  " "(" "  (" " " ")\n"] 7 "\n\n(fn EVALD FUNC ARGS FLAGS)"])
  debug-early-backtrace()
  debug-early(error (error "(require cl-macs) while preparing to dump"))
  require(cl-macs)

byte-code("\300\301!\210\302\303\304\305#\204\24\0\306\307!\210\306\310!\210\300\207"
[provide cl-lib load "cl-loaddefs" noerror quiet require cl-macs cl-seq] 4)
  load("emacs-lisp/cl-lib")
  load("loadup.el")
(require cl-macs) while preparing to dump
make[1]: *** [Makefile:682: emacs.pdmp] Error 127
make[1]: Leaving directory '/h/emacs-dev/emacs-30/src'
make: *** [Makefile:554: src] Error 2


>
> Yes that's correct, so far we never supported re-dumping Emacs with
> elns.
>
> I think it should be rather easy tho, I'll try to investigate a bit.
>
>
Thanks for looking into this!

[-- Attachment #2: Type: text/html, Size: 3231 bytes --]

Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Eli Zaretskii wrote:

> No, I said that loading stuff that _is_not_necessary_ will
> _cause_ bloat. Please don't misrepresent what I wrote.
>
>> That is disrespectful and also not true, it is useful code,
>> and where it is or isn't added or loaded doesn't change it
>> into something it isn't.

> See above.

Yeah, okay but I think it is better to avoid that word?

>> Also, instead of the general "slippery slope" model of
>> thought and offered explanation, it is better to either
>> have a clear policy "we add stuff that [...]".
>
> The policy exists and is clear: we don' preload anything
> that isn't needed by Emacs either during the dumping process
> or when it starts after dumping.

OK, you have a policy! Good :)

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Björn Bidar]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Björn Bidar <bjorn.bidar@thaodan.de>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  Gregory Heytings <gregory@heytings.org>,
>>   rms@gnu.org,  emacs-devel@gnu.org
>> Date: Tue, 24 Oct 2023 11:10:06 +0300
>> 
>> Stefan Kangas <stefankangas@gmail.com> writes:
>> 
>> > Eli Zaretskii <eliz@gnu.org> writes:
>> >
>> >> Preloading unnecessary stuff is a slippery slope that once we start
>> >> there's no way of knowing where we end.  So let's not.
>> >
>> > AFAIK, the main drawback is increased memory consumption.  The main
>> > benefit is that Emacs starts faster.
>> 
>> About how much memory are we talking here? All this sounds like talk
>>  about micro optimizations.
>
> If we ignore such "micro optimizations", we will quickly bloat Emacs,
> since those small amounts add up.  We have a lot of relatively small
> potential additions that we make a point of not adding, because if we
> decide adding one is okay, then why not add all the rest?  We don't
> add such stuff to subr.el or to simple.el or to files.el, and keep
> them separate, precisely because adding them all will not be
> insignificant.

I get that point but in this case we talk about doing something
slightly different to not load cl-lib while eventually cl-lib is loaded
anyway.

>
>> Emacs speed is more limited about it being single threaded most of the
>>  time than anything else.
>
> I don't see the relevance, sorry.  We can work on making Emacs faster
> without bloating it.

From my pov Emacs start up time or speed is more limted by Emacs being
single threaded rather than loading a few kbs of code that will be
loaded later anyway.

Re: Lisp files that load cl-lib in problematical ways

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

>>> Introduction of symbols that are not needed.
>>
>> Sure, but that's rather subjective.
>
> No, it's completely objective: we add symbols to obarray ...

This part is objective, indeed.

> ... that don't need to be there.

But this part seems more subjective (based on a very strict definition
of "need", a lot of now-preloaded things don't really need to be).

>> One could also argue that it is a good thing to have certain symbols
>> available immediately, for example because they are typically
>> needed.
>
> We have autoloading to take care of that.

Yes, of course.  But we also have preloading.  ;-)

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Björn Bidar <bjorn.bidar@thaodan.de>
> Cc: stefankangas@gmail.com,  gregory@heytings.org,  rms@gnu.org,
>   emacs-devel@gnu.org
> Date: Tue, 24 Oct 2023 18:29:55 +0300
> 
> > If we ignore such "micro optimizations", we will quickly bloat Emacs,
> > since those small amounts add up.  We have a lot of relatively small
> > potential additions that we make a point of not adding, because if we
> > decide adding one is okay, then why not add all the rest?  We don't
> > add such stuff to subr.el or to simple.el or to files.el, and keep
> > them separate, precisely because adding them all will not be
> > insignificant.
> 
> I get that point but in this case we talk about doing something
> slightly different to not load cl-lib while eventually cl-lib is loaded
> anyway.

How is cl-lib different from any other package that is extremely
likely to be loaded close to a beginning of a production session?

The rationale for not loading anything we don't strictly need is that
different users have different use patterns, and Emacs can be used in
different modes: a GUI or TTY interactive session (with various user
customizations that load packages), a batch command, in a script using
--script, as a daemon, etc.  Each one of these needs different
optional modes needs to load different packages; preloading things
that are frequently loaded is therefore likely to load stuff needed by
some other mode/configuration, and that is simply not clean, besides
the fact that it "punishes" users by having them load stuff they don't
need.  E.g., my production session loads no less than 214 packages
right when it starts.  Why would we do that when everything works
without it?  What kind of problem are we trying to solve here?  I see
no problem.

> >> Emacs speed is more limited about it being single threaded most of the
> >>  time than anything else.
> >
> > I don't see the relevance, sorry.  We can work on making Emacs faster
> > without bloating it.
> 
> >From my pov Emacs start up time or speed is more limted by Emacs being
> single threaded rather than loading a few kbs of code that will be
> loaded later anyway.

So our POVs differ.  Lat's agree to disagree about that.

Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Eli Zaretskii wrote:

> E.g., my production session loads no less than 214 packages
> right when it starts. Why would we do that when everything
> works without it? What kind of problem are we trying to
> solve here? I see no problem.

We are trying to understand why guys grep the source and
traces this huge software component down to individual
instances of its use, and like, "hey, can we remove that?".

Gonna do that to those other 213 packages as well?

So let me ask, what is the special attention to cl-lib? To me
it looks like - not saying it is like that - ideology, if you
don't like CL maybe you see cl-lib like some infiltration from
CL? And for this reason don't like it? Well, if so, sorry to
say guys, but you lost the battle then! We are not getting rid
of this infiltration.

Might and Glory to Elisp(cl)!

:P

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Andrea Corallo]

Andrea Corallo <acorallo@gnu.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Ihor Radchenko <yantar92@posteo.net>
>>> Cc: stefankangas@gmail.com, gregory@heytings.org, rms@gnu.org,
>>>  emacs-devel@gnu.org
>>> Date: Tue, 24 Oct 2023 13:33:01 +0000
>>> 
>>> ----- /tmp/dump.el -----
>>> (require 'cl-lib)
>>> (dump-emacs-portable "/tmp/emacs-dumped")
>>> -------- end -----------
>>> 
>>> and got surprised by
>>> 
>>> > ./src/emacs -Q --batch -l /tmp/dump.el 
>>> Dumping fingerprint: 83db7cb9b32874a4c4a0b923b2ca94f0d08a9a22ff41e18a2f92290276ef7d07
>>> 
>>> Error: error ("Trying to dump non fixed-up eln file")
>>>   mapbacktrace(#f(compiled-function (evald func args flags) #<bytecode -0x154b0bb3d851781b>))
>>>   debug-early-backtrace()
>>>   debug-early(error (error "Trying to dump non fixed-up eln file"))
>>>   dump-emacs-portable("/tmp/emacs-dumped")
>>>   eval-buffer(#<buffer  *load*> nil "/tmp/dump.el" nil t)
>>>   load-with-code-conversion("/tmp/dump.el" "/tmp/dump.el" nil t)
>>>   load("/tmp/dump.el" nil t)
>>>   command-line-1(("-l" "/tmp/dump.el"))
>>>   command-line()
>>>   normal-top-level()
>>> Trying to dump non fixed-up eln file
>>> 
>>> Is it a bug? Or do I miss something important?
>>
>> I think you are trying to do something that isn't supported.  Andrea,
>> am I right?
>>
>> Your original idea was to add stuff to loadup, not to dump an already
>> dumped Emacs, and with AOT *.eln stuff on top of that.  Preloaded
>> *.eln files are handled specially by loadup.el during the dumping
>> process, and you tried doing that without observing the protocol of
>> *.eln dumping.
>
>
> Yes that's correct, so far we never supported re-dumping Emacs with
> elns.
>
> I think it should be rather easy tho, I'll try to investigate a bit.

Okay with f7d88f4a047 on master I'm now able to re-dump a native
compiled Emacs!  I followed essentially what Ihor attemped so I loaded
cl-lib before re-dumping as well.

Not 100% sure loadup.el is the best place to host the definition of the
the function and the two new variables I've introduced.  Happy to move
things around if necessary.

Thanks

  Andrea

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Tue, 24 Oct 2023 19:54:11 +0200
> 
> Eli Zaretskii wrote:
> 
> So let me ask, what is the special attention to cl-lib?

For the gazillionth time: it isn't!  Not as far as preloading it is
considered.

> To me it looks like - not saying it is like that - ideology, if you
> don't like CL maybe you see cl-lib like some infiltration from CL?
> And for this reason don't like it? Well, if so, sorry to say guys,
> but you lost the battle then! We are not getting rid of this
> infiltration.

Not relevant to the issue of loading cl-lib.

Re: Lisp files that load cl-lib in problematical ways

[Ihor Radchenko]

Andrea Corallo <acorallo@gnu.org> writes:

>> I think it should be rather easy tho, I'll try to investigate a bit.
>
> Okay with f7d88f4a047 on master I'm now able to re-dump a native
> compiled Emacs!  I followed essentially what Ihor attemped so I loaded
> cl-lib before re-dumping as well.

I confirm.
I was able to dump all the extra built-in libraries Emacs loads, except
Org mode without errors on the latest master.
Now, experiencing various funny glitches :)

But even pre-loading extra built-in libraries reduced startup time
noticeably:

;; With dumper: "7.715609 seconds"
;; Without dumper: "9.866784 seconds"

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > frameset.el makes moderately heavy use of cl-lib, including at least two
  > externally visible cl-defun's where keyword parameters are used.  It
  > would be a lot of work to replace cl-lib in this file

cl-defun is a macro; using it ought to work without causing cl-lib to
be loaded.  So if frameset.el does load cl-lib, the next question is
what it is in the file that actually does so.

We might discover an underlying bug by which some construction loads
cl-lib but shouldn't do so.  Or we might find a few things frameset.el
which load cl-lib for clear reasons, but which would be a small
change.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Loading cl-lib when the user doesn't really need it is wasteful.
To load it unconditionally would only be more so.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Even calling M-: (backtrace) causes cl-lib to get loaded.

  > That's because we use cl-print to display the backtrace.  That change
  > was done intentionally and for good reasons.  So that is not a
  > "problem" that needs to be "fixed", from where I stand.

Can this load cl-print.el instead of cl-lib.el?  Would that do the job?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > It would not have helped to define cl-pushnew on cl-macs, because
  > cl-pushnew calls cl-adjoin, which is a function defined on cl-lib.  So
  > basically cl-pushnew is a (rather thin) wrapper around cl-adjoin, and
  > thus it is correctly defined on the same file where cl-adjoin is
  > defined.

That is a good point -- for that particular solution I proposed.

But if we let ourselves be a little more flexible, how about this
solution: install this

        (while (string-match "\\W" abbrev pos)
          (or (memq (aref abbrev (match-beginning 0)) badchars)
              (push (aref abbrev (match-beginning 0)) badchars))
          (setq pos (1+ pos)))

instead of the current code:

        (while (string-match "\\W" abbrev pos)
          (cl-pushnew (aref abbrev (match-beginning 0)) badchars)
          (setq pos (1+ pos)))

A CL fan might say the latter is simpler, but for me, who isn't sure
just how to use cl-pushnew, the former seems conceptually simpler.

The two ecxpressions have two different kinds of complexity.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > I investigated byte-run.el.  I don't think it uses any cl- facility.
  > > Rather, it declares a let-variable whose name starts with cl-.

  > byte-run.el indeed doesn't need cl-lib, and loading byte-run.elc
  > doesn't load cl-lib.  So I see no problem here, either.  IOW, this is
  > a false positive due to the naïve search pattern used to detect this.

That's right.  I suggest renaming the variable to eliminate the false positive.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Richard Stallman wrote:

> But if we let ourselves be a little more flexible, how about
> this solution: install this
>
>         (while (string-match "\\W" abbrev pos)
>           (or (memq (aref abbrev (match-beginning 0)) badchars)
>               (push (aref abbrev (match-beginning 0)) badchars))
>           (setq pos (1+ pos)))
>
> instead of the current code:
>
>         (while (string-match "\\W" abbrev pos)
>           (cl-pushnew (aref abbrev (match-beginning 0)) badchars)
>           (setq pos (1+ pos)))
>
> A CL fan might say the latter is simpler [...]

Yes, but none of those codes are super-simple, to be fair,
are they?

BTW the CL case can be improved like this - if it isn't that
much simpler, then at least it is much neater, I'd say.

(while (string-match "\\W" abbrev pos)
  (cl-pushnew (aref abbrev (match-beginning 0)) badchars)
  (cl-incf pos) )

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Stefan Kangas]

Richard Stallman <rms@gnu.org> writes:

> But if we let ourselves be a little more flexible, how about this
> solution: install this
>
>         (while (string-match "\\W" abbrev pos)
>           (or (memq (aref abbrev (match-beginning 0)) badchars)
>               (push (aref abbrev (match-beginning 0)) badchars))
>           (setq pos (1+ pos)))
>
> instead of the current code:
>
>         (while (string-match "\\W" abbrev pos)
>           (cl-pushnew (aref abbrev (match-beginning 0)) badchars)
>           (setq pos (1+ pos)))
>
> A CL fan might say the latter is simpler, but for me, who isn't sure
> just how to use cl-pushnew, the former seems conceptually simpler.

The `cl-pushnew' docstring explains:

    Add X to the list stored in PLACE unless X is already in the list.

> The two ecxpressions have two different kinds of complexity.

The only difference I see is that new version is harder to read, and
more prone to bugs (as you have to write the same code twice).

I'd suggest avoiding these types of changes.

Re: Lisp files that load cl-lib in problematical ways

[Gerd Möllmann]

Stefan Kangas <stefankangas@gmail.com> writes:

> Richard Stallman <rms@gnu.org> writes:
>
>> But if we let ourselves be a little more flexible, how about this
>> solution: install this
>>
>>         (while (string-match "\\W" abbrev pos)
>>           (or (memq (aref abbrev (match-beginning 0)) badchars)
>>               (push (aref abbrev (match-beginning 0)) badchars))
>>           (setq pos (1+ pos)))
>>
>> instead of the current code:
>>
>>         (while (string-match "\\W" abbrev pos)
>>           (cl-pushnew (aref abbrev (match-beginning 0)) badchars)
>>           (setq pos (1+ pos)))
>>
>> A CL fan might say the latter is simpler, but for me, who isn't sure
>> just how to use cl-pushnew, the former seems conceptually simpler.
>
> The `cl-pushnew' docstring explains:
>
>     Add X to the list stored in PLACE unless X is already in the list.
>
>> The two ecxpressions have two different kinds of complexity.
>
> The only difference I see is that new version is harder to read, and
> more prone to bugs (as you have to write the same code twice).
>
> I'd suggest avoiding these types of changes.

+1

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: eliz@gnu.org, emacs-devel@gnu.org
> Date: Tue, 24 Oct 2023 22:44:49 -0400
> 
>   > frameset.el makes moderately heavy use of cl-lib, including at least two
>   > externally visible cl-defun's where keyword parameters are used.  It
>   > would be a lot of work to replace cl-lib in this file
> 
> cl-defun is a macro; using it ought to work without causing cl-lib to
> be loaded.  So if frameset.el does load cl-lib, the next question is
> what it is in the file that actually does so.

It uses several cl-lib functions: cl-every, cl-find-if,
cl-delete-if-not, and some others.

This code was written before we had seq.el, which is nowadays
preloaded.  So I guess we could rewrite frameset.el to use seq.el
instead (and a few cl-macs macros it uses).  If someone wants to work
on this, please do.

In any case, I see no reason to continue asking this kind of questions
on the list.  Each such question can be easily answered by using
M-. and "C-h f", so anyone who is interested in understanding why a
given Lisp package loads cl-lib already has the answers at their
fingertips.  There's no reason to ask questions on the list that can
be easily answered in a few seconds.

I also very much doubt that what people will find by using "C-h f" and
M-. will be "bugs", since such bugs would have been found long ago.
It bothers the heck out of me to hear assessments like that which
clearly hint that the quality of our code is deemed to be so low; it
is not.

And again, let's not keep this discussion, as it just wastes bandwidth
and energy on something that can be found within seconds by anyone who
wants to know.

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: acm@muc.de, emacs-devel@gnu.org
> Date: Tue, 24 Oct 2023 22:46:53 -0400
> 
>   > > Even calling M-: (backtrace) causes cl-lib to get loaded.
> 
>   > That's because we use cl-print to display the backtrace.  That change
>   > was done intentionally and for good reasons.  So that is not a
>   > "problem" that needs to be "fixed", from where I stand.
> 
> Can this load cl-print.el instead of cl-lib.el?  Would that do the job?

No.  If you look at cl-print.el, you will immediately see why: it uses
cl-incf and cl-find-class.

Once again: we do pay attention to this stuff, so the chance of
finding some random example where cl-lib is loaded due to a bug are
slim.  And in any case, it is not necessary to discuss this here, one
can just look in the code using "C-h f".

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: acm@muc.de, emacs-devel@gnu.org
> Date: Tue, 24 Oct 2023 22:47:23 -0400
> 
>   > It would not have helped to define cl-pushnew on cl-macs, because
>   > cl-pushnew calls cl-adjoin, which is a function defined on cl-lib.  So
>   > basically cl-pushnew is a (rather thin) wrapper around cl-adjoin, and
>   > thus it is correctly defined on the same file where cl-adjoin is
>   > defined.
> 
> That is a good point -- for that particular solution I proposed.
> 
> But if we let ourselves be a little more flexible, how about this
> solution: install this
> 
>         (while (string-match "\\W" abbrev pos)
>           (or (memq (aref abbrev (match-beginning 0)) badchars)
>               (push (aref abbrev (match-beginning 0)) badchars))
>           (setq pos (1+ pos)))
> 
> instead of the current code:
> 
>         (while (string-match "\\W" abbrev pos)
>           (cl-pushnew (aref abbrev (match-beginning 0)) badchars)
>           (setq pos (1+ pos)))

If someone wants to work on such rewrites, I'm not sure I will object,
provided that the result is clean, tested, and is not horribly
complicated.  But personally, I see no need for investing any effort
in such rewrites, since there's nothing wrong with this code and
nothing wrong with loading cl-lib when its facilities are put to a
good use.

Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Eli Zaretskii wrote:

> I also very much doubt that what people will find by using
> "C-h f" and M-. will be "bugs", since such bugs would have
> been found long ago. It bothers the heck out of me to hear
> assessments like that which clearly hint that the quality of
> our code is deemed to be so low; it is not.

?

Who is saying that?

And what code is that?

> And again, let's not keep this discussion, as it just wastes
> bandwidth and energy on something that can be found within
> seconds by anyone who wants to know.

Indeed, let's drop the cl-lib discussion. Our collective
investigation into the matter has shown that we should all let
it be. Muck around with it further, possible problems ahead
for what in practice will be very close to zero gain.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Corwin Brust]

[-- Attachment #1: Type: text/plain, Size: 937 bytes --]

On Tue, Oct 24, 2023 at 2:43 PM Ihor Radchenko <yantar92@posteo.net> wrote:

> Andrea Corallo <acorallo@gnu.org> writes:
>
> >> I think it should be rather easy tho, I'll try to investigate a bit.
> >
> > Okay with f7d88f4a047 on master I'm now able to re-dump a native
> > compiled Emacs!  I followed essentially what Ihor attemped so I loaded
> > cl-lib before re-dumping as well.
>
> I confirm.
>

It worked for me also, trying from a MSYS2/MINGW64 shell, on Windows.

$ bin/emacs --batch -Q -eval '(let ((custom-theme-load-path (list
"<path>/.emacs.d/elpa/cyberpunk-2019-theme-20191008.1133/"))) (require
'"'"'erc) (load-theme '"'"'cyberpunk-2019 t) (enable-theme
'"'"'cyberpunk-2019) (dump-emacs-portable "erc.pdmp"))'

After which this works:

$ bin/runemacs.exe --dump-file erc.pdmp -fs -l <my-erc-setup>

I've been connected for around 13 hours, so far, so I think it's working :)

Thank you Andrea.

[-- Attachment #2: Type: text/html, Size: 1719 bytes --]

Re: Lisp files that load cl-lib in problematical ways

[Andrea Corallo]

Andrea Corallo <acorallo@gnu.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Andrea Corallo <acorallo@gnu.org>
>>> Cc: Stephen Berman <stephen.berman@gmx.net>,  incal@dataswamp.org,
>>>   emacs-devel@gnu.org
>>> Date: Thu, 19 Oct 2023 09:44:51 -0400
>>> 
>>> Okay, I confirm that comp.el loads cl-lib, so any jit compilation
>>> triggered loads that.
>>
>> This is OK, not a problem.
>>
>>> OTOH one thing we could do, if that's important, is to split the code
>>> that only drives the async compilation (that actually happens in a
>>> subprocess) so we don't load cl-lib in the Emacs the user is actually
>>> using.  This should not be that hard (optimism mode on).
>>
>> I don't see the need, but I will not object if you want to make such
>> changes.
>
> Okay I'll think about it, might be a positive change even leaving aside
> the cl-lib discussion.

So I did some experimentation (scratch/comp-run branch) where I splitted
the code needed to run the (async) native compiler at runtime into a
separete file (comp-run.el).

I think it's a good change as:

1- instead of loading almost 6000 lines of compiler only code
   (comp.el+comp-cstr.el not including dependencies) a normal user now
   has to load only ~500 LOC.  The rest will be loaded only by the async
   workers.

2- it is conceptually correct to divide the code needed at runtime from
   the one necessary to actually compile.

3- at the first startup on my configuration only gv.el gets native
   compiled!

Limitations so far are that:

1- if any of the async compilation will have to report a warning,
   warnings.el will require icons.el that will still require cl-lib.el.

2- if cl-lib.el and other dependencies of the native-compiler are not
   native compiled because of some other package gets loaded, they might
   stay bytecode.  In this case the native compilers running in
   background _might_ have some performance degradation.  I guess is
   more theoretical than practical but we might want to have a look
   before committing.

Bests!

  Andrea

Re: Lisp files that load cl-lib in problematical ways

[Andrea Corallo]

Andrea Corallo <acorallo@gnu.org> writes:

> Andrea Corallo <acorallo@gnu.org> writes:
>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>>>> From: Andrea Corallo <acorallo@gnu.org>
>>>> Cc: Stephen Berman <stephen.berman@gmx.net>,  incal@dataswamp.org,
>>>>   emacs-devel@gnu.org
>>>> Date: Thu, 19 Oct 2023 09:44:51 -0400
>>>> 
>>>> Okay, I confirm that comp.el loads cl-lib, so any jit compilation
>>>> triggered loads that.
>>>
>>> This is OK, not a problem.
>>>
>>>> OTOH one thing we could do, if that's important, is to split the code
>>>> that only drives the async compilation (that actually happens in a
>>>> subprocess) so we don't load cl-lib in the Emacs the user is actually
>>>> using.  This should not be that hard (optimism mode on).
>>>
>>> I don't see the need, but I will not object if you want to make such
>>> changes.
>>
>> Okay I'll think about it, might be a positive change even leaving aside
>> the cl-lib discussion.
>
> So I did some experimentation (scratch/comp-run branch) where I splitted
> the code needed to run the (async) native compiler at runtime into a
> separete file (comp-run.el).
>
> I think it's a good change as:
>
> 1- instead of loading almost 6000 lines of compiler only code
>    (comp.el+comp-cstr.el not including dependencies) a normal user now
>    has to load only ~500 LOC.  The rest will be loaded only by the async
>    workers.
>
> 2- it is conceptually correct to divide the code needed at runtime from
>    the one necessary to actually compile.
>
> 3- at the first startup on my configuration only gv.el gets native
>    compiled!

That's a little more sorry for the mistake, in this conf the list of
native compiled files at startup is: gv.el compile.el ansi-color.el
comint.el text-property-search.el ring.el ansi-osc.el
display-line-numbers.el

  Andrea

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

What's wrong with using cl-lib in many files, for example abbrev.el ad
debug.el, is that each one is a step towards making Emacs depend
pervasively on cl-lib.

With each step in that direction, it becomes gradually harder and
harder to work on Emacs at all without being familiar with the CL functions.
That general dependence is the problem I want to prevent.

So I've decided to start fixing some of those files not to use the
run-time CL features..

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Can this load cl-print.el instead of cl-lib.el?  Would that do the job?

  > No.  If you look at cl-print.el, you will immediately see why: it uses
  > cl-incf and cl-find-class.

Thanks for telling me about the problem at that next stage.
If I had tried that change I would have see why it did not work,
but I had only got as far as asking you.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > A CL fan might say the latter is simpler, but for me, who isn't sure
  > > just how to use cl-pushnew, the former seems conceptually simpler.

  > The `cl-pushnew' docstring explains:

  >     Add X to the list stored in PLACE unless X is already in the list.

Yes, it does -- but what I said is valid nonetheless.
We're talking about the complexity of different ways of writing code.

A reference to a function that you don't know has a kind of complexity.
The fact that you can look up that function doesn't eliminate that complexity.

  > The only difference I see is that new version is harder to read, and
  > more prone to bugs (as you have to write the same code twice).

Do you frequently use the cl- facilities such as `cl-pushnew'?  If so,
that might explain why you didn't _see_ the extra complexity that
comes from calling `cl-pushnew'.  That extra complexity exists for
people who don't habitually use cl facilities, and not for those who
do habitually use them.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Andrea Corallo <acorallo@gnu.org>
> Cc: stephen.berman@gmx.net,  incal@dataswamp.org,  emacs-devel@gnu.org
> Date: Wed, 25 Oct 2023 18:02:52 -0400
> 
> So I did some experimentation (scratch/comp-run branch) where I splitted
> the code needed to run the (async) native compiler at runtime into a
> separete file (comp-run.el).
> 
> I think it's a good change as:
> 
> 1- instead of loading almost 6000 lines of compiler only code
>    (comp.el+comp-cstr.el not including dependencies) a normal user now
>    has to load only ~500 LOC.  The rest will be loaded only by the async
>    workers.
> 
> 2- it is conceptually correct to divide the code needed at runtime from
>    the one necessary to actually compile.
> 
> 3- at the first startup on my configuration only gv.el gets native
>    compiled!

Sounds good to me, thanks.

> Limitations so far are that:
> 
> 1- if any of the async compilation will have to report a warning,
>    warnings.el will require icons.el that will still require cl-lib.el.

I don't see this as a significant problem.  Here's my data point: I
use desktop.el to save/restore my production sessions, which tend to
be large, so when a new version of Emacs starts, I have more than 200
packages natively-compiled in the background during the first 2
minutes, and not a single warning emitted.  Granted, those are all
bundled packages, but why shouldn't we expect third-party packages to
live up to the same standards?

And in addition, users can always turn off the warnings if loading
cl-lib bothers them.

> 2- if cl-lib.el and other dependencies of the native-compiler are not
>    native compiled because of some other package gets loaded, they might
>    stay bytecode.  In this case the native compilers running in
>    background _might_ have some performance degradation.  I guess is
>    more theoretical than practical but we might want to have a look
>    before committing.

Doesn't happen here: I see that cl-lib is natively-compiled in the
very first group of files compiled after Emacs starts.

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Andrea Corallo <acorallo@gnu.org>
> Cc: stephen.berman@gmx.net,  incal@dataswamp.org,  emacs-devel@gnu.org
> Date: Wed, 25 Oct 2023 18:31:52 -0400
> 
> That's a little more sorry for the mistake, in this conf the list of
> native compiled files at startup is: gv.el compile.el ansi-color.el
> comint.el text-property-search.el ring.el ansi-osc.el
> display-line-numbers.el

Can you figure out why is display-line-numbers compiled in that case?

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: acm@muc.de, emacs-devel@gnu.org
> Date: Wed, 25 Oct 2023 22:27:32 -0400
> 
> What's wrong with using cl-lib in many files, for example abbrev.el ad
> debug.el, is that each one is a step towards making Emacs depend
> pervasively on cl-lib.

I don't understand how cl-lib here is different from other packages
that are used pervasively in Emacs.  We have quite a few of them that
aren't preloaded.

> With each step in that direction, it becomes gradually harder and
> harder to work on Emacs at all without being familiar with the CL functions.
> That general dependence is the problem I want to prevent.

This ship has sailed long ago.  cl-lib is used in many packages, both
those bundled with Emacs and those unbundled.  It is considered a
convenience library by many Lisp programmers who contribute to Emacs.
In this situation, rejecting code that uses cl-lib is
counter-productive and detrimental to attracting new contributors.

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: acm@muc.de, emacs-devel@gnu.org
> Date: Wed, 25 Oct 2023 22:27:34 -0400
> 
>   > > Can this load cl-print.el instead of cl-lib.el?  Would that do the job?
> 
>   > No.  If you look at cl-print.el, you will immediately see why: it uses
>   > cl-incf and cl-find-class.
> 
> Thanks for telling me about the problem at that next stage.
> If I had tried that change I would have see why it did not work,
> but I had only got as far as asking you.

There's no need to try.  I didn't try, I just typed "C-x cl- C-s C-s..."
and looked at the hits, invoking "C-h f" where I didn't recognize the
symbol from memory.

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: eliz@gnu.org, acm@muc.de, emacs-devel@gnu.org
> Date: Wed, 25 Oct 2023 22:27:54 -0400
> 
>   > The only difference I see is that new version is harder to read, and
>   > more prone to bugs (as you have to write the same code twice).
> 
> Do you frequently use the cl- facilities such as `cl-pushnew'?  If so,
> that might explain why you didn't _see_ the extra complexity that
> comes from calling `cl-pushnew'.  That extra complexity exists for
> people who don't habitually use cl facilities, and not for those who
> do habitually use them.

I think the need to be familiar with the cl-lib and cl-macs
functionalities is nowadays a requirement for any Emacs maintainer.
It can take time to familiarize oneself with them, but it isn't rocket
science.

Btw, some of the cl-lib facilities should not be used in new code if
they have equivalent implementations ins seq.el (which is nowadays
preloaded) and in map.el.  cl-generic.el is also preloaded, so should
be used in preference to the cl-lib alternatives.  And some frequently
needed cl-lib functions were moved to cl-preloaded.el, and are also
always available.

Re: Lisp files that load cl-lib in problematical ways

[Adam Porter]

Hi Eli,

I'm curious about this:

> Btw, some of the cl-lib facilities should not be used in new code if
> they have equivalent implementations ins seq.el (which is nowadays
> preloaded) and in map.el.

Since seq.el and map.el use generic methods, don't they have some 
additional overhead compared to type-specific functions, like cl-lib 
ones that are specific to lists?  Or is it small enough now that it 
doesn't matter anymore?

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> Date: Thu, 26 Oct 2023 04:31:14 -0500
> Cc: acm@muc.de, emacs-devel@gnu.org, rms@gnu.org, stefankangas@gmail.com
> From: Adam Porter <adam@alphapapa.net>
> 
> > Btw, some of the cl-lib facilities should not be used in new code if
> > they have equivalent implementations ins seq.el (which is nowadays
> > preloaded) and in map.el.
> 
> Since seq.el and map.el use generic methods, don't they have some 
> additional overhead compared to type-specific functions, like cl-lib 
> ones that are specific to lists?  Or is it small enough now that it 
> doesn't matter anymore?

I don't know.  Feel free to investigate and post the results.  If
there's a significant run-time penalty, we should work on making these
packages faster, since we consider them to be part of our
infrastructure.

Re: Lisp files that load cl-lib in problematical ways

[Dmitry Gutov]

On 26/10/2023 12:31, Adam Porter wrote:
> Hi Eli,
> 
> I'm curious about this:
> 
>> Btw, some of the cl-lib facilities should not be used in new code if
>> they have equivalent implementations ins seq.el (which is nowadays
>> preloaded) and in map.el.
> 
> Since seq.el and map.el use generic methods, don't they have some 
> additional overhead compared to type-specific functions, like cl-lib 
> ones that are specific to lists?  Or is it small enough now that it 
> doesn't matter anymore?

For most functions, the dispatch overhead might only show up on the 
radar for small sequences (otherwise it's dwarfed by useful work). And I 
haven't yet seen a case where it was specifically a problem.

cl-lib has its own performance problems, so the real answer is for the 
programmer to measure all alternatives and sometimes contribute 
performance fixes upstream ;)

Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Eli Zaretskii wrote:

>> Thanks for telling me about the problem at that next stage.
>> If I had tried that change I would have see why it did not
>> work, but I had only got as far as asking you.
>
> There's no need to try. I didn't try, I just typed "C-x cl-
> C-s C-s..." and looked at the hits, invoking "C-h f" where
> I didn't recognize the symbol from memory.

Oh, the help loads cl-lib. Do 'emacs -Q' and then C-h
f cl-loop ... it auto-completes and you can use it after that,
because the help itself loaded it.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Alan Mackenzie]

Hello, Eli.

On Thu, Oct 26, 2023 at 09:55:45 +0300, Eli Zaretskii wrote:
> > From: Richard Stallman <rms@gnu.org>
> > Cc: eliz@gnu.org, acm@muc.de, emacs-devel@gnu.org
> > Date: Wed, 25 Oct 2023 22:27:54 -0400

> >   > The only difference I see is that new version is harder to read, and
> >   > more prone to bugs (as you have to write the same code twice).

> > Do you frequently use the cl- facilities such as `cl-pushnew'?  If so,
> > that might explain why you didn't _see_ the extra complexity that
> > comes from calling `cl-pushnew'.  That extra complexity exists for
> > people who don't habitually use cl facilities, and not for those who
> > do habitually use them.

> I think the need to be familiar with the cl-lib and cl-macs
> functionalities is nowadays a requirement for any Emacs maintainer.

For me, that is not possible.  I simply don't have the brain power to
memorize such a morass of arbitrary, poorly named, poorly documented
stuff.  I am forever cursed by having to look up doc strings each time I
encounter these objects.  Maybe not things like cl-incf, but
monstrosities such as cl-labels just leave me groaning.  I frequently
spend, perhaps, half of my debugging time trying to understand what some
cl-* does rather than concentrating on the problem to be debugged.

> It can take time to familiarize oneself with them, but it isn't rocket
> science.

How many contributors are actually familiar with the bulk of cl-lib?
I suspect, very few indeed.  Maybe Stefan M.  The people who have written
strongly in favour of cl-lib over the last week or two have not been
those who spend lots of time maintaining others' code.

I further suspect that many contributions using cl-lib are from people
who know a small part of cl-lib, find the functionality intriguing, and
thus use it.  The trouble is, each such contributor knows a different
part of it, so in aggregate the part that an Emacs maintainer needs to
know (or continually to look up) is large.

Note that Common Lisp never took off as a popular language.  There must
be reasons for this.

> Btw, some of the cl-lib facilities should not be used in new code if
> they have equivalent implementations ins seq.el (which is nowadays
> preloaded) and in map.el.  cl-generic.el is also preloaded, so should
> be used in preference to the cl-lib alternatives.  And some frequently
> needed cl-lib functions were moved to cl-preloaded.el, and are also
> always available.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Lisp files that load cl-lib in problematical ways

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <acorallo@gnu.org>
>> Cc: stephen.berman@gmx.net,  incal@dataswamp.org,  emacs-devel@gnu.org
>> Date: Wed, 25 Oct 2023 18:31:52 -0400
>>
>> That's a little more sorry for the mistake, in this conf the list of
>> native compiled files at startup is: gv.el compile.el ansi-color.el
>> comint.el text-property-search.el ring.el ansi-osc.el
>> display-line-numbers.el
>
> Can you figure out why is display-line-numbers compiled in that case?

Sorry I had to improve my methodology to see what's really compiled just
starting up emacs -Q.  Depending on the methodology used to perform the
measure something else can be compiled.  This is the correct list I
believe:

gv.el comint.el compile.el ansi-osc.el ansi-color.el ring.el
text-property-search.el comp-run.el

Thanks

  Andrea

Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Alan Mackenzie wrote:

> How many contributors are actually familiar with the bulk of
> cl-lib? I suspect, very few indeed. Maybe Stefan M.
> The people who have written strongly in favour of cl-lib
> over the last week or two have not been those who spend lots
> of time maintaining others' code.
>
> I further suspect that many contributions using cl-lib are
> from people who know a small part of cl-lib, find the
> functionality intriguing, and thus use it. The trouble is,
> each such contributor knows a different part of it, so in
> aggregate the part that an Emacs maintainer needs to know
> (or continually to look up) is large.

It is the nature of innovation. People like to be at the
front. They like to try out exotic new gear. And the people at
the rear understand this - even if it means they have to build
new facilities to accomodate for the new stuff once in
a while.

If we tell people at the front to stick with the old ways, the
old gear, they will simply leave our section for something
more interesting to do. And we won't get anywhere.

> Note that Common Lisp never took off as a popular language.
> There must be reasons for this.

Maintaining it is too difficult?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Bob Rogers]

   From: Emanuel Berg <incal@dataswamp.org>
   Date: Thu, 26 Oct 2023 13:32:43 +0200

   Alan Mackenzie wrote:

   . . .

   > Note that Common Lisp never took off as a popular language.  There
   > must be reasons for this.

   Maintaining it is too difficult?

Even before CL, Lisp systems tended to be memory-hungry.  Common Lisp
was designed as a large language, for a future where memory would be
cheap.  That future eventually came, but before it came the PC boom,
when tiny systems with relatively tiny amounts of RAM ruled the roost.
And Lisp was neglected even on more powerful workstations; some of you
will rememeber the Sun Sparcstation Pmeg problem that caused page table
thrashing if you tried to allocate more than 16MB.  (This is an over-
simplification for why Lisp popularity declined, of course.)

   This complexity and the resulting complications, I believe, are what
Richard was trying to avoid when he designed Emacs Lisp to be a simpler
dialect of Lisp, and why he sees cl-lib.el as a trojan horse that is
changing Emacs Lisp.  (And why I hope this bit of history is not quite
off-topic.)

					-- Bob Rogers
					   http://www.rgrjr.com/

Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Bob Rogers wrote:

> Even before CL, Lisp systems tended to be memory-hungry.
> Common Lisp was designed as a large language, for a future
> where memory would be cheap. That future eventually came,
> but before it came the PC boom, when tiny systems with
> relatively tiny amounts of RAM ruled the roost. And Lisp was
> neglected even on more powerful workstations; some of you
> will rememeber the Sun Sparcstation Pmeg problem that caused
> page table thrashing if you tried to allocate more than
> 16MB. (This is an over- simplification for why Lisp
> popularity declined, of course.)

Before the PC: too memory-hungry
On the PCs: not enough RAM
On the workstations: neglected

> This complexity and the resulting complications, I believe,
> are what Richard was trying to avoid when he designed Emacs
> Lisp to be a simpler dialect of Lisp, and why he sees
> cl-lib.el as a trojan horse that is changing Emacs Lisp.

If so, it has certainly succeeded! There used to be 'labels',
'incf', and 'decf' in Elisp. Now, not only do they face
competition from `cl-labels', `cl-incf', and `cl-decf' - they
don't exist anymore.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > How is cl-lib different from any other package that is extremely
  > likely to be loaded close to a beginning of a production session?

I think that is the crucial question -- it shows the deep difference.

Most packages do a specific kind of job and have just a few entry
points that relate to that job.  Consider sendmail.el, for instance.
It Has commands to initialize a message and to send it, and some
relating to editing one.  Most of the package is internals
that you wouldn't need to know about in order to _use_ sendmail.

CL makes a big contrast with that. It is mainly entry points.  And
many of these entry points try to be as multi-purpose as possible.

What are the implications of this?

If your package FROB wants to send an email, and it uses sendmail.el
to do that, that won't add much complexity to Emacs or FROB.  Using
sendmail.tl won't make FROB harder to maintain, because as a
maintainer of FROB there is only a little you need to know as a
consequnce of its use of sendmail.  Use of sendmail.el from the few
places that want to send an email just is not a problem.

This is why I'm concered with how many packages use CL, but not
concerned with how many use sendmail.el.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: bjorn.bidar@thaodan.de, stefankangas@gmail.com,
> 	gregory@heytings.org, emacs-devel@gnu.org
> Date: Thu, 26 Oct 2023 22:12:18 -0400
> 
>   > How is cl-lib different from any other package that is extremely
>   > likely to be loaded close to a beginning of a production session?
> 
> I think that is the crucial question -- it shows the deep difference.
> 
> Most packages do a specific kind of job and have just a few entry
> points that relate to that job.  Consider sendmail.el, for instance.
> It Has commands to initialize a message and to send it, and some
> relating to editing one.  Most of the package is internals
> that you wouldn't need to know about in order to _use_ sendmail.
> 
> CL makes a big contrast with that. It is mainly entry points.  And
> many of these entry points try to be as multi-purpose as possible.

We have a lot of the CL variety as well.  Look at lisp/emacs-lisp/
directory: it is full of such "entry-point-only" packages, and most of
them aren't preloaded, they load on demand.  That's typical of any
Lisp package that provides infrastructure rather than applications,
like sendmail.el does.  cl-lib is one of those infrastructure
packages, but it isn't the only one, far from that.  We take care not
to preload such infrastructure packages unless they are needed by the
code which runs at build time, or are needed at startup.

So I again say: cl-lib is not different from any other infrastructure
package that is likely to be loaded because some Lisp program needs to
use it.

> This is why I'm concered with how many packages use CL, but not
> concerned with how many use sendmail.el.

Are you concerned with any of the following?

  easy-mmode.el
  eieio.el
  generic.el
  icons.el
  let-alist.el
  map.el
  rx.el
  subr-x.el
  unsafep.el
  warnings.el

They are all of the same nature: they provide utility functions to be
used by Lisp programs.

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >    This complexity and the resulting complications, I believe, are what
  > Richard was trying to avoid when he designed Emacs Lisp to be a simpler
  > dialect of Lisp, and why he sees cl-lib.el as a trojan horse that is
  > changing Emacs Lisp.

I think you've put your finger on it.

I see nothing wrong with having an optional extension which implements
(more or less) Common Lisp in Emacs Lisp.  That's what I thought we
had done.

However, I object to incorporating (more or less) Common Lisp an essential part
of Emacs Lisp.  And it looks like we've been drifting into that.

There is no sharp line between the one and the other, and no simple
test to determine which of those two our current situation actually
is.  Rather, there is a spectrum that runs from "CL support is
available but you can ignore it" to "the CL constructs are an
essential and unavoidable part of Emacs Lisp."

The specific practical questions I've asked are efforts to evaluate
where we are now along that spectrum.  Of course, the answer to that
isn't precise either.  But I was very surprised to learn how far
Emacs has gone towards the latter end.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

What's missing in ELisp that makes people want to use cl-lib?

[Stefan Kangas]

Richard Stallman <rms@gnu.org> writes:

> The specific practical questions I've asked are efforts to evaluate
> where we are now along that spectrum.  Of course, the answer to that
> isn't precise either.  But I was very surprised to learn how far
> Emacs has gone towards the latter end.

It seems to me that the extent to which Emacs has shifted "towards the
latter end" is still open to interpretation.

More fundamentally, I believe the question we should be asking isn't the
quantity of cl-lib used in Emacs, but rather why these abstractions from
Common Lisp appeal to ELisp developers.  Most ELisp developers, I
presume, do not have a background in Common Lisp, indicating that their
choice to use cl-lib.el isn't driven by a mere affinity for CL.

Is it possible that certain abstractions or functions are filling a gap
in Emacs Lisp itself?  In the case of `cl-pushnew', I happen to think
that the answer is yes.

Discussing `cl-pushnew' and other cases like it might help us identify
areas for improvement in Emacs Lisp proper.  Perhaps we can borrow the
CL name/behavior, or perhaps we can come up with something even better.
That would be more constructive, and more helpful, than spending our
time changing code to not use cl-lib.el just for the sake of it.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Stefan Kangas wrote:

> More fundamentally, I believe the question we should be
> asking isn't the quantity of cl-lib used in Emacs, but
> rather why these abstractions from Common Lisp appeal to
> ELisp developers. Most ELisp developers, I presume, do not
> have a background in Common Lisp, indicating that their
> choice to use cl-lib.el isn't driven by a mere affinity
> for CL.

Good idea, I tried this to find out. Maybe the command can be
improved ...

$ grep -E -h -o 'cl-[-a-z]*' **/*.el | sort | uniq -c | sort -r -b -n | head -n 10

   1416 cl-defmethod
    979 cl-loop
    720 cl-incf
    614 cl-lib
    486 cl-assert
    309 cl-letf
    303 cl-seq
    281 cl-defstruct
    257 cl-defgeneric
    227 cl-case

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Sat, 28 Oct 2023 18:18:53 +0200
> 
> Stefan Kangas wrote:
> 
> > More fundamentally, I believe the question we should be
> > asking isn't the quantity of cl-lib used in Emacs, but
> > rather why these abstractions from Common Lisp appeal to
> > ELisp developers. Most ELisp developers, I presume, do not
> > have a background in Common Lisp, indicating that their
> > choice to use cl-lib.el isn't driven by a mere affinity
> > for CL.
> 
> Good idea, I tried this to find out. Maybe the command can be
> improved ...
> 
> $ grep -E -h -o 'cl-[-a-z]*' **/*.el | sort | uniq -c | sort -r -b -n | head -n 10
> 
>    1416 cl-defmethod
>     979 cl-loop
>     720 cl-incf
>     614 cl-lib
>     486 cl-assert
>     309 cl-letf
>     303 cl-seq
>     281 cl-defstruct
>     257 cl-defgeneric
>     227 cl-case

Some of these are macros in cl-macs, so not relevant to the discussion
about cl-lib.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Eli Zaretskii wrote:

>> $ grep -E -h -o 'cl-[-a-z]*' **/*.el | sort | uniq -c | sort -r -b -n |
>>  head -n 10
>> 
>>    1416 cl-defmethod
>>     979 cl-loop
>>     720 cl-incf
>>     614 cl-lib
>>     486 cl-assert
>>     309 cl-letf
>>     303 cl-seq
>>     281 cl-defstruct
>>     257 cl-defgeneric
>>     227 cl-case
>
> Some of these are macros in cl-macs, so not relevant to the
> discussion about cl-lib.

It is everything, remove the last 'head -n 10' to get it.

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Jim Porter]

On 10/28/2023 8:39 AM, Stefan Kangas wrote:
> Is it possible that certain abstractions or functions are filling a gap
> in Emacs Lisp itself?  In the case of `cl-pushnew', I happen to think
> that the answer is yes.

I agree with this. I can speak about Eshell in particular, since it's 
historically used a bit of cl-lib, and I've added more usage to it.

By far the number one cl-lib thing Eshell uses is 'cl-assert'. This 
would be reasonably easy to do without cl-lib, but it's a pretty 
convenient syntax for, well... asserting some precondition for a 
function. That's been useful for helping to identify subtle bugs in 
Eshell's logic.

After that, the next most important cl-lib thing (in my opinion) is 
generic functions. I introduced this in Eshell to break apart some 
monolithic I/O code that had large 'cond' forms for each I/O target. 
Now, with generic functions, they're separate, and it's much easier for 
third parties to add new I/O target types. In the future, I'll likely 
use generic functions for a few more bits in Eshell.

Related to the above is 'cl-defstruct'. That's useful for working with 
generic functions, but I think there's also some benefit to using them 
instead of lists (or alists/plists) when you want something with a 
very-specific set of members (similar to a C struct or C++ class). 
However, I could do without CL-style structs for this case, excluding 
the generic function side.

Finally, Eshell uses a few miscellaneous cl-lib functions, some of which 
are probably necessary, and some of which are just conveniences. The 
(probable) necessities are: cl-progv and cl-mapc. The conveniences are: 
cl-loop, cl-flet, cl-list*, cl-remove-if, cl-incf, and cl-decf.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> Date: Sat, 28 Oct 2023 12:10:23 -0700
> Cc: emacs-devel@gnu.org
> From: Jim Porter <jporterbugs@gmail.com>
> 
> On 10/28/2023 8:39 AM, Stefan Kangas wrote:
> > Is it possible that certain abstractions or functions are filling a gap
> > in Emacs Lisp itself?  In the case of `cl-pushnew', I happen to think
> > that the answer is yes.
> 
> I agree with this. I can speak about Eshell in particular, since it's 
> historically used a bit of cl-lib, and I've added more usage to it.
> 
> By far the number one cl-lib thing Eshell uses is 'cl-assert'.

cl-assert is a macro in cl-macs, so it isn't relevant to this
discussion.

> After that, the next most important cl-lib thing (in my opinion) is 
> generic functions.

If you mean cl-defgeneric and cl-defmethod, they again are not in
cl-lib, so not relevant to this discussion.  Moreover, they are
defined in cl-generic.el, which is preloaded.

> Related to the above is 'cl-defstruct'.

cl-defstruct is again a macro in cl-macs, not in cl-lib.

> Finally, Eshell uses a few miscellaneous cl-lib functions, some of which 
> are probably necessary, and some of which are just conveniences. The 
> (probable) necessities are: cl-progv and cl-mapc. The conveniences are: 
> cl-loop, cl-flet, cl-list*, cl-remove-if, cl-incf, and cl-decf.

These are the only ones relevant to this discussion.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Sat, 28 Oct 2023 21:08:33 +0200
> 
> Eli Zaretskii wrote:
> 
> >> $ grep -E -h -o 'cl-[-a-z]*' **/*.el | sort | uniq -c | sort -r -b -n |
> >>  head -n 10
> >> 
> >>    1416 cl-defmethod
> >>     979 cl-loop
> >>     720 cl-incf
> >>     614 cl-lib
> >>     486 cl-assert
> >>     309 cl-letf
> >>     303 cl-seq
> >>     281 cl-defstruct
> >>     257 cl-defgeneric
> >>     227 cl-case
> >
> > Some of these are macros in cl-macs, so not relevant to the
> > discussion about cl-lib.
> 
> It is everything, remove the last 'head -n 10' to get it.

What I wrote is valid even without 'head -n 10'.  My point is that
this kind of list is not useful for this discussion.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Eli Zaretskii wrote:

>>>> $ grep -E -h -o 'cl-[-a-z]*' **/*.el | sort | uniq -c | sort -r -b -n |
>>>>  head -n 10
>>>> 
>>>>    1416 cl-defmethod
>>>>     979 cl-loop
>>>>     720 cl-incf
>>>>     614 cl-lib
>>>>     486 cl-assert
>>>>     309 cl-letf
>>>>     303 cl-seq
>>>>     281 cl-defstruct
>>>>     257 cl-defgeneric
>>>>     227 cl-case
>>>
>>> Some of these are macros in cl-macs, so not relevant to the
>>> discussion about cl-lib.
>> 
>> It is everything, remove the last 'head -n 10' to get it.
>
> What I wrote is valid even without 'head -n 10'. My point is
> that this kind of list is not useful for this discussion.

Of course what you say is valid, since some part of everything
is always something.

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Jim Porter]

On 10/28/2023 12:14 PM, Eli Zaretskii wrote:
>> Date: Sat, 28 Oct 2023 12:10:23 -0700
>> Cc: emacs-devel@gnu.org
>> From: Jim Porter <jporterbugs@gmail.com>
>>
>> I agree with this. I can speak about Eshell in particular, since it's
>> historically used a bit of cl-lib, and I've added more usage to it.
>>
>> By far the number one cl-lib thing Eshell uses is 'cl-assert'.
> 
> cl-assert is a macro in cl-macs, so it isn't relevant to this
> discussion.

Thanks. I wasn't aware of this distinction. (I see that it was discussed 
elsewhere in the parent thread, but that thread is very big and I hadn't 
been following it closely.)

Is there any benefit in aliasing macros like this to non 'cl-FOO' names 
anyway? As a mere consumer of the various Common Lisp code in Emacs, 
it's not always clear to me what's ok to use versus what should ideally 
be avoided.

>> Finally, Eshell uses a few miscellaneous cl-lib functions, some of which
>> are probably necessary, and some of which are just conveniences. The
>> (probable) necessities are: cl-progv and cl-mapc. The conveniences are:
>> cl-loop, cl-flet, cl-list*, cl-remove-if, cl-incf, and cl-decf.
> 
> These are the only ones relevant to this discussion.

I see that 'cl-progv' is in cl-macs.el, so I guess that's not relevant. 
'cl-mapc' is in cl-extra.el, which I suppose is relevant here? I find it 
very helpful, but I'd be happy with any function/macro that would make 
it easy to iterate over multiple sequences at once.

Of the conveniences, I think the ones I'd miss the most are 'cl-incf' 
and 'cl-decf', which I use in Eshell for recounting some handles. (I 
could possibly rewrite that code to let the garbage collector do all the 
work, but that code interacts with some of the trickier bits in Eshell, 
so maybe it's best not to change it too much.)

'cl-remove-if' should be easy to replace with 'seq-filter' or somesuch. 
Likewise 'cl-list*' with 'nconc'. I know there's some controversy around 
'cl-loop' (I'm not a big fan myself), and I've thought about adding some 
new hooks to Eshell that would probably obviate the need for 'cl-loop' 
(which is used in em-extpipe.el).

So in conclusion, I guess the relevant functions that I care about are 
'cl-mapc', and to a lesser degree 'cl-incf'/'cl-decf'. Everything else 
could go away if we wanted to avoid them.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> Date: Sat, 28 Oct 2023 14:03:29 -0700
> Cc: stefankangas@gmail.com, rms@gnu.org, rogers@rgrjr.com, emacs-devel@gnu.org
> From: Jim Porter <jporterbugs@gmail.com>
> 
> Is there any benefit in aliasing macros like this to non 'cl-FOO' names 
> anyway?

I don't have an opinion on this, so I'll let others chime in.  In
general, introducing aliases has a minor downside: people will need to
learn two names instead of just one.

> I see that 'cl-progv' is in cl-macs.el, so I guess that's not relevant. 
> 'cl-mapc' is in cl-extra.el, which I suppose is relevant here?

Yes, cl-extra is just another part of cl-lib.

> I find it very helpful, but I'd be happy with any function/macro
> that would make it easy to iterate over multiple sequences at once.

Try looking in seq.el or map.el.

> 'cl-remove-if' should be easy to replace with 'seq-filter' or somesuch. 
> Likewise 'cl-list*' with 'nconc'. I know there's some controversy around 
> 'cl-loop' (I'm not a big fan myself), and I've thought about adding some 
> new hooks to Eshell that would probably obviate the need for 'cl-loop' 
> (which is used in em-extpipe.el).
> 
> So in conclusion, I guess the relevant functions that I care about are 
> 'cl-mapc', and to a lesser degree 'cl-incf'/'cl-decf'. Everything else 
> could go away if we wanted to avoid them.

Since seq.el is preloaded, I think we should prefer it to cl-lib.
Likewise map.el, although that isn't preloaded.

Re: Lisp files that load cl-lib in problematical ways

[João Távora]

On Sat, Oct 28, 2023 at 4:20 AM Richard Stallman <rms@gnu.org> wrote:

> I think you've put your finger on it.
>
> I see nothing wrong with having an optional extension which implements
> (more or less) Common Lisp in Emacs Lisp.  That's what I thought we
> had done.
>
> However, I object to incorporating (more or less) Common Lisp an essential part
> of Emacs Lisp.  And it looks like we've been drifting into that.
>
> There is no sharp line between the one and the other, and no simple
> test to determine which of those two our current situation actually
> is.  Rather, there is a spectrum that runs from "CL support is
> available but you can ignore it" to "the CL constructs are an
> essential and unavoidable part of Emacs Lisp."
>
> The specific practical questions I've asked are efforts to evaluate
> where we are now along that spectrum.  Of course, the answer to that
> isn't precise either.  But I was very surprised to learn how far
> Emacs has gone towards the latter end.

I'm going add my 2c here.  I think this whole discussion, which I haven't
followed very closely, might be slanted due to a misunderstanding.

Most, maybe all, programming languages have two different ways they can
evolve.  One is to add or modify the API offered by their standard library
by using the language itself, and another is to add or modify properties
of the fundamental mechanics of the language by changing any of the
runtime, interpreter or the compiler.

The latter can't be done using the language itself.  Now, Common Lisp
has a number of features that belong to the latter group that Emacs Lisp
doesn't have and there's no significant motion in that direction.
Examples are a namespacing system based on packages, a programmable
reader, a much more advanced condition handling system.  I'm sure
I'm forgetting some.

In fact the only way that Emacs Lisp, the core language, has approached
Common Lisp in recent years to any degree was through the addition of
lexical binding, and I think that most people here would agree that
was a very good thing.

Well, maybe native compilation also counts.  Then again, another very
good thing, I suppose.

cl-lib.el, like any other library, doesn't add -- because it really
can't -- any features of that calibre to Emacs Lisp.  It just adds
functions and macros much in the way that lots of other libraries
add functions and macros.  For example, the seq.el library is another
much later library (2014) that offers much of the same functionality of
cl-lib.el in wholly new abstractions that, say, the 2000's Emacs Lisp
coder won't be familiar with.  Another such library is pcase.
Does use of these libraries mutate Emacs Lisp programs to the point
that they are unrecognizable as Emacs Lisp programs?  Are the pcase
macros fundamentally easier to read than, say, cl-case macros?  Is
seq's sequence manipulating dictionary better than CL's?  Is
seq-some really better than cl-some? Isn't that entirely a question
of personal preference?

seq.el is preloaded and nobody seems to care (and I also don't, to
be honest).  Anyway, the point is that to hack on long-lived files
such as lisp/minibuffer.el, one can't really "ignore" the new
dictionary of seq.el anymore.

If one were to compare apples to apples one could argue that CL's
functions are _more_  accessible and quite well documented, since
both the thought put into the design of those interfaces and the use
they've seen by Lisp programmers of various breeds far exceed those
of seq.el and pcase.el.

Just my 2c,
João

RE: [External] : Re: Lisp files that load cl-lib in problematical ways

[Drew Adams]

> If one were to compare apples to apples one could
> argue that CL's functions are _more_  accessible
> and quite well documented,

Yes, if you mean their documentation in the
Common Lisp reference doc (esp. CLTL2, but even
the Hyperspec).

But no, not so well documented, if you mean the
Emacs doc for them.  (Just one opinion.)

> both the thought put into the design of those
> interfaces and the use they've seen by Lisp
> programmers of various breeds far exceed those
> of seq.el and pcase.el.

+1.  Far exceed.

Personally, I'd prefer that the Common Lisp
sequence functions just be added to Elisp, in
place of those in seq.el - keeping anything
truly Elisp-specific and found to be sorely
missing from C.L.  (Of course, this might mean
supporting their keyword args.)

I can't speak so definitively about `pcase.el'.
Some C.L. things could be used directly in place
of some of what `pcase' etc. do.  But patterned
binding, which pcase also does, is very useful.

(Patterned binding could usefully be separate
from any conditional behavior, but that's just
a personal opinion, and far down the river
that's passed under the bridge.)

By "just add to Elisp" I mean essentially (1)
get rid of the `cl-' prefix and (2) add the
construct, preferably "full-blown" but limited
in some ways if necessary, to Elisp.  Some
constructs could be preloaded, some autoloaded.
___

Every few months or so we hear input about
vastly multiplying the number of Emacs users
by changing this or that behavior or name to
something that potential newcomers might be
more familiar with.

There might be something to say, in terms of
leveraging existing familiarity with C.L. (and
perhaps in terms of reuse of some existing
C.L. code out there), for integrating more
C.L. stuff directly into Elisp.  The C.L.
"community" might not be on the order of, say,
the JavaScript or Python "community"..., but
still.  (And I'm pretty sure that many C.L.
programmers already use Emacs.)

Again, just one opinion.

(You might note that I'm both (1) for _not_
requiring `cl-lib.el' gratuitously in my
libraries and (2) integrating some more of
`cl-*.el' into Elisp.  For me, that's not a
contradiction.  If it seems to be one for
someone else, so be it.)

Re: [External] : Re: Lisp files that load cl-lib in problematical ways

[João Távora]

Drew Adams <drew.adams@oracle.com> writes:

> But no, not so well documented, if you mean the
> Emacs doc for them.  (Just one opinion.)

You're right, of course, but only because the Hyperspec and CLTL2 and
are top-class quality documentation.  The Emacs docs for this library
aren't up to that standard, but they're not terrible either.  Nothing
that can't be plugged, anyway.  Each function could use a link to the
Hyperspec maybe.

> I can't speak so definitively about `pcase.el'.
> Some C.L. things could be used directly in place
> of some of what `pcase' etc. do.  But patterned
> binding, which pcase also does, is very useful.

Sure, of course.  I use pcase sometimes, though cl-destructure-case is
also very useful.  The main advantage of pcase, as far as I udnerstand,
is that it is programmable, but I've not been able to get it.  So in
Eglot I rolled my own macros for fancier destructuring (see eglot.el's
eglot-dbind, for example)

I also agree many things could be brought into no-prefix Elisp
functions.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Of the conveniences, I think the ones I'd miss the most are 'cl-incf' 
  > and 'cl-decf', which I use in Eshell for recounting some handles.

If this pattern extends to many files -- if getf, incf and decf are
particularly useful among CL constructs -- making them standard parts of Emacs Lisp would be better than inducing lots of files to load cl-lib.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Is it possible that certain abstractions or functions are filling a gap
  > in Emacs Lisp itself?

Yes, it could well be true for some of them.

                           In the case of `cl-pushnew', I happen to think
  > that the answer is yes.

Do people really want `cl-pushnew', with its keyword arguments?
Or would a simple `pushnew' be even more pleasing?

To define `pushnew' in Emacs Lisp, and document it in the Emacs Lisp
Reference Manual, would be much cleaner than spreading the use of
cl-lib to more files.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [External] : Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

João Távora wrote:

> I also agree many things could be brought into no-prefix
> Elisp functions.

They can, but won't that just be a new name, and a new place?

Maybe some people will feel better about it that way but
I think to a lot of people it won't matter because they feel
that in terms of technology cl-lib is already Elisp and see no
harm using it whenever it is useful.

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: Lisp files that load cl-lib in problematical ways

[João Távora]

On Mon, Oct 30, 2023 at 2:44 PM Emanuel Berg <incal@dataswamp.org> wrote:
>
> João Távora wrote:
>
> > I also agree many things could be brought into no-prefix
> > Elisp functions.
>
> They can, but won't that just be a new name, and a new place?

Presumably the autoload embargo is lifted for those.

> Maybe some people will feel better about it that way but
> I think to a lot of people it won't matter because they feel
> that in terms of technology cl-lib is already Elisp and see no
> harm using it whenever it is useful.

As do I, but I can't use it in preloaded files.

João

Re: [External] : Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

João Távora wrote:

> As do I, but I can't use it in preloaded files.

Yes, so it will be a little win then.

But what will happen with the cl- versions, are we to keep
them? If not, maybe the people who wrote the library will
protest, and also, all the code that is currently cl- prefixed
will have to be changed.

Maybe reimplement them _and_ keep the cl- ones in their
current place?

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

>> As do I, but I can't use it in preloaded files.
>
> Yes, so it will be a little win then.
>
> But what will happen with the cl- versions, are we to keep
> them? If not, maybe the people who wrote the library will
> protest, and also, all the code that is currently cl-
> prefixed will have to be changed.
>
> Maybe reimplement them _and_ keep the cl- ones in their
> current place?

Here is the top 100 list.

A few hits don't make sense since the command is imperfect,
but other than that the list should be a pretty good
reflection of the situation.

Good luck with this :)

$ grep -E -h -o 'cl-[-a-z]*' **/*.el | sort | uniq -c | sort -r -b -n | head -n 100

   1416 cl-defmethod
    979 cl-loop
    720 cl-incf
    614 cl-lib
    486 cl-assert
    309 cl-letf
    303 cl-seq
    281 cl-defstruct
    257 cl-defgeneric
    227 cl-case
    218 cl-list
    209 cl-end
    189 cl-call-next-method
    172 cl-start
    171 cl-pushnew
    149 cl-defun
    143 cl-decf
    142 cl-p
    139 cl-keys
    135 cl-flet
    132 cl-return
    125 cl--loop-args
    114 cl--generic-
    106 cl-block
    104 cl-macs
    103 cl-destructuring-bind
     99 cl-check-type
     95 cl-some
     89 cl-
     89 cl-kwds
     88 cl-typep
     76 cl-prin
     72 cl-getf
     71 cl-extra
     69 cl-x
     69 cl-callf
     67 cl-pred
     67 cl--find-class
     66 cl-remove-if-not
     66 cl-defmacro
     65 cl-labels
     63 cl-second
     62 cl-return-from
     62 cl-mode
     62 cl-count
     61 cl-or-cast
     60 cl-set-difference
     53 cl-remove-if
     53 cl-generic
     51 cl-plusp
     51 cl-first
     51 cl-ecase
     50 cl-cvs
     49 cl-res
     49 cl-position
     48 cl-item
     47 cl-third
     47 cl-program
     47 cl-mapcan
     47 cl-every
     46 cl-tree
     45 cl-tempo-comma
     45 cl-substitute
     44 cl-current-ic
     43 cl-subseq
     42 cl-key
     40 cl-mapcar
     39 cl-tempo-right-paren
     39 cl-tempo-left-paren
     39 cl-rest
     39 cl-function
     39 cl-func
     39 cl-dolist
     38 cl-print-object
     38 cl-f
     37 cl-tempo-tags
     37 cl--slot-descriptor-name
     37 cl-macrolet
     37 cl-do
     36 cl-progv
     36 cl-find-if
     35 cl-embed-code
     34 cl-new
     34 cl-evenp
     33 cl-gensym
     33 cl-defsubst
     33 cl-check-register
     32 cl-print
     32 cl-pos
     32 cl-get-next-code
     31 cl-adjoin
     30 cl-remove-duplicates
     30 cl--loop-body
     30 cl-block-end-regexp
     30 cl-block-begin-regexp
     30 cl-basic-offset
     29 cl-remove
     29 cl-i
     28 cl-union
     28 cl-old-struct-compat-mode

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Jim Porter]

On 10/29/2023 7:09 PM, Richard Stallman wrote:
>    > Of the conveniences, I think the ones I'd miss the most are 'cl-incf'
>    > and 'cl-decf', which I use in Eshell for recounting some handles.
> 
> If this pattern extends to many files -- if getf, incf and decf are
> particularly useful among CL constructs -- making them standard parts of Emacs Lisp would be better than inducing lots of files to load cl-lib.

I don't think I'd fight tooth and nail to get incf/decf, but I'd 
certainly be happy to see them in standard Elisp.

I'm not sure how much others want to see setf-like constructs in 
Elisp[1], but I like them quite a bit. They fit in neatly alongside the 
C/C++ parts of my brain. Specifically, I find this style:

   (cl-incf (cdr output-target))
   - or -
   output_target.tail++;

a lot nicer than this:

   (setcdr output-target (1+ (cdr output-target)))
   - or -
   output_target.tail = output_target.tail + 1;

[1] I recall that some generalized values for setf were deprecated 
recently, but I think that was mainly for things where the setter had 
some non-obvious behavior.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Adam Porter]

> Do people really want `cl-pushnew', with its keyword arguments?
> Or would a simple `pushnew' be even more pleasing?

It might be more aesthetically or ideologically pleasing, but it would 
be much less useful.  :test is often necessary for using EQUAL (as EQ is 
often the wrong function to use); without it, a hypothetical non-CL 
PUSHNEW function would often be unusable.  And :key makes it easy to use 
with, e.g. structs' accessor functions (which are very helpful for 
developers as they allow compile-time checking, completion, etc).

When CL-PUSHNEW is called without keyword arguments, it's as simple as a 
hypothetical non-CL PUSHNEW that offered no keyword arguments.  The 
keyword arguments provide optional flexibility; without them, one would 
have to write a much larger, more complex form to do the same things.

Finally, if there were an additional, non-CL PUSHNEW function added, I 
would think that would be much more confusing to users, especially new 
ones, for there to be a difference in behavior between PUSHNEW and 
CL-PUSHNEW.

So I would ask that CL-PUSHNEW become PUSHNEW as-is.

> To define `pushnew' in Emacs Lisp, and document it in the Emacs Lisp
> Reference Manual, would be much cleaner than spreading the use of
> cl-lib to more files.

I would be happy to see more cl-lib functions "moved" back into "core" 
Elisp.  Using Elisp without cl-lib feels restrictive to me.

By the way, with regard to CL-PUSHNEW, it seems surprising that no one's 
mentioned the existing counterpart in "core" Elisp, ADD-TO-LIST. 
Notably, its docstring says:

   This is handy to add some elements to configuration variables,
   but please do not abuse it in Elisp code, where you are usually
   better off using ‘push’ or ‘cl-pushnew’.

So even "core", non-cl-lib Elisp recommends the use of some cl-lib 
functions, i.e. CL-PUSHNEW.  That certainly suggests a "gap" in core 
Elisp functionality.

--Adam

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> Date: Mon, 30 Oct 2023 21:31:41 -0500
> Cc: emacs-devel@gnu.org, stefankangas@gmail.com
> From: Adam Porter <adam@alphapapa.net>
> 
> By the way, with regard to CL-PUSHNEW, it seems surprising that no one's 
> mentioned the existing counterpart in "core" Elisp, ADD-TO-LIST. 
> Notably, its docstring says:
> 
>    This is handy to add some elements to configuration variables,
>    but please do not abuse it in Elisp code, where you are usually
>    better off using ‘push’ or ‘cl-pushnew’.
> 
> So even "core", non-cl-lib Elisp recommends the use of some cl-lib 
> functions, i.e. CL-PUSHNEW.  That certainly suggests a "gap" in core 
> Elisp functionality.

This kind of remarks in Emacs doc strings are never about "gaps" of
any kind, they are about cleaner coding practices.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Tue, Oct 31, 2023 at 2:32 AM Adam Porter <adam@alphapapa.net> wrote:
>
> > Do people really want `cl-pushnew', with its keyword arguments?
> > Or would a simple `pushnew' be even more pleasing?
>
> It might be more aesthetically or ideologically pleasing, but it would
> be much less useful.  :test is often necessary for using EQUAL (as EQ is
> often the wrong function to use); without it, a hypothetical non-CL
> PUSHNEW function would often be unusable.  And :key makes it easy to use
> with, e.g. structs' accessor functions (which are very helpful for
> developers as they allow compile-time checking, completion, etc).
>
> When CL-PUSHNEW is called without keyword arguments, it's as simple as a
> hypothetical non-CL PUSHNEW that offered no keyword arguments.  The
> keyword arguments provide optional flexibility; without them, one would
> have to write a much larger, more complex form to do the same things.

Of course.

The other day I noticed we have a C function called 'car-less-than-car'
whose only point is to optimize a super-specific use of 'sort'.  Maybe
in 1993 we didn't have compiler macros but now we do, so this is the
wrong way to go about it.

We should just write

  (cl-sort foo #'< :key #'car)

which is much more modular, readable, reusable, and doesn't require
the programmer to know of this specific vocabulary.

Sure, "but what about the optimization?", I hear you ask.  Well if
you must have it then just use a compiler macro to have your cake and
eat it too.

In this case, something like

  (defun cl-sort-compiler-macro (form &rest args)
    (if (equal (cdr args) '(#'< :key #'car))
        `(cl-sort ,(car args) #'car-less-than-car)
      form))

and add it to the cl-sort declaration with

  (declare (compiler-macro 'cl-sort-compiler-macro))

> Finally, if there were an additional, non-CL PUSHNEW function added, I
> would think that would be much more confusing to users, especially new
> ones, for there to be a difference in behavior between PUSHNEW and
> CL-PUSHNEW.

Exactly.  Please don't do this, we have enough such cases already.
Either migrate the fully functional version or don't.

Also, it's not too late to add keyword arguments to our 'sort'

João

Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

You seem to be judging features in terms of how much fundamental
change it makes in Emacs Lisp.  What you say about these changes, at
that level, seems to be valid, but it's a different issue, thus a
change of subject.

The problem of widespread use of cl-lib is that it adds a lot of
details to what one needs to know to work on Elisp programs.

  > In fact the only way that Emacs Lisp, the core language, has approached
  > Common Lisp in recent years to any degree was through the addition of
  > lexical binding,

Lexical binding, as a language feature, adds little complexity of
details to Emacs Lisp.  Once you kow what it is, the only added detail
is how to enable it for a given source file, and that is very simple.

  > Well, maybe native compilation also counts.

That too adds little complexity, expecially if you don't turn it on
when you build Emacs.  I haven't had to learn anything to understand
Elisp programs because of the existence of native compilation.

  > cl-lib.el, like any other library, doesn't add -- because it really
  > can't -- any features of that calibre to Emacs Lisp.  It just adds
  > functions and macros much in the way that lots of other libraries
  > add functions and macros.

This is what maks cl-lib such a pain: it adds many functions, which
have many details.  When lots of Emacs Lisp programs use cl-lib, that
means all those details become more things everyone working on Emacs
Lisp programs needs to know.

In other words, you're judging various changes by how deeply they
change the Emacs Lisp language, but the problem I'm concerned with is
about how many superficial changes they make.

  >   Anyway, the point is that to hack on long-lived files
  > such as lisp/minibuffer.el, one can't really "ignore" the new
  > dictionary of seq.el anymore.

That's exactly the problem.

  > If one were to compare apples to apples one could argue that CL's
  > functions are _more_  accessible and quite well documented,

The documentation that counts, for Emacs Lisp, is documentation that
is suitable to integrate into Emacs.  It has to be technically
suitable (written in Texinfo) and legally suitable (under a free
license compatile with the GFDL).  If it isn't in Texinfo, integrating
it into Emacs would be a substantial job, If it isn't
license-compatible, we would have to rewrite it starting from zero.

If it isn't free, we cannot refer to it as documentation at all.

Is there clear and well-written documentation for cl-lib which fits
those criteria?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [External] : Re: Lisp files that load cl-lib in problematical ways

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Here is the top 100 list.

  > A few hits don't make sense since the command is imperfect,
  > but other than that the list should be a pretty good
  > reflection of the situation.

Would someoe like to filter that list to remove everything not defined
in cl-lib.el itself?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > It might be more aesthetically or ideologically pleasing, but it would 
  > be much less useful.

If we add a function `pushnew' to Emacs Lisp, it will not have a :text
argument.  We are not going to start adding those arguments to
ordinary Emavs Lisp functions.

We could define `pushnew-equal' as well as `pushnew'.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > It might be more aesthetically or ideologically pleasing, but it would 
  > be much less useful.

If we add a function `pushnew' to Emacs Lisp, it will have a calling convention
like other Emacs Lisp functions -- not using CL-style keyword arguments.

We could define two functions, `pushnew' and `pushnq', if they are
both worth adding.  Or maybe the version that uses `equal' would DTRT
for most calls and we would only need that one.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [External] : Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Richard Stallman wrote:

>> Here is the top 100 list.
>>
>> A few hits don't make sense since the command is imperfect,
>> but other than that the list should be a pretty good
>> reflection of the situation.
>
> Would someoe like to filter that list to remove everything
> not defined in cl-lib.el itself?

One can do this in zsh by means of an extended glob pattern.

We see that `cl-pushnew' is on position #15, used 169 times.
So it is popular, but perhaps less so than expected?

`cl-loop' (#2, 979 times) and `cl-incf' (#3, 715 times) are as
expected very popular.

And there are more stuff that has been mentioned that appears
on the top 10 part of the list.

It is the list processing langauge :P

#! /bin/zsh
#
# this file:
#   https://dataswamp.org/~incal/conf/.zsh/emacs

setopt extendedglob

cle () {
    cd ~/src/emacs
    grep -E -h -o 'cl-[-a-z]+' **/*.el~lisp/emacs-lisp/cl-lib.el |
        sort          |
        uniq -c       |
        sort -r -b -n |
        head -n 100
}

$ cle
   1416 cl-defmethod
    979 cl-loop
    715 cl-incf
    611 cl-lib
    486 cl-assert
    309 cl-letf
    301 cl-seq
    279 cl-defstruct
    257 cl-defgeneric
    227 cl-case
    209 cl-end
    208 cl-list
    189 cl-call-next-method
    172 cl-start
    169 cl-pushnew
    149 cl-defun
    142 cl-p
    142 cl-decf
    135 cl-flet
    132 cl-return
    130 cl-keys
    125 cl--loop-args
    114 cl--generic-
    106 cl-block
    103 cl-destructuring-bind
    101 cl-macs
     99 cl-check-type
     95 cl-some
     89 cl-kwds
     88 cl-typep
     76 cl-prin
     72 cl-getf
     67 cl-pred
     67 cl--find-class
     66 cl-remove-if-not
     66 cl-defmacro
     66 cl-callf
     65 cl-labels
     65 cl-extra
     63 cl-x
     62 cl-second
     62 cl-return-from
     62 cl-mode
     62 cl-count
     61 cl-or-cast
     60 cl-set-difference
     53 cl-remove-if
     53 cl-generic
     51 cl-ecase
     50 cl-plusp
     50 cl-first
     50 cl-cvs
     49 cl-position
     47 cl-program
     47 cl-mapcan
     47 cl-every
     46 cl-third
     46 cl-res
     45 cl-tempo-comma
     45 cl-substitute
     44 cl-current-ic
     43 cl-subseq
     42 cl-key
     41 cl-item
     39 cl-tempo-right-paren
     39 cl-tempo-left-paren
     39 cl-function
     39 cl-dolist
     38 cl-print-object
     38 cl-mapcar
     37 cl-tempo-tags
     37 cl--slot-descriptor-name
     37 cl-macrolet
     37 cl-do
     36 cl-progv
     36 cl-find-if
     35 cl-f
     35 cl-embed-code
     34 cl-tree
     34 cl-func
     33 cl-gensym
     33 cl-evenp
     33 cl-defsubst
     33 cl-check-register
     32 cl-rest
     32 cl-print
     32 cl-pos
     32 cl-get-next-code
     30 cl-remove-duplicates
     30 cl--loop-body
     30 cl-block-end-regexp
     30 cl-block-begin-regexp
     30 cl-basic-offset
     29 cl-remove
     29 cl-i
     28 cl-union
     28 cl-member
     28 cl-adjoin
     27 cl-new
     27 cl-mismatch

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: rogers@rgrjr.com, incal@dataswamp.org, emacs-devel@gnu.org
> Date: Tue, 31 Oct 2023 21:48:57 -0400
> 
> Is there clear and well-written documentation for cl-lib which fits
> those criteria?

We have cl.texi.  Whether it's clear and well-written is a matter of
opinion.

Re: [External] : Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Wed, 01 Nov 2023 12:16:26 +0100
> 
> Richard Stallman wrote:
> 
> > Would someoe like to filter that list to remove everything
> > not defined in cl-lib.el itself?
> 
> One can do this in zsh by means of an extended glob pattern.
> 
> We see that `cl-pushnew' is on position #15, used 169 times.
> So it is popular, but perhaps less so than expected?
> 
> `cl-loop' (#2, 979 times) and `cl-incf' (#3, 715 times) are as
> expected very popular.
> 
> And there are more stuff that has been mentioned that appears
> on the top 10 part of the list.
> 
> It is the list processing langauge :P
> 
> #! /bin/zsh
> #
> # this file:
> #   https://dataswamp.org/~incal/conf/.zsh/emacs
> 
> setopt extendedglob
> 
> cle () {
>     cd ~/src/emacs
>     grep -E -h -o 'cl-[-a-z]+' **/*.el~lisp/emacs-lisp/cl-lib.el |
>         sort          |
>         uniq -c       |
>         sort -r -b -n |
>         head -n 100
> }
> 
> $ cle
>    1416 cl-defmethod
>     979 cl-loop
>     715 cl-incf
>     611 cl-lib
>     486 cl-assert
>     309 cl-letf
>     301 cl-seq
>     279 cl-defstruct
>     257 cl-defgeneric
>     227 cl-case
>     209 cl-end
>     208 cl-list
>     189 cl-call-next-method
>     172 cl-start
>     169 cl-pushnew
>     149 cl-defun
>     142 cl-p
>     142 cl-decf
>     135 cl-flet
>     132 cl-return
>     130 cl-keys
>     125 cl--loop-args
>     114 cl--generic-
>     106 cl-block
>     103 cl-destructuring-bind
>     101 cl-macs
>      99 cl-check-type
>      95 cl-some
>      89 cl-kwds
>      88 cl-typep
>      76 cl-prin
>      72 cl-getf
>      67 cl-pred
>      67 cl--find-class
>      66 cl-remove-if-not
>      66 cl-defmacro
>      66 cl-callf
>      65 cl-labels
>      65 cl-extra
>      63 cl-x
>      62 cl-second
>      62 cl-return-from
>      62 cl-mode
>      62 cl-count
>      61 cl-or-cast
>      60 cl-set-difference
>      53 cl-remove-if
>      53 cl-generic
>      51 cl-ecase
>      50 cl-plusp
>      50 cl-first
>      50 cl-cvs
>      49 cl-position
>      47 cl-program
>      47 cl-mapcan
>      47 cl-every
>      46 cl-third
>      46 cl-res
>      45 cl-tempo-comma
>      45 cl-substitute
>      44 cl-current-ic
>      43 cl-subseq
>      42 cl-key
>      41 cl-item
>      39 cl-tempo-right-paren
>      39 cl-tempo-left-paren
>      39 cl-function
>      39 cl-dolist
>      38 cl-print-object
>      38 cl-mapcar
>      37 cl-tempo-tags
>      37 cl--slot-descriptor-name
>      37 cl-macrolet
>      37 cl-do
>      36 cl-progv
>      36 cl-find-if
>      35 cl-f
>      35 cl-embed-code
>      34 cl-tree
>      34 cl-func
>      33 cl-gensym
>      33 cl-evenp
>      33 cl-defsubst
>      33 cl-check-register
>      32 cl-rest
>      32 cl-print
>      32 cl-pos
>      32 cl-get-next-code
>      30 cl-remove-duplicates
>      30 cl--loop-body
>      30 cl-block-end-regexp
>      30 cl-block-begin-regexp
>      30 cl-basic-offset
>      29 cl-remove
>      29 cl-i
>      28 cl-union
>      28 cl-member
>      28 cl-adjoin
>      27 cl-new
>      27 cl-mismatch

This list is not what Richard asked for, AFAIU.  Richard asked for a
list of functions defined in cl-lib.el, and only functions defined in
cl-lib.el.  The list above shows stuff that doesn't fit the request;
some examples of irrelevant symbols:

  cl-defmethod
  cl-defgeneric
  cl-loop
  cl-assert
  cl-defstruct
  cl-case
  cl-end
  cl-list

and many more.

I conclude that the above script is not a good way of producing a
useful list that Richard asked for.

Re: [External] : Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Eli Zaretskii wrote:

> This list is not what Richard asked for, AFAIU.
> Richard asked for a list of functions defined in cl-lib.el,

There are 20 `defun' in cl-lib.el, however one, "last*", is
commented out, so that leaves the following 19:

cl--defalias
cl--do-subst
cl--old-struct-type-of
cl--set-buffer-substring
cl--set-substring
cl-acons
cl-adjoin
cl-constantly
cl-copy-list
cl-digit-char-p
cl-evenp
cl-ldiff
cl-list*
cl-mapcar
cl-oddp
cl-pairlis
cl-proclaim
cl-subst
cl-values-list

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Wed, 01 Nov 2023 13:49:57 +0100
> 
> There are 20 `defun' in cl-lib.el, however one, "last*", is
> commented out, so that leaves the following 19:
> 
> cl--defalias
> cl--do-subst
> cl--old-struct-type-of
> cl--set-buffer-substring
> cl--set-substring
> cl-acons
> cl-adjoin
> cl-constantly
> cl-copy-list
> cl-digit-char-p
> cl-evenp
> cl-ldiff
> cl-list*
> cl-mapcar
> cl-oddp
> cl-pairlis
> cl-proclaim
> cl-subst
> cl-values-list

Thanks.  The "cl--*" ones should also be removed, I think (as they
are not supposed to be called from outsider cl-lib.el), but other than
that, I think this is the list.  And I cannot fail to point out that
the list is quite short, not nearly as long as the discussion seemed
to indicate.  Maybe we need to add to the list the functions in
cl-extra.el and cl-seq.el?

Re: [External] : Re: Lisp files that load cl-lib in problematical ways

[Emanuel Berg]

Eli Zaretskii wrote:

> And I cannot fail to point out that the list is quite short,
> not nearly as long as the discussion seemed to indicate.
> Maybe we need to add to the list the functions in
> cl-extra.el and cl-seq.el?

`how-many' "defun" in cl-lib related files:

72 cl-macs.el
64 cl-seq.el
55 cl-extra.el
46 cl-generic.el
31 cl-indent.el
20 cl-lib.el
15 cl-loaddefs.el
14 cl-print.el
 9 cl-preloaded.el

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: Lisp files that load cl-lib in problematical ways

[Alan Mackenzie]

Hello, Eli.

On Wed, Nov 01, 2023 at 15:07:31 +0200, Eli Zaretskii wrote:
> > From: Emanuel Berg <incal@dataswamp.org>
> > Date: Wed, 01 Nov 2023 13:49:57 +0100

> > There are 20 `defun' in cl-lib.el, however one, "last*", is
> > commented out, so that leaves the following 19:

> > cl--defalias
> > cl--do-subst
> > cl--old-struct-type-of
> > cl--set-buffer-substring
> > cl--set-substring
> > cl-acons
> > cl-adjoin
> > cl-constantly
> > cl-copy-list
> > cl-digit-char-p
> > cl-evenp
> > cl-ldiff
> > cl-list*
> > cl-mapcar
> > cl-oddp
> > cl-pairlis
> > cl-proclaim
> > cl-subst
> > cl-values-list

> Thanks.  The "cl--*" ones should also be removed, I think (as they
> are not supposed to be called from outsider cl-lib.el), but other than
> that, I think this is the list.  And I cannot fail to point out that
> the list is quite short, not nearly as long as the discussion seemed
> to indicate.  Maybe we need to add to the list the functions in
> cl-extra.el and cl-seq.el?

I think so.  Richard's concern is the sheer bulk of cl-*, which adds
complexity to the language by virtue of the mass of detail which both
new and experienced Emacs hackers have to cope with.  That complexity
exists regardless of when those functions and macros get loaded.

So, from that point of view, cl-extra.el and cl-seq.el are certainly
relevant, as is cl-macs.el, and maybe one or two others.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [External] : Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> Date: Wed, 1 Nov 2023 14:09:15 +0000
> Cc: Emanuel Berg <incal@dataswamp.org>, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > Maybe we need to add to the list the functions in
> > cl-extra.el and cl-seq.el?
> 
> I think so.  Richard's concern is the sheer bulk of cl-*, which adds
> complexity to the language by virtue of the mass of detail which both
> new and experienced Emacs hackers have to cope with.  That complexity
> exists regardless of when those functions and macros get loaded.
> 
> So, from that point of view, cl-extra.el and cl-seq.el are certainly
> relevant, as is cl-macs.el, and maybe one or two others.

No, cl-macs is not relevant, since those are just macros, they leave
no trace after compiling.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

It might be ok to add some keyword arguments to `sort', which are more
unusual and complex to use, but not to simple constructs like
`pushnew'.  This is Emacs Lisp, not Common Lisp.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Jim Porter]

On 11/1/2023 7:28 PM, Richard Stallman wrote:
> It might be ok to add some keyword arguments to `sort', which are more
> unusual and complex to use, but not to simple constructs like
> `pushnew'.  This is Emacs Lisp, not Common Lisp.

I'm not advocating for (or against) this, but if we wanted something 
like 'pushnew' and didn't want to use keyword arguments, then what about 
these two functions?

   (defun pushnew (value place &optional testfun) ...)
   (defun pushnew-key (value place keyfun) ...)

There's no reason I can see that we'd *need* to have both forms in a 
single function. (Unless we specifically wanted to match the Common Lisp 
API, and that's what cl-lib is for.)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Philip Kaludercic]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> It might be ok to add some keyword arguments to `sort', which are more
> unusual and complex to use, but not to simple constructs like
> `pushnew'.  This is Emacs Lisp, not Common Lisp.

What does that last sentence mean?  I hope this is not a too
philosophical question, but what constitutes "Emacs Lisp"?  It would
seem peculiar if it were to be defined by the arbitrary decisions of the
past, constrained by the contingent circumstances of the time.
Certainly part of it is that it is part of a tradition of Lisp
implementations, that gives it a distinctive flavour, but to me the main
aspects and attractions of Emacs Lisp (in contrast to CL and Scheme)
have been:

1. Not standardised; it is possible to extend the language without
   having to worry about how many implementations will follow along

2. Accepting Unix; not trying to abstract over various different
   operating systems and file systems, making it easier to work with GNU
   and other POSIX-based environments.

The first point would be relevant here.  Just like if-let or
thread-first, Emacs Lisp can learn from interesting ideas that other
languages provide, adapt and add them, making them available to
everyone.  If a form expresses something that people frequently want to
do, but either have to elaborate using (unless (memq ...) (push ...)),
then we are making Emacs more useful and expressive by providing the
functionality OOTB.  And isn't that the real point of Emacs Lisp?

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: João Távora <joaotavora@gmail.com>,
>  adam@alphapapa.net, emacs-devel@gnu.org,  stefankangas@gmail.com
> Date: Thu, 02 Nov 2023 08:55:54 +0000
> 
> Richard Stallman <rms@gnu.org> writes:
> 
> > It might be ok to add some keyword arguments to `sort', which are more
> > unusual and complex to use, but not to simple constructs like
> > `pushnew'.  This is Emacs Lisp, not Common Lisp.
> 
> What does that last sentence mean?

AFAIU, it means that Emacs Lisp traditionally doesn't use keyword
arguments, except as relatively rare exceptions.

> what constitutes "Emacs Lisp"?  It would
> seem peculiar if it were to be defined by the arbitrary decisions of the
> past, constrained by the contingent circumstances of the time.

Those "arbitrary decisions" are what got us to where we are now, 40
years later.  So some respect for those "arbitrary decisions" is due,
I think.

> to me the main aspects and attractions of Emacs Lisp (in contrast to
> CL and Scheme) have been:
> 
> 1. Not standardised; it is possible to extend the language without
>    having to worry about how many implementations will follow along

IMNSHO, extending Emacs Lisp as the language is not the main goal of
Emacs development.  Emacs Lisp is not a programming language on its
own, it is a language for implementing and extending features and
extensions in Emacs.  Thus, the main goal of Emacs development is to
develop applications and application-level features, and provide more
opportunities for extending the core where that is considered useful.
What we have in Emacs Lisp is IMO ample for that purpose.  Moreover,
most participants in Emacs development are not experts in programming
languages, their expertise is elsewhere (which is definitely a Good
Thing).

Objectively, adding new syntax and semantics to Emacs Lisp does make
the source code harder to read and maintain, because it makes the
language larger and requires familiarization with those new language
features, which more often than not look and feel completely different
from the "traditional" Emacs Lisp.  So even if we conclude that these
additions are worthwhile, we should not pretend they come without a
price, and IMO we should think carefully whether their use is
justified before we use them in each and every case.

> Emacs Lisp can learn from interesting ideas that other
> languages provide, adapt and add them, making them available to
> everyone.

It certainly can.  The question is: should it?  Since we are not
driven by any standard, it is completely up to us to make those
decisions, and we should IMO make them judiciously and carefully,
taking the downsides into consideration.  In particular, I hope people
agree that making a language too large and complex is not a good
thing in the long run.

> If a form expresses something that people frequently want to
> do, but either have to elaborate using (unless (memq ...) (push ...)),
> then we are making Emacs more useful and expressive by providing the
> functionality OOTB.  And isn't that the real point of Emacs Lisp?

AFAIU, what Richard says is that if some form is frequently used, we
should indeed consider adding a primitive or an API for it, but that
primitive or API doesn't have to look like CL.  Cf the pushnew
argument.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

[ Richard, with this message, I am also replying to your message in the
other thread ]

On Wed, Nov 1, 2023 at 1:48 AM Richard Stallman <rms@gnu.org> wrote:

>   >   Anyway, the point is that to hack on long-lived files
>   > such as lisp/minibuffer.el, one can't really "ignore" the new
>   > dictionary of seq.el anymore.
>
> That's exactly the problem.

Then why this laser-focus on cl-lib.el?  Why not criticize the use of
seq.el and map.el and pcase.el all of which "add many functions, which
have many details"?  Why are these libraries seemingly exempt from this
discussion?

The above question is mostly rhetorical, I am _not_ criticizing these
libraries or advocating restricting seq.el and map.el and pcase.el.

Reading certain parts of Emacs core has become impossible without
getting well acquainted with Stefan's Monnier pcase.el which I had to do
some 7-8 years ago.  It was an entirely alien creature when I first
saw it.  It is directly inspired by pattern matching of the ML language.

When reading a pcase,  I occasionally think to myself that it's overkill
complexity, that "in the old days" it was all much simpler.  In many
other cases I've learned to appreciate, and now it's part of my Lisp
toolbox too.

I think it's important to allow the programmers and maintainers working
on specific sections of the code a certain freedom to recommend
or select certain abstractions.  But Emacs's Lisp code base is now
very extensive and any kind of core-wide ban or rewrites of certain
patterns is counter-productive and highly contentious.  Let's be
very careful when going down that path.

On Thu, Nov 2, 2023 at 2:28 AM Richard Stallman <rms@gnu.org> wrote:
>
> It might be ok to add some keyword arguments to `sort', which are more
> unusual and complex to use, but not to simple constructs like
> `pushnew'.  This is Emacs Lisp, not Common Lisp.

Adding keyword arguments to a given function doesn't transform
Emacs Lisp into Common Lisp in any meaningful way.  Just as
translating a work by Shakespeare into Portuguese doesn't transform
Portuguese into English.  It just allows Portuguese speakers
to enjoy some very decent, maybe even essential, poetry.

Has Emacs Lisp become ML through the use of pcase.el?  Of course not.
It's become easier to write many things in Emacs Lisp, if anything.

Frequently, the argument advanced for not wanting to import feature
A or B from this other language, is "Emacs Lisp is to be kept
simple and not have this complexity".  Fair enough, programmers' craft
is foremost about managing complexity.  So very often a decision to
simplify here and now breeds enormous complexity elsewhere later.
Simplicity that proved effective and beneficial 30 years ago, in
a different context, might not really be those things today, or
30 years from now.

For example, the early decision not to have namespaces in Emacs Lisp,
taken in the name of simplicity, may have facilitated development of
Emacs features in a time where there were reasonably few packages.
But as the ecosystem grew, it created -- and still creates  -- many
challenges.

For example, one of the challenges it creates is this discussion
right here:  to be able to easily delimit what that "core" or
"standard" Emacs Lisp is -- which is what Philip Addressed.  There
is in fact no such thing.

This is _not_ necessarily advocating for namespaces/packages in Emacs
Lisp, it's just my way of pointing out that the argument of "simplicity"
is much more nuanced.  Is a keyword-argument-less 'pushnew' simpler
than 'cl-pushnew'?  Yes it is, undoubtedly.  Will the former breed
less complexity than the latter?  Very unlikely IMO.

Instead we should look at how much more complex than the hypothetical
'eq'-comparing version of 'pushnew' is `cl-pushnew` in the end.  In
the argumentless version it's exactly the same.  When a keyword
argument _is_ needed (say we need to test with #'equal or
#'my-special-comp)  doesn't that increase in complexity pay off
three-fold almost immediately?  IOW versatility of a given utility
must obviously be taken in consideration.

By the way, if it hasn't yet become apparent to you or other readers,
the keywords usually accepted to cl-sort, cl-pushnew, etc all belong
to a fairly small dictionary with consistent meaning.  For example,
the ':key' accepted by 'cl-sort' tells the thing to consider when
comparing, and the ':key' accepted by  'cl-pushnew' also tells
the thing to consider when comparing.  It is not more useful to
'cl-sort' than it is to 'cl-pushnew'.

Keyword arguments are IMHO one of the most elegant mechanisms to
promote maintainability, and generally typing less code.
Of course, there might be other possible, unique and original ways of
achieving the same of course, just as pcase.el showed me there were
other ways of doing destructuring.   But choosing any one option
just for the sake of "not being like any other language" is not the
right path to take.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

João Távora wrote:

>>> Anyway, the point is that to hack on long-lived files such
>>> as lisp/minibuffer.el, one can't really "ignore" the new
>>> dictionary of seq.el anymore.
>>
>> That's exactly the problem.
>
> Then why this laser-focus on cl-lib.el? Why not criticize
> the use of seq.el and map.el and pcase.el all of which "add
> many functions, which have many details"? Why are these
> libraries seemingly exempt from this discussion?

Indeed, this has been mentioned several times now and the
answer we get is "There is no special treatment of cl-lib.el".
Hard to believe.

> The above question is mostly rhetorical, I am _not_
> criticizing these libraries or advocating restricting seq.el
> and map.el and pcase.el.
>
> Reading certain parts of Emacs core has become impossible
> without getting well acquainted with Stefan's Monnier
> pcase.el which I had to do some 7-8 years ago. It was an
> entirely alien creature when I first saw it. It is directly
> inspired by pattern matching of the ML language.
>
> When reading a pcase, I occasionally think to myself that
> it's overkill complexity, that "in the old days" it was all
> much simpler. In many other cases I've learned to
> appreciate, and now it's part of my Lisp toolbox too.
>
> I think it's important to allow the programmers and
> maintainers working on specific sections of the code
> a certain freedom to recommend or select certain
> abstractions. But Emacs's Lisp code base is now very
> extensive and any kind of core-wide ban or rewrites of
> certain patterns is counter-productive and highly
> contentious. Let's be very careful when going down
> that path.

Very much so.

> Has Emacs Lisp become ML through the use of pcase.el?
> Of course not. It's become easier to write many things in
> Emacs Lisp, if anything.

It has become the SME(cl), the Standard Meta Emacs
Common Lisp. Deal with it.

-- 
underground experts united
https://dataswamp.org/~incal

RE: [External] : Re: What's missing in ELisp that makes people want to use cl-lib?

[Drew Adams]

My 2c:

There's a difference between (1) adding &key to
Elisp generally and (2) letting more functions
take advantage of :key and :test, in particular.

(We have :test for `make-hash-table', but I
think that's about it.)

Needing multiple functions to allow for what
:key and :test provide isn't really "simpler".
Likewise for needing positional KEY and TEST
optional args.  :key and :test are convenient.

Heresy or not, I'd welcome more allowance of
them, regardless of any coincidence, or lack
of it, with existing Common Lisp functions.

It would be nice to be able to provide for
their use in user-defined functions, without
any need to load anything from `cl-*'.

Just one stodgy opinion.

Re: [External] : Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Thu, Nov 2, 2023 at 3:20 PM Drew Adams <drew.adams@oracle.com> wrote:

> There's a difference between (1) adding &key to
> Elisp generally and (2) letting more functions
> take advantage of :key and :test, in particular.

Good point.  Very good point.  It's just
that (1) happens to be a very good way to do (2).

But you're 100% right for sure it's not the only
way and (2) is ultimately what I'm after as well.

> (We have :test for `make-hash-table', but I
> think that's about it.)

We have keyword arguments for more functions,
make-process, json-serialize are two that
come to mind, pretty sure there are others.

> It would be nice to be able to provide for
> their use in user-defined functions, without
> any need to load anything from `cl-*'.

So why not introduce a 'definitelynotcl-defun'
macro with support for '&definitelynotkey' that
does that? ;-)

João

Re: [External] : Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

João Távora wrote:

> So why not introduce a 'definitelynotcl-defun' macro with
> support for '&definitelynotkey' that does that? ;-)

We don't want do stop anyone from introducing new stuff.
If they manage to get it into Emacs, it can't be that bad.
If they also get other people to use it, it must be
even better.

For the people who are maintaining it, I think to some extent
it would come with the job to handle code that isn't
necessarily written as the maintainer would like it to be.
Since it is originally written by another person, as
I understand the case to be in terms of this discussion.
A person more friendly to cl-lib?

But nothing prevents the maintainer from not using cl-lib
him/herself. And, to what extent can code that relies heavily
on cl-lib be left alone by the maintainer? As long as it
works? Do we have bugs that typically require big re-writes of
existing code?

If so, I agree the maintainer's job isn't easy.

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: João Távora <joaotavora@gmail.com>,
>>  adam@alphapapa.net, emacs-devel@gnu.org,  stefankangas@gmail.com
>> Date: Thu, 02 Nov 2023 08:55:54 +0000
>> 
>> Richard Stallman <rms@gnu.org> writes:
>> 
>> > It might be ok to add some keyword arguments to `sort', which are more
>> > unusual and complex to use, but not to simple constructs like
>> > `pushnew'.  This is Emacs Lisp, not Common Lisp.
>> 
>> What does that last sentence mean?
>
> AFAIU, it means that Emacs Lisp traditionally doesn't use keyword
> arguments, except as relatively rare exceptions.

It is used rather prominently in `define-minor-mode' or
`define-derived-mode', or do you specifically mean keyword arguments to
functions?

>> what constitutes "Emacs Lisp"?  It would
>> seem peculiar if it were to be defined by the arbitrary decisions of the
>> past, constrained by the contingent circumstances of the time.
>
> Those "arbitrary decisions" are what got us to where we are now, 40
> years later.  So some respect for those "arbitrary decisions" is due,
> I think.

No disrespect meant, but I am not sure we are thinking of the same
things.  An "arbitrary decision" usually doesn't matter much, like
calling a function rplacd or setcdr.  If a decision got us to where we
are now, I would say it wasn't that arbitrary, but a good one?

>> to me the main aspects and attractions of Emacs Lisp (in contrast to
>> CL and Scheme) have been:
>> 
>> 1. Not standardised; it is possible to extend the language without
>>    having to worry about how many implementations will follow along
>
> IMNSHO, extending Emacs Lisp as the language is not the main goal of
> Emacs development.  Emacs Lisp is not a programming language on its
> own, it is a language for implementing and extending features and
> extensions in Emacs.  Thus, the main goal of Emacs development is to
> develop applications and application-level features, and provide more
> opportunities for extending the core where that is considered useful.
> What we have in Emacs Lisp is IMO ample for that purpose.  Moreover,
> most participants in Emacs development are not experts in programming
> languages, their expertise is elsewhere (which is definitely a Good
> Thing).

Of course not extending it for its own sake, but I would assume that
making it easier for users to write practical and useful code should be
something that is valued.

> Objectively, adding new syntax and semantics to Emacs Lisp does make
> the source code harder to read and maintain, because it makes the
> language larger and requires familiarization with those new language
> features, which more often than not look and feel completely different
> from the "traditional" Emacs Lisp.  So even if we conclude that these
> additions are worthwhile, we should not pretend they come without a
> price, and IMO we should think carefully whether their use is
> justified before we use them in each and every case.

Could you explain what you mean by "traditional" Emacs Lisp?  But yes, I
am not advocating for senselessly adding whatever seems remotely
interesting, of course.

>> Emacs Lisp can learn from interesting ideas that other
>> languages provide, adapt and add them, making them available to
>> everyone.
>
> It certainly can.  The question is: should it?  Since we are not
> driven by any standard, it is completely up to us to make those
> decisions, and we should IMO make them judiciously and carefully,
> taking the downsides into consideration.  In particular, I hope people
> agree that making a language too large and complex is not a good
> thing in the long run.
>
>> If a form expresses something that people frequently want to
>> do, but either have to elaborate using (unless (memq ...) (push ...)),
>> then we are making Emacs more useful and expressive by providing the
>> functionality OOTB.  And isn't that the real point of Emacs Lisp?
>
> AFAIU, what Richard says is that if some form is frequently used, we
> should indeed consider adding a primitive or an API for it, but that
> primitive or API doesn't have to look like CL.  Cf the pushnew
> argument.

Right, I agree with everything here.  The functionality, not the style
is of interest.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Bob Rogers]

   From: Philip Kaludercic <philipk@posteo.net>
   Date: Thu, 02 Nov 2023 21:26:00 +0000

   Eli Zaretskii <eliz@gnu.org> writes:

   >> From: Philip Kaludercic <philipk@posteo.net>
   >> Cc: João Távora <joaotavora@gmail.com>,
   >>  adam@alphapapa.net, emacs-devel@gnu.org,  stefankangas@gmail.com
   >> Date: Thu, 02 Nov 2023 08:55:54 +0000
   >> 
   >> Richard Stallman <rms@gnu.org> writes:
   >> 
   >> > It might be ok to add some keyword arguments to `sort', which
   >> > are more unusual and complex to use, but not to simple
   >> > constructs like `pushnew'.  This is Emacs Lisp, not Common Lisp.
   >> 
   >> What does that last sentence mean?
   >
   > . . .

   . . .

   >> what constitutes "Emacs Lisp"?  It would seem peculiar if it were
   >> to be defined by the arbitrary decisions of the past, constrained
   >> by the contingent circumstances of the time.
   >
   > Those "arbitrary decisions" are what got us to where we are now, 40
   > years later.  So some respect for those "arbitrary decisions" is due,
   > I think.

   No disrespect meant, but I am not sure we are thinking of the same
   things.  An "arbitrary decision" usually doesn't matter much, like
   calling a function rplacd or setcdr.  If a decision got us to where we
   are now, I would say it wasn't that arbitrary, but a good one?

I think we are not really talking about arbitrary decisions here, but
about language style decisions, which may seem arbitrary (especially if
you disagree with the style of the resulting language!) but do matter to
the consistency and coherence of the resulting language.  And, although
my experience only goes back to Emacs 18, I think the original Emacs
Lisp had a definite style that set it apart from other Lisp dialects.
But that was a long time ago, and Emacs Lisp has grown enormously (as
someone else pointed out early in the original thread) to support the
enormous growth in Emacs, so that original style is now much harder to
see.  (Indeed, I notice it mostly when updating my old code.)

   So, Richard, I see that you are fighting to preserve something real
and important -- your vision of Emacs Lisp as a coherent language -- and
I believe that may be your prerogative as its creator.  But I also
believe you may be a decade or two too late.

					-- Bob Rogers
					   http://www.rgrjr.com/

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Bob Rogers wrote:

> I think we are not really talking about arbitrary decisions
> here, but about language style decisions, which may seem
> arbitrary (especially if you disagree with the style of the
> resulting language!) but do matter to the consistency and
> coherence of the resulting language. And, although my
> experience only goes back to Emacs 18, I think the original
> Emacs Lisp had a definite style that set it apart from other
> Lisp dialects. But that was a long time ago, and Emacs Lisp
> has grown enormously (as someone else pointed out early in
> the original thread) to support the enormous growth in
> Emacs, so that original style is now much harder to see.
> (Indeed, I notice it mostly when updating my old code.)

So let's hear it then, what is it that sets Emacs 18 Elisp
apart from Emacs 30 Elisp

> So, Richard, I see that you are fighting to preserve
> something real and important -- your vision of Emacs Lisp as
> a coherent language --

and how is the Emacs 30 Elisp less coherent than the
Emacs 18 Elisp?

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Philip Kaludercic wrote:

> Could you explain what you mean by "traditional" Emacs Lisp?

Traditional Elisp does not use concepts from ML and CL - it
merely implements them.

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: rms@gnu.org,  joaotavora@gmail.com,  adam@alphapapa.net,
>   emacs-devel@gnu.org,  stefankangas@gmail.com
> Date: Thu, 02 Nov 2023 21:26:00 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > AFAIU, it means that Emacs Lisp traditionally doesn't use keyword
> > arguments, except as relatively rare exceptions.
> 
> It is used rather prominently in `define-minor-mode' or
> `define-derived-mode', or do you specifically mean keyword arguments to
> functions?

I did mention exceptions, didn't I?

And define-derived-mode supports keyword arguments only since Emacs
22; the original implementation didn't.

> >> what constitutes "Emacs Lisp"?  It would
> >> seem peculiar if it were to be defined by the arbitrary decisions of the
> >> past, constrained by the contingent circumstances of the time.
> >
> > Those "arbitrary decisions" are what got us to where we are now, 40
> > years later.  So some respect for those "arbitrary decisions" is due,
> > I think.
> 
> No disrespect meant, but I am not sure we are thinking of the same
> things.  An "arbitrary decision" usually doesn't matter much, like
> calling a function rplacd or setcdr.  If a decision got us to where we
> are now, I would say it wasn't that arbitrary, but a good one?

Exactly my point.  So what did you mean by "It would seem peculiar if
[what constitutes Emacs Lisp] were to be defined by the arbitrary
decisions of the past"?

> > IMNSHO, extending Emacs Lisp as the language is not the main goal of
> > Emacs development.  Emacs Lisp is not a programming language on its
> > own, it is a language for implementing and extending features and
> > extensions in Emacs.  Thus, the main goal of Emacs development is to
> > develop applications and application-level features, and provide more
> > opportunities for extending the core where that is considered useful.
> > What we have in Emacs Lisp is IMO ample for that purpose.  Moreover,
> > most participants in Emacs development are not experts in programming
> > languages, their expertise is elsewhere (which is definitely a Good
> > Thing).
> 
> Of course not extending it for its own sake, but I would assume that
> making it easier for users to write practical and useful code should be
> something that is valued.

We should consider such additions carefully, weighing their advantages
against the disadvantages: introducing "alien" syntax, making the
language larger, etc.

> > Objectively, adding new syntax and semantics to Emacs Lisp does make
> > the source code harder to read and maintain, because it makes the
> > language larger and requires familiarization with those new language
> > features, which more often than not look and feel completely different
> > from the "traditional" Emacs Lisp.  So even if we conclude that these
> > additions are worthwhile, we should not pretend they come without a
> > price, and IMO we should think carefully whether their use is
> > justified before we use them in each and every case.
> 
> Could you explain what you mean by "traditional" Emacs Lisp?

Basically, the language as it is, without macros whose syntax is
different from Emacs Lisp.  For example, cl-loop has syntax that to my
eyes is starkly not Emacs Lisp, because it uses many keyword-like
parts that look like they were lifted from Fortran.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: rms@gnu.org,  joaotavora@gmail.com,  adam@alphapapa.net,
>>   emacs-devel@gnu.org,  stefankangas@gmail.com
>> Date: Thu, 02 Nov 2023 21:26:00 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > AFAIU, it means that Emacs Lisp traditionally doesn't use keyword
>> > arguments, except as relatively rare exceptions.
>> 
>> It is used rather prominently in `define-minor-mode' or
>> `define-derived-mode', or do you specifically mean keyword arguments to
>> functions?
>
> I did mention exceptions, didn't I?
>
> And define-derived-mode supports keyword arguments only since Emacs
> 22; the original implementation didn't.

Ok, I interpreted this as in "few usages", not in "few definitions".

>> >> what constitutes "Emacs Lisp"?  It would
>> >> seem peculiar if it were to be defined by the arbitrary decisions of the
>> >> past, constrained by the contingent circumstances of the time.
>> >
>> > Those "arbitrary decisions" are what got us to where we are now, 40
>> > years later.  So some respect for those "arbitrary decisions" is due,
>> > I think.
>> 
>> No disrespect meant, but I am not sure we are thinking of the same
>> things.  An "arbitrary decision" usually doesn't matter much, like
>> calling a function rplacd or setcdr.  If a decision got us to where we
>> are now, I would say it wasn't that arbitrary, but a good one?
>
> Exactly my point.  So what did you mean by "It would seem peculiar if
> [what constitutes Emacs Lisp] were to be defined by the arbitrary
> decisions of the past"?

That "setcar" is supposed to be more Elisp-ish than "rplacd", or things
like that. 

>> > IMNSHO, extending Emacs Lisp as the language is not the main goal of
>> > Emacs development.  Emacs Lisp is not a programming language on its
>> > own, it is a language for implementing and extending features and
>> > extensions in Emacs.  Thus, the main goal of Emacs development is to
>> > develop applications and application-level features, and provide more
>> > opportunities for extending the core where that is considered useful.
>> > What we have in Emacs Lisp is IMO ample for that purpose.  Moreover,
>> > most participants in Emacs development are not experts in programming
>> > languages, their expertise is elsewhere (which is definitely a Good
>> > Thing).
>> 
>> Of course not extending it for its own sake, but I would assume that
>> making it easier for users to write practical and useful code should be
>> something that is valued.
>
> We should consider such additions carefully, weighing their advantages
> against the disadvantages: introducing "alien" syntax, making the
> language larger, etc.

What I had in mind were extensions like the recent `with-restriction'.
Defining something like this came to my mine on more than one occasion
when combining `save-restriction' and `narrow-to-region' in exactly the
same pattern as I had done many times before, because it seems natural
to have it combined in a single form.  I don't think this is necessarily
alien, perhaps even natural in the long term.

>> > Objectively, adding new syntax and semantics to Emacs Lisp does make
>> > the source code harder to read and maintain, because it makes the
>> > language larger and requires familiarization with those new language
>> > features, which more often than not look and feel completely different
>> > from the "traditional" Emacs Lisp.  So even if we conclude that these
>> > additions are worthwhile, we should not pretend they come without a
>> > price, and IMO we should think carefully whether their use is
>> > justified before we use them in each and every case.
>> 
>> Could you explain what you mean by "traditional" Emacs Lisp?
>
> Basically, the language as it is, without macros whose syntax is
> different from Emacs Lisp.  For example, cl-loop has syntax that to my
> eyes is starkly not Emacs Lisp, because it uses many keyword-like
> parts that look like they were lifted from Fortran.

Then again (cl-)loop is a peculiar example; even among CL programmers
there are many that dislike using it on the same grounds.  As a macro,
it doesn't really even attempt to blend into the surrounding language,
but uses special keywords and syntax parsing, that has both its
advantages and disadvantages, but it is still used if only because CL
doesn't have a simple `while' macro.

This is a much more stark example than pushnew.  I could imagine that
pushnew could be implemented without a keyword argument, if we accepted
three arguments (newelt place compare-fn).

But there are a plenty of definitions in cl-lib that in some form or
another I think would be nice to have in Elisp proper as well: list*,
flet, defmacro/defgeneric, defstruct would be some examples that come to
mind.  This is not only my personal perspective, but the feeling I get
from reviewing Elisp packages that include cl-lib for only two or three
definitions.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: rms@gnu.org,  joaotavora@gmail.com,  adam@alphapapa.net,
>   emacs-devel@gnu.org,  stefankangas@gmail.com
> Date: Fri, 03 Nov 2023 07:48:21 +0000
> 
> >> >> what constitutes "Emacs Lisp"?  It would
> >> >> seem peculiar if it were to be defined by the arbitrary decisions of the
> >> >> past, constrained by the contingent circumstances of the time.
> >> >
> >> > Those "arbitrary decisions" are what got us to where we are now, 40
> >> > years later.  So some respect for those "arbitrary decisions" is due,
> >> > I think.
> >> 
> >> No disrespect meant, but I am not sure we are thinking of the same
> >> things.  An "arbitrary decision" usually doesn't matter much, like
> >> calling a function rplacd or setcdr.  If a decision got us to where we
> >> are now, I would say it wasn't that arbitrary, but a good one?
> >
> > Exactly my point.  So what did you mean by "It would seem peculiar if
> > [what constitutes Emacs Lisp] were to be defined by the arbitrary
> > decisions of the past"?
> 
> That "setcar" is supposed to be more Elisp-ish than "rplacd", or things
> like that. 

I didn't mean traditional names, I meant traditional syntax and
semantics of constructs.

> >> Of course not extending it for its own sake, but I would assume that
> >> making it easier for users to write practical and useful code should be
> >> something that is valued.
> >
> > We should consider such additions carefully, weighing their advantages
> > against the disadvantages: introducing "alien" syntax, making the
> > language larger, etc.
> 
> What I had in mind were extensions like the recent `with-restriction'.

I doubt that this is what bothers Richard (and me, to some extent).

> >> Could you explain what you mean by "traditional" Emacs Lisp?
> >
> > Basically, the language as it is, without macros whose syntax is
> > different from Emacs Lisp.  For example, cl-loop has syntax that to my
> > eyes is starkly not Emacs Lisp, because it uses many keyword-like
> > parts that look like they were lifted from Fortran.
> 
> Then again (cl-)loop is a peculiar example; even among CL programmers
> there are many that dislike using it on the same grounds.

Grepping our own sources for "(cl-loop" comes back with more than 760
hits.  That's more than one use for every 2 Lisp files in our tree.

And cl-loop is not the only example of additions whose syntax deviates
from Emacs Lisp traditions.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> >> Could you explain what you mean by "traditional" Emacs Lisp?
>> >
>> > Basically, the language as it is, without macros whose syntax is
>> > different from Emacs Lisp.  For example, cl-loop has syntax that to my
>> > eyes is starkly not Emacs Lisp, because it uses many keyword-like
>> > parts that look like they were lifted from Fortran.
>> 
>> Then again (cl-)loop is a peculiar example; even among CL programmers
>> there are many that dislike using it on the same grounds.
>
> Grepping our own sources for "(cl-loop" comes back with more than 760
> hits.  That's more than one use for every 2 Lisp files in our tree.
>
> And cl-loop is not the only example of additions whose syntax deviates
> from Emacs Lisp traditions.

It certainly has fans, though I believe this is more due to the utility
and flexibility than the syntax.  If there were a Elisp'ish alternative
(in the sense in which pcase is also an alternative to cl-case --
assuming that you see pcase as Elisp'ish?), then I honestly believe it
could be more popular than cl-loop.  I would be disappointed if being
Elisp'ish would mean that these kinds of language features per se
wouldn't be permissible.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Bob Rogers]

   From: Emanuel Berg <incal@dataswamp.org>
   Date: Fri, 03 Nov 2023 03:21:30 +0100

   Bob Rogers wrote:

   > I think we are not really talking about arbitrary decisions here,
   > but about language style decisions . . .

   So let's hear it then, what is it that sets Emacs 18 Elisp
   apart from Emacs 30 Elisp . . .

Briefly, it's now a much larger language . . .

   and how is the Emacs 30 Elisp less coherent than the Emacs 18 Elisp?

   -- 
   underground experts united
   https://dataswamp.org/~incal

. . . and the additions have come from diverse sources.  It started as a
Maclisp subset with extensions to support editing.  The issue is that
the programming features have grown, and not only from Common Lisp,
moving it away from its Maclisp roots.  (This, I believe, is the gist of
Richard's complaint.)  And the language editing features have (not
surprisingly) grown with Emacs' growth as an editor, as they have had to
do, which has unavoidably added to the learning curve you have to climb
before you can contribute to Emacs.  It has also made the language more
about editing and less about programming, which IMHO makes the style
question less acute.

					-- Bob

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Bob Rogers wrote:

>> So let's hear it then, what is it that sets Emacs 18 Elisp
>> apart from Emacs 30 Elisp . . .
>
> Briefly, it's now a much larger language . . .
>
>> and how is the Emacs 30 Elisp less coherent than the Emacs
>> 18 Elisp?
>
> . . . and the additions have come from diverse sources.
> It started as a Maclisp subset with extensions to support
> editing. The issue is that the programming features have
> grown, and not only from Common Lisp, moving it away from
> its Maclisp roots. (This, I believe, is the gist of
> Richard's complaint.) And the language editing features have
> (not surprisingly) grown with Emacs' growth as an editor, as
> they have had to do, which has unavoidably added to the
> learning curve you have to climb before you can contribute
> to Emacs. It has also made the language more about editing
> and less about programming, which IMHO makes the style
> question less acute.

I understand that it has expanded in size, this is the nature
of technology systems in general and perhaps even more so with
Lisp programming, since it is so easy and fun to do.

But I am still uncertain how this has changed the style and
reduced the coherence of Elisp.

Case in point, I don't see any such difference when I look at
cl-lib.el?

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Bob Rogers]

   From: Emanuel Berg <incal@dataswamp.org>
   Date: Fri, 03 Nov 2023 21:32:15 +0100

   Bob Rogers wrote:

   >> So let's hear it then, what is it that sets Emacs 18 Elisp
   >> apart from Emacs 30 Elisp . . .
   >
   > Briefly, it's now a much larger language . . .
   >
   >> and how is the Emacs 30 Elisp less coherent than the Emacs
   >> 18 Elisp?
   >
   > . . . and the additions have come from diverse sources . . .

   I understand that it has expanded in size, this is the nature
   of technology systems in general and perhaps even more so with
   Lisp programming, since it is so easy and fun to do.

But, just to point out the obvious, Emacs Lisp is unusual in that the
language and the application are inseparable.  In other language systems
(e.g. Python, Perl) the language, the application, and packages used to
build the application (even when distributed as part of the language)
are all distinct; in Emacs the boundaries are much fuzzier, undoubtedly
another consequence of Lisp.

   But I am still uncertain how this has changed the style and
   reduced the coherence of Elisp.

I'm sorry I'm apparently unable to be more concrete about explaining my
impressions.  Part of this is due to the elapsed time.  ISTR that, when
I started reading Elisp in the mid-90's, it had a "retro" feel compared
to the other Lisps I was using.  As time went on, I've had to hit C-h f
more and more to understand what was going on, not only for language
features (pcase has been mentioned), but for editing features like
indirect buffers and overlays as they were added -- not strictly part of
the language, but again, where's do you draw the boundary?  (I've also
had to read more C code to follow what's going on, but that's a separate
issue.)

   Case in point, I don't see any such difference when I look at
   cl-lib.el?

   -- 
   underground experts united
   https://dataswamp.org/~incal

Between cl-lib.el and what?

					-- Bob

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Bob Rogers wrote:

> I'm sorry I'm apparently unable to be more concrete about
> explaining my impressions. Part of this is due to the
> elapsed time. ISTR that, when I started reading Elisp in the
> mid-90's, it had a "retro" feel compared to the other Lisps
> I was using. As time went on, I've had to hit C-h f more and
> more to understand what was going on, not only for language
> features (pcase has been mentioned), but for editing
> features like indirect buffers and overlays as they were
> added -- not strictly part of the language, but again,
> where's do you draw the boundary?

To be honest, I have not seen any.

>> Case in point, I don't see any such difference when I look
>> at cl-lib.el?
>
> Between cl-lib.el and what?

We have to get the Emacs 18 source so we can compare :)

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Stephen Berman]

On Fri, 03 Nov 2023 23:46:22 +0100 Emanuel Berg <incal@dataswamp.org> wrote:

> Bob Rogers wrote:
>
>> I'm sorry I'm apparently unable to be more concrete about
>> explaining my impressions. Part of this is due to the
>> elapsed time. ISTR that, when I started reading Elisp in the
>> mid-90's, it had a "retro" feel compared to the other Lisps
>> I was using. As time went on, I've had to hit C-h f more and
>> more to understand what was going on, not only for language
>> features (pcase has been mentioned), but for editing
>> features like indirect buffers and overlays as they were
>> added -- not strictly part of the language, but again,
>> where's do you draw the boundary?
>
> To be honest, I have not seen any.
>
>>> Case in point, I don't see any such difference when I look
>>> at cl-lib.el?
>>
>> Between cl-lib.el and what?
>
> We have to get the Emacs 18 source so we can compare :)

You can find several versions here:

https://github.com/larsbrinkhoff/emacs-history

Steve Berman

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Stephen Berman wrote:

>> We have to get the Emacs 18 source so we can compare :)
>
> You can find several versions here:
>
> https://github.com/larsbrinkhoff/emacs-history

Several, including Emacs 18!

I followed the link to this page:

  http://bitsavers.org/bits/MIT/gnu/emacs_18.41.tar.gz

Now we can see clearly how the style has changed :)

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Andrea Corallo]

Andrea Corallo <acorallo@gnu.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Andrea Corallo <acorallo@gnu.org>
>>> Cc: Stephen Berman <stephen.berman@gmx.net>,  incal@dataswamp.org,
>>>   emacs-devel@gnu.org
>>> Date: Thu, 19 Oct 2023 09:44:51 -0400
>>> 
>>> Okay, I confirm that comp.el loads cl-lib, so any jit compilation
>>> triggered loads that.
>>
>> This is OK, not a problem.
>>
>>> OTOH one thing we could do, if that's important, is to split the code
>>> that only drives the async compilation (that actually happens in a
>>> subprocess) so we don't load cl-lib in the Emacs the user is actually
>>> using.  This should not be that hard (optimism mode on).
>>
>> I don't see the need, but I will not object if you want to make such
>> changes.
>
> Okay I'll think about it, might be a positive change even leaving aside
> the cl-lib discussion.

Okay in scratch/comp-run we have a branch that does not require to load
comp.el for jit compilations and for installing already existing
trampolines (compiling a new trampolines indeed require to load the
compiler).

It achieves that adding comp-run.el for runtime dependencies.

Still the compiler is loaded for C-h f, to solve that I think we'd need
to add a third file (ex comp-common.el), not sure it's worth ATM.

BR

  Andrea

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >    So, Richard, I see that you are fighting to preserve something real
  > and important -- your vision of Emacs Lisp as a coherent language -- and
  > I believe that may be your prerogative as its creator.  But I also
  > believe you may be a decade or two too late.

It is not too late if we find a few more contributors who appreciate
what is at stake.

The consistent style of Emacs Lisp is one part of it.
But there is a more practical part, which I've explained recently:
the duplication, even triplication of features for the same job.
The result of this is to multiply what one needs to know
to understand the code in Emacs.

If Emacs Lisp had only one interface for doing a particular job, it
would be enough to learn that one.  But when it has several different
interfaces for doing the same job, you end up needing to know all
those interfaces.   Any one file probably uses one of the interfaces;
to understand that file, knowing that one interface is emnough.
But the next file you look at probably uses a different interface,
and the next one yet another interface -- so in practice you need to
know all the redundant interfaces.

This puts an avoidable extra a burden on everyone who wants to read
the Lisp code of Emacs.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [External] : Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > There's a difference between (1) adding &key to
  > Elisp generally and (2) letting more functions
  > take advantage of :key and :test, in particular.

That is a valid distinction.

I would be happy to see a keyword argument facility provided for
function definitions in Emacs Lisp.  There are occasions where it is
good for a function to accept keyword arguments -- especially those
functions that need to accept lots of arguments.

What I object to are the keyword arguments used in Common Lisp to call
simple, everyday functions.  For instance, the sequence functions.
(For this reason, I prefer the seq- functions to the Common Lisp
sequence functions.)

But we can split that issue by making another distinction, between
(2a) sequence functions that _accept_ the Common Lisp sequence
function keywords, such as :key and :test, and
(2b) frequently needing to pass those arguments.

(2a) does not make most code any more clumsy.  It affects only the
_definitions_ of these sequence functions.  What makes code ugly is
(2b).

Logically, (2a) does not force (2b), but the two are directly related.
The designers of Common Lisp saw (2a) as providing an opportunity to
"simplify" by eliminating the predefined functions to compare using
`equal' and to compare using `eq'.  By default they use `eql',
which says "a pox on both `memq' and `member'."

In practice, that makes the Common Lisp functions inconvenient to use.

I would not be unhappy if Emacs Lisp functions started to support :key
and ;test where they make sense, provided they did this in an Emacs
Lisp way: default comparison is `equal'.

(We should take care to implement it without a slowdown in the case
where the keyword args are not specified.)

This means that `member' would accept ;test, but it would still
compare using `equal' by default (if :test was nil or not specified).
And there would still be `memq' to compare with `eq'.

When I say "new functions", that includes any Common Lisp functions
that are currently available in cl-seq.el with a cl- prefix but NOT
treated as part of Emacs Lisp.  To treat them as part of Emacs Lisp,
we should make them compatible, we should (1) make them default to
Emacs Lisp tradition and (2) accompany them with a corresponding q
function, if comparing using `eq' is useful for that function.

We would also rename the function, as part of the bargain, to avoid an
incompatibility.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

RE: [External] : Re: What's missing in ELisp that makes people want to use cl-lib?

[Drew Adams]

>   > There's a difference between (1) adding &key to
>   > Elisp generally and (2) letting more functions
>   > take advantage of :key and :test, in particular.
> 
> That is a valid distinction.
> 
> I would be happy to see a keyword argument facility provided for
> function definitions in Emacs Lisp.  There are occasions where it is
> good for a function to accept keyword arguments -- especially those
> functions that need to accept lots of arguments.
> 
> What I object to are the keyword arguments used in Common Lisp to call
> simple, everyday functions.  For instance, the sequence functions.
> (For this reason, I prefer the seq- functions to the Common Lisp
> sequence functions.)
> 
> But we can split that issue by making another distinction, between
> (2a) sequence functions that _accept_ the Common Lisp sequence
> function keywords, such as :key and :test, and
> (2b) frequently needing to pass those arguments.
> 
> (2a) does not make most code any more clumsy.  It affects only the
> _definitions_ of these sequence functions.  What makes code ugly is
> (2b).
> 
> Logically, (2a) does not force (2b), but the two are directly related.
> The designers of Common Lisp saw (2a) as providing an opportunity to
> "simplify" by eliminating the predefined functions to compare using
> `equal' and to compare using `eq'.  By default they use `eql',
> which says "a pox on both `memq' and `member'."
> 
> In practice, that makes the Common Lisp functions inconvenient to use.
> 
> I would not be unhappy if Emacs Lisp functions started to support :key
> and ;test where they make sense, provided they did this in an Emacs
> Lisp way: default comparison is `equal'.
> 
> (We should take care to implement it without a slowdown in the case
> where the keyword args are not specified.)
> 
> This means that `member' would accept ;test, but it would still
> compare using `equal' by default (if :test was nil or not specified).
> And there would still be `memq' to compare with `eq'.
> 
> When I say "new functions", that includes any Common Lisp functions
> that are currently available in cl-seq.el with a cl- prefix but NOT
> treated as part of Emacs Lisp.  To treat them as part of Emacs Lisp,
> we should make them compatible, we should (1) make them default to
> Emacs Lisp tradition and (2) accompany them with a corresponding q
> function, if comparing using `eq' is useful for that function.
> 
> We would also rename the function, as part of the bargain, to avoid an
> incompatibility.

FWIW, to me, all of what you say there is welcome news.

When incorporating whatever you describe as incorporating
from cl-* into Elisp (using `equal' as default), I hope
you mean giving it some other name than the CL name it
has (ignoring the `cl-' prefix or not).  I think that's
what you have in mind, when you say this:

  > We would also rename the function, as part of the
  > bargain, to avoid an incompatibility.

IOW, to me it would be fine to get such a function for
Elisp, but _not_ to have it keep the CL name but with
altered semantics (e.g. `equal' default, not `eql').

I've long argued that if we have a Common Lisp emulation
library then it should be a reasonable emulation, and
not gratuitously deviate from CL behavior/semantics.

It would not be good to keep the CL name for a function
but change its behavior from defaulting with `eql' to
defaulting with `equal'.  Better to give it a different
name.  At _least_ drop the `cl-' prefix, but better to
avoid any confusion by giving it an altogether different
name.

RE: [External] : Re: What's missing in ELisp that makes people want to use cl-lib?

[Bob Rogers]

   From: Drew Adams <drew.adams@oracle.com>
   Date: Tue, 7 Nov 2023 16:39:16 +0000

   >   > There's a difference between (1) adding &key to
   >   > Elisp generally and (2) letting more functions
   >   > take advantage of :key and :test, in particular.
   > 
   > That is a valid distinction.
   > 
   > I would be happy to see a keyword argument facility provided for
   > function definitions in Emacs Lisp.  There are occasions where it is
   > good for a function to accept keyword arguments -- especially those
   > functions that need to accept lots of arguments . . .

   FWIW, to me, all of what you say there is welcome news . . .

Likewise.  I have especially wished for a native &key; I think it will
greatly help Elisp code readability.

					-- Bob

Re: [External] : Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Richard Stallman <rms@gnu.org> writes:

> We would also rename the function, as part of the bargain, to avoid an
> incompatibility.

I don't know what this bargain is about, but I somehow got the vibes of
a yet another new interface being on the horizon. Just saying.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Richard Stallman wrote:

>> So, Richard, I see that you are fighting to preserve
>> something real and important -- your vision of Emacs Lisp
>> as a coherent language -- and I believe that may be your
>> prerogative as its creator. But I also believe you may be
>> a decade or two too late.
>
> It is not too late if we find a few more contributors who
> appreciate what is at stake [...]

Maybe, but how?

We have `cl-incf' and can't remove it as that would break the
CL emulation interface.

To the people who use it, and don't care about the CL
emulation part necessarily, they just think it is useful, we
can't say instead use

  (setq var (1+ var))

because we can't make the case that that is in any way better
than

  (cl-incf var)

We can however add a new function, "incf", that does the same
thing, only does not rely upon cl-lib in any way.

But how would this help the maintainers? Instead of
familiarizing themselves with `cl-incf', and all other
functions provided by cl-lib that are in use and people will
continue to use, they will have to learn the new functions
replacing them? So what difference will that make?

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Björn Bidar]

Emanuel Berg <incal@dataswamp.org> writes:

> But how would this help the maintainers? Instead of
> familiarizing themselves with `cl-incf', and all other
> functions provided by cl-lib that are in use and people will
> continue to use, they will have to learn the new functions
> replacing them? So what difference will that make?

It does sound more and more the fight for some kind of language purity
to not use cl-lib. Removed from reality like Don Quixote.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Alan Mackenzie]

Hello, Björn.

On Wed, Nov 08, 2023 at 17:00:06 +0200, Björn Bidar wrote:
> Emanuel Berg <incal@dataswamp.org> writes:

> > But how would this help the maintainers? Instead of
> > familiarizing themselves with `cl-incf', and all other
> > functions provided by cl-lib that are in use and people will
> > continue to use, they will have to learn the new functions
> > replacing them? So what difference will that make?

> It does sound more and more the fight for some kind of language purity
> to not use cl-lib. Removed from reality like Don Quixote.

That is verging on the offensively patronising, and I don't think it is
appropriate language for this mailing list.

Just take one look at this "reality" you're so supportive of: the
widespread use of cl-lib, not just in people's own projects, but
throughout the core of Emacs, has multiplied the size of Lisp language
part of Emacs by a factor of around 3.   This is a gross increase in
complexity for maintainers that is not justified by the slight increase
in facility that cl-lib (along with things like seq.el and oclosures)
gives.

Throughout this long discussion, this indiscriminate use of cl-lib has
been supported only by occasional contributers.  Those who actually
maintain other people's code, apart from (I think) Eli, Richard and
myself, have been conspicuously silent.  None of us three have favoured
such use of cl-lib.

Occasional contributors may be fascinated by cl-lib, and learn enough of
it to use random bits of it in their code.  The trouble is, each such
contributor uses a different piece of cl-lib, with the result that those
who end up maintaining it need to know a far greater part of it just to
cope.

This factor of 3 is, I believe, a significant barrier to new programmers
coming into Emacs; Elisp is just that much more difficult than it was in
the past.  And it isn't just for newcomers that it is more difficult.  I
spend a significant amount of debugging time having to look up doc
strings and manual pages for obscure cl-lib (etc.) functions.  This is
lost time, representing a diminution of my capacity to maintain Emacs.
I suspect it is similar for other maintainers.

That is the current reality.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Alan Mackenzie wrote:

> Occasional contributors may be fascinated by cl-lib, and
> learn enough of it to use random bits of it in their code.
> The trouble is, each such contributor uses a different piece
> of cl-lib, with the result that those who end up maintaining
> it need to know a far greater part of it just to cope.

Yes, this sounds like a good explanation how the situation has
appeared. We see how two worlds collide.

But how do you propose to improve the situation, then?

Maybe maintaining can be distributed so that the "occasional
contributors" are maintaining their own code?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Andrea Corallo]

Andrea Corallo <acorallo@gnu.org> writes:

> Andrea Corallo <acorallo@gnu.org> writes:
>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>>>> From: Andrea Corallo <acorallo@gnu.org>
>>>> Cc: Stephen Berman <stephen.berman@gmx.net>,  incal@dataswamp.org,
>>>>   emacs-devel@gnu.org
>>>> Date: Thu, 19 Oct 2023 09:44:51 -0400
>>>> 
>>>> Okay, I confirm that comp.el loads cl-lib, so any jit compilation
>>>> triggered loads that.
>>>
>>> This is OK, not a problem.
>>>
>>>> OTOH one thing we could do, if that's important, is to split the code
>>>> that only drives the async compilation (that actually happens in a
>>>> subprocess) so we don't load cl-lib in the Emacs the user is actually
>>>> using.  This should not be that hard (optimism mode on).
>>>
>>> I don't see the need, but I will not object if you want to make such
>>> changes.
>>
>> Okay I'll think about it, might be a positive change even leaving aside
>> the cl-lib discussion.
>
> Okay in scratch/comp-run we have a branch that does not require to load
> comp.el for jit compilations and for installing already existing
> trampolines (compiling a new trampolines indeed require to load the
> compiler).
>
> It achieves that adding comp-run.el for runtime dependencies.
>
> Still the compiler is loaded for C-h f, to solve that I think we'd need
> to add a third file (ex comp-common.el), not sure it's worth ATM.

I think what is now in scratch/comp-run does what we want.  If there are
no objections I'll push it to master after some polishing in the
following days.

Bests

  Andrea

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 08/11/2023 19:18, Alan Mackenzie wrote:
> Throughout this long discussion, this indiscriminate use of cl-lib has
> been supported only by occasional contributers.  Those who actually
> maintain other people's code, apart from (I think) Eli, Richard and
> myself, have been conspicuously silent.

I like cl-lib (it could be optimized better still, but it's good for 
certain uses), and this discussion doesn't look like it will reach any 
constructive conclusion, so it didn't seem worth it to spend time on it.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 992 bytes --]

On Wed, Nov 8, 2023, 20:20 Dmitry Gutov <dmitry@gutov.dev> wrote:

> On 08/11/2023 19:18, Alan Mackenzie wrote:
> > Throughout this long discussion, this indiscriminate use of cl-lib has
> > been supported only by occasional contributers.  Those who actually
> > maintain other people's code, apart from (I think) Eli, Richard and
> > myself, have been conspicuously silent.
>
> I like cl-lib (it could be optimized better still, but it's good for
> certain uses), and this discussion doesn't look like it will reach any
> constructive conclusion, so it didn't seem worth it to spend time on it.
>

If it needed any confirmation, I too like cl-lib and I too help maintain
other people's code in the Emacs core tree as well as maintaining a number
of libraries I have authored. I read many styles of Elisp here and am not
really phased by any particular programming technique being discussed here
(cl-lib, seq, pcase, verbose while loops with catch/throw, etc etc).

João

[-- Attachment #2: Type: text/html, Size: 1645 bytes --]

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

João Távora wrote:

>>> Throughout this long discussion, this indiscriminate use
>>> of cl-lib has been supported only by occasional
>>> contributers. Those who actually maintain other people's
>>> code, apart from (I think) Eli, Richard and myself, have
>>> been conspicuously silent.
>>
>> I like cl-lib (it could be optimized better still, but it's
>> good for certain uses), and this discussion doesn't look
>> like it will reach any constructive conclusion, so it
>> didn't seem worth it to spend time on it.
>
> If it needed any confirmation, I too like cl-lib and I too
> help maintain other people's code in the Emacs core tree as
> well as maintaining a number of libraries I have authored.
> I read many styles of Elisp here and am not really phased by
> any particular programming technique being discussed here
> (cl-lib, seq, pcase, verbose while loops with catch/throw,
> etc etc).

I like cl-lib and I maintain an ELPA package [1] - ELPA, as we
know, is part of GNU Emacs. It doesn't use cl-lib. Okay,
so bad example then.

[1] https://elpa.gnu.org/packages/wrap-search.html

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > When incorporating whatever you describe as incorporating
  > from cl-* into Elisp (using `equal' as default), I hope
  > you mean giving it some other name than the CL name it
  > has (ignoring the `cl-' prefix or not).

Yes, for sure.  (I thought I had made that explicit.)  The natural
name to use is the existing Emacs Lisp name -- for instance, `member'.

I'm not arguing for adding the :key and :test keyword args to every
function that could support them, or to any functions at all.  But IF
and WHEN we want to have functions supporting those keyword arguments
as parts of Emacs Lisp that programmers need to know about, this is
the clean way to do it.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Michael Heerdegen]

Dmitry Gutov <dmitry@gutov.dev> writes:

> I like cl-lib (it could be optimized better still, but it's good for
> certain uses), and this discussion doesn't look like it will reach any
> constructive conclusion, so it didn't seem worth it to spend time on
> it.

I also find it a bit exaggerating: cl-lib is large, but the majority of
its uses in Emacs are from a small subset.  In cases where it's
objectively beneficial, using its stuff can make the code actually
better readable.  But everybody should, as always, just select the
simplest tools when coding.  Please let's avoid a discussion about what
different opinions about readability and simplicity we have.

But of course: when we continue to migrate the most useful stuff in
cl-lib out to other libraries (like we did with generic functions,
generalized variables, sequence functions, ...), answering this question
might become harder in the future.

Michael.

Re: What's missing in ELisp that makes people want to use cl-lib?

[T.V Raman]

Michael Heerdegen <michael_heerdegen@web.de> writes:


One more fan of cl, and cl-libs is a good solution and a nice compromise
 between having the expressiveness of cl, without loading all of cl at
 runtime.

Creating and migrating to cl-libs was a large effort:

A. For the creators of cl-libs who carefully did the refactoring,
   and for
   B: Package authors who paid attention to the "cl used at runtime "
   warning to properly namespace-qualify cl calls,  and remove  (require
   'cl) 


So I dont quite understand the apparent desire to negate all of that work?
   
>
>> I like cl-lib (it could be optimized better still, but it's good for
>> certain uses), and this discussion doesn't look like it will reach any
>> constructive conclusion, so it didn't seem worth it to spend time on
>> it.
>
> I also find it a bit exaggerating: cl-lib is large, but the majority of
> its uses in Emacs are from a small subset.  In cases where it's
> objectively beneficial, using its stuff can make the code actually
> better readable.  But everybody should, as always, just select the
> simplest tools when coding.  Please let's avoid a discussion about what
> different opinions about readability and simplicity we have.
>
> But of course: when we continue to migrate the most useful stuff in
> cl-lib out to other libraries (like we did with generic functions,
> generalized variables, sequence functions, ...), answering this question
> might become harder in the future.
>
> Michael.
>
>

-- 

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Alan Mackenzie <acm@muc.de> writes:

> That is the current reality.

Maybe you could elaborate on what the plan then could look like?
Or is it not about a plan, but something else? 

Re: What's missing in ELisp that makes people want to use cl-lib?

[Alan Mackenzie]

Hello, João

On Wed, Nov 08, 2023 at 21:10:00 +0000, João Távora wrote:
>    On Wed, Nov 8, 2023, 20:20 Dmitry Gutov <[1]dmitry@gutov.dev> wrote:

>      On 08/11/2023 19:18, Alan Mackenzie wrote:
>>      > Throughout this long discussion, this indiscriminate use of
>>      > cl-lib has been supported only by occasional contributers.Â
>>      > Those who actually maintain other people's code, apart from (I
>>      > think) Eli, Richard and myself, have been conspicuously
>>      > silent.

>>      I like cl-lib (it could be optimized better still, but it's good
>>      for certain uses), and this discussion doesn't look like it will
>>      reach any constructive conclusion, so it didn't seem worth it to
>>      spend time on it.

>    If it needed any confirmation, I too like cl-lib and I too help
>    maintain other people's code in the Emacs core tree as well as
>    maintaining a number of libraries I have authored.

There's a difference between liking cl-lib and advocating its
indiscriminate use.  I don't think you've done the latter in this (and
related) threads.

Nobody who likes cl-lib has yet addressed the point made by Richard and
(less eloquently) by me, namely that the incorporation and use of cl-lib
swells the size and complexity of Emacs Lisp to the point of making
maintenance difficult.  What is your view on this important point?

>    I read many styles of Elisp here and am not really phased by any
>    particular programming technique being discussed here (cl-lib, seq,
>    pcase, verbose while loops with catch/throw, etc etc).  João

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: What's missing in ELisp that makes people want to use cl-lib?

[Alan Mackenzie]

On Thu, Nov 09, 2023 at 08:24:14 +0100, Gerd Möllmann wrote:
> Alan Mackenzie <acm@muc.de> writes:

>>Just take one look at this "reality" you're so supportive of: the
>>widespread use of cl-lib, not just in people's own projects, but
>>throughout the core of Emacs, has multiplied the size of Lisp language
>>part of Emacs by a factor of around 3.   This is a gross increase in
>>complexity for maintainers that is not justified by the slight increase
>>in facility that cl-lib (along with things like seq.el and oclosures)
>>gives.

>>Throughout this long discussion, this indiscriminate use of cl-lib has
>>been supported only by occasional contributers.  Those who actually
>>maintain other people's code, apart from (I think) Eli, Richard and
>>myself, have been conspicuously silent.  None of us three have favoured
>>such use of cl-lib.

>>Occasional contributors may be fascinated by cl-lib, and learn enough
>>of it to use random bits of it in their code.  The trouble is, each
>>such contributor uses a different piece of cl-lib, with the result that
>>those who end up maintaining it need to know a far greater part of it
>>just to cope.

>>This factor of 3 is, I believe, a significant barrier to new
>>programmers coming into Emacs; Elisp is just that much more difficult
>>than it was in the past.  And it isn't just for newcomers that it is
>>more difficult.  I spend a significant amount of debugging time having
>>to look up doc strings and manual pages for obscure cl-lib (etc.)
>>functions.

> > That is the current reality.

> Maybe you could elaborate on what the plan then could look like?
> Or is it not about a plan, but something else? 

You stripped all the context.  I've put it back again.

As a concrete plan, I would propose the following for discussion:

We should deprecate those functions/macros/variables in cl-lib that have
no doc string, or a substandard one.  This includes "internal" functions,
too.  Also to be deprecated are obscure functions/m/v (such as
cl-labels).

Having done this, we recode code currently using those deprecated f/m/v.

(Here a "substandard" doc string is contrasted with an adequate one,
which does all of the following:
(i) It says what the function/macro _does_, or what the variable _is_.
(ii) It describes the form and meaning of each parameter, and its
  relationship to (i).
(iii) If the return value is significant, it describes this.
(iv) It describes all effects on the global state, such as where it
  writes results to, and suchlike.)

This would reduce the size of cl-lib to something more manageable than
what we have at the moment.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Thu, Nov 9, 2023 at 10:05 AM Alan Mackenzie <acm@muc.de> wrote:

> >    If it needed any confirmation, I too like cl-lib and I too help
> >    maintain other people's code in the Emacs core tree as well as
> >    maintaining a number of libraries I have authored.
>
> There's a difference between liking cl-lib and advocating its
> indiscriminate use.  I don't think you've done the latter in this (and
> related) threads.

Yes, you're right.  Indeed I don't' advocate for its indiscriminate use,
just as I don't advocate for indiscriminate use of anything, except
perhaps drinking water and brushing teeth.

> Nobody who likes cl-lib has yet addressed the point made by Richard and
> (less eloquently) by me, namely that the incorporation and use of cl-lib
> swells the size and complexity of Emacs Lisp to the point of making
> maintenance difficult.  What is your view on this important point?

That it doesn't make maintenance any more difficult than any other
Elisp construct, be it very old and curiously named like 'rplacd' or
much, much newer like `seq-do` or `pcase-lambda`.

My specific view on cl-lib.el is that it brings us a small part of
the results of  non-trivial design work put in when important figures
in the Lisp world met  regularly for many years to deliver what has
proved to become excellent,  battle-tested, widely understood and
impeccably documented programming abstractions.

What I'm reading so far in this long discussion is that the argument
of its detractors isn't really that cl-lib isn't good, but that
it is superfluous and that learning it is a burden on maintainers.
Well, it's just as superfluous as all of Elisp apart from two handfuls
of primitives, I guess.  Or any programming language for that matter, if
you know enough machine code.  Or any other programming abstraction I
happen not to be familiar with.

Also I seem to hear that Elisp has some kind of ideal hard-to-define
identity or fingerprint and that it shouldn't try to become anything
else.  But this argument is very strange given Elisp is, like any
decent language, easy to extend.  Not to mention I struggle to see
the technical advantage in uniqueness for uniqueness sake.  A good
part of Elisp is about a special purpose function: editing buffers,
and an the equally important part is about doing things with bits in
memory, there's no reason to advocate for singularity here.

I also hear Elisp shouldn't become Common Lisp, but not only is the
use of cl-lib.el nowhere a step in that direction, but also -- somewhat
ironically -- if Elisp _were_ Common Lisp, then that hard-to-define
identity would be much easier to define and language extension would be
much easier to organize into compartments to facilitate policy-making.

Again, the only thing that has brought Elisp any closer to Common
Lisp significantly, was the introduction of lexical binding some 10
years ago.  Elisp looks a lot different now than it did in the 90's.
Closures everywhere, higher-order functions!  Shudder!

There's even talk of a continuation-passing style library, to
be called future.el or promise.el or something.  Oh dear, what
will be of us that we will have to evolve like any other language
in the world!

So I propose we let programmers use their judgement.  Really
respect people who write code for no money and give the copyright
away to the FSF.  Maybe suggest guidelines such as not introduce
cl-loop where a dolist would do the job just as economically and
elegantly. Don't use higher-order functions just because it looks
cool. But maybe do suggest to use cl-position with a :key and a
:test instead of a complex while loop with an auxiliary variable.
Or cl-set-difference instead of nested loops.  Suggest to express
intent, use abstractions.   Suggest to be consistent, be scrutinous,
be "discriminate", be mindful of the style of the current area
you are working on.

But don't suggest anything too hard, especially if it's not
modifying code you have authored.  Don't use arguments of authority
when you can point to specific things.  Be generally respectful of
people putting in any good work even if you don't like the style,
and try to learn a new thing or two every now and then.

BTW here are some nice generic suggestions from the Lisp world,
written by two fenomenal programmers.  I love reading this from
time to time:

   https://www.cs.umd.edu/~nau/cmsc421/norvig-lisp-style.pdf

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Thu, 9 Nov 2023 11:06:02 +0000
> Cc: Dmitry Gutov <dmitry@gutov.dev>,
>  Björn Bidar <bjorn.bidar@thaodan.de>, 
>  emacs-devel <emacs-devel@gnu.org>
> 
> So I propose we let programmers use their judgement.

That's fair enough, provided that those same programmers step up to
become the Emacs maintainers, and thus share the burden to at least
some extent.  Especially those of them who consider witty sarcasm a
useful style in a discussion like this one.

The following is left as an exercise for the interested readers: take
a non-trivial piece of code that you never in your life have laid your
eyes upon, and decide what to do with a serious bug report about that
code, with or without a patch proposed by someone.  A recent case in
point: bug#66970.  Bonus points for doing this with code in a
programming language you don't command 110%, like C if you are mainly
a Lisp programmer or vice versa.

Re: Lisp files that load cl-lib in problematical ways

[Andrea Corallo]

Andrea Corallo <acorallo@gnu.org> writes:

> Andrea Corallo <acorallo@gnu.org> writes:
>
>> Andrea Corallo <acorallo@gnu.org> writes:
>>
>>> Eli Zaretskii <eliz@gnu.org> writes:
>>>
>>>>> From: Andrea Corallo <acorallo@gnu.org>
>>>>> Cc: Stephen Berman <stephen.berman@gmx.net>,  incal@dataswamp.org,
>>>>>   emacs-devel@gnu.org
>>>>> Date: Thu, 19 Oct 2023 09:44:51 -0400
>>>>> 
>>>>> Okay, I confirm that comp.el loads cl-lib, so any jit compilation
>>>>> triggered loads that.
>>>>
>>>> This is OK, not a problem.
>>>>
>>>>> OTOH one thing we could do, if that's important, is to split the code
>>>>> that only drives the async compilation (that actually happens in a
>>>>> subprocess) so we don't load cl-lib in the Emacs the user is actually
>>>>> using.  This should not be that hard (optimism mode on).
>>>>
>>>> I don't see the need, but I will not object if you want to make such
>>>> changes.
>>>
>>> Okay I'll think about it, might be a positive change even leaving aside
>>> the cl-lib discussion.
>>
>> Okay in scratch/comp-run we have a branch that does not require to load
>> comp.el for jit compilations and for installing already existing
>> trampolines (compiling a new trampolines indeed require to load the
>> compiler).
>>
>> It achieves that adding comp-run.el for runtime dependencies.
>>
>> Still the compiler is loaded for C-h f, to solve that I think we'd need
>> to add a third file (ex comp-common.el), not sure it's worth ATM.
>
> I think what is now in scratch/comp-run does what we want.  If there are
> no objections I'll push it to master after some polishing in the
> following days.

Okay I've installed the changes.

To summarize now comp.el is loaded only when a trampoline need to be
compiled (there's no other way around for sync compilation to happen
indeed).  In all other cases (jit compilation, installation of exinting
trampoline, C-h f) comp.el is *not* loaded in the main Emacs process.

As a consequence of this cl-lib will not be loaded as well, the only
caveat is when a warning has to be reported (form the jit compilation
for instance) cl-lib will be loaded as it's an indirect dependecy of
warnings.el.

Best Regards

  Andrea

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 09/11/2023 12:34, Alan Mackenzie wrote:
> As a concrete plan, I would propose the following for discussion:
> 
> We should deprecate those functions/macros/variables in cl-lib that have
> no doc string, or a substandard one.  This includes "internal" functions,
> too.  Also to be deprecated are obscure functions/m/v (such as
> cl-labels).

I'm not sure we can remove cl-labels -- it's useful enough for the cases 
where it does help, and there are callers inside and outside of Emacs.

Whatever other removals, would probably need to be discussed case-by-case.

> Having done this, we recode code currently using those deprecated f/m/v.
> 
> (Here a "substandard" doc string is contrasted with an adequate one,
> which does all of the following:
> (i) It says what the function/macro_does_, or what the variable_is_.
> (ii) It describes the form and meaning of each parameter, and its
>    relationship to (i).
> (iii) If the return value is significant, it describes this.
> (iv) It describes all effects on the global state, such as where it
>    writes results to, and suchlike.)

Improving cl-lib's documentation would be a welcome effort.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Alan Mackenzie <acm@muc.de> writes:

> On Thu, Nov 09, 2023 at 08:24:14 +0100, Gerd Möllmann wrote:
>> Alan Mackenzie <acm@muc.de> writes:
>
>>>Just take one look at this "reality" you're so supportive of: the
>>>widespread use of cl-lib, not just in people's own projects, but
>>>throughout the core of Emacs, has multiplied the size of Lisp language
>>>part of Emacs by a factor of around 3.   This is a gross increase in
>>>complexity for maintainers that is not justified by the slight increase
>>>in facility that cl-lib (along with things like seq.el and oclosures)
>>>gives.
>
>>>Throughout this long discussion, this indiscriminate use of cl-lib has
>>>been supported only by occasional contributers.  Those who actually
>>>maintain other people's code, apart from (I think) Eli, Richard and
>>>myself, have been conspicuously silent.  None of us three have favoured
>>>such use of cl-lib.
>
>>>Occasional contributors may be fascinated by cl-lib, and learn enough
>>>of it to use random bits of it in their code.  The trouble is, each
>>>such contributor uses a different piece of cl-lib, with the result that
>>>those who end up maintaining it need to know a far greater part of it
>>>just to cope.
>
>>>This factor of 3 is, I believe, a significant barrier to new
>>>programmers coming into Emacs; Elisp is just that much more difficult
>>>than it was in the past.  And it isn't just for newcomers that it is
>>>more difficult.  I spend a significant amount of debugging time having
>>>to look up doc strings and manual pages for obscure cl-lib (etc.)
>>>functions.
>
>> > That is the current reality.
>
>> Maybe you could elaborate on what the plan then could look like?
>> Or is it not about a plan, but something else? 
>
> You stripped all the context.  I've put it back again.
>
> As a concrete plan, I would propose the following for discussion:

Thanks for that. I find it much easier to digest than this clean/unclean
thing.

And wow, that will not be popular :-).

>
> We should deprecate those functions/macros/variables in cl-lib that have
> no doc string, or a substandard one.  This includes "internal" functions,
> too.  Also to be deprecated are obscure functions/m/v (such as
> cl-labels).

Am I right in assuming that this is not about the documentation itself,
but just some selection criterium for reducing the size of cl-lib?

> Having done this, we recode code currently using those deprecated
> f/m/v.

What would recode mean? Using seq/map?

>
> (Here a "substandard" doc string is contrasted with an adequate one,
> which does all of the following:
> (i) It says what the function/macro _does_, or what the variable _is_.
> (ii) It describes the form and meaning of each parameter, and its
>   relationship to (i).
> (iii) If the return value is significant, it describes this.
> (iv) It describes all effects on the global state, such as where it
>   writes results to, and suchlike.)
>
> This would reduce the size of cl-lib to something more manageable than
> what we have at the moment.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Thu, Nov 9, 2023 at 11:38 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Thu, 9 Nov 2023 11:06:02 +0000
> > Cc: Dmitry Gutov <dmitry@gutov.dev>,
> >  Björn Bidar <bjorn.bidar@thaodan.de>,
> >  emacs-devel <emacs-devel@gnu.org>
> >
> > So I propose we let programmers use their judgement.
>
> That's fair enough, provided that those same programmers step up to
> become the Emacs maintainers, and thus share the burden to at least
> some extent.  Especially those of them who consider witty sarcasm a
> useful style in a discussion like this one.

Did you read the rest of the paragraph from which you extracted that
one line?  I'm advocating for being discriminate and

Also, I would hope this is not a suggestion that I don't step to share the
burden of maintaining things?  I do, certainly for all the features that I
have I added to core in the past 10 years and frequently also to parts
of the code those features interact with.  Do you often have to fix bugs in
flymake.el, elec-pair.el, icomplete.el, eldoc.el, jsonrpc.el, etc...
that I didn't care to take a look at in a reasonable time frame?  More
importantly, to what extent are you and others significantly or often
hindered by the style I use there?

I can't address what I don't know about.  I can't track the bug tracker
as closely as you do.  If I'm missing or late about bug reports about
such things, just ping me like you usually do.

Or are you suggesting I should dumb down my style for when I'm dead and
my code survives me?  Just so you know, I think that's not entirely
unreasonable.  In fact if you read the pdf I linked to, it's a valid
criteria for making decisions.   But it's definitely not the only one.
Who knows if Emacs 40's maintainer wouldn't have appreciated it if I
had smartened it up even more and use _only_ cl-lib.

Anyway, if there's some code of mine that's bothering you and you
find inescrutable, point me to it and we can discuss it, of course.

> The following is left as an exercise for the interested readers: take
> a non-trivial piece of code that you never in your life have laid your
> eyes upon, and decide what to do with a serious bug report about that
> code, with or without a patch proposed by someone.  A recent case in
> point: bug#66970.  Bonus points for doing this with code in a
> programming language you don't command 110%, like C if you are mainly
> a Lisp programmer or vice versa.

Do you need help reading some Elisp construct or debugging?
Is this meaningfully related to style in Lisp code?  You want help
reading hangul.el?  (which doesn't seem to use cl-lib.el btw)

Would you like help rewriting hangul-insert-character to support
overwrite-mode, perhaps by leveraging post-self-insert-hook?  I can
help with that.  If there is some deep required specific domain
knowledge about input methods or how the Korean language works, you
or someone else may have to help out with that.  If C code is involved,
it's not a problem.  I'll have a better look later.

The problem of maintaining other people's code (which btw is 95%
of all my professional experience) is not about programming techniques
or even languages at all, it's about domain knowledge.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Thu, Nov 9, 2023 at 11:49 AM Dmitry Gutov <dmitry@gutov.dev> wrote:

> Improving cl-lib's documentation would be a welcome effort.

For sure, and not a hard one as well, as all those functions and
macros are pretty good, often flawless emulations of CL functions that
are impeccably documented in

  http://www.lispworks.com/documentation/HyperSpec/Front/

Which is of free access (though not of a compatible license, I
think).  But if people can point to the 5 most confusing functions
they think are poorly documented, I volunteer to rewrite the
docstrings for them.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Alan Mackenzie]

Hello, Dmitry.

On Thu, Nov 09, 2023 at 13:48:34 +0200, Dmitry Gutov wrote:

> On 09/11/2023 12:34, Alan Mackenzie wrote:
> > As a concrete plan, I would propose the following for discussion:

> > We should deprecate those functions/macros/variables in cl-lib that have
> > no doc string, or a substandard one.  This includes "internal" functions,
> > too.  Also to be deprecated are obscure functions/m/v (such as
> > cl-labels).

> I'm not sure we can remove cl-labels -- it's useful enough for the cases 
> where it does help, and there are callers inside and outside of Emacs.

> Whatever other removals, would probably need to be discussed case-by-case.

There are probably too many such functions/macros/variables to make that
practicable.  If we did do this, the discussion for each f/m/v would
likely balloon out into a thread as long as this one, and nothing much
would get done.

> > Having done this, we recode code currently using those deprecated f/m/v.

> > (Here a "substandard" doc string is contrasted with an adequate one,
> > which does all of the following:
> > (i) It says what the function/macro_does_, or what the variable_is_.
> > (ii) It describes the form and meaning of each parameter, and its
> >    relationship to (i).
> > (iii) If the return value is significant, it describes this.
> > (iv) It describes all effects on the global state, such as where it
> >    writes results to, and suchlike.)

> Improving cl-lib's documentation would be a welcome effort.

Yes, people have been saying this for years.  I wrote a chapter on
cl-print.el for cl.info a couple of months back.  Other than that nobody
seems prepared to do anything much on this front.

What is needed is a sustained effort from a cl-lib enthusiast to fix the
50 - 100 doc strings which are missing or substandard.  Well, João has
offered to start off with 5 doc strings, which is good.  Maybe something
will come of that.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: What's missing in ELisp that makes people want to use cl-lib?

[Alan Mackenzie]

Hello, João.

On Thu, Nov 09, 2023 at 12:40:24 +0000, João Távora wrote:
> On Thu, Nov 9, 2023 at 11:49 AM Dmitry Gutov <dmitry@gutov.dev> wrote:

> > Improving cl-lib's documentation would be a welcome effort.

> For sure, and not a hard one as well, as all those functions and
> macros are pretty good, often flawless emulations of CL functions that
> are impeccably documented in

>   http://www.lispworks.com/documentation/HyperSpec/Front/

> Which is of free access (though not of a compatible license, I
> think).  But if people can point to the 5 most confusing functions
> they think are poorly documented, I volunteer to rewrite the
> docstrings for them.

How much are you prepared to do?  I don't have a list of the _most_
confusing doc strings, there are too many to chose from.  But starting
at the start of cl-macs.el, we have:

(i) cl--compiler-macro-list*; completely undocumented.
(ii) cl--simple-expr-p: Talks about "side effects", but not what they
  are side effects of.  Doesn't describe it's parameters or return
  value.  It's unclear what it is that "executes quickly".
(iii) cl--expr-contains: It's unclear what X and Y are, and what "refers
  to" means.
(iv) cl--expr-contains-any; completely undocumented.
(v) cl--expr-depends-p: It's unclear what X and Y are, though Y appears
  to be some sort of container of symbols.  It's unclear what sort of
  "dependency" the function handles, or what "may" means in the context.

There are many more.

> João

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Thu, Nov 9, 2023 at 12:34 PM João Távora <joaotavora@gmail.com> wrote:

> > The following is left as an exercise for the interested readers: take
> > a non-trivial piece of code that you never in your life have laid your
> > eyes upon, and decide what to do with a serious bug report about that
> > code, with or without a patch proposed by someone.  A recent case in
> > point: bug#66970.  Bonus points for doing this with code in a
> > programming language you don't command 110%, like C if you are mainly
> > a Lisp programmer or vice versa.

>  I'll have a better look later.

At first sight, didn't seem terribly hard.  I gave it some basic testing
seems to work like the OP in that bug would like it to.  Could be a start
I guess.  Let me know if you don't like the style.

diff --git a/lisp/leim/quail/hangul.el b/lisp/leim/quail/hangul.el
index 46a2e5a6ba2..c97362b1250 100644
--- a/lisp/leim/quail/hangul.el
+++ b/lisp/leim/quail/hangul.el
@@ -146,21 +146,26 @@ hangul-insert-character
       (progn
         (delete-region (region-beginning) (region-end))
         (deactivate-mark)))
-  (quail-delete-region)
-  (let ((first (car queues)))
-    (insert
-     (hangul-character
-      (+ (aref first 0) (hangul-djamo 'cho (aref first 0) (aref first 1)))
-      (+ (aref first 2) (hangul-djamo 'jung (aref first 2) (aref first 3)))
-      (+ (aref first 4) (hangul-djamo 'jong (aref first 4) (aref first 5))))))
-  (move-overlay quail-overlay (overlay-start quail-overlay) (point))
-  (dolist (queue (cdr queues))
-    (insert
-     (hangul-character
-      (+ (aref queue 0) (hangul-djamo 'cho (aref queue 0) (aref queue 1)))
-      (+ (aref queue 2) (hangul-djamo 'jung (aref queue 2) (aref queue 3)))
-      (+ (aref queue 4) (hangul-djamo 'jong (aref queue 4) (aref queue 5)))))
-    (move-overlay quail-overlay (1+ (overlay-start quail-overlay)) (point))))
+  (let* ((things-to-insert
+          (with-temp-buffer
+            (dolist (queue queues (mapcar #'identity (buffer-string)))
+              (insert
+               (hangul-character
+                (+ (aref queue 0) (hangul-djamo 'cho (aref queue 0)
(aref queue 1)))
+                (+ (aref queue 2) (hangul-djamo 'jung (aref queue 2)
(aref queue 3)))
+                (+ (aref queue 4) (hangul-djamo 'jong (aref queue 4)
(aref queue 5))))))))
+         (may-have-to-overwrite-p
+          (or (= (overlay-start quail-overlay) (overlay-end quail-overlay))
+              (cdr things-to-insert))))
+    (quail-delete-region)
+    (dolist (c things-to-insert)
+      (let ((last-command-event c)
+            (overwrite-mode (and overwrite-mode
+                                 may-have-to-overwrite-p
+                                 overwrite-mode)))
+        (self-insert-command 1)
+        (setq may-have-to-overwrite-p nil)))
+    (move-overlay quail-overlay (1- (point)) (point))))

 (defun hangul-djamo (jamo char1 char2)
   "Return the double Jamo index calculated from the arguments.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Thu, Nov 9, 2023 at 1:36 PM Alan Mackenzie <acm@muc.de> wrote:
>
> Hello, João.
>
> On Thu, Nov 09, 2023 at 12:40:24 +0000, João Távora wrote:
> > On Thu, Nov 9, 2023 at 11:49 AM Dmitry Gutov <dmitry@gutov.dev> wrote:
>
> > > Improving cl-lib's documentation would be a welcome effort.
>
> > For sure, and not a hard one as well, as all those functions and
> > macros are pretty good, often flawless emulations of CL functions that
> > are impeccably documented in
>
> >   http://www.lispworks.com/documentation/HyperSpec/Front/
>
> > Which is of free access (though not of a compatible license, I
> > think).  But if people can point to the 5 most confusing functions
> > they think are poorly documented, I volunteer to rewrite the
> > docstrings for them.
>
> How much are you prepared to do?  I don't have a list of the _most_
> confusing doc strings, there are too many to chose from.  But starting
> at the start of cl-macs.el, we have:
>
> (i) cl--compiler-macro-list*; completely undocumented.
> (ii) cl--simple-expr-p: Talks about "side effects", but not what they
>   are side effects of.  Doesn't describe it's parameters or return
>   value.  It's unclear what it is that "executes quickly".
> (iii) cl--expr-contains: It's unclear what X and Y are, and what "refers
>   to" means.
> (iv) cl--expr-contains-any; completely undocumented.
> (v) cl--expr-depends-p: It's unclear what X and Y are, though Y appears
>   to be some sort of container of symbols.  It's unclear what sort of
>   "dependency" the function handles, or what "may" means in the context.
>
> There are many more.

These are all internal functions and implementation details.  They're
not necessary at all for users of cl-lib.el, only for its developers.
What problem are you trying to solve by enhancing these docstrings?  I thought
the problem here was code that _used_ cl-lib.el, not hacking on
cl-lib.el itself.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 09/11/2023 15:36, Alan Mackenzie wrote:
> (i) cl--compiler-macro-list*; completely undocumented.
> (ii) cl--simple-expr-p: Talks about "side effects", but not what they
>    are side effects of.  Doesn't describe it's parameters or return
>    value.  It's unclear what it is that "executes quickly".
> (iii) cl--expr-contains: It's unclear what X and Y are, and what "refers
>    to" means.
> (iv) cl--expr-contains-any; completely undocumented.
> (v) cl--expr-depends-p: It's unclear what X and Y are, though Y appears
>    to be some sort of container of symbols.  It's unclear what sort of
>    "dependency" the function handles, or what "may" means in the context.

These aren't public-facing functions, though.

Adding docs wouldn't hurt, but it's not like these could be removed due 
to the absence of them (that should depend on which public callers they 
have).

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> That it doesn't make maintenance any more difficult than any other
> Elisp construct, be it very old and curiously named like 'rplacd' or
> much, much newer like `seq-do` or `pcase-lambda`.
>
> My specific view on cl-lib.el is that it brings us a small part of
> the results of  non-trivial design work put in when important figures
> in the Lisp world met  regularly for many years to deliver what has
> proved to become excellent,  battle-tested, widely understood and
> impeccably documented programming abstractions.
>
> What I'm reading so far in this long discussion is that the argument
> of its detractors isn't really that cl-lib isn't good, but that
> it is superfluous and that learning it is a burden on maintainers.
> Well, it's just as superfluous as all of Elisp apart from two handfuls
> of primitives, I guess.  Or any programming language for that matter, if
> you know enough machine code.  Or any other programming abstraction I
> happen not to be familiar with.
>
> Also I seem to hear that Elisp has some kind of ideal hard-to-define
> identity or fingerprint and that it shouldn't try to become anything
> else.  But this argument is very strange given Elisp is, like any
> decent language, easy to extend.  Not to mention I struggle to see
> the technical advantage in uniqueness for uniqueness sake.  A good
> part of Elisp is about a special purpose function: editing buffers,
> and an the equally important part is about doing things with bits in
> memory, there's no reason to advocate for singularity here.
>
> I also hear Elisp shouldn't become Common Lisp, but not only is the
> use of cl-lib.el nowhere a step in that direction, but also -- somewhat
> ironically -- if Elisp _were_ Common Lisp, then that hard-to-define
> identity would be much easier to define and language extension would be
> much easier to organize into compartments to facilitate policy-making.
>
> Again, the only thing that has brought Elisp any closer to Common
> Lisp significantly, was the introduction of lexical binding some 10
> years ago.  Elisp looks a lot different now than it did in the 90's.
> Closures everywhere, higher-order functions!  Shudder!
>
> There's even talk of a continuation-passing style library, to
> be called future.el or promise.el or something.  Oh dear, what
> will be of us that we will have to evolve like any other language
> in the world!
>
> So I propose we let programmers use their judgement.  Really
> respect people who write code for no money and give the copyright
> away to the FSF.  Maybe suggest guidelines such as not introduce
> cl-loop where a dolist would do the job just as economically and
> elegantly. Don't use higher-order functions just because it looks
> cool. But maybe do suggest to use cl-position with a :key and a
> :test instead of a complex while loop with an auxiliary variable.
> Or cl-set-difference instead of nested loops.  Suggest to express
> intent, use abstractions.   Suggest to be consistent, be scrutinous,
> be "discriminate", be mindful of the style of the current area
> you are working on.

Such casuistry and non-sequiturs are the reasons I'm glad I have thus
far given this conversation a generously wide berth.  But one's
restraint has its limits, and I suppose mine have been overstepped,
since I can't keep myself from delving head-first into this madhouse.
Here goes...

> Well, it's just as superfluous as all of Elisp apart from two handfuls
> of primitives, I guess.  Or any programming language for that matter, if
> you know enough machine code.  Or any other programming abstraction I
> happen not to be familiar with.

cl-lib does certainly provide some value for money, by way of constructs
that lend themselves to many general situations, but it does not provide
this value in the lion's share of the circumstances these constructs are
used in.  Consider the blithe misapplication of CL generics throughout
the window system code:

(cl-defmethod gui-backend-set-selection (type value
                                              &context (window-system android))
  ;; First, try to turn value into a string.
  ;; Don't set anything if that did not work.
  (when-let ((string (android-encode-select-string value)))
    (cond ((eq type 'CLIPBOARD)
           (android-set-clipboard string))
          ((eq type 'PRIMARY)
           (setq android-primary-selection string))
          ((eq type 'SECONDARY)
           (setq android-secondary-selection string)))))

do we really expect window-system to be anything _but_ android, so long
as a window system has been initialized, when android-win.el is loaded?
Why was it necessary to convolute the code when g-b-s-s (and its ilk)
could as easily have been defined as an ordinary function in each of the
window system files?

For a second example, how about cl-list*?  Is there really a purpose
for this function, in light of how it can always be replaced by:

  (nconc (list a b) c)

and demands callers load cl-lib.el?  The call to this function in
desktop.el is particularly dubitable: as far as the eye can see, it is
the singular reason that file requires cl-lib, its remainder being
implemented with the cut and dry Emacs Lisp builtins that are well known
to us all.  This abject misuse of cl-lib extensions, not an unqualified
opposition to extending Emacs Lisp, is the catalyst of most misgivings
towards it.

Like begets like, even imitations of like: as the corpus of files which
load cl-lib for legitimate purposes burgeons, so does that of files
which load it for positively frivolous reasons.  Thus usage of cl-lib
should be discouraged in general, and users firmly exhorted to seek
replacements.  Even if there are situations where it is not superfluous,
their existence doesn't outweigh the proliferation of that where it is,
and ultimately it stands us in better stead to treat it as such in
general.

> That it doesn't make maintenance any more difficult than any other
> Elisp construct, be it very old and curiously named like 'rplacd' or
> much, much newer like `seq-do` or `pcase-lambda`.

pcase-lambda and seq-do (plus the boatload of seq and pcase functions)
are not "other Elisp constructs," not in my book.  seq functions are all
generic functions, and their documentation is nothing short of
inscrutable; take the example of seq-do's:

Apply FUNCTION to each element of SEQUENCE.
Presumably, FUNCTION has useful side effects.
Return SEQUENCE.

What is a useful side effect?  Does this imply FUNCTION can modify
SEQUENCE as it is being iterated through?

and each time I read the documentation for pcase, I wind up bewildered
by the needlessly laconic style it is written in.  Most of its users
don't display a persuasive need for it either, a demographic of which
frameset.el is representative: much of its work is also replicated in
the *fns.c files and frame.c, which seem to fare equally well, despite
being authored in a language that largely enjoys an absence of byzantine
constructs.

If doc strings are meant to be incomprehensible without recourse to the
Lisp reference manual, then I suggest they not be written at all.  A doc
string that alludes to and accentuates concepts such as "useful side
effects" without providing the reader with a definition or a reference
as to where he can obtain such a definition is not edifying; it leaves
an unpleasant vacuum in the reader's mind that rankles until he bows to
fate and reads the manual anyway.

rplacd and rplaca are aliases to setcar and setcdr; Emacs Lisp code
ought to use the latter, and aliases and the wholesale introduction of
functions from other languages are plainly incommensurable.

> My specific view on cl-lib.el is that it brings us a small part of
> the results of  non-trivial design work put in when important figures
> in the Lisp world met  regularly for many years to deliver what has
> proved to become excellent,  battle-tested, widely understood and
> impeccably documented programming abstractions.

It's a fair bet that Common Lisp users today number fewer than Emacs
users; of course this does not immediately reflect on the character of
features present in Common Lisp, but it is a consideration to bear in
mind.

The quality of the documentation, which for cl-lib is far from
irreproachable, cannot allay the challenge posed by learning to read and
write a language that is in plenty of respects at variance with the
language Emacs users and developers have learned.  When the latter is in
fact documented beyond reproach (save for rough spots such as the said
pcase), more than adequate for programming in Emacs, and understood by
all Emacs users, the profundity of the deliberations underlying the
design of Common Lisp and the eminence of its designers are not
justifications for its pervading Emacs code, let alone to the degree
that it has by now.

> Also I seem to hear that Elisp has some kind of ideal hard-to-define
> identity or fingerprint and that it shouldn't try to become anything
> else.  But this argument is very strange given Elisp is, like any
> decent language, easy to extend.  Not to mention I struggle to see
> the technical advantage in uniqueness for uniqueness sake.  A good
> part of Elisp is about a special purpose function: editing buffers,
> and an the equally important part is about doing things with bits in
> memory, there's no reason to advocate for singularity here.

The relative ease of extension implies that we should be more
circumspect in judging which extensions to allow--that is the basis of
much of the argumentation here, not the unsupportable notion that there
is an identity to Emacs Lisp which must be preserved at all costs.

> I also hear Elisp shouldn't become Common Lisp, but not only is the
> use of cl-lib.el nowhere a step in that direction, but also -- somewhat
> ironically -- if Elisp _were_ Common Lisp, then that hard-to-define
> identity would be much easier to define and language extension would be
> much easier to organize into compartments to facilitate policy-making.

This is self-evidently wooden language.  I expect the likes of
"compartmentalizing a program for policy-making" from suits, not from
free software developers.

> Again, the only thing that has brought Elisp any closer to Common
> Lisp significantly, was the introduction of lexical binding some 10
> years ago.  Elisp looks a lot different now than it did in the 90's.

[...]

> Closures everywhere, higher-order functions!  Shudder!

This is not ipso facto a net positive and is also beside the point.

> There's even talk of a continuation-passing style library, to
> be called future.el or promise.el or something.  Oh dear, what
> will be of us that we will have to evolve like any other language
> in the world!

I certainly hope never to see the day that becomes widespread in Emacs
code.

> So I propose we let programmers use their judgement.  Really
> respect people who write code for no money and give the copyright
> away to the FSF.  Maybe suggest guidelines such as not introduce
> cl-loop where a dolist would do the job just as economically and
> elegantly. Don't use higher-order functions just because it looks
> cool. But maybe do suggest to use cl-position with a :key and a
> :test instead of a complex while loop with an auxiliary variable.
> Or cl-set-difference instead of nested loops.  Suggest to express
> intent, use abstractions.   Suggest to be consistent, be scrutinous,
> be "discriminate", be mindful of the style of the current area
> you are working on.

Respect for volunteers is no doubt laudable, but it is very much an
position to be considered alone which everything you've presented above
does not impact and nobody takes exception to.  From it, and even
supposing for the sake of argument that everything else you've said is
incontrovertible fact, it doesn't really follow that we should not seek
to reduce the incidence of cl-lib usage in core code.  Now I can't be
certain that is the object of your militating, if you will, though
promoting cl-set-difference and cl-position as alternatives to hallowed
Emacs Lisp constructs does suggest so, but the principle argument
against these constructs is that they are alien to the small group of
individuals charged with trudging through code where people want to make
use of them.  So no contentions resting on their virtues and
distinctions rather than their prevalence among us can hold water...

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

João Távora wrote:

> These are all internal functions and implementation details.
> They're not necessary at all for users of cl-lib.el, only
> for its developers. What problem are you trying to solve by
> enhancing these docstrings? I thought the problem here was
> code that _used_ cl-lib.el, not hacking on cl-lib.el itself.

It is still nuch better than removing or rewriting it, so
let's do that.

We can use checkdoc to find out what docstrings are not
submitted or not correctly formated. Those should be pretty
quick fixes, to get away with those warnings.

After we have done that, if someone says some specific
docstring is still lacking we can focus on such individual
cases in more detail.

Here, to get the list from, e.g., cl-lib.el

(defun check-package-style ()
  (interactive)
  (let ((msg "Style check..."))
    (message msg)
    (checkdoc-current-buffer t)
    (message "%sdone" msg) ))

And the list is

cl-lib.el:133: Argument ‘keys’ should appear (as KEYS) in the doc string
cl-lib.el:160: All variables and subroutines might as well have a documentation string
cl-lib.el:166: All variables and subroutines might as well have a documentation string
cl-lib.el:197: All variables and subroutines might as well have a documentation string
cl-lib.el:197: All variables and subroutines might as well have a documentation string
cl-lib.el:226: Arguments occur in the doc string out of order
cl-lib.el:258: Argument ‘specs’ should appear (as SPECS) in the doc string
cl-lib.el:357: Argument ‘cl-func’ should appear (as CL-FUNC) in the doc string
cl-lib.el:450: Argument ‘rest’ should appear (as REST) in the doc string
cl-lib.el:488: Argument ‘cl-item’ should appear (as CL-ITEM) in the doc string
cl-lib.el:501: Argument ‘cl-new’ should appear (as CL-NEW) in the doc string
cl-lib.el:510: All variables and subroutines might as well have a documentation string
cl-lib.el:544: All variables and subroutines might as well have a documentation string
cl-lib.el:582: Probably "returns" should be imperative "return"

Here are all the cl-lib files

cl-extra.el
cl-generic.el
cl-indent.el
cl-lib.el
cl-loaddefs.el
cl-macs.el
cl-preloaded.el
cl-print.el
cl-seq.el

GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, cairo version 1.16.0)
of 2023-11-01 [commit
8eb9b82ce58a8323af21f52625a401e19e279a9c]

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Thu, Nov 9, 2023 at 1:46 PM Po Lu <luangruo@yahoo.com> wrote:
>
> João Távora <joaotavora@gmail.com> writes:

> This is self-evidently wooden language.

And all the time I thought I was being cheerful.

> I expect the likes of
> "compartmentalizing a program for policy-making" from suits, not from
> free software developers.

Are _you_ the suit?  I missed the point of most of your very
elegant text, but isn't policy-making exactly what you (and I) are
trying to do, i.e. discourage some things and encourage others?

What's the gripe with compartments?  We don't have all of Elisp in
a single library?  I was talking about namespaces by the way,
if you didn't catch the reference (admittedly obscure)

Also, is really cl-set-difference to take the difference
of two sets worse than writing a "hallowed" multi-line
construct??  Maybe I'm not seeing what the construct is, so
what is your preferred Elisp hallowed way to take the
difference of two sets, say make a list  that contains
all items that appear in LIST1 but not LIST2?  Or to find
the index of a certain element of a sequence...

These comparisons, taken one by one, are the hard evidence of that
allows us to determine what is "alien" or not, not theological
consecrations of personal tastes of a given group, _any_ group.
For example, I don't think your 'nconc'  form is more readable
than 'cl-list*'.  It's slightly less readable IMO.  We could alias
`list*` to nconc though, even better.  But I can agree it's not
worth requiring cl-lib.el just for cl-list* alone.

Much like you prefer setcar to rplacd. Which I also do, btw.  But
I prefer (setf (car ...))  even more.  But you'll frequently see
me using setcar and setcdr (or even rplacwahtevercannevertypeit)
if that's the style of the file I'm working on.  Because I too
have my personal tastes, but contrary to others I can let go of
them without too much fuss.

Stefan M's pcase in particular is great.  Its docstring is tough,
I agree, and could use some examples (I always end up grepping
for examples).  But that macro is a lifesaver.

The way you talk about seq.el and pcase.el makes me think you
want to ban _all the things_, not just cl-lib.  I'm confused about
what you want Elisp code in core to look like.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Thu, Nov 9, 2023 at 2:28 PM João Távora <joaotavora@gmail.com> wrote:

> than 'cl-list*'.  It's slightly less readable IMO.  We could alias
> `list*` to nconc though, even better.

oops, no we couldn't.  Same job, different calling convention.
I just don't use either that much, so here again my suggestion
is to just use what fits better with the style of surrounding code.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Alan Mackenzie]

Hello, Gerd.

On Thu, Nov 09, 2023 at 13:19:28 +0100, Gerd Möllmann wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > On Thu, Nov 09, 2023 at 08:24:14 +0100, Gerd Möllmann wrote:
> >> Alan Mackenzie <acm@muc.de> writes:

> >>>Just take one look at this "reality" you're so supportive of: the
> >>>widespread use of cl-lib, not just in people's own projects, but
> >>>throughout the core of Emacs, has multiplied the size of Lisp language
> >>>part of Emacs by a factor of around 3.   This is a gross increase in
> >>>complexity for maintainers that is not justified by the slight increase
> >>>in facility that cl-lib (along with things like seq.el and oclosures)
> >>>gives.

> >>>Throughout this long discussion, this indiscriminate use of cl-lib has
> >>>been supported only by occasional contributers.  Those who actually
> >>>maintain other people's code, apart from (I think) Eli, Richard and
> >>>myself, have been conspicuously silent.  None of us three have favoured
> >>>such use of cl-lib.

> >>>Occasional contributors may be fascinated by cl-lib, and learn enough
> >>>of it to use random bits of it in their code.  The trouble is, each
> >>>such contributor uses a different piece of cl-lib, with the result that
> >>>those who end up maintaining it need to know a far greater part of it
> >>>just to cope.

> >>>This factor of 3 is, I believe, a significant barrier to new
> >>>programmers coming into Emacs; Elisp is just that much more difficult
> >>>than it was in the past.  And it isn't just for newcomers that it is
> >>>more difficult.  I spend a significant amount of debugging time having
> >>>to look up doc strings and manual pages for obscure cl-lib (etc.)
> >>>functions.

> >> > That is the current reality.

> >> Maybe you could elaborate on what the plan then could look like?
> >> Or is it not about a plan, but something else? 

> > You stripped all the context.  I've put it back again.

> > As a concrete plan, I would propose the following for discussion:

> Thanks for that. I find it much easier to digest than this clean/unclean
> thing.

> And wow, that will not be popular :-).

It will be a mixed popular/unpopular, with probably few people in the
middle.  Back when you were the Maintainer, I think there was a ban on
the use of cl.el and friends, except for at compilation time.  I don't
think that caused any problems.

> > We should deprecate those functions/macros/variables in cl-lib that have
> > no doc string, or a substandard one.  This includes "internal" functions,
> > too.  Also to be deprecated are obscure functions/m/v (such as
> > cl-labels).

> Am I right in assuming that this is not about the documentation itself,
> but just some selection criterium for reducing the size of cl-lib?

The most troublesome cl-lib functions are those without adequate
documentation.  They're the ones that waste unbounded amounts of time
when trying to debug something which uses them.  That's why I'd like to
do away with them.  There are few people willing to fix the doc strings
in cl-lib, though João has volunteered to make a start.

> > Having done this, we recode code currently using those deprecated
> > f/m/v.

> What would recode mean? Using seq/map?

I hadn't really thought of seq.el or map.el.  What do they do?  But it's
clear that working code can be written without these or cl-lib.

> > (Here a "substandard" doc string is contrasted with an adequate one,
> > which does all of the following:
> > (i) It says what the function/macro _does_, or what the variable _is_.
> > (ii) It describes the form and meaning of each parameter, and its
> >   relationship to (i).
> > (iii) If the return value is significant, it describes this.
> > (iv) It describes all effects on the global state, such as where it
> >   writes results to, and suchlike.)

> > This would reduce the size of cl-lib to something more manageable than
> > what we have at the moment.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Thu, 9 Nov 2023 12:34:09 +0000
> Cc: acm@muc.de, dmitry@gutov.dev, bjorn.bidar@thaodan.de, emacs-devel@gnu.org
> 
> On Thu, Nov 9, 2023 at 11:38 AM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > > From: João Távora <joaotavora@gmail.com>
> > > Date: Thu, 9 Nov 2023 11:06:02 +0000
> > > Cc: Dmitry Gutov <dmitry@gutov.dev>,
> > >  Björn Bidar <bjorn.bidar@thaodan.de>,
> > >  emacs-devel <emacs-devel@gnu.org>
> > >
> > > So I propose we let programmers use their judgement.
> >
> > That's fair enough, provided that those same programmers step up to
> > become the Emacs maintainers, and thus share the burden to at least
> > some extent.  Especially those of them who consider witty sarcasm a
> > useful style in a discussion like this one.
> 
> Did you read the rest of the paragraph from which you extracted that
> one line?

You should by now know me well enough to realize that I read
everything here.  However, I respond only to the parts where I have
something useful and non-trivial to say.

> Also, I would hope this is not a suggestion that I don't step to share the
> burden of maintaining things?  I do, certainly for all the features that I
> have I added to core in the past 10 years and frequently also to parts
> of the code those features interact with.  Do you often have to fix bugs in
> flymake.el, elec-pair.el, icomplete.el, eldoc.el, jsonrpc.el, etc...
> that I didn't care to take a look at in a reasonable time frame?  More
> importantly, to what extent are you and others significantly or often
> hindered by the style I use there?

I said nothing about your style.  Neither do I doubt that you know
your own code very well.  We all know our own code well.  I'm asking
you not to judge the burden of the maintainers in maintaining code
they didn't write nor even see, until you get to our place and do it
for a while.  Only then you will be able to realize the role of
obscure programming style and use of unfamiliar syntax in making the
job harder than it has to be.

> The problem of maintaining other people's code (which btw is 95%
> of all my professional experience) is not about programming techniques
> or even languages at all, it's about domain knowledge.

It is both.  In particular, when you need to understand why some code
fragment misbehaves, you need to understand the implementation and
what it intends to do, not only why.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Alan Mackenzie wrote:

> Just take one look at this "reality" you're so supportive
> of: the widespread use of cl-lib, not just in people's own
> projects, but throughout the core of Emacs, has multiplied
> the size of Lisp language part of Emacs by a factor of
> around 3.

So say I write a function like this, which has nothing to do
with cl-lib BTW.

(defun nand (&rest conditions)
  (when (member nil conditions)
    t) )

Some would say it is consistent, since we have `not' and
`and', so why not NAND as well?

Others would say it is cleaner in source than (not (and ...))
- especially if you find use of that recurring, since it is
a single function, not a combination of two.

Let's then say I put it in a package that makes its way into
Emacs, and some people start to use it since they also find it
consistent, clean, and useful.

What has then happened?

Have I introduced something that is clean, consistent,
and useful?

Or have I increased the size of the Lisp language by 1, making
it harder for the maintainers to do their work?

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Alan Mackenzie wrote:

> The most troublesome cl-lib functions are those without
> adequate documentation. They're the ones that waste
> unbounded amounts of time when trying to debug something
> which uses them. That's why I'd like to do away with them.

Why not instead provide them with docstrings, I think everyone
can agree they should have them?

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

João Távora wrote:

> But I can agree it's not worth requiring cl-lib.el just for
> cl-list* alone.

Some people will say one should write as good code as
possible, and `require' whatever necessary for that.

I can agree bringing in external packages and such should be
avoided if one can alter just some detail so to not have to do
that, but cl-lib.el is not external in any way. In this
directory, where the cl-lib files are

  /usr/local/share/emacs/30.0.50/lisp/emacs-lisp

are 102 .el files.

Are we gonna have individual rules for 102 files? What files
are to be used and what aren't?

I think there should be only one rule, write as good code you
can and 'require' any file you need to achieve this.
These files are there because they are useful and there is no
harm interconnecting them.

-- 
underground experts united
https://dataswamp.org/~incal

Re: What's missing in ELisp that makes people want to use cl-lib?

[Emanuel Berg]

Eli Zaretskii wrote:

> We all know our own code well. I'm asking you not to judge
> the burden of the maintainers in maintaining code they
> didn't write nor even see [...]

We should distribute the maintenance effort so that to
a higher degree, people who wrote the code are maintaining it,
and, whenever that cannot be achieved, at least people who
take on "I'll maintain this package", it will then be
understood, for example, if the file contains a lot of cl-lib,
the maintainer who takes on that file will have to deal
with it, because it comes with the task.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Andrea Corallo <acorallo@gnu.org>
> Cc: stephen.berman@gmx.net,  incal@dataswamp.org,  emacs-devel@gnu.org
> Date: Thu, 09 Nov 2023 06:41:03 -0500
> 
> > I think what is now in scratch/comp-run does what we want.  If there are
> > no objections I'll push it to master after some polishing in the
> > following days.
> 
> Okay I've installed the changes.

Thanks, but please in the future allow a bit longer than just 17 hours
for people to review the changes and post comments.

I see some warnings when compiling the new files in a build without
native-compilation:

  In comp-eln-load-path-eff:
  emacs-lisp/comp-common.el:519:31: Warning: reference to free variable `comp-native-version-dir'
  emacs-lisp/comp-common.el:522:11: Warning: reference to free variable `native-comp-eln-load-path'

  In native-compile-async-skip-p:
  emacs-lisp/comp-run.el:124:54: Warning: reference to free variable `comp--no-native-compile'
  In comp-run-async-workers:
  emacs-lisp/comp-run.el:309:14: Warning: reference to free variable `comp-deferred-pending-h'
  In comp-subr-trampoline-install:
  emacs-lisp/comp-run.el:336:21: Warning: reference to free variable `native-comp-enable-subr-trampolines'
  emacs-lisp/comp-run.el:338:34: Warning: reference to free variable `comp-installed-trampolines-h'

  In end of data:
  emacs-lisp/comp-run.el:342:8: Warning: the function `comp--install-trampoline' is not known to be defined.
  emacs-lisp/comp-run.el:291:37: Warning: the function `native-elisp-load' is not known to be defined.
  emacs-lisp/comp-run.el:211:35: Warning: the function `comp-el-to-eln-filename' is not known to be defined.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Thu, Nov 9, 2023 at 3:05 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > that I didn't care to take a look at in a reasonable time frame?  More
> > importantly, to what extent are you and others significantly or often
> > hindered by the style I use there?
>
> I said nothing about your style.  Neither do I doubt that you know
> your own code very well.  We all know our own code well.  I'm asking
> you not to judge the burden of the maintainers in maintaining code
> they didn't write nor even see, until you get to our place and do it
> for a while.

Okay, and what I tried to say is that I _do_ have a valid judgement
of exactly this burden, to some quite significant degree, because not only
do I maintain "my" code, I have to wade through a lot of code that
I didn't write.  I didn't write any of icomplete.el and yet managed
to significantly enhance it, for example.  Or elisp-mode.el, or
minibuffer.el.  Or flymake.el, which I rewrote, backward compatibly.
Or eldoc.el, where I did likewise.  Or electric.el or so many other
pieces of code I have to visit and read to make sense of Emacs.
These aren't trivial files they're much larger.  I go
wherever M-. takes me.

So I have to read and understand lots of "alien" code in styles that
wouldn't be my first preference if I had written it from scratch,
often styles that lead to many "blerghs" and "yucks" which I know
we all do.  And sometimes that alien code was written by myself a
longer time ago.

And what I'm trying to say is that all those difficulties are much
much less importnat than shortcomings in domain knowledge.

> Only then you will be able to realize the role of
> obscure programming style and use of unfamiliar syntax in making the
> job harder than it has to be.

I am prepared to admit that that impact is heavier to someone
like you who is the first responder for any and every report to
parts of Emacs you didn't even know existed.  But I don't think
a uniform programming style, if that even exists (let alone
being feasible to enforce) would make your job much easier.

Maybe _somewhat_ easier, but to what loss?  Aggravated contributors,
endless hair-splitting of what exactly good style is.   And style
preferences change anyway: what I "liked" 10 years ago is not what
I like now. It's really a bad idea.

But it's definitely a good idea to review any _new_ contributions to
basic Elisp constructs clearly.  I'm not proposing a free-for-all.

I've said this often without any kind of irony or sarcasm that I think
you're an excellent maintainer.  This is not to "butter you up": I want
you to be happy maintaining Emacs for a long time.  IMHO, if I am allowed,
you should keep doing what you do, which is try to ping the specialists
in each domain, and prompt non-specialists too like you did with
bug#66970, to which I hope to have provided at least the start of
a fix.

João Távora

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Thu, 09 Nov 2023 16:29:45 +0100
> 
> Eli Zaretskii wrote:
> 
> > We all know our own code well. I'm asking you not to judge
> > the burden of the maintainers in maintaining code they
> > didn't write nor even see [...]
> 
> We should distribute the maintenance effort so that to
> a higher degree, people who wrote the code are maintaining it,

This is a volunteer-based project, so we have no way of "distributing
the maintenance effort", unless people volunteer to do it.  In
particular, for absolute majority of the Emacs code, the people who
wrote the code are no longer on board.  That's the reality.

If you mean that the authors who are still here should be asked to
take care of their code, then this is already being done, always, and
cannot be done to any higher degree.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Thu, 9 Nov 2023 15:39:48 +0000
> Cc: acm@muc.de, dmitry@gutov.dev, bjorn.bidar@thaodan.de, emacs-devel@gnu.org
> 
> And what I'm trying to say is that all those difficulties are much
> much less importnat than shortcomings in domain knowledge.

Not IME, FWIW.

> Maybe _somewhat_ easier, but to what loss?  Aggravated contributors,
> endless hair-splitting of what exactly good style is.   And style
> preferences change anyway: what I "liked" 10 years ago is not what
> I like now. It's really a bad idea.

You will not find any message from me that says we should ban cl-lib.
Maybe you are confusing me with someone else.  All I'm saying is that
people should not dismiss this aspect so easily, when they consider
the subject.  It is not just a coding style issue.

Re: Lisp files that load cl-lib in problematical ways

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <acorallo@gnu.org>
>> Cc: stephen.berman@gmx.net,  incal@dataswamp.org,  emacs-devel@gnu.org
>> Date: Thu, 09 Nov 2023 06:41:03 -0500
>> 
>> > I think what is now in scratch/comp-run does what we want.  If there are
>> > no objections I'll push it to master after some polishing in the
>> > following days.
>> 
>> Okay I've installed the changes.
>
> Thanks, but please in the future allow a bit longer than just 17 hours
> for people to review the changes and post comments.

You're right sorry, I had the impression this branch of the thread was
becoming a monologue and not many people were interested so I proceeded,
also the weekend I'll not be responsive so I wanted to have it in with
some fixing for fixing in case.  Noted for the next time.

> I see some warnings when compiling the new files in a build without
> native-compilation:
>
>   In comp-eln-load-path-eff:
>   emacs-lisp/comp-common.el:519:31: Warning: reference to free variable `comp-native-version-dir'
>   emacs-lisp/comp-common.el:522:11: Warning: reference to free variable `native-comp-eln-load-path'
>
>   In native-compile-async-skip-p:
>   emacs-lisp/comp-run.el:124:54: Warning: reference to free variable `comp--no-native-compile'
>   In comp-run-async-workers:
>   emacs-lisp/comp-run.el:309:14: Warning: reference to free variable `comp-deferred-pending-h'
>   In comp-subr-trampoline-install:
>   emacs-lisp/comp-run.el:336:21: Warning: reference to free variable `native-comp-enable-subr-trampolines'
>   emacs-lisp/comp-run.el:338:34: Warning: reference to free variable `comp-installed-trampolines-h'
>
>   In end of data:
>   emacs-lisp/comp-run.el:342:8: Warning: the function `comp--install-trampoline' is not known to be defined.
>   emacs-lisp/comp-run.el:291:37: Warning: the function `native-elisp-load' is not known to be defined.
>   emacs-lisp/comp-run.el:211:35: Warning: the function `comp-el-to-eln-filename' is not known to be defined.

Thanks 13a1797d5b6 fixes all of these here.

  Andrea

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Thu, Nov 9, 2023 at 3:51 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Thu, 9 Nov 2023 15:39:48 +0000
> > Cc: acm@muc.de, dmitry@gutov.dev, bjorn.bidar@thaodan.de, emacs-devel@gnu.org
> >
> > And what I'm trying to say is that all those difficulties are much
> > much less importnat than shortcomings in domain knowledge.
>
> Not IME, FWIW.

Fair enough.  If you ever find my code whose style you think
needs rewriting, or need help translating another piece of
code to more familiar Lisp, let me know.

João

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Andrea Corallo <acorallo@gnu.org>
> Cc: stephen.berman@gmx.net,  incal@dataswamp.org,  emacs-devel@gnu.org
> Date: Thu, 09 Nov 2023 11:10:13 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Thanks, but please in the future allow a bit longer than just 17 hours
> > for people to review the changes and post comments.
> 
> You're right sorry, I had the impression this branch of the thread was
> becoming a monologue and not many people were interested so I proceeded,
> also the weekend I'll not be responsive so I wanted to have it in with
> some fixing for fixing in case.  Noted for the next time.

Silence is not necessarily a sign that no one is paying attention...

> Thanks 13a1797d5b6 fixes all of these here.

Hmm... I still see warnings:

  In native-compile-async-skip-p:
  emacs-lisp/comp-run.el:124:54: Warning: reference to free variable `comp--no-native-compile'

  In comp-run-async-workers:
  emacs-lisp/comp-run.el:309:14: Warning: reference to free variable `comp-deferred-pending-h'

  In comp-subr-trampoline-install:
  emacs-lisp/comp-run.el:336:21: Warning: reference to free variable `native-comp-enable-subr-trampolines'
  emacs-lisp/comp-run.el:338:34: Warning: reference to free variable `comp-installed-trampolines-h'

  In end of data:
  emacs-lisp/comp-run.el:342:8: Warning: the function `comp--install-trampoline' is not known to be defined.
  emacs-lisp/comp-run.el:291:37: Warning: the function `native-elisp-load' is not known to be defined.
  emacs-lisp/comp-run.el:211:35: Warning: the function `comp-el-to-eln-filename' is not known to be defined.

  In comp--func-arity:
  emacs-lisp/comp.el:1179:26: Warning: reference to free variable `comp-subr-arities-h'

  In comp-final:
  emacs-lisp/comp.el:3146:56: Warning: reference to free variable `native-comp-eln-load-path'

  In comp--trampoline-abs-filename:
  emacs-lisp/comp.el:3217:29: Warning: reference to free variable `native-comp-enable-subr-trampolines'
  emacs-lisp/comp.el:3221:46: Warning: reference to free variable `comp-native-version-dir'

  In comp-compile-all-trampolines:
  emacs-lisp/comp.el:3409:21: Warning: reference to free variable `native-comp-eln-load-path'

  In comp-lookup-eln:
  emacs-lisp/comp.el:3421:15: Warning: reference to free variable `native-comp-eln-load-path'
  emacs-lisp/comp.el:3423:48: Warning: reference to free variable `comp-native-version-dir'

  In batch-native-compile:
  emacs-lisp/comp.el:3458:25: Warning: reference to free variable `native-comp-eln-load-path'

  In batch-byte+native-compile:
  emacs-lisp/comp.el:3502:24: Warning: reference to free variable `native-comp-eln-load-path'

  In native-compile-prune-cache:
  emacs-lisp/comp.el:3515:25: Warning: reference to free variable `native-comp-eln-load-path'
  emacs-lisp/comp.el:3527:32: Warning: reference to free variable `comp-native-version-dir'

  In end of data:
  emacs-lisp/comp.el:3420:25: Warning: the function `comp-el-to-eln-rel-filename' is not known to be defined.
  emacs-lisp/comp.el:3389:18: Warning: the function `native-elisp-load' is not known to be defined.
  emacs-lisp/comp.el:3115:6: Warning: the function `comp--release-ctxt' is not known to be defined.
  emacs-lisp/comp.el:3112:4: Warning: the function `comp--init-ctxt' is not known to be defined.
  emacs-lisp/comp.el:3109:6: Warning: the function `comp--compile-ctxt-to-file' is not known to be defined.
  emacs-lisp/comp.el:804:12: Warning: the function `comp-el-to-eln-filename' is not known to be defined.

Re: Lisp files that load cl-lib in problematical ways

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <acorallo@gnu.org>
>> Cc: stephen.berman@gmx.net,  incal@dataswamp.org,  emacs-devel@gnu.org
>> Date: Thu, 09 Nov 2023 11:10:13 -0500
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > Thanks, but please in the future allow a bit longer than just 17 hours
>> > for people to review the changes and post comments.
>> 
>> You're right sorry, I had the impression this branch of the thread was
>> becoming a monologue and not many people were interested so I proceeded,
>> also the weekend I'll not be responsive so I wanted to have it in with
>> some fixing for fixing in case.  Noted for the next time.
>
> Silence is not necessarily a sign that no one is paying attention...
>
>> Thanks 13a1797d5b6 fixes all of these here.
>
> Hmm... I still see warnings:

Apologies again, my test was just wrong, 9c9b87639f9 should do a better
job.

Thanks

  Andrea

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Alan Mackenzie <acm@muc.de> writes:

>> Thanks for that. I find it much easier to digest than this clean/unclean
>> thing.
>
>> And wow, that will not be popular :-).
>
> It will be a mixed popular/unpopular, with probably few people in the
> middle.  Back when you were the Maintainer, I think there was a ban on
> the use of cl.el and friends, except for at compilation time.  I don't
> think that caused any problems.

Not that I remember, but it's of course a long time ago.

But somehow, CL discussions and the clean/unclean thing from Richard
were always present. At least I seem to remember that.

>> > We should deprecate those functions/macros/variables in cl-lib that have
>> > no doc string, or a substandard one.  This includes "internal" functions,
>> > too.  Also to be deprecated are obscure functions/m/v (such as
>> > cl-labels).
>
>> Am I right in assuming that this is not about the documentation itself,
>> but just some selection criterium for reducing the size of cl-lib?
>
> The most troublesome cl-lib functions are those without adequate
> documentation.  They're the ones that waste unbounded amounts of time
> when trying to debug something which uses them.  That's why I'd like to
> do away with them.  There are few people willing to fix the doc strings
> in cl-lib, though João has volunteered to make a start.

Ok, thanks.

>> > Having done this, we recode code currently using those deprecated
>> > f/m/v.
>
>> What would recode mean? Using seq/map?
>
> I hadn't really thought of seq.el or map.el.  What do they do?

I would describe both as sets of generic functions for sequence handling
and mapping over sequences. I'm personally not using them, as I wrote here:

  https://lists.gnu.org/archive/html/emacs-devel/2023-11/msg00122.html

That spawned answers from Eli and a sub-thread from Richard, which I
personally found unconvincing.

But I'd recommend in any case that you build your own opinion.

> But it's
> clear that working code can be written without these or cl-lib.
>
>> > (Here a "substandard" doc string is contrasted with an adequate one,
>> > which does all of the following:
>> > (i) It says what the function/macro _does_, or what the variable _is_.
>> > (ii) It describes the form and meaning of each parameter, and its
>> >   relationship to (i).
>> > (iii) If the return value is significant, it describes this.
>> > (iv) It describes all effects on the global state, such as where it
>> >   writes results to, and suchlike.)
>
>> > This would reduce the size of cl-lib to something more manageable than
>> > what we have at the moment.

Re: Lisp files that load cl-lib in problematical ways

[Eli Zaretskii]

> From: Andrea Corallo <acorallo@gnu.org>
> Cc: stephen.berman@gmx.net,  incal@dataswamp.org,  emacs-devel@gnu.org
> Date: Thu, 09 Nov 2023 13:34:06 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Thanks 13a1797d5b6 fixes all of these here.
> >
> > Hmm... I still see warnings:
> 
> Apologies again, my test was just wrong, 9c9b87639f9 should do a better
> job.

Compiles cleanly now, thanks.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Bob Rogers]

   From: Emanuel Berg <incal@dataswamp.org>
   Date: Thu, 09 Nov 2023 16:13:29 +0100

   Alan Mackenzie wrote:

   > The most troublesome cl-lib functions are those without
   > adequate documentation. They're the ones that waste
   > unbounded amounts of time when trying to debug something
   > which uses them. That's why I'd like to do away with them.

   Why not instead provide them with docstrings, I think everyone
   can agree they should have them?

As a long-time CL/EL hacker, I will also undertake to document cl-lib
functions.  I think that is a much easier, and safer, alternative to
trying to recode cl-* usage.

   FWIW, I am also of the philosophy that simple functions should be
used where possible and cl-* functions where necessary, all in the name
of keeping my own code concise and readable.  But, alas, readability is
in the eye of the beholder.

					-- Bob Rogers
					   http://www.rgrjr.com/

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> On Thu, Nov 9, 2023 at 2:28 PM João Távora <joaotavora@gmail.com>
> wrote:
>
>> than 'cl-list*'.  It's slightly less readable IMO.  We could alias
>> `list*` to nconc though, even better.
>
> oops, no we couldn't.  Same job, different calling convention.
> I just don't use either that much, so here again my suggestion
> is to just use what fits better with the style of surrounding code.

There is no reason to require a 21 KB file for a six line function.
None.  Yet some of our files do just that; surely this is a
consideration on equal footing with one's own perception of style?

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> Are _you_ the suit?  I missed the point of most of your very
> elegant text, but isn't policy-making exactly what you (and I) are
> trying to do, i.e. discourage some things and encourage others?

I'm not advocating partitioning Elisp into compartments, and to give
countenance to some of them on the grounds of the stature of their
designers.  That's your proposal.  My synthesis is roughly as follows:

  1. cl-lib is a comparatively large file.

  2. cl-lib is frequently loaded to provide trivial functions
     which could effortlessly be replaced by built-in constructs.

  3. cl-lib also defines a number of functions imported from a different
     programming language, the beliefs underlying whose design don't
     align with those of Emacs.

Therefore, we should discourage cl-lib from being introduced in the
first place, because its proliferation will render programmers more
eager to use it without considering its implications.

> What's the gripe with compartments?  We don't have all of Elisp in
> a single library?  I was talking about namespaces by the way,
> if you didn't catch the reference (admittedly obscure)

Because Elisp isn't naturally partitioned into "compartments."  The
basic unit into which our code is divided is the file, and all
considerations must be framed on that basis.

> Also, is really cl-set-difference to take the difference
> of two sets worse than writing a "hallowed" multi-line
> construct??  Maybe I'm not seeing what the construct is, so
> what is your preferred Elisp hallowed way to take the
> difference of two sets, say make a list  that contains
> all items that appear in LIST1 but not LIST2?  Or to find
> the index of a certain element of a sequence...

dolist with a catch/throw, or dolist, member and push?

> These comparisons, taken one by one, are the hard evidence of that
> allows us to determine what is "alien" or not, not theological
> consecrations of personal tastes of a given group, _any_ group.
> For example, I don't think your 'nconc' form is more readable
> than 'cl-list*'.  It's slightly less readable IMO.  We could alias
> `list*` to nconc though, even better.  But I can agree it's not
> worth requiring cl-lib.el just for cl-list* alone.
>
> Much like you prefer setcar to rplacd. Which I also do, btw.  But
> I prefer (setf (car ...))  even more.  But you'll frequently see
> me using setcar and setcdr (or even rplacwahtevercannevertypeit)
> if that's the style of the file I'm working on.  Because I too
> have my personal tastes, but contrary to others I can let go of
> them without too much fuss.

My point was that comparing rplaca to cl-lib is much the same as
comparing apples to oranges, since rplaca is an _alias_ to setcar.

> Stefan M's pcase in particular is great.  Its docstring is tough,
> I agree, and could use some examples (I always end up grepping
> for examples).  But that macro is a lifesaver.
>
> The way you talk about seq.el and pcase.el makes me think you
> want to ban _all the things_, not just cl-lib.  I'm confused about
> what you want Elisp code in core to look like.

Well, touch-screen.el is a good example of that.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

Po Lu <luangruo@yahoo.com> writes:

> Respect for volunteers is no doubt laudable, but it is very much an
> position to be considered alone which everything you've presented above
> does not impact and nobody takes exception to.  From it, and even
> supposing for the sake of argument that everything else you've said is
> incontrovertible fact, it doesn't really follow that we should not seek
> to reduce the incidence of cl-lib usage in core code.  Now I can't be
> certain that is the object of your militating, if you will, though
> promoting cl-set-difference and cl-position as alternatives to hallowed
> Emacs Lisp constructs does suggest so, but the principle argument
                                                 ---------

Principal, sorry.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 10, 2023 at 12:32 AM Po Lu <luangruo@yahoo.com> wrote:
>
> João Távora <joaotavora@gmail.com> writes:
>
> > Are _you_ the suit?  I missed the point of most of your very
> > elegant text, but isn't policy-making exactly what you (and I) are
> > trying to do, i.e. discourage some things and encourage others?
>
> I'm not advocating partitioning Elisp into compartments, and to give
> countenance to some of them on the grounds of the stature of their
> designers.  That's your proposal.

What the heck?  What I said, is that if Elisp _had_ better
ways of making compartments, such as namespaces, managing which
compartments are preferred in certain parts of code would be easier.
I stated multiple times that I tend to trust developers to pick
the tools they find most appropriate for the job.  I simply stated
that it's funny and ironic that the language whose influence you're
trying to exorcize is precisely the one with one of the best
tools to do that exorcizing.

Maybe you didn't understand this because you're not familiar with
CL packages, so I can explain.  In CL you can actually write a program
in a package that doesn't ':USE' the :COMMON-LISP package at all, or
that only imports a handful of symbols from it (maybe the sacred
Elisp subset).  Or you can make your own packages with just the
symbols you want and maybe use that.  Meaning you'd be effectively
insulated from CL or at least the parts you don't like and could
write your perfect purist Lisp program.   Come on, isn't that funny?

>   3. cl-lib also defines a number of functions imported from a different
>      programming language, the beliefs underlying whose design don't
>      align with those of Emacs.

There are no "beliefs" in Common Lisp's, neither should there be
in Elisp.  I don't think it helps your case to repeatedly evoke
religion.

> Therefore, we should discourage cl-lib from being introduced in the
> first place, because its proliferation will render programmers more
> eager to use it without considering its implications.

Every other choice also has "implications".  Most people favourable
to cl-lib in this discussion know why they chose to use its functions.
I wouldn't be so condescending.

> > Also, is really cl-set-difference to take the difference
> > of two sets worse than writing a "hallowed" multi-line
> > construct??  Maybe I'm not seeing what the construct is, so
> > what is your preferred Elisp hallowed way to take the
> > difference of two sets, say make a list  that contains
> > all items that appear in LIST1 but not LIST2?  Or to find
> > the index of a certain element of a sequence...
>
> dolist with a catch/throw, or dolist, member and push?

Hmmm, a bit vague, no?  Humor me: if it's really so easy and so
readable please write out your preferred equivalent of, say

   (cl-set-difference l1 l2) ; hopefully obvious, like l1 - l2 in python

and

   ;; iterate through someseq and find index of first element
   ;; whose car matches probe according to 'probe-equals'.
   (cl-position probe someseq :key #'car :test #'probe-equals)

Shouldn't take long.  I can try myself, of course, but I think it's
fair to give the proponent of the alternate approach a shot before
comparing the two approaches.

> My point was that comparing rplaca to cl-lib is much the same as
> comparing apples to oranges, since rplaca is an _alias_ to setcar.

Some get confused by cl-labels, some by archaic things like rplaca,
some by new stuff like static-if.  Doesn't matter what is alias to
what, when you see both these things  in code you don't know what
they do until you look them up.  And many times you'll think "there's
a much better way".

> Well, touch-screen.el is a good example of that.

OK.  So, basically, your code :-) '(if x (progn ...))' instead
of 'cond', and lots of handrolled catch/throw with while t. OK fine.

I have a feeling I think you'd love cl-loop, maybe even too much :-)

Also quite some long functions there.  300 lines? Maybe typing
"touch-screen" each time to define and call functions
discourages you from modularizing more?  Or maybe it doesn't
bother you at all, so fine.

Don't get me wrong, I am very thankful for your contributions,
I really am.  Write Elisp code in the manner you find most
comfortable.  But there are programmers working on problem domains
in the core other than touch-screen.el, maybe with different
non-C backgrounds.  Many of them are historical contributors to
Emacs.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 10, 2023 at 12:03 AM Po Lu <luangruo@yahoo.com> wrote:
>

> There is no reason to require a 21 KB file for a six line function.
> None.  Yet some of our files do just that; surely this is a
> consideration on equal footing with one's own perception of style?

I said I agree with you.  So go there and replace cl-list* with nconc.
Or put it in a much smaller preloaded file, it doesn't have
dependencies.  Or just define it in terms of nconc if it's really equivalent.

But cl-lib usage and usefulness is really much more than that tiny aspect
you're focusing on.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> I said I agree with you.  So go there and replace cl-list* with nconc.
> Or put it in a much smaller preloaded file, it doesn't have
> dependencies.  Or just define it in terms of nconc if it's really
> equivalent.
>
> But cl-lib usage and usefulness is really much more than that tiny
> aspect you're focusing on.

cl-list* is just one example of a theme that constantly repeats itself
throughout our code.  Removing this one instance is futile, since
there's plenty more where it came from.  The only remedy is for everyone
to develop a general aversion to cl-lib.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> What the heck?  What I said, is that if Elisp _had_ better
> ways of making compartments, such as namespaces, managing which
> compartments are preferred in certain parts of code would be easier.
> I stated multiple times that I tend to trust developers to pick
> the tools they find most appropriate for the job.  I simply stated
> that it's funny and ironic that the language whose influence you're
> trying to exorcize is precisely the one with one of the best
> tools to do that exorcizing.

Which is tantamount to, or at least comes off as, arguing in favor of
that language's influence.

> Maybe you didn't understand this because you're not familiar with
> CL packages, so I can explain.  In CL you can actually write a program
> in a package that doesn't ':USE' the :COMMON-LISP package at all, or
> that only imports a handful of symbols from it (maybe the sacred
> Elisp subset).  Or you can make your own packages with just the
> symbols you want and maybe use that.  Meaning you'd be effectively
> insulated from CL or at least the parts you don't like and could
> write your perfect purist Lisp program.   Come on, isn't that funny?

I know how packages function under Common Lisp, but they are not
salvation.  My objective is not to write code free of Common Lisp
influence; this I already do.  It is for everyone else to write them, or
provide a 95% effective guarantee that I (and many others) will never
have to read them, which is easier said than done; a leitmotif in Emacs
development is for active developers to vanish overnight, leaving a
small corps of developers with reponsibility for their upkeep in the
years that follow.

Granted, the foregoing is something of an overstatement.  But the
alacrity with which programmers seize at opportunities to employ cl-lib
does call for one.

> There are no "beliefs" in Common Lisp's, neither should there be
> in Elisp.  I don't think it helps your case to repeatedly evoke
> religion.

Really?  Here is one example to the contrary: keyword arguments.  No
Emacs Lisp built-in takes them, yet it is scarcely possible to name a
Common Lisp list manipulation function that does not.

> Every other choice also has "implications".  Most people favourable
> to cl-lib in this discussion know why they chose to use its functions.
> I wouldn't be so condescending.

Yet few of these people consider the impact their decision makes to
others, who will in due course have the responsibility for reading this
code pawned off them.

> Hmmm, a bit vague, no?  Humor me: if it's really so easy and so
> readable please write out your preferred equivalent of, say
>
>    (cl-set-difference l1 l2) ; hopefully obvious, like l1 - l2 in python

(let ((list nil))
  (dolist (x l1)
    (unless (member x l2)
      (push x list)))
  (dolist (x l2)
    (unless (member x l1)
      (push x list)))
  list)

> and
>
>    ;; iterate through someseq and find index of first element
>    ;; whose car matches probe according to 'probe-equals'.
>    (cl-position probe someseq :key #'car :test #'probe-equals)
>
> Shouldn't take long.  I can try myself, of course, but I think it's
> fair to give the proponent of the alternate approach a shot before
> comparing the two approaches.

(catch 'tag
  (let ((index 0))
    (dolist (tem someseq)
      (when (eq (car tem) probe)
        (throw 'tag index))
      (setq index (1+ index)))))

> Some get confused by cl-labels, some by archaic things like rplaca,
> some by new stuff like static-if.  Doesn't matter what is alias to
> what, when you see both these things  in code you don't know what
> they do until you look them up.  And many times you'll think "there's
> a much better way".

There's not much of a point in arguing against such patent absurdities
as considering the operation of setcar as complex as that of cl-labels,
is there?

> OK.  So, basically, your code :-) '(if x (progn ...))' instead
> of 'cond', and lots of handrolled catch/throw with while t. OK fine.
>
> I have a feeling I think you'd love cl-loop, maybe even too much :-)

I think not.

> Also quite some long functions there.  300 lines? Maybe typing
> "touch-screen" each time to define and call functions
> discourages you from modularizing more?  Or maybe it doesn't
> bother you at all, so fine.

It doesn't bother me that functions for whom there are no reasons to use
their constituents elsewhere have yet to be balkanized, no.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

João Távora <joaotavora@gmail.com> writes:

> On Thu, Nov 9, 2023 at 10:05 AM Alan Mackenzie <acm@muc.de> wrote:
>
>> >    If it needed any confirmation, I too like cl-lib and I too help
>> >    maintain other people's code in the Emacs core tree as well as
>> >    maintaining a number of libraries I have authored.
>>
>> There's a difference between liking cl-lib and advocating its
>> indiscriminate use.  I don't think you've done the latter in this (and
>> related) threads.
>
> Yes, you're right.  Indeed I don't' advocate for its indiscriminate use,
> just as I don't advocate for indiscriminate use of anything, except
> perhaps drinking water and brushing teeth.
>
>> Nobody who likes cl-lib has yet addressed the point made by Richard and
>> (less eloquently) by me, namely that the incorporation and use of cl-lib
>> swells the size and complexity of Emacs Lisp to the point of making
>> maintenance difficult.  What is your view on this important point?
>
> That it doesn't make maintenance any more difficult than any other
> Elisp construct, be it very old and curiously named like 'rplacd' or
> much, much newer like `seq-do` or `pcase-lambda`.
>
> My specific view on cl-lib.el is that it brings us a small part of
> the results of  non-trivial design work put in when important figures
> in the Lisp world met  regularly for many years to deliver what has
> proved to become excellent,  battle-tested, widely understood and
> impeccably documented programming abstractions.
>
> What I'm reading so far in this long discussion is that the argument
> of its detractors isn't really that cl-lib isn't good, but that
> it is superfluous and that learning it is a burden on maintainers.
> Well, it's just as superfluous as all of Elisp apart from two handfuls
> of primitives, I guess.  Or any programming language for that matter, if
> you know enough machine code.  Or any other programming abstraction I
> happen not to be familiar with.
>
> Also I seem to hear that Elisp has some kind of ideal hard-to-define
> identity or fingerprint and that it shouldn't try to become anything
> else.  But this argument is very strange given Elisp is, like any
> decent language, easy to extend.  Not to mention I struggle to see
> the technical advantage in uniqueness for uniqueness sake.  A good
> part of Elisp is about a special purpose function: editing buffers,
> and an the equally important part is about doing things with bits in
> memory, there's no reason to advocate for singularity here.
>
> I also hear Elisp shouldn't become Common Lisp, but not only is the
> use of cl-lib.el nowhere a step in that direction, but also -- somewhat
> ironically -- if Elisp _were_ Common Lisp, then that hard-to-define
> identity would be much easier to define and language extension would be
> much easier to organize into compartments to facilitate policy-making.
>
> Again, the only thing that has brought Elisp any closer to Common
> Lisp significantly, was the introduction of lexical binding some 10
> years ago.  Elisp looks a lot different now than it did in the 90's.
> Closures everywhere, higher-order functions!  Shudder!
>
> There's even talk of a continuation-passing style library, to
> be called future.el or promise.el or something.  Oh dear, what
> will be of us that we will have to evolve like any other language
> in the world!
>
> So I propose we let programmers use their judgement.  Really
> respect people who write code for no money and give the copyright
> away to the FSF.  Maybe suggest guidelines such as not introduce
> cl-loop where a dolist would do the job just as economically and
> elegantly. Don't use higher-order functions just because it looks
> cool. But maybe do suggest to use cl-position with a :key and a
> :test instead of a complex while loop with an auxiliary variable.
> Or cl-set-difference instead of nested loops.  Suggest to express
> intent, use abstractions.   Suggest to be consistent, be scrutinous,
> be "discriminate", be mindful of the style of the current area
> you are working on.
>
> But don't suggest anything too hard, especially if it's not
> modifying code you have authored.  Don't use arguments of authority
> when you can point to specific things.  Be generally respectful of
> people putting in any good work even if you don't like the style,
> and try to learn a new thing or two every now and then.
>
> BTW here are some nice generic suggestions from the Lisp world,
> written by two fenomenal programmers.  I love reading this from
> time to time:
>
>    https://www.cs.umd.edu/~nau/cmsc421/norvig-lisp-style.pdf

Thanks, that was well written, João. I agree.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Po Lu <luangruo@yahoo.com> writes:

> Respect for volunteers is no doubt laudable, but it is very much an
> position to be considered alone which everything you've presented above
> does not impact and nobody takes exception to.  From it, and even
> supposing for the sake of argument that everything else you've said is
> incontrovertible fact, it doesn't really follow that we should not seek
> to reduce the incidence of cl-lib usage in core code.  Now I can't be
> certain that is the object of your militating, if you will, though
> promoting cl-set-difference and cl-position as alternatives to hallowed
> Emacs Lisp constructs does suggest so, but the principle argument
> against these constructs is that they are alien to the small group of
> individuals charged with trudging through code where people want to make
> use of them.  So no contentions resting on their virtues and
> distinctions rather than their prevalence among us can hold water...

(Sorry if I don't cite the whole mail. Please re-add what you find
important.)

Could you perhaps formulate a plan for what should be done? I personally
find something concrete much easier to handle than a back-and-forth of
arguments.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Could you perhaps formulate a plan for what should be done? I personally
> find something concrete much easier to handle than a back-and-forth of
> arguments.

First off, I think people should make it clear that new code saturated
with calls to cl-lib will not be accepted.  Once people are coaxed out
of this habit, it will be possible to start removing calls to cl-lib
from existing code.

This will also afford us opportunities to move frequently used cl-lib
constructs that aren't all that adventitious to subr.el.  If all goes
well, it will be feasible to use Emacs without loading cl-lib at all.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 10, 2023 at 3:17 AM Po Lu <luangruo@yahoo.com> wrote:
>
> João Távora <joaotavora@gmail.com> writes:
>
> > What the heck?  What I said, is that if Elisp _had_ better
> > ways of making compartments, such as namespaces, managing which
> > compartments are preferred in certain parts of code would be easier.
> > I stated multiple times that I tend to trust developers to pick
> > the tools they find most appropriate for the job.  I simply stated
> > that it's funny and ironic that the language whose influence you're
> > trying to exorcize is precisely the one with one of the best
> > tools to do that exorcizing.
>
> Which is tantamount to, or at least comes off as, arguing in favor of
> that language's influence.

No, not mounting any tantas, I was just making a joke.  But yes, of course
I like Common Lisp.  I like C++ too (not as much, tho) and it pisses off
C people, too.

> Granted, the foregoing is something of an overstatement.  But the
> alacrity with which programmers seize at opportunities to employ cl-lib
> does call for one.

But that leitmotif is everywhere (and us much less strong in open-source,
btw).  Surely if you've ever worked at a large enough company you'll see
bizarre code from people not working there anymore.  And that goes for
everything.  You think everyone will forever be grateful to read your
300-line inline-all-the-things functions in touch-screen.el when you're
not around to explain what they do?  They won't.  It makes no sense
to single out any given library for this evil, especially not cl-lib.el
which is a well-understood stable piece of kit.  In Emacs core and
just as strongly outside it.

> > There are no "beliefs" in Common Lisp's, neither should there be
> > in Elisp.  I don't think it helps your case to repeatedly evoke
> > religion.
>
> Really?  Here is one example to the contrary: keyword arguments.  No
> Emacs Lisp built-in takes them, yet it is scarcely possible to name a
> Common Lisp list manipulation function that does not.

Many examples of Emacs Lisp functions take them.  Even Richard seems
to have come around to their usefulness.  CL functions take them in
consistent ways, but you can use them without keyword arguments if
you're stubborn enough.  Anyway, it's not a "belief", it's just a
good way to provide versatility for functions.

> > Every other choice also has "implications".  Most people favourable
> > to cl-lib in this discussion know why they chose to use its functions.
> > I wouldn't be so condescending.
>
> Yet few of these people consider the impact their decision makes to
> others, who will in due course have the responsibility for reading this
> code pawned off them.

Why do you exempt yourself from this irresponsibility?  Can't you see
this is all subjective and it seems a bit arrogant to say "my code is
so responsible"??  All code is bad.

> > Hmmm, a bit vague, no?  Humor me: if it's really so easy and so
> > readable please write out your preferred equivalent of, say
> >
> >    (cl-set-difference l1 l2) ; hopefully obvious, like l1 - l2 in python
>
> (let ((list nil))
>   (dolist (x l1)
>     (unless (member x l2)
>       (push x list)))
>   (dolist (x l2)
>     (unless (member x l1)
>       (push x list)))
>   list)

>
> (catch 'tag
>   (let ((index 0))
>     (dolist (tem someseq)
>       (when (eq (car tem) probe)
>         (throw 'tag index))
>       (setq index (1+ index)))))

Nuff said :-)  .  So if multiple set difference operations or multiple
index-finding operations are needed you write these choo-choo trains
again and again.  That sure explains the 300 lines.

> > Some get confused by cl-labels, some by archaic things like rplaca,
> > some by new stuff like static-if.  Doesn't matter what is alias to
> > what, when you see both these things  in code you don't know what
> > they do until you look them up.  And many times you'll think "there's
> > a much better way".
>
> There's not much of a point in arguing against such patent absurdities
> as considering the operation of setcar as complex as that of cl-labels,
> is there?

I guess not for human versions of --finline-functions who think,
or rather "believe" that such common practices for code reuse
as subroutine encapsulation is "balkanization".  It's probably "patently absurd"
for them yes.

João Távora

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 10, 2023 at 2:43 AM Po Lu <luangruo@yahoo.com> wrote:

> cl-list* is just one example of a theme that constantly repeats itself
> throughout our code.  Removing this one instance is futile,

You're just contradicting yourself.  First, you argue that it's easy
to replace and has these immediate benefits and then when someone agrees
with you, you say you don't want to.

Also btw, it might not be easy to replace at all cl-list* can be used
with higher order #'apply in contexts where nconc can't.  Perhaps
you despise the programmer who wrote

  (apply #'cl-list* input-data)

But perhaps they did it to avoid writing a multi-line raw loop choo choo
train again.

So it's not true what you said originally that everytime you find a
cl-list* it can be cleanly replaced by nconc.  Not if you understand
or even appreciate function composition.

> since there's plenty more where it came from.   The only remedy is
> for everyone to develop a general aversion to cl-lib.

Be consistent: find the places using simple forms of cl-list* that
presumably (as you said) are the only motivations for the whole
cl-lib.el 27kb loading.  I didn't check but this was your opening
argument after all.  So do that and you'll have effectively and
consensually helped Emacs's code base.  You'll be closer to your
dream goal, using Emacs without loading cl-lib.el.   Be consistent
with your words, else you're just hand-waving and advocating
for irrationality.  "develop a general aversion"??? I like to think
of programmers as scientists, not acolytes to some sect.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 10, 2023 at 10:42 AM Po Lu <luangruo@yahoo.com> wrote:
>
> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
> > Could you perhaps formulate a plan for what should be done? I personally
> > find something concrete much easier to handle than a back-and-forth of
> > arguments.
>
> First off, I think people should make it clear that new code saturated
> with calls to cl-lib will not be accepted.  Once people are coaxed out
> of this habit, it will be possible to start removing calls to cl-lib
> from existing code.
>
> This will also afford us opportunities to move frequently used cl-lib
> constructs that aren't all that adventitious to subr.el.  If all goes
> well, it will be feasible to use Emacs without loading cl-lib at all.

Just a quick question.  Does all of this go for seq.el, map.el and
pcase.el as well, or just cl-lib?

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> Just a quick question.  Does all of this go for seq.el, map.el and
> pcase.el as well, or just cl-lib?

It's not a requirement for pcase to be loaded in code which makes use of
it, and the latter two are (I believe) preloaded.  So only cl-lib.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Alan Mackenzie]

Hello, João.

On Thu, Nov 09, 2023 at 13:41:48 +0000, João Távora wrote:
> On Thu, Nov 9, 2023 at 1:36 PM Alan Mackenzie <acm@muc.de> wrote:

> > Hello, João.

> > On Thu, Nov 09, 2023 at 12:40:24 +0000, João Távora wrote:
> > > On Thu, Nov 9, 2023 at 11:49 AM Dmitry Gutov <dmitry@gutov.dev> wrote:

> > > > Improving cl-lib's documentation would be a welcome effort.

> > > For sure, and not a hard one as well, as all those functions and
> > > macros are pretty good, often flawless emulations of CL functions that
> > > are impeccably documented in

> > >   http://www.lispworks.com/documentation/HyperSpec/Front/

> > > Which is of free access (though not of a compatible license, I
> > > think).  But if people can point to the 5 most confusing functions
> > > they think are poorly documented, I volunteer to rewrite the
> > > docstrings for them.

> > How much are you prepared to do?  I don't have a list of the _most_
> > confusing doc strings, there are too many to chose from.  But starting
> > at the start of cl-macs.el, we have:

> > (i) cl--compiler-macro-list*; completely undocumented.
> > (ii) cl--simple-expr-p: Talks about "side effects", but not what they
> >   are side effects of.  Doesn't describe it's parameters or return
> >   value.  It's unclear what it is that "executes quickly".
> > (iii) cl--expr-contains: It's unclear what X and Y are, and what "refers
> >   to" means.
> > (iv) cl--expr-contains-any; completely undocumented.
> > (v) cl--expr-depends-p: It's unclear what X and Y are, though Y appears
> >   to be some sort of container of symbols.  It's unclear what sort of
> >   "dependency" the function handles, or what "may" means in the context.

> > There are many more.

> These are all internal functions and implementation details.  They're
> not necessary at all for users of cl-lib.el, only for its developers.

So, you're not prepared to rewrite their doc strings as you promised
yesterday, any more.

> What problem are you trying to solve by enhancing these docstrings?

Being able to debug Emacs problems.

> I thought the problem here was code that _used_ cl-lib.el, not hacking
> on cl-lib.el itself.

cl-lib uses itself, and needs debugging too.  Debugging code that uses
abstractions often requires penetrating these abstractions and checking
their innards.

Still, if you're not prepared to fix doc strings in internal functions,
here are five external functions with inadequate doc strings:

(i) cl-fill: The doc string misnames some of the parameters.  It is
unclear what "fill" means, and the forms of CL-SEQ and CL-ITEM aren't
specified.  Is each element of CL-SEQ "filled" with the entirety of
CL-ITEM, or what?  Finally, there is no documentation of CL-KEYS, beyond
stating that two particular keywords are "supported", whatever that
means.  What do :start and :end actually do?  The return value is not
specified.

(ii) cl-replace: Much the same problems as for cl-fill.

(iii) cl-remove: Same again.  Additionally, it is not specified what an
"occurrence" is; this presumably involves some equality test, and which
one is used is not specified.  Again the effect of the keywords is
unspecified.  There are keywords used in its code which aren't listed in
the doc string, namely :if and :if-not.

(iv) cl-remove-if: Much the same again.

(v) cl-remove-if-not: ... and again.

I found these simply by starting at the beginning of cl-seq and listing
pretty much each external function I found in a simple search.  There are
a lot more like these.

Still, even having just those five things properly documented would be
worthwhile.  Thanks for volunteering to do this!

> João

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> You're just contradicting yourself.  First, you argue that it's easy
> to replace and has these immediate benefits and then when someone agrees
> with you, you say you don't want to.

I didn't say I don't want to, I said that removing this one instance
will not solve the underlying readiness to resort to _non-preloaded_
files, of which cl-lib is one, and an egregious offender.

> Also btw, it might not be easy to replace at all cl-list* can be used
> with higher order #'apply in contexts where nconc can't.  Perhaps
> you despise the programmer who wrote
>
>   (apply #'cl-list* input-data)
>
> But perhaps they did it to avoid writing a multi-line raw loop choo choo
> train again.
>
> So it's not true what you said originally that everytime you find a
> cl-list* it can be cleanly replaced by nconc.  Not if you understand
> or even appreciate function composition.

I don't appreciate "function combination," whatever that is.

In a language such as C, the function of a single block is clear.  Each
block commences with a list of one declarator for each variable.

Variables are supplied to and operated on by a list of arithmetic or
data operators and procedures, subject to control constructs all
programmers learn, and a value derived from one or more expressions
holding these variables is returned to complete the block's procedure.

Thus as data changes hands between operators, functions and variables,
its movement can be discerned with only a handful of operators and
control constructs in memory.  This is how a computer operates, or what
_you_ do when working out long division on a piece of paper, or when you
set yourself to any other task entailing the observation of a series of
steps while responding to changing circumstances in the process.

Emacs Lisp is well capable of aligning to such requirements as well;
today many would implement the old float-to-string with cl-loop and
floating point numbers with structs, but in the past they were
implemented with cons cells and while loops, without us being any the
worse for wear.

(defun float-to-string (fnum &optional sci)
  "Convert the floating point number to a decimal string.
Optional second argument non-nil means use scientific notation."
  (let* ((value (fabs fnum)) (sign (< (car fnum) 0))
	 (power 0) (result 0) (str "") 
	 (temp 0) (pow10 _f1))

    (if (f= fnum _f0)
	"0"
      (if (f>= value _f1)			; find largest power of 10 <= value
	  (progn				; value >= 1, power is positive
	    (while (f<= (setq temp (f* pow10 highest-power-of-10)) value)
	      (setq pow10 temp
		    power (+ power decimal-digits)))
	    (while (f<= (setq temp (f* pow10 _f10)) value)
	      (setq pow10 temp
		    power (1+ power))))
	(progn				; value < 1, power is negative
	  (while (f> (setq temp (f/ pow10 highest-power-of-10)) value)
	    (setq pow10 temp
		  power (- power decimal-digits)))
	  (while (f> pow10 value)
	    (setq pow10 (f/ pow10 _f10)
		  power (1- power)))))
					  ; get value in range 100000 to 999999
      (setq value (f* (f/ value pow10) all-decimal-digs-minval)
	    result (ftrunc value))
      (let (int)
	(if (f> (f- value result) _f1/2)	; round up if remainder > 0.5
	    (setq int (1+ (fint result)))
	  (setq int (fint result)))
	(setq str (int-to-string int))
	(if (>= int 1000000)
	    (setq power (1+ power))))

      (if sci				; scientific notation
	  (setq str (concat (substring str 0 1) "." (substring str 1)
			    "E" (int-to-string power)))

					  ; regular decimal string
	(cond ((>= power (1- decimal-digits))
					  ; large power, append zeroes
	       (let ((zeroes (- power decimal-digits)))
		 (while (natnump zeroes)
		   (setq str (concat str "0")
			 zeroes (1- zeroes)))))

					  ; negative power, prepend decimal
	      ((< power 0)		; point and zeroes
	       (let ((zeroes (- (- power) 2)))
		 (while (natnump zeroes)
		   (setq str (concat "0" str)
			 zeroes (1- zeroes)))
		 (setq str (concat "0." str))))

	      (t				; in range, insert decimal point
	       (setq str (concat
			  (substring str 0 (1+ power))
			  "."
			  (substring str (1+ power)))))))

      (if sign				; if negative, prepend minus sign
	  (concat "-" str)
	str))))

If it is to these that you apply the moniker "multi-line raw loop choo
choo trains," then they are precisely what should be written more, not
less.

The "Lisp way" (more felicitously the cl-lib way) is to provide
functions to truckloads of other functions which interface them with one
another.

Each such function is a unique control construct.  To glean what a
function does, one must not only commit each of these control constructs
to memory, but as is all too often the case when the functions provided
to the control constructs are beyond its control, also examine each of
its callers, with labyrinths that put C++ to shame into the bargain,
like pcase-lambda and macros which rewrite the control flow of a form
beyond recognition.

> Be consistent: find the places using simple forms of cl-list* that
> presumably (as you said) are the only motivations for the whole
> cl-lib.el 27kb loading.  I didn't check but this was your opening
> argument after all.  So do that and you'll have effectively and
> consensually helped Emacs's code base.  You'll be closer to your
> dream goal, using Emacs without loading cl-lib.el.   Be consistent
> with your words, else you're just hand-waving and advocating
> for irrationality.  "develop a general aversion"??? I like to think
> of programmers as scientists, not acolytes to some sect.

If such invective and an unwillingness to reason from the specific to
the general is the best response you have, I won't condescend to answer.

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 10, 2023 at 12:13 PM Alan Mackenzie <acm@muc.de> wrote:
>
> Hello, João.
>
> On Thu, Nov 09, 2023 at 13:41:48 +0000, João Távora wrote:
> > On Thu, Nov 9, 2023 at 1:36 PM Alan Mackenzie <acm@muc.de> wrote:
>
> > > Hello, João.
>
> > > On Thu, Nov 09, 2023 at 12:40:24 +0000, João Távora wrote:
> > > > On Thu, Nov 9, 2023 at 11:49 AM Dmitry Gutov <dmitry@gutov.dev> wrote:
>
> > > > > Improving cl-lib's documentation would be a welcome effort.
>
> > > > For sure, and not a hard one as well, as all those functions and
> > > > macros are pretty good, often flawless emulations of CL functions that
> > > > are impeccably documented in
>
> > > >   http://www.lispworks.com/documentation/HyperSpec/Front/
>
> > > > Which is of free access (though not of a compatible license, I
> > > > think).  But if people can point to the 5 most confusing functions
> > > > they think are poorly documented, I volunteer to rewrite the
> > > > docstrings for them.
>
> > > How much are you prepared to do?  I don't have a list of the _most_
> > > confusing doc strings, there are too many to chose from.  But starting
> > > at the start of cl-macs.el, we have:
>
> > > (i) cl--compiler-macro-list*; completely undocumented.
> > > (ii) cl--simple-expr-p: Talks about "side effects", but not what they
> > >   are side effects of.  Doesn't describe it's parameters or return
> > >   value.  It's unclear what it is that "executes quickly".
> > > (iii) cl--expr-contains: It's unclear what X and Y are, and what "refers
> > >   to" means.
> > > (iv) cl--expr-contains-any; completely undocumented.
> > > (v) cl--expr-depends-p: It's unclear what X and Y are, though Y appears
> > >   to be some sort of container of symbols.  It's unclear what sort of
> > >   "dependency" the function handles, or what "may" means in the context.
>
> > > There are many more.
>
> > These are all internal functions and implementation details.  They're
> > not necessary at all for users of cl-lib.el, only for its developers.
>
> So, you're not prepared to rewrite their doc strings as you promised
> yesterday, any more.

I thought the whole problem is that code using cl-lib.el is hard to use.
You said cl-labels was "obscure", so I was prepared to enhance its
docstring, yes.  That's a public macro.  Along with 4 other "obscure"
public ones.  Documentation cl-lib's implementation details is something
else entirely.  It doesn't hurt if there's suspicion that cl-lib.el is
misbehaving in function or efficiency.   But that's not the matter in
discussion here and it doesn't solve the presumed problem of not
understanding code that makes uses cl-lib.

> > What problem are you trying to solve by enhancing these docstrings?
> Being able to debug Emacs problems.

What problem specifically are you having trouble with?  What do you
ultimately want to achieve that you can't without docstrings for
cl-lib.el's implementation details.  This might be an XY problem
(https://en.wikipedia.org/wiki/XY_problem)

> cl-lib uses itself, and needs debugging too.  Debugging code that uses
> abstractions often requires penetrating these abstractions and checking
> their innards.

Well, no.  I don't go read OS source code whenever I have doubts
about how to use a system call or how to read a given piece of
code that uses that system call, I read the manual page for the system
call itself.  Just as I don't pull up a magnifying glass to check if
my CPU's transistors are doing what they should, I trust the data sheet
describing the instructions.  So, no, at least not desirably.  That's
the whole argument for good docstrings: that it relieves you from
having to penetrate innards  to understand a function's interface.
Feels a bit odd to be defending age-old ideas of encapsulation and
information-hiding here.

To summarize, I thought it would be absolutely clear that I volunteered
to write docstrings for public functions, public interfaces.  I'm
amazed it wasn't, but I apologize for the misunderstanding nonetheless.
Docstrings for things that the user sees, is confused by, finds
obscure, etc. Give me 5 of  those, and I'll rework their docstrings for
clarity.  I suggest cl-labels since you singled it out and some
sequence manipulating functions which just say: "Keywords supported:
:test :test-not :key :start :end :from-end" but don't explain what
each of those keyword arguments do.

Another useful thing would be to extend eldoc support in Elisp mode
to be able to highlight the currently active keyword argument.  I'll
also look into that, but no promises.

Thanks,
João

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 10, 2023 at 12:52 PM Po Lu <luangruo@yahoo.com> wrote:

> > So it's not true what you said originally that everytime you find a
> > cl-list* it can be cleanly replaced by nconc.  Not if you understand
> > or even appreciate function composition.
>
> I don't appreciate "function combination," whatever that is.

Composition, not "combination" You find it in mathematics.
Pretty useful many contexts.  IMHO of course.

Anyway, I don't think this argument has any end.  You don't seem to
have any concept of information hiding, abstraction reuse, etc.
Nor do you seem to have any inclination to start actually acting
on that purported plan of yours, even where there is clear
agreement on specific parts of it.  So we don't have a common basis
for discussion in this matter.   That's fine, what can I say?
Go assembly, go machine code!

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> But that leitmotif is everywhere (and us much less strong in open-source,
> btw).  Surely if you've ever worked at a large enough company you'll see
> bizarre code from people not working there anymore.

Where I work, our practices and policies prevent these difficulties from
ever materializing.  Think a standardized coding style, mandatory quotas
for documentation length, expositions written and, hmm, anthologized
independently of the code itself, and review by individuals who have,
needless to say, never seen the code before in their lives, and will not
so much as bat an eyelid before striking down code they cannot easily
understand.  The first and third are what's being proposed here for
Emacs, unless I'm greatly mistaken.

> And that goes for everything.  You think everyone will forever be
> grateful to read your 300-line inline-all-the-things functions in
> touch-screen.el when you're not around to explain what they do?  They
> won't.

It's well-documented and in line with other Emacs Lisp code, so I think
they will.  Indeed one individual is already implementing the
recognition of panning and zoom gestures with it as a basis, without any
counsel from me besides referring him to the file itself.

> It makes no sense to single out any given library for this evil,
> especially not cl-lib.el which is a well-understood stable piece of
> kit.  In Emacs core and just as strongly outside it.

See what I explained earlier.

> Many examples of Emacs Lisp functions take them.

Like?  Which list manipulation functions do?
Contrast cl-member to member.

> Even Richard seems to have come around to their usefulness.  CL
> functions take them in consistent ways, but you can use them without
> keyword arguments if you're stubborn enough.  Anyway, it's not a
> "belief", it's just a good way to provide versatility for functions.

A "belief" in this sense is a model or general scheme that is adhered to
when designing a function, of course, so let's not descend into
quibbling.

> Why do you exempt yourself from this irresponsibility?  Can't you see
> this is all subjective and it seems a bit arrogant to say "my code is
> so responsible"??

Thus far there have been no complaints that code _without_ cl-lib,
pcase, seq or elt is unreadable.  Most of our code (this time measured
by lines) falls squarely within that category.

> All code is bad.

With that outlook, there's not much of a purpose in writing any code for
Emacs, is there?

>> > Hmmm, a bit vague, no?  Humor me: if it's really so easy and so
>> > readable please write out your preferred equivalent of, say
>> >
>> >    (cl-set-difference l1 l2) ; hopefully obvious, like l1 - l2 in python
>>
>> (let ((list nil))
>>   (dolist (x l1)
>>     (unless (member x l2)
>>       (push x list)))
>>   (dolist (x l2)
>>     (unless (member x l1)
>>       (push x list)))
>>   list)
>
>>
>> (catch 'tag
>>   (let ((index 0))
>>     (dolist (tem someseq)
>>       (when (eq (car tem) probe)
>>         (throw 'tag index))
>>       (setq index (1+ index)))))
>
> Nuff said :-)  .  So if multiple set difference operations or multiple
> index-finding operations are needed you write these choo-choo trains
> again and again.  That sure explains the 300 lines.

Sure, but catch, let, dolist, when, throw, setq and member are
elementary operations under Emacs Lisp everyone learns.  They are also
few in number.

> I guess not for human versions of --finline-functions who think, or
> rather "believe" that such common practices for code reuse as
> subroutine encapsulation is "balkanization".  It's probably "patently
> absurd" for them yes.

But the Emacs Lisp compiler doesn't inline functions, does it?  At any
rate you have conflated one of my statements with the other, so read
again.

Thanks.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Alan Mackenzie]

Hello, João.

On Thu, Nov 09, 2023 at 11:06:02 +0000, João Távora wrote:
> On Thu, Nov 9, 2023 at 10:05 AM Alan Mackenzie <acm@muc.de> wrote:

> > >    If it needed any confirmation, I too like cl-lib and I too help
> > >    maintain other people's code in the Emacs core tree as well as
> > >    maintaining a number of libraries I have authored.
> >
> > There's a difference between liking cl-lib and advocating its
> > indiscriminate use.  I don't think you've done the latter in this (and
> > related) threads.

> Yes, you're right.  Indeed I don't' advocate for its indiscriminate use,
> just as I don't advocate for indiscriminate use of anything, except
> perhaps drinking water and brushing teeth.

> > Nobody who likes cl-lib has yet addressed the point made by Richard and
> > (less eloquently) by me, namely that the incorporation and use of cl-lib
> > swells the size and complexity of Emacs Lisp to the point of making
> > maintenance difficult.  What is your view on this important point?

> That it doesn't make maintenance any more difficult than any other
> Elisp construct, be it very old and curiously named like 'rplacd' or
> much, much newer like `seq-do` or `pcase-lambda`.

OK, thanks for answering the question.  Please bare in mind that other
people's experience of maintaining code using cl-* is very different.

> My specific view on cl-lib.el is that it brings us a small part of
> the results of  non-trivial design work put in when important figures
> in the Lisp world met  regularly for many years to deliver what has
> proved to become excellent,  battle-tested, widely understood and
> impeccably documented programming abstractions.

> What I'm reading so far in this long discussion is that the argument
> of its detractors isn't really that cl-lib isn't good, but that
> it is superfluous and that learning it is a burden on maintainers.
> Well, it's just as superfluous as all of Elisp apart from two handfuls
> of primitives, I guess.

No, it's completely superfluous, even if it can be convenient.  Learning
it is a burden because its objects and abstractions are, well, very
abstract as well as being badly documented.  They have little
relationship to anything concrete in Emacs, and they are typically
complicated with many seemingly random details in a way that Emacs
primitives are not.

> Or any programming language for that matter, if you know enough
> machine code.  Or any other programming abstraction I happen not to be
> familiar with.

I am familiar with machine code.  It is perhaps pertinent to remark that
modern processors tend to the reduced instruction set model - that the
complicated "do it in one single complicated instruction" notion doesn't
help the writing of good compilers, or the running of programs
efficiently.  I suspect that analagous logic applies to programming
languages too.

[ .... ]

> João

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Fri, 10 Nov 2023 12:53:10 +0000
> Cc: Dmitry Gutov <dmitry@gutov.dev>,
>  Gerd Möllmann <gerd.moellmann@gmail.com>, 
>  Björn Bidar <bjorn.bidar@thaodan.de>, emacs-devel@gnu.org
> 
> > cl-lib uses itself, and needs debugging too.  Debugging code that uses
> > abstractions often requires penetrating these abstractions and checking
> > their innards.
> 
> Well, no.  I don't go read OS source code whenever I have doubts
> about how to use a system call or how to read a given piece of
> code that uses that system call, I read the manual page for the system
> call itself.

This misses the point which I think Alan wanted to make.  The point is
that when some cl-lib function seems to return an unexpected result,
one needs to debug that function.  And since Edebug has only
rudimentary and frequently completely inadequate support for debugging
cl-lib abstractions, one must then attempt at analyzing and
understanding those abstractions as part of debugging.

And if you never had such moments when some OS syscall produced an
unexpected result and the man pages failed to explain why, then I
guess you have yet a few TIL situations in your future ;-)

> Just as I don't pull up a magnifying glass to check if
> my CPU's transistors are doing what they should, I trust the data sheet
> describing the instructions.

Strawmen aside, this flies in the face of one of the great advantages
of working with Free Software: if you want and need to, you can find
out what happens on any level, including the lowest level of OS system
calls.  Trying to laugh it away is a non-starter around here.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> Composition, not "combination" You find it in mathematics.
> Pretty useful many contexts.  IMHO of course.
>
> Anyway, I don't think this argument has any end.  You don't seem to
> have any concept of information hiding, abstraction reuse, etc.

Yes, why should I chase chimeras, when I could as easily get down to
work?

To mention an example that is perhaps a tadge closer to home,
c-forward-decl-or-cast-1 is a function you would certainly advocate the
balkanization of.  But if it is ultimately divvied up into dozens of
private functions, of which it remains the only caller, what use will
the CC Mode user have gotten out of the process?  CC Mode will only have
become that much slower, and, where CC Mode is concerned, speed is like
gold dust.

> Nor do you seem to have any inclination to start actually acting
> on that purported plan of yours, even where there is clear
> agreement on specific parts of it.

Where's the agreement?

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 10, 2023 at 1:14 PM Po Lu <luangruo@yahoo.com> wrote:

> needless to say, never seen the code before in their lives, and will not
> so much as bat an eyelid before striking down code they cannot easily
> understand.

Wow, sounds like a fantastic healthy work environment.  Do people
ever leave the company?  Like, alive?

> > Many examples of Emacs Lisp functions take them.
>
> Like?  Which list manipulation functions do?
> Contrast cl-member to member.

You didn't say "list manipulation function" . You wrote "No Emacs
built-in takes them".  That's false.  Take make-hash-table.  Oh noes,
looks just like CL.  Guess you're going to hand-roll hash tables
now, right? :-)

Take make-process or json-serialize.  There are more.  Keyword
arguments are in other Non-lisp languages too, as you probably
know, they're called named parameters sometimes.  Heck C++
has been pining for them for so long it's sad.

> > All code is bad.
>
> With that outlook, there's not much of a purpose in writing any code for
> Emacs, is there?

There is, because bad as it is, code gets jobs done that humans can't do.

> But the Emacs Lisp compiler doesn't inline functions, does it?

No, but you do, in actual source code!  That was my point!  You
have these fine abstractions to common operations like intersecting
sets, finding indices, and so much more, and you prefer to handroll
your expanded versions each time instead of going through a function
of an established library.

Very rarely, IME, are there for doing this manual inlining
even more rarely in Elisp which has a compiler-macro mechanism.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Manuel Giraud via Emacs development discussions.]

Po Lu <luangruo@yahoo.com> writes:

[...]

> If it is to these that you apply the moniker "multi-line raw loop choo
> choo trains," then they are precisely what should be written more, not
> less.

Just my 2 cents, but here you have chosen the code for parsing a
floating representation which is, of course, full of peculiarities and
corner cases.  It would not be that much different should it be written
in « pure » CL.

But at the other end of the spectrum, there is lots of code which is
just: walk through this collection and do "this" if "that".
-- 
Manuel Giraud

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 10, 2023 at 1:16 PM Alan Mackenzie <acm@muc.de> wrote:

> > Or any programming language for that matter, if you know enough
> > machine code.  Or any other programming abstraction I happen not to be
> > familiar with.
>
> I am familiar with machine code.  It is perhaps pertinent to remark that
> modern processors tend to the reduced instruction set model - that the
> complicated "do it in one single complicated instruction" notion doesn't
> help the writing of good compilers, or the running of programs
> efficiently.  I suspect that analagous logic applies to programming
> languages too.

That's a clever and interesting perspective actually, but I would
tend to say no.  Programming languages are still written and read
by humans.  Human brains aren't designed the same way processors
are.  For one they aren't "designed" at all.  And we don't consume
information the same way a metal likes to, we don't unroll loops,
and at least I don't prefetch all of the supermarket shelf when
someone tells me to get olives, etc, etc.  Plus I'm horrible at
parallelism.  And we don't speak with reduced instruction sets,
we have many words with nuanced meanings.

João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Alan Mackenzie]

Hello, João.

On Fri, Nov 10, 2023 at 12:53:10 +0000, João Távora wrote:
> On Fri, Nov 10, 2023 at 12:13 PM Alan Mackenzie <acm@muc.de> wrote:

> > On Thu, Nov 09, 2023 at 13:41:48 +0000, João Távora wrote:
> > > On Thu, Nov 9, 2023 at 1:36 PM Alan Mackenzie <acm@muc.de> wrote:

> > > > On Thu, Nov 09, 2023 at 12:40:24 +0000, João Távora wrote:
> > > > > On Thu, Nov 9, 2023 at 11:49 AM Dmitry Gutov <dmitry@gutov.dev> wrote:

> > > > > > Improving cl-lib's documentation would be a welcome effort.

> > > > > For sure, and not a hard one as well, as all those functions and
> > > > > macros are pretty good, often flawless emulations of CL functions that
> > > > > are impeccably documented in

> > > > >   http://www.lispworks.com/documentation/HyperSpec/Front/

> > > > > Which is of free access (though not of a compatible license, I
> > > > > think).  But if people can point to the 5 most confusing functions
> > > > > they think are poorly documented, I volunteer to rewrite the
> > > > > docstrings for them.

> > > > How much are you prepared to do?  I don't have a list of the _most_
> > > > confusing doc strings, there are too many to chose from.  But starting
> > > > at the start of cl-macs.el, we have:

> > > > (i) cl--compiler-macro-list*; completely undocumented.
> > > > (ii) cl--simple-expr-p: Talks about "side effects", but not what they
> > > >   are side effects of.  Doesn't describe it's parameters or return
> > > >   value.  It's unclear what it is that "executes quickly".
> > > > (iii) cl--expr-contains: It's unclear what X and Y are, and what "refers
> > > >   to" means.
> > > > (iv) cl--expr-contains-any; completely undocumented.
> > > > (v) cl--expr-depends-p: It's unclear what X and Y are, though Y appears
> > > >   to be some sort of container of symbols.  It's unclear what sort of
> > > >   "dependency" the function handles, or what "may" means in the context.

> > > > There are many more.

> > > These are all internal functions and implementation details.  They're
> > > not necessary at all for users of cl-lib.el, only for its developers.

> > So, you're not prepared to rewrite their doc strings as you promised
> > yesterday, any more.

> I thought the whole problem is that code using cl-lib.el is hard to use.

More precisely, it's hard to understand and to debug, precisely because
it uses cl-* functions.

> You said cl-labels was "obscure", so I was prepared to enhance its
> docstring, yes.  That's a public macro.  Along with 4 other "obscure"
> public ones.

Ah, so your offer of fixing 5 doc strings is limited to obscure objects.
That wasn't clear from the outset, and wasn't what you said.

Documentation won't help cl-labels become any less obscure.  I've
debugged code using it, but I havn't any clue what it does.  There is
nothing about its functionality which sticks in memory.  Its name is
poor - "labels" isn't a verb, and what it does has nothing to do with
labels.

> Documentation cl-lib's implementation details is something else
> entirely.  It doesn't hurt if there's suspicion that cl-lib.el is
> misbehaving in function or efficiency.   But that's not the matter in
> discussion here and it doesn't solve the presumed problem of not
> understanding code that makes uses cl-lib.

It would solve part of the problem.

> > > What problem are you trying to solve by enhancing these docstrings?
> > Being able to debug Emacs problems.

> What problem specifically are you having trouble with?  What do you
> ultimately want to achieve that you can't without docstrings for
> cl-lib.el's implementation details.  This might be an XY problem
> (https://en.wikipedia.org/wiki/XY_problem)

I'm not having any specific problem at the moment.  The need for accurate
readable doc strings, particularly for code as difficult and obscure as
cl-*, should be self-evident.

I don't know what an "XY problem" is and can't be bothered to look up
your reference.  Thanks for not explaining clearly what you mean.

> > cl-lib uses itself, and needs debugging too.  Debugging code that uses
> > abstractions often requires penetrating these abstractions and checking
> > their innards.

> Well, no.  I don't go read OS source code whenever I have doubts
> about how to use a system call or how to read a given piece of
> code that uses that system call, I read the manual page for the system
> call itself.

One Emacs bug I solved involved hacking into the Linux kernel.  The
solution to the bug was to upgrade to a specific kernel version or later.

> Just as I don't pull up a magnifying glass to check if my CPU's
> transistors are doing what they should, I trust the data sheet
> describing the instructions.  So, no, at least not desirably.  That's
> the whole argument for good docstrings: that it relieves you from
> having to penetrate innards  to understand a function's interface.
> Feels a bit odd to be defending age-old ideas of encapsulation and
> information-hiding here.

Total strawman argument.

> To summarize, I thought it would be absolutely clear that I volunteered
> to write docstrings for public functions, public interfaces.

No, it wasn't clear.

> I'm amazed it wasn't, but I apologize for the misunderstanding
> nonetheless.  Docstrings for things that the user sees, is confused by,
> finds obscure, etc. Give me 5 of  those, and I'll rework their
> docstrings for clarity.

I did so in my last post, and you've just snipped them without comment.
That's twice I've given you five bad doc strings, and you aren't acting
on either set.  That took a significant amount of my time.

Thanks.

It seems to me you are going back on your undertaking to amend these five
doc strings.

> I suggest cl-labels since you singled it out and some sequence
> manipulating functions which just say: "Keywords supported: :test
> :test-not :key :start :end :from-end" but don't explain what each of
> those keyword arguments do.

That would certainly be useful, yes.

> Another useful thing would be to extend eldoc support in Elisp mode
> to be able to highlight the currently active keyword argument.  I'll
> also look into that, but no promises.

That is something else entirely, unrelated to the poor quality of the doc
strings that we're talking about.

> Thanks,
> João

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: What's missing in ELisp that makes people want to use cl-lib?

[Manuel Giraud via Emacs development discussions.]

João Távora <joaotavora@gmail.com> writes:

[...]

> and at least I don't prefetch all of the supermarket shelf when
> someone tells me to get olives, etc, etc.

😂 you should!  It sure takes some space at home but then you really
save time.
-- 
Manuel Giraud

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 10, 2023 at 2:25 PM Alan Mackenzie <acm@muc.de> wrote:

> I did so in my last post, and you've just snipped them without comment.
> That's twice I've given you five bad doc strings, and you aren't acting
> on either set.  That took a significant amount of my time.

Those are not public functions, Alan.  Even Dmitry pointed that out
immediately.

> It seems to me you are going back on your undertaking to amend these five
> doc strings.

Stop insinuating that.  I'm sorry you lost time looking at those
functions (by why???).  But it's not my fault you have no distinction of
public/private interface.  Or at least not one I share.  But I do, and
the Emacs documentation does.  The Emacs and Elisp manual doesn't
describe '--' functions, period. Even checkdoc.el has this concept.
Docstrings for internal functions don't hurt, but they're not the
essential parts to describe a library.  Surely not the first things
you should focus on, because internals are by definition prone to
one could re-implement all of cl-lib.el keeping the same interfaces
and their docstrings with wholly new internals.  I don't know why
am I'm having to clarify this.

Anyway, take care, Alan.
João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Po Lu <luangruo@yahoo.com> writes:

> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
>> Could you perhaps formulate a plan for what should be done? I personally
>> find something concrete much easier to handle than a back-and-forth of
>> arguments.
>
> First off, I think people should make it clear that new code saturated
> with calls to cl-lib will not be accepted.  Once people are coaxed out
> of this habit, it will be possible to start removing calls to cl-lib
> from existing code.
>
> This will also afford us opportunities to move frequently used cl-lib
> constructs that aren't all that adventitious to subr.el.  If all goes
> well, it will be feasible to use Emacs without loading cl-lib at all.

Thanks!

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Po Lu <luangruo@yahoo.com> writes:

> João Távora <joaotavora@gmail.com> writes:
>
>> Just a quick question.  Does all of this go for seq.el, map.el and
>> pcase.el as well, or just cl-lib?
>
> It's not a requirement for pcase to be loaded in code which makes use of
> it, and the latter two are (I believe) preloaded.  So only cl-lib.

I don't understand that. If cl-lib were preloaded, it would also be no
problem? 

Re: What's missing in ELisp that makes people want to use cl-lib?

[Alan Mackenzie]

Hello again, João.

On Fri, Nov 10, 2023 at 14:35:38 +0000, João Távora wrote:
> On Fri, Nov 10, 2023 at 2:25 PM Alan Mackenzie <acm@muc.de> wrote:

> > I did so in my last post, and you've just snipped them without comment.
> > That's twice I've given you five bad doc strings, and you aren't acting
> > on either set.  That took a significant amount of my time.

> Those are not public functions, Alan.  Even Dmitry pointed that out
> immediately.

The second set were public functions.  And as Eli pointed out,
non-public functions need documenting too.

> > It seems to me you are going back on your undertaking to amend these
> > five doc strings.

> Stop insinuating that.

You made an undertaking, the text of which was still in this thread
until you snipped it from your last post.

I now accept you won't be amending the doc strings I suggested, or
probably any others.  I think that's a shame, and I'm disappointed.

> I'm sorry you lost time looking at those functions (by why???).  But
> it's not my fault you have no distinction of public/private interface.

What sort of a smear is that meant to be?  It's just that I'm aware of
the need to document internal functions, you seem not to be.  As a
cl-lib enthusiast, you would have been best placed to fix this doc.

> Or at least not one I share.  But I do, and the Emacs documentation
> does.  The Emacs and Elisp manual doesn't describe '--' functions,
> period. Even checkdoc.el has this concept.  Docstrings for internal
> functions don't hurt, but they're not the essential parts to describe
> a library.  Surely not the first things you should focus on, because
> internals are by definition prone to one could re-implement all of
> cl-lib.el keeping the same interfaces and their docstrings with wholly
> new internals.  I don't know why am I'm having to clarify this.

I don't think there's much point continuing this bit of the conversation
any further.

> Anyway, take care, Alan.
> João

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 10, 2023 at 3:11 PM Alan Mackenzie <acm@muc.de> wrote:

> > Those are not public functions, Alan.  Even Dmitry pointed that out
> > immediately.
>
> The second set were public functions.  And as Eli pointed out,
> non-public functions need documenting too.

Oh then I missed that second set in all this jumble.  Can you please
post it again?

Thanks,
João

Re: What's missing in ELisp that makes people want to use cl-lib?

[Alan Mackenzie]

On Fri, Nov 10, 2023 at 15:14:05 +0000, João Távora wrote:
> On Fri, Nov 10, 2023 at 3:11 PM Alan Mackenzie <acm@muc.de> wrote:

> > > Those are not public functions, Alan.  Even Dmitry pointed that out
> > > immediately.
> >
> > The second set were public functions.  And as Eli pointed out,
> > non-public functions need documenting too.

> Oh then I missed that second set in all this jumble.  Can you please
> post it again?

It's in my post in this thread posted at Date: Fri, 10 Nov 2023 12:13:25
+0000.

Or to be found at
https://lists.gnu.org/archive/html/emacs-devel/2023-11/msg00439.html.

> Thanks,
> João

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: What's missing in ELisp that makes people want to use cl-lib?

[João Távora]

On Fri, Nov 10, 2023 at 3:47 PM Alan Mackenzie <acm@muc.de> wrote:
>
> On Fri, Nov 10, 2023 at 15:14:05 +0000, João Távora wrote:
> > On Fri, Nov 10, 2023 at 3:11 PM Alan Mackenzie <acm@muc.de> wrote:
>
> > > > Those are not public functions, Alan.  Even Dmitry pointed that out
> > > > immediately.
> > >
> > > The second set were public functions.  And as Eli pointed out,
> > > non-public functions need documenting too.
>
> > Oh then I missed that second set in all this jumble.  Can you please
> > post it again?
>
> It's in my post in this thread posted at Date: Fri, 10 Nov 2023 12:13:25
> +0000.
>
> Or to be found at
> https://lists.gnu.org/archive/html/emacs-devel/2023-11/msg00439.html.

Alright, here are two of the five functions documented in a patch.
Will look at the other 3 later.  By the way, if you have a web browser,
the so-called Common Lisp hyperspec is an excellent resource
to learn to work with CL functions.

Here is the doc for FILL and REPLACE:

  http://www.lispworks.com/documentation/HyperSpec/Body/f_fill.htm
  http://www.lispworks.com/documentation/HyperSpec/Body/f_replac.htm

Disclamer:  I'm open to tweaking these docstrings I wrote to
some reasonable degree, like grammar fixes, reordering sentences,
 etc.  Constructive criticism only. I will not indulge endless
hair-splitting, bikeshedding or deep personal considerations of
what proper docstrings should contain and look like, especially
if I suspect it is fueled by any deep disdain (or "generic aversion"
as someone put it) of the cl-lib.el library, such as has been patent
so far in some posts of this discussion so far.

IOW I agree with you that the current docstrings are subpar, so
this is why I volunteered to enhance them, but this is my honest
writing and my own considerations of what a good docstring should
look like, based on many years experience with Lisp and Emacs, and
how it may help a honest user of these functions put them to good
use or read code containing them.

And now the patch for 2 of the 5 functions you mentioned,
João

diff --git a/lisp/emacs-lisp/cl-seq.el b/lisp/emacs-lisp/cl-seq.el
index ec481121ae6..dd9f37b0a82 100644
--- a/lisp/emacs-lisp/cl-seq.el
+++ b/lisp/emacs-lisp/cl-seq.el
@@ -162,9 +162,19 @@ cl-reduce

 ;;;###autoload
 (defun cl-fill (cl-seq cl-item &rest cl-keys)
-  "Fill the elements of SEQ with ITEM.
-\nKeywords supported:  :start :end
-\n(fn SEQ ITEM [KEYWORD VALUE]...)"
+  "Replace elements of SEQ between START and END with ITEM.
+SEQ is a Lisp sequence.  It is destructively modified and
+returned.
+
+START and END are indexes like in `aref' or `elt'.  They
+designate the subsequence of SEQ to operate on.  If END is nil,
+process the sequence to the end. They default to 0 and nil,
+respectively, meaning process the whole SEQ.
+
+START and END are keyword arguments.  See info node `(cl) Program
+Structure > Argument Lists' for details.
+
+\n(fn SEQ ITEM &key START END...)"
   (cl--parsing-keywords ((:start 0) :end) ()
     (if (listp cl-seq)
  (let ((p (nthcdr cl-start cl-seq))
@@ -182,10 +192,30 @@ cl-fill

 ;;;###autoload
 (defun cl-replace (cl-seq1 cl-seq2 &rest cl-keys)
-  "Replace the elements of SEQ1 with the elements of SEQ2.
-SEQ1 is destructively modified, then returned.
-\nKeywords supported:  :start1 :end1 :start2 :end2
-\n(fn SEQ1 SEQ2 [KEYWORD VALUE]...)"
+  "Replace the elements of SEQ1 with elements of SEQ2.
+SEQ1 and SEQ2 are both Lisp sequences.  SEQ1 is destructively
+modified and returned.
+
+START1, END1, START2 and END2 can be indexes like in `aref' or
+`elt'.  Each pair designates two subsequences of SEQ1 and SEQ2,
+respectively, to operate on.  If END1 or END2 is nil, consider
+the respective sequences to the end.
+
+Consecutive elements of the subsequence of SEQ1 are replaced by
+consecutive elements of the subsequence of SEQ2.
+
+If the subsequences vary in length, the shorter one determines
+how many elements are replaced.  Extra elements in either
+subsequence are ignored.
+
+START1 and START2 default to 0, END1 and END2 default to nil,
+meaning replace as much of SEQ1 as possible with elements from
+SEQ2.
+
+START1, END1, START2 and END2 are keyword arguments.  See info
+node `(cl) Program Structure > Argument Lists' for details.
+
+\n(fn SEQ1 SEQ2 &key START1 END1 START2 END2...)"
   (cl--parsing-keywords ((:start1 0) :end1 (:start2 0) :end2) ()
     (if (and (eq cl-seq1 cl-seq2) (<= cl-start2 cl-start1))
  (or (= cl-start1 cl-start2)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Alan Mackenzie]

Hello, João.

On Fri, Nov 10, 2023 at 16:10:36 +0000, João Távora wrote:

[ .... ]

> Alright, here are two of the five functions documented in a patch.
> Will look at the other 3 later.

Thanks!  These two look like just the job!

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 10/11/2023 14:52, Po Lu wrote:
> (defun float-to-string (fnum &optional sci)
>    "Convert the floating point number to a decimal string.
> Optional second argument non-nil means use scientific notation."
>    (let* ((value (fabs fnum)) (sign (< (car fnum) 0))
> 	 (power 0) (result 0) (str "")
> 	 (temp 0) (pow10 _f1))
> 
>      (if (f= fnum _f0)
> 	"0"
>        (if (f>= value _f1)			; find largest power of 10 <= value
> 	  (progn				; value >= 1, power is positive
> 	    (while (f<= (setq temp (f* pow10 highest-power-of-10)) value)
 >      .............
>        (if sign				; if negative, prepend minus sign
> 	  (concat "-" str)
> 	str))))
> 
> If it is to these that you apply the moniker "multi-line raw loop choo
> choo trains," then they are precisely what should be written more, not
> less.

I don't have the corresponding high-level implementation to compare to, 
but the main thing about the code you posted, is that one has to, 
basically, read every line (of which there are ~70) to understand what 
the function is doing and/or to find any irregularities in its 
implementation (e.g. when looking for a bug).

> The "Lisp way" (more felicitously the cl-lib way) is to provide
> functions to truckloads of other functions which interface them with one
> another.

A high-level approach is to use a set of agreed-upon abstractions 
(functional programming in general, across languages, has a basic set, 
but there can be additions) to write programs shorter.

> Each such function is a unique control construct.  To glean what a
> function does, one must not only commit each of these control constructs
> to memory, but as is all too often the case when the functions provided
> to the control constructs are beyond its control, also examine each of
> its callers, with labyrinths that put C++ to shame into the bargain,
> like pcase-lambda and macros which rewrite the control flow of a form
> beyond recognition.

There is, indeed, a mental cost to understanding the code which uses 
them -- one has to know the abstractions and be able to connect the 
names with the concepts. There aren't too many of them in use, though. 
We can argue whether cl-lib is an optimal set, or could be reduced 
(there are indeed many functions and macros in there that I never saw 
the point of using). But your proposed goal (full eventual removal of 
cl-lib, or anything like it) is a non-starter from my POV.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 10/11/2023 05:17, Po Lu wrote:
>> Hmmm, a bit vague, no?  Humor me: if it's really so easy and so
>> readable please write out your preferred equivalent of, say
>>
>>     (cl-set-difference l1 l2) ; hopefully obvious, like l1 - l2 in python
> (let ((list nil))
>    (dolist (x l1)
>      (unless (member x l2)
>        (push x list)))
>    (dolist (x l2)
>      (unless (member x l1)
>        (push x list)))
>    list)

Just to take this example, there are about 2 major bugs: 1) the list 
will come out reversed, 2) the code does XOR rather than subtraction.

To be clear, this is not to take a dig at your coding ability (which I'm 
convinced is quite high), but rather to make a point about the potential 
for problems when one uses lower-level tools and has to reinvent the wheel.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 10/11/2023 15:14, Po Lu wrote:
>> Why do you exempt yourself from this irresponsibility?  Can't you see
>> this is all subjective and it seems a bit arrogant to say "my code is
>> so responsible"??
> Thus far there have been no complaints that code_without_  cl-lib,
> pcase, seq or elt is unreadable.  Most of our code (this time measured
> by lines) falls squarely within that category.

Without having a specific enemy to blame, when the function is long, 
people just either admit out loud they find the code difficult or 
silently avoid changing it.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> I don't understand that. If cl-lib were preloaded, it would also be no
> problem? 

That's an invitation to use it, which will lead to more cryptic cl-lib
code.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> Wow, sounds like a fantastic healthy work environment.  Do people
> ever leave the company?  Like, alive?

Sarcasm aside, most people have no trouble assenting to these
procedures.  Not when they yield good code whose meaning remains clear
for generations to come.

Dismissing the intrinsic merit to readable code should be a no-starter
hereabouts.

> You didn't say "list manipulation function" . You wrote "No Emacs
> built-in takes them".  That's false.  Take make-hash-table.  Oh noes,
> looks just like CL.  Guess you're going to hand-roll hash tables
> now, right? :-)

Hash tables don't count, because they were designed to be compatible
with the old CL hash table functions, just all that faster.

> Take make-process or json-serialize.  There are more.  Keyword
> arguments are in other Non-lisp languages too, as you probably
> know, they're called named parameters sometimes.  Heck C++
> has been pining for them for so long it's sad.

I don't know why make-process was introduced, but it enjoys the excuse
of accepting plenty of arguments.  json-serialize does not: its two
optional parameters don't even encounter the 7-argument limit for C
DEFUNs.

It is lamentable that the keyword argument illness has spread into C,
but that doesn't palliate its deficiencies.

> There is, because bad as it is, code gets jobs done that humans can't
> do.

I don't think anyone seriously believes what you claim here, in essence
that it is OK to install dismal code while being well aware of its
nature as such.

> No, but you do, in actual source code!  That was my point!  You
> have these fine abstractions to common operations like intersecting
> sets, finding indices, and so much more, and you prefer to handroll
> your expanded versions each time instead of going through a function
> of an established library.

These "fine abstractions" are also abstruse, while the set of primitives
you never hesitate to impugn have carried Emacs through its entire
existence.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

Dmitry Gutov <dmitry@gutov.dev> writes:

> I don't have the corresponding high-level implementation to compare
> to, but the main thing about the code you posted, is that one has to,
> basically, read every line (of which there are ~70) to understand what
> the function is doing and/or to find any irregularities in its
> implementation (e.g. when looking for a bug).

By contrast, in the archetypal cl-lib user, one must read every line,
the doc strings of every cl-lib function it calls, and the several lines
adjacent to each place where it is called.  To speak nothing of cl-lib
macros which recast a form's control flow into unrecognizability.

> A high-level approach is to use a set of agreed-upon abstractions
> (functional programming in general, across languages, has a basic set,
> but there can be additions) to write programs shorter.

In this respect functional programming can be likened to poetry--after
writing a witty, caustic epigram inveighing against some perceived
iniquity, a poet is always very pleased with himself.  But should he
apply his manner of writing to communication with real people, as by
speaking in epigrams before a clerk at an unemployment office, it is
likely he will starve.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

Dmitry Gutov <dmitry@gutov.dev> writes:

> Without having a specific enemy to blame, when the function is long,
> people just either admit out loud they find the code difficult or
> silently avoid changing it.

The length of the function plays a far lesser role in one's perception
of a function's complexity than the quantity (or scarcity) of commentary
and the number of unfamiliar constructs it employs do.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

Dmitry Gutov <dmitry@gutov.dev> writes:

> Just to take this example, there are about 2 major bugs: 1) the list
> will come out reversed, 2) the code does XOR rather than subtraction.

Hmm, reading its doc string, I think a more illustrative name for
cl-set-difference would be cl-set-complement.  In this case, the second
loop can be removed.

If the order is significant (it seldom is), the list can be nreversed
afterwards.  Or alternatively a local variable can be maintained holding
the last cdr of the list, and insertion performed on that instead.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> That's a clever and interesting perspective actually, but I would
> tend to say no.  Programming languages are still written and read
> by humans.  Human brains aren't designed the same way processors
> are.  For one they aren't "designed" at all.  And we don't consume
> information the same way a metal likes to, we don't unroll loops,
> and at least I don't prefetch all of the supermarket shelf when
> someone tells me to get olives, etc, etc.  Plus I'm horrible at
> parallelism.  And we don't speak with reduced instruction sets,
> we have many words with nuanced meanings.

No, but humans are less still adapted for processing dozens of
obliquely-named constructs which wreck havoc on the control and data
flow of many a form.

Suppose that to retrieve a shopping cart, it were mandatory to connect
half a dozen unmarked pulleys and pneumatic tubes to one another in a
particular arrangement before depositing your coin in one end of the
contrivance, rather than to merely insert the coin in a small
receptacle.  Would you not be so deterred from availing yourself of the
gratis trolley rentals?

Re: What's missing in ELisp that makes people want to use cl-lib?

[Dmitry Gutov]

On 11/11/2023 03:11, Po Lu wrote:
>> A high-level approach is to use a set of agreed-upon abstractions
>> (functional programming in general, across languages, has a basic set,
>> but there can be additions) to write programs shorter.
> In this respect functional programming can be likened to poetry--after
> writing a witty, caustic epigram inveighing against some perceived
> iniquity, a poet is always very pleased with himself.  But should he
> apply his manner of writing to communication with real people, as by
> speaking in epigrams before a clerk at an unemployment office, it is
> likely he will starve.

Poets don't use a common intermediate language which can be defined, 
studied and use by anybody who has the inclination and time to 
understand it. And makes sentences shorter.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

Dmitry Gutov <dmitry@gutov.dev> writes:

> Poets don't use a common intermediate language which can be defined,
> studied and use by anybody who has the inclination and time to
> understand it. And makes sentences shorter.

Poetic license can be defined, studied and employed by anyone with a
sufficient impetus to do so, which is why it is analogous to functional
programming, or to any other vogue for a particular programming practice
that enjoys a spot in the limelight once in a while.  But such time and
inclination are always in short supply.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >    Why not instead provide them with docstrings, I think everyone
  >    can agree they should have them?

  > As a long-time CL/EL hacker, I will also undertake to document cl-lib
  > functions.  I think that is a much easier, and safer, alternative to
  > trying to recode cl-* usage.

Those functions certainly should have doc strings, and adding some is
a good thing to do.  But that will not cure the larger problem
caused by too much usage of those functions.  They were meant to be used
in a few obsure corners, not used so often that maintainers need to know
about them.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: What's missing in ELisp that makes people want to use cl-lib?

[Michael Heerdegen]

Po Lu <luangruo@yahoo.com> writes:

> I don't appreciate "function combination," whatever that is.
>
> In a language such as C, the function of a single block is clear.  Each
> block commences with a list of one declarator for each variable.
> [...]
> If it is to these that you apply the moniker "multi-line raw loop choo
> choo trains," then they are precisely what should be written more, not
> less.

Side note: I'm not a "fan" of cl-lib, and I hate cl-loop.  I hate it
when I need to understand code that uses it.

But this discussion has some aspects that I don't like - forcing other
programmers to share the own likes and ways of programming, "without
discussing each decision" although these decision have a very broad
reach, are backward incompatible and throw away the long-year work of
very committed and not unintelligent people.  Be aware that some things
in cl-lib are very well written and you'll need a very long time to to
develop a replacement of even approximate quality.  This discussion has
too much from "Personally I don't like this, let's just throw it all
away and start from scratch" that I only know from maybe a bit naive
people in emacs-help.

I understand most of your points of criticism.  A lot of parts of cl-lib
are not very useful for core Emacs, are harmful when used, should be
avoided as much as possible.  OTOH, there also are the useful things,
`cl-labels' (it had been mentioned as a main example) is definitely
among them for me.  Simple, well-defined and used to implement other
things...  I can't believe that it is an expectation that somebody's
personal preferences and likes should be the only guide of how to
proceed.

cl-lib started as emulation package for Common Lisp.  It was a huge work
to get it where it is now.  We should definitely _not_ simply throw away
this achievement.

I see that some parts of cl-lib are not well documented, and there are
problems when debugging it's stuff.  But honestly, other parts of Emacs
have the same problems.  And the Edebug support of cl stuff has also
improved in the last time - this is possible (this question only applies
to macros anyway).  FWIW, just as a random opinion, I found the original
sparse documentation of `pcase' very fine (and yes, I understood it),
so, please note, we are different people having a different background.
Please, let's discuss this with a bit tolerance and a bit less anger and
sarcasm.  It is an important decision.

My goal would be like this:

 - keep cl-lib available to it's current extent for end users

 - try to get unnecessary cl-lib uses out of the code base

 - make useful cl ideas available in the core language

And: one main problem we have is that cl-lib is more or less a huge
monolithic block (as a `require'able library).  Why not split it up?  I
don't see why it is necessary that when I want to use a small useful cl
thing that cl-loop must be made available.  Such things could go to its
own library, name it cl-obscure when you want.

But I don't see a necessity to, for example, kick keyword support out of
reach for the Emacs code base because, there are still, few, use cases
where it fits.  We don't want to end in a situation where we have 15
places that all reinvent the same wheel in subtly different ways.  This
would not be an improvement.  There is no need to reinvent at all.  This
stuff is well tested and integrated.  As long as the fundamental
guideline is clear that using such features is limited to spare
situations where it really makes things simpler, I don't see a problem,
apart maybe from the fact that cl-lib is big.  I would like us to try to
factor it when the size is a problem (is it a practical, objective, or
more or less mainly only a psychological problem btw? - that has
actually been one of my main questions when following this thread).


Michael.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Michael Heerdegen]

João Távora <joaotavora@gmail.com> writes:

> I suggest cl-labels since you singled it out and some
> sequence manipulating functions which just say: "Keywords supported:
> :test :test-not :key :start :end :from-end" but don't explain what
> each of those keyword arguments do.

I dunno if this would fit in our concept, but it would be nice if those
functions supporting keywords had a link (or an initially folded
subsection) where the keywords are explained in a quick way.  One needs
them too seldom to remember all the details until the next time.

Michael. 

Re: What's missing in ELisp that makes people want to use cl-lib?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 569 bytes --]

On Sat, Nov 11, 2023 at 09:23:26AM +0800, Po Lu wrote:
> Dmitry Gutov <dmitry@gutov.dev> writes:
> 
> > Just to take this example, there are about 2 major bugs: 1) the list
> > will come out reversed, 2) the code does XOR rather than subtraction.
> 
> Hmm, reading its doc string, I think a more illustrative name for
> cl-set-difference would be cl-set-complement.  In this case, the second
> loop can be removed.

Yes, "difference" is, surprisingly, ambiguous. Your implementation is
usually called "symmetric difference" to avoid that.

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: What's missing in ELisp that makes people want to use cl-lib?

[Michael Heerdegen]

Alan Mackenzie <acm@muc.de> writes:

> I don't think there's much point continuing this bit of the
> conversation any further.

There is much point.  Please, again, let's calm down a bit.  João is not
personally responsible for the things you criticize.  He will maybe not
be able to fix all of these points in a couple of days.  And he does not
have to prove that he can.  This doesn't mean it is not possible in a
longer term.

Alan, I totally understand that you are annoyed.  But impatience is not
helpful in this case, it will rather prevent that we get things done in
a good way.

Michael.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Michael Heerdegen]

Po Lu <luangruo@yahoo.com> writes:

> Dmitry Gutov <dmitry@gutov.dev> writes:
>
> > Just to take this example, there are about 2 major bugs: 1) the list
> > will come out reversed, 2) the code does XOR rather than subtraction.
>
> Hmm, reading its doc string, I think a more illustrative name for
> cl-set-difference would be cl-set-complement.  In this case, the second
> loop can be removed.

BTW, a "set.el" library is on our wishlist, complementing seq.el and
map.el.

Michael.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Gerd Möllmann]

Po Lu <luangruo@yahoo.com> writes:

> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
>> I don't understand that. If cl-lib were preloaded, it would also be no
>> problem? 
>
> That's an invitation to use it, which will lead to more cryptic cl-lib
> code.

Aha, but seq/map are okay because they are preloaded.
I guess I don't have to understand everything.

Re: What's missing in ELisp that makes people want to use cl-lib?

[Po Lu]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Aha, but seq/map are okay because they are preloaded.
> I guess I don't have to understand everything.

I don't think they are, but their incursion into our code is
accomplished fact.  So we can only bow to fate...

Re: What's missing in ELisp that makes people want to use cl-lib?

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Date: Sat, 11 Nov 2023 07:01:39 +0100
> 
> My goal would be like this:
> 
>  - keep cl-lib available to it's current extent for end users
> 
>  - try to get unnecessary cl-lib uses out of the code base
> 
>  - make useful cl ideas available in the core language

These are exactly our goals.  We've been doing this for years: that's
how stuff like 'when', 'unless', 'push', 'pop', and others got into
the core.

IOW, the way to make some cl-lib functionality freely available in
core is to have a similar or identical implementation in the likes of
subr.el or seq.el, using our naming convention instead of cl-FOO.
We've been doing that for many years, and we will continue doing i