Should we just start dumping cl-lib?

Should we just start dumping cl-lib?

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 308 bytes --]

Commit d605a539d4863c7e7c66aaea6e538ae14c47f3d7 broke Emacs when winner
was enabled without cl-lib being loaded. Nobody seems to have noticed
until now, which suggests that either almost nobody uses winner, or
almost everybody has cl-lib loaded anyway. Maybe it's time to just dump
cl-lib with Emacs.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Should we just start dumping cl-lib?

[John Wiegley]

>>>>> Daniel Colascione <dancol@dancol.org> writes:

> Commit d605a539d4863c7e7c66aaea6e538ae14c47f3d7 broke Emacs when winner was
> enabled without cl-lib being loaded. Nobody seems to have noticed until now,
> which suggests that either almost nobody uses winner, or almost everybody
> has cl-lib loaded anyway. Maybe it's time to just dump cl-lib with Emacs.

I use winner, but always have cl-lib loaded. As for dumping it into Emacs; is
there a precedent for including Lisp libraries of this type into core?

John

Re: Should we just start dumping cl-lib?

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 723 bytes --]

On 10/01/2015 10:12 PM, John Wiegley wrote:
>>>>>> Daniel Colascione <dancol@dancol.org> writes:
> 
>> Commit d605a539d4863c7e7c66aaea6e538ae14c47f3d7 broke Emacs when winner was
>> enabled without cl-lib being loaded. Nobody seems to have noticed until now,
>> which suggests that either almost nobody uses winner, or almost everybody
>> has cl-lib loaded anyway. Maybe it's time to just dump cl-lib with Emacs.
> 
> I use winner, but always have cl-lib loaded.

So does almost everyone else, apparently. So let's save everyone some
initialization CPU time and preload cl-lib.

> As for dumping it into Emacs; is
> there a precedent for including Lisp libraries of this type into core?

XEmacs does it.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Should we just start dumping cl-lib?

[Eli Zaretskii]

> From: Daniel Colascione <dancol@dancol.org>
> Date: Thu, 1 Oct 2015 22:28:53 -0700
> 
> >> Commit d605a539d4863c7e7c66aaea6e538ae14c47f3d7 broke Emacs when winner was
> >> enabled without cl-lib being loaded. Nobody seems to have noticed until now,
> >> which suggests that either almost nobody uses winner, or almost everybody
> >> has cl-lib loaded anyway. Maybe it's time to just dump cl-lib with Emacs.
> > 
> > I use winner, but always have cl-lib loaded.
> 
> So does almost everyone else, apparently.

Only those who use winner, apparently.  This incident tells us nothing
about those who don't use winner, right?

> So let's save everyone some initialization CPU time and preload
> cl-lib.

Up until now, the rules for pre-loading package X were that either
(a) some preloaded package requires X, or (b) X will be auto-loaded at
the beginning of every session by some very frequent operation, like
visiting the first file or creating a window or a frame.

Since winner is itself an optional package that not everyone uses
(e.g., I don't), this case doesn't seem to qualify, IMO.

Re: Should we just start dumping cl-lib?

[Oleh Krehel]

Daniel Colascione <dancol@dancol.org> writes:

> So does almost everyone else, apparently. So let's save everyone some
> initialization CPU time and preload cl-lib.

I'm all for it. I can't imagine using Emacs without loading cl-lib.

    Oleh

Re: Should we just start dumping cl-lib?

[Eli Zaretskii]

> Date: Fri, 02 Oct 2015 10:19:12 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> Up until now, the rules for pre-loading package X were that either
> (a) some preloaded package requires X, or (b) X will be auto-loaded at
> the beginning of every session by some very frequent operation, like
> visiting the first file or creating a window or a frame.
> 
> Since winner is itself an optional package that not everyone uses
> (e.g., I don't), this case doesn't seem to qualify, IMO.

OTOH, visiting a C file or a file in Outline mode loads cl-lib, so
perhaps it's indeed time to preload it.

Re: Should we just start dumping cl-lib?

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 626 bytes --]

> Up until now, the rules for pre-loading package X were that either
> (a) some preloaded package requires X, or (b) X will be auto-loaded at
> the beginning of every session by some very frequent operation, like
> visiting the first file or creating a window or a frame.
>
> Since winner is itself an optional package that not everyone uses
> (e.g., I don't), this case doesn't seem to qualify, IMO.

This might still be an instance of (b), given how many packages require
cl-lib. It would be useful if someone grepped together a list of all such
packages so we could figure how often it'll just be loaded at startup
anyway.

[-- Attachment #2: Type: text/html, Size: 722 bytes --]

Re: Should we just start dumping cl-lib?

[Mark Oteiza]

Daniel Colascione <dancol@dancol.org> writes:

> Commit d605a539d4863c7e7c66aaea6e538ae14c47f3d7 broke Emacs when winner
> was enabled without cl-lib being loaded. Nobody seems to have noticed
> until now, which suggests that either almost nobody uses winner, or
> almost everybody has cl-lib loaded anyway. Maybe it's time to just dump
> cl-lib with Emacs.

Bug#21549

I'd be happy to have more useful functions in "core". (Can we have
`filter` yet?)

Re: Should we just start dumping cl-lib?

[Andreas Schwab]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> This might still be an instance of (b), given how many packages require
> cl-lib.

They should require it only during compiling.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Should we just start dumping cl-lib?

[Artur Malabarba]

2015-10-02 13:21 GMT+01:00 Andreas Schwab <schwab@linux-m68k.org>:
> They should require it only during compiling.

That's only viable if they only use macros. Many of them use functions
from cl-lib so they need a proper require.

Below is a list of all files which grep for ^(require 'cl-lib), which
means they require it at runtime. If any of these are expected to be
loaded at the start of every session, then we should either fix that
require or preload cl-lib.
The ones tat catch my attention are vc.el and help-mode.el.

man.el:92
emacs-lisp/cl.el:30
emacs-lisp/cl-extra.el:40
emacs-lisp/edebug.el:56
emacs-lisp/cl-seq.el:44
emacs-lisp/ert.el:60
emacs-lisp/generator.el:79
emacs-lisp/cl-macs.el:46
emacs-lisp/eieio-core.el:34
dom.el:27
calendar/parse-time.el:37
calendar/todo-mode.el:57
doc-view.el:139
frameset.el:41
wid-edit.el:58
edmacro.el:66
url/url-http.el:28
midnight.el:39
shadowfile.el:78
vc/vc.el:710
vc/ediff-init.el:27
eshell/em-ls.el:29
eshell/em-term.el:34
image-dired.el:159
profiler.el:29
desktop.el:138
progmodes/sql.el:229
progmodes/hideif.el:107
progmodes/ebrowse.el:36
progmodes/xref.el:53
progmodes/gdb-mi.el:94
progmodes/python.el:262
thumbs.el:60
help-mode.el:33
help-fns.el:35
net/gnutls.el:38
net/rcirc.el:46
net/nsm.el:27
net/sasl-scram-rfc.el:40
woman.el:417

Re: Should we just start dumping cl-lib?

[Tassilo Horn]

Mark Oteiza <mvoteiza@udel.edu> writes:

>> Commit d605a539d4863c7e7c66aaea6e538ae14c47f3d7 broke Emacs when
>> winner was enabled without cl-lib being loaded. Nobody seems to have
>> noticed until now, which suggests that either almost nobody uses
>> winner, or almost everybody has cl-lib loaded anyway. Maybe it's time
>> to just dump cl-lib with Emacs.
>
> Bug#21549
>
> I'd be happy to have more useful functions in "core". (Can we have
> `filter` yet?)

There are `seq-filter', `seq-remove', `seq-map', `seq-mapcat',
`seq-reduce', etc. in the new seq.el, and a nice API for associative
structures in map.el which are in "core" although not dumped just like
cl-lib.

Many seq.el functions have cl-lib counterparts which do pretty much the
same.  So the decision if cl-lib should be dumped is also a strategic
decision.  Do we want to advocate seq.el or cl-lib?  (Well, right now
seq.el depends on cl-lib so we couldn't dump seq.el without cl-lib but
anyway...)

Bye,
Tassilo

Re: Should we just start dumping cl-lib?

[Eli Zaretskii]

> From: Mark Oteiza <mvoteiza@udel.edu>
> Date: Fri, 02 Oct 2015 07:46:46 -0400
> 
> I'd be happy to have more useful functions in "core". (Can we have
> `filter` yet?)

Please always accompany such suggestions with rationale.  Adding to
the core just because "why not?" is IMO not a good methodology.  Some
people still care about the memory footprint of programs, so we don't
want to bloat that unless there are good reasons.  Such reasons can
only be discussed on a case by case basis.

Re: Should we just start dumping cl-lib?

[Mark Oteiza]

On 02/10/15 at 04:45pm, Eli Zaretskii wrote:
> > From: Mark Oteiza <mvoteiza@udel.edu>
> > Date: Fri, 02 Oct 2015 07:46:46 -0400
> > 
> > I'd be happy to have more useful functions in "core". (Can we have
> > `filter` yet?)
> 
> Please always accompany such suggestions with rationale.  Adding to
> the core just because "why not?" is IMO not a good methodology.  Some
> people still care about the memory footprint of programs, so we don't
> want to bloat that unless there are good reasons.  Such reasons can
> only be discussed on a case by case basis.

Understood, thanks.

Regarding the breakage with winner.el, I don't understand why under some
circumstances nothing seems to break if cl- functions are used at
runtime in things with only `(eval-when-compile (require 'cl-lib))`. For
instance, git-grepping for 'cl-remove-if' I see package.el and
checkdoc.el don't require cl-lib at runtime.

Re: Should we just start dumping cl-lib?

[Eli Zaretskii]

> Date: Fri, 2 Oct 2015 10:14:52 -0400
> From: Mark Oteiza <mvoteiza@udel.edu>
> Cc: emacs-devel@gnu.org
> 
> Regarding the breakage with winner.el, I don't understand why under some
> circumstances nothing seems to break if cl- functions are used at
> runtime in things with only `(eval-when-compile (require 'cl-lib))`. For
> instance, git-grepping for 'cl-remove-if' I see package.el and
> checkdoc.el don't require cl-lib at runtime.

Are the cl-lib functions they use auto-loaded, perhaps?

Re: Should we just start dumping cl-lib?

[Mark Oteiza]

On 02/10/15 at 05:18pm, Eli Zaretskii wrote:
> > Date: Fri, 2 Oct 2015 10:14:52 -0400
> > From: Mark Oteiza <mvoteiza@udel.edu>
> > Cc: emacs-devel@gnu.org
> > 
> > Regarding the breakage with winner.el, I don't understand why under some
> > circumstances nothing seems to break if cl- functions are used at
> > runtime in things with only `(eval-when-compile (require 'cl-lib))`. For
> > instance, git-grepping for 'cl-remove-if' I see package.el and
> > checkdoc.el don't require cl-lib at runtime.
> 
> Are the cl-lib functions they use auto-loaded, perhaps?

Yes. However, it was an issue in winner.el--why not in these libraries?

Re: Should we just start dumping cl-lib?

[Rasmus]

Hi,

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> Below is a list of all files which grep for ^(require 'cl-lib), which
> means they require it at runtime. If any of these are expected to be
> loaded at the start of every session, then we should either fix that
> require or preload cl-lib.

Plus some libraries "reimplement" cl functions.  Org for sure, but I
/think/ also Gnus (but I'm guessing here).  This is not necessarily bad as
not all Emacsens have cl-lib; still it would be nicer to rely on cl-lib.

Rasmus

-- 
Sådan en god dansk lagereddike kan man slet ikke bruge mere

Re: Should we just start dumping cl-lib?

[Eli Zaretskii]

> Date: Fri, 2 Oct 2015 10:22:40 -0400
> From: Mark Oteiza <mvoteiza@udel.edu>
> Cc: emacs-devel@gnu.org
> 
> On 02/10/15 at 05:18pm, Eli Zaretskii wrote:
> > > Date: Fri, 2 Oct 2015 10:14:52 -0400
> > > From: Mark Oteiza <mvoteiza@udel.edu>
> > > Cc: emacs-devel@gnu.org
> > > 
> > > Regarding the breakage with winner.el, I don't understand why under some
> > > circumstances nothing seems to break if cl- functions are used at
> > > runtime in things with only `(eval-when-compile (require 'cl-lib))`. For
> > > instance, git-grepping for 'cl-remove-if' I see package.el and
> > > checkdoc.el don't require cl-lib at runtime.
> > 
> > Are the cl-lib functions they use auto-loaded, perhaps?
> 
> Yes. However, it was an issue in winner.el--why not in these libraries?

The issue in winner.el, AFAIU, was under very specific circumstances.
Perhaps under those circumstances, those other packages will also have
some problem.

I'd suggest to dig into the problem with winner and understand why it
happens in that specific situation.  E.g., perhaps load-path is not
yet set when the custom file is loaded?

Re: Should we just start dumping cl-lib?

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 648 bytes --]

Mark Oteiza <mvoteiza@udel.edu> writes:

> Daniel Colascione <dancol@dancol.org> writes:
>
>> Commit d605a539d4863c7e7c66aaea6e538ae14c47f3d7 broke Emacs when winner
>> was enabled without cl-lib being loaded. Nobody seems to have noticed
>> until now, which suggests that either almost nobody uses winner, or
>> almost everybody has cl-lib loaded anyway. Maybe it's time to just dump
>> cl-lib with Emacs.
>
> Bug#21549
>
> I'd be happy to have more useful functions in "core". (Can we have
> `filter` yet?)

It's not in "core", it's not preloaded, but seq.el provides useful
additional sequence-manipulation functions such as `seq-filter'.

Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: Should we just start dumping cl-lib?

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 1384 bytes --]

I agree. seq-filter is fine and seq.el is bundled with Emacs, so it should
be enough. There's also an equivalent in cl under a different name.

We have enough unprefixed symbols already, let's not start adding more
unless they are very basic things like `def` macros or flow control macros.

On 2 Oct 2015 2:37 pm, "Tassilo Horn" <tsdh@gnu.org> wrote:
>
> Mark Oteiza <mvoteiza@udel.edu> writes:
>
> >> Commit d605a539d4863c7e7c66aaea6e538ae14c47f3d7 broke Emacs when
> >> winner was enabled without cl-lib being loaded. Nobody seems to have
> >> noticed until now, which suggests that either almost nobody uses
> >> winner, or almost everybody has cl-lib loaded anyway. Maybe it's time
> >> to just dump cl-lib with Emacs.
> >
> > Bug#21549
> >
> > I'd be happy to have more useful functions in "core". (Can we have
> > `filter` yet?)
>
> There are `seq-filter', `seq-remove', `seq-map', `seq-mapcat',
> `seq-reduce', etc. in the new seq.el, and a nice API for associative
> structures in map.el which are in "core" although not dumped just like
> cl-lib.
>
> Many seq.el functions have cl-lib counterparts which do pretty much the
> same.  So the decision if cl-lib should be dumped is also a strategic
> decision.  Do we want to advocate seq.el or cl-lib?  (Well, right now
> seq.el depends on cl-lib so we couldn't dump seq.el without cl-lib but
> anyway...)
>
> Bye,
> Tassilo
>

[-- Attachment #2: Type: text/html, Size: 1875 bytes --]

Re: Should we just start dumping cl-lib?

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> Please always accompany such suggestions with rationale. Adding to the core
> just because "why not?" is IMO not a good methodology. Some people still
> care about the memory footprint of programs, so we don't want to bloat that
> unless there are good reasons. Such reasons can only be discussed on a case
> by case basis.

Although personally I wouldn't mind seeing cl-lib in core, for the benefit of
Emacs I have to agree with Eli. It's not something we should do just because
it sounds like a good idea, but because not doing it would cost us something.
Shaving a few milliseconds from startup is not enough of a gain to be worth
bloating core.

John

Re: Should we just start dumping cl-lib?

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1258 bytes --]

On 10/02/2015 09:07 AM, John Wiegley wrote:
>>>>>> Eli Zaretskii <eliz@gnu.org> writes:
> 
>> Please always accompany such suggestions with rationale. Adding to the core
>> just because "why not?" is IMO not a good methodology. Some people still
>> care about the memory footprint of programs, so we don't want to bloat that
>> unless there are good reasons. Such reasons can only be discussed on a case
>> by case basis.
> 
> Although personally I wouldn't mind seeing cl-lib in core, for the benefit of
> Emacs I have to agree with Eli. It's not something we should do just because
> it sounds like a good idea, but because not doing it would cost us something.
> Shaving a few milliseconds from startup is not enough of a gain to be worth
> bloating core.

What's the cost of increasing the size of the dumped image? We don't
take COW faults on most of the pages, so we can discard them at will.
Because the OS can discard clean pages without writing them to the
pagefile and reconstitute them at will, transferring code from Emacs
private allocations to the dumped Emacs image _decreases_ memory
footprint. That we also shave a few milliseconds off startup is another
benefit.

If not cl-lib, then what? What is the actual bar?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Should we just start dumping cl-lib?

[Eli Zaretskii]

> From: Daniel Colascione <dancol@dancol.org>
> Date: Fri, 2 Oct 2015 17:19:45 -0700
> 
> On 10/02/2015 09:07 AM, John Wiegley wrote:
> >>>>>> Eli Zaretskii <eliz@gnu.org> writes:
> > 
> >> Please always accompany such suggestions with rationale. Adding to the core
> >> just because "why not?" is IMO not a good methodology. Some people still
> >> care about the memory footprint of programs, so we don't want to bloat that
> >> unless there are good reasons. Such reasons can only be discussed on a case
> >> by case basis.
> > 
> > Although personally I wouldn't mind seeing cl-lib in core, for the benefit of
> > Emacs I have to agree with Eli. It's not something we should do just because
> > it sounds like a good idea, but because not doing it would cost us something.
> > Shaving a few milliseconds from startup is not enough of a gain to be worth
> > bloating core.
> 
> What's the cost of increasing the size of the dumped image? We don't
> take COW faults on most of the pages, so we can discard them at will.
> Because the OS can discard clean pages without writing them to the
> pagefile and reconstitute them at will, transferring code from Emacs
> private allocations to the dumped Emacs image _decreases_ memory
> footprint. That we also shave a few milliseconds off startup is another
> benefit.
> 
> If not cl-lib, then what? What is the actual bar?

Did you miss my further message where I said we should probably
preload cl-lib, because many popular commands in a fresh session load
it anyway?

Re: Should we just start dumping cl-lib?

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 2156 bytes --]

On 10/03/2015 12:22 AM, Eli Zaretskii wrote:
>> From: Daniel Colascione <dancol@dancol.org>
>> Date: Fri, 2 Oct 2015 17:19:45 -0700
>>
>> On 10/02/2015 09:07 AM, John Wiegley wrote:
>>>>>>>> Eli Zaretskii <eliz@gnu.org> writes:
>>>
>>>> Please always accompany such suggestions with rationale. Adding to the core
>>>> just because "why not?" is IMO not a good methodology. Some people still
>>>> care about the memory footprint of programs, so we don't want to bloat that
>>>> unless there are good reasons. Such reasons can only be discussed on a case
>>>> by case basis.
>>>
>>> Although personally I wouldn't mind seeing cl-lib in core, for the benefit of
>>> Emacs I have to agree with Eli. It's not something we should do just because
>>> it sounds like a good idea, but because not doing it would cost us something.
>>> Shaving a few milliseconds from startup is not enough of a gain to be worth
>>> bloating core.
>>
>> What's the cost of increasing the size of the dumped image? We don't
>> take COW faults on most of the pages, so we can discard them at will.
>> Because the OS can discard clean pages without writing them to the
>> pagefile and reconstitute them at will, transferring code from Emacs
>> private allocations to the dumped Emacs image _decreases_ memory
>> footprint. That we also shave a few milliseconds off startup is another
>> benefit.
>>
>> If not cl-lib, then what? What is the actual bar?
> 
> Did you miss my further message where I said we should probably
> preload cl-lib, because many popular commands in a fresh session load
> it anyway?

I did, sorry. Anyway, let's wait until we have a new maintainer and
release Emacs 25 before making this change, but it sounds like we
definitely want to proceed in the direction of dumping cl-lib.

Trying to do it locally, I ran into a very weird problem: functions we
call at compile time, like cl--defalias, raise errors when called:  the
bytecode vector ends up being the number zero instead of the expected
string. Turning cl--defalias and similar functions into macros makes the
build work, but it doesn't solve the real problem.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Should we just start dumping cl-lib?

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Daniel Colascione <dancol@dancol.org>
> Date: Sat, 3 Oct 2015 18:24:30 -0700
> 
> > Did you miss my further message where I said we should probably
> > preload cl-lib, because many popular commands in a fresh session load
> > it anyway?
> 
> I did, sorry. Anyway, let's wait until we have a new maintainer and
> release Emacs 25 before making this change, but it sounds like we
> definitely want to proceed in the direction of dumping cl-lib.

I see no reason to wait until Emacs 25.1 is released.  The release
date is sufficiently far away for us to be able to do that ASAP.

> Trying to do it locally, I ran into a very weird problem: functions we
> call at compile time, like cl--defalias, raise errors when called:  the
> bytecode vector ends up being the number zero instead of the expected
> string. Turning cl--defalias and similar functions into macros makes the
> build work, but it doesn't solve the real problem.

If you have time, please try fixing this.

Thanks.