moving more cl seq/mapping support into core

moving more cl seq/mapping support into core

[MON KEY]

,----
| :FROM:                      Chong Yidong
| :SUBJECT:                   Re: Problems with xml-parse-string
| :DATE:                      Fri, 24 Sep 2010 12:26:21 -0400
|
| It is illogical to criticize sxml for wasting conses, while arguing for
| wrapping each text node in a cons.
|
| Anyway, it is difficult to see how real the problem is without a
| concrete example.  Could you provide one?  I suspect that the real
| problem, if one exists, is Elisp's relatively weak support for list
| mapping and reduction; if that's the case, the correct solution is to
| pull in some of the relevant functions from the CL package.
|
| (URL `http://lists.gnu.org/archive/html/emacs-devel/2010-09/msg01250.html')
`----

,----
|
| :FROM    RMS
| :SUBJECT Re: Emacs Package Management
| :DATE    Tue, 15 Sep 2009 03:16:50 -0400
|
|
|     Could we then privide aliases like "cl-loop" for "loop" ?
|
| I am not against it.
|
|     Maybe we could include Cl functions in the Emacs core
|     incrementaly then?  Each CL function moved to the core would
|     require an agreement on the functions inclusion and a
|     documentation patch.
|
| That's more or less what we've been doing.  Originally I strove very
| hard to keep Emacs itself small.  Many basic and obviously useful
| functions were not standardly available, but they were in CL. Since
| then we have made a number of them standard, and we could certainly
| do this for more of them in the future when it seems best.
|
| But some of the CL facilities are overly complex.  And some,
| specifically setf and friends, are not implemented quite right in
| the Emacs context, which makes them ugly to include.
|
| {...}
|
| :SEE (URL `http://lists.gnu.org/archive/html/emacs-devel/2009-09/msg00342.html')
`----


Following is a list of symbols in other packages wich cl.el provides and which
are duplicated either verbatim or indirectly to achieve the same affect
and presumably to avoid cl.el runtime package requirements.
These are without the `&keys` lambda-lists but some provide predicate
type arguments to accomplish similiarly to :test

`edmacro-subseq'            <- `subseq' emacs-lisp/cl-extra.el
`smtpmail-intersection'     <- `intersection' emacs-lisp/cl-seq.el
`dired-file-set-difference' <- `set-difference'  lisp/dired-aux.el

`edebug-gensym-index' <- `*gensym-counter*' emacs-lisp/cl-macs.el
`edebug-gensym'       <- `gensym'  emacs-lisp/cl-macs.el

`edebug-lambda-list-keywordp' could be improved by changing from:

  (and (symbolp object)
       (= ?& (aref (symbol-name object) 0)))

to:
 (and (memq object lambda-list-keywords) t)

e.g.
 (and (memq '&aux lambda-list-keywords) t)

`lambda-list-keywords' is a constant in emacs-lisp/cl-macs.el

erc/erc-compat.el
`erc-member-if', `erc-delete-if', `erc-remove-if-not'
`erc-subseq' is a verbatim duplicate of `subseq'.

gnus-util.el has `gnus-merge' which is a modified version of cl.el's `merge'
It does a noop for its TYPE arg (but includes it for compatibility) and omits
the "&rest cl-keys" from its lambda-list.

It also provides a function `gnus-mapcar' which is a variant of `mapcar*'
without `cl-mapcar-many' dependence.

`gnus-remove-if' is similar to `remove-if' without CL keywords.

Are there any others like this that I've missed?

Following is a list of cl.el derived feautres which don't appear (as best I can
tell) to rely on cl keywords. I've attempted to enumerate where a given symbol
carries a `cl-compiler-macro` property or `cl-byte-compile-compiler-macro` and
`byte-compile-inline-expand` property values.

Surely some of these are good canditdates for moving into core.  They're
certainly tested, documented and some of them are significantly optimized away
by the byte-compiler.

;; artithmetic
`gcd'            ;(symbol-plist 'gcd)
                 ;=> side-effect-free t

`lcm'            ;(symbol-plist 'lcm)
                 ;=> side-effect-free t

`isqrt'          ;(symbol-plist 'isqrt)
                 ;=> side-effect-free t

`signum'         ;(symbol-plist 'signum)
                 ;=> side-effect-free t

`equalp'         ;(symbol-plist 'equalp)
                 ;=> side-effect-free error-free

`plusp'  ;(symbol-plist 'plusp)
         ;=> side-effect-free t
         ;   byte-compile cl-byte-compile-compiler-macro
         ;   cl-compiler-macro (lambda (w x) (list (quote >) x 0))

`minusp' ;(symbol-plist 'minusp)
         ;=> side-effect-free t byte-compile
         ;    cl-byte-compile-compiler-macro
         ;    cl-compiler-macro (lambda (w x) (list (quote <) x 0)))

`oddp'   ;(symbol-plist 'oddp)
         ;=> side-effect-free t

`evenp'  ;(symbol-plist 'evenp)
         ;=> side-effect-free t

;; generalized
`cl-set-frame-visible-p'

;; type
`coerce'

;; matching
`every'
`some'
`notevery'  ;(symbol-plist 'notevery)
            ;=> byte-optimizer byte-compile-inline-expand

`notany'    ;(symbol-plist 'notany)
            ;=> byte-optimizer byte-compile-inline-expand

;; mapping
`map'       ;(symbol-plist 'map)
            ;=> byte-optimizer byte-compile-inline-expand
`mapcon'
`mapl'
`cl-mapc'
`maplist'
`cl-mapcar-many'
`cl-map-keymap-recursively'
`cl-map-intervals'
`cl-map-overlays'

;; seqs
`concatenate' ;(symbol-plist 'concatenate)
              ;=> byte-optimizer byte-compile-inline-expand

`revappend'   ;(symbol-plist 'revappend)
              ;=> byte-optimizer byte-compile-inline-expand

`nreconc'     ;(symbol-plist 'nreconc)
              ;=> byte-optimizer byte-compile-inline-expand

`list-length' ;(symbol-plist 'list-length)
              ;=> side-effect-free t

`endp'       ;(symbol-plist 'endp);
             ;=> side-effect-free t
             ;   byte-compile cl-byte-compile-compiler-macro
             ;   cl-compiler-macro (lambda (w x) (list (quote null) x)))


`acons'  ; (symbol-plist 'acons)
         ;=> side-effect-free error-free
         ;   byte-optimizer byte-compile-inline-expand

`list*'  ;(symbol-plist 'list*)
         ;=> side-effect-free error-free
         ;   byte-compile cl-byte-compile-compiler-macro
         ;   compiler-macro-file "cl-macs.el"
         ;   cl-compiler-macro #[ ...byte-code-vector elided ]

`ldiff'    ;(symbol-plist 'ldiff)
           ;=> side-effect-free t

`pairlis'  ;(symbol-plist 'pairlis)
           ;=> side-effect-free t

;; These don't have `cl-' namespacing for symbol-name or the local vars.
`copy-list'
`tailp'

;; This one doesn't have `cl-' namespacing for the local vars.
`cl-maclisp-member'

--
/s_P\

Re: moving more cl seq/mapping support into core

[Leo]

On 2010-09-24 23:48 +0100, MON KEY wrote:
> Following is a list of symbols in other packages wich cl.el provides
> and which are duplicated either verbatim or indirectly to achieve the
> same affect and presumably to avoid cl.el runtime package
> requirements.

Those functions are not there overnight they probably came from people's
years of experience using and designing lisp. So they will be reinvented
from time to time for writing any non-trivial elisp packages except now
we have slightly different versions here and there.

My wish is all of them should be incorporated into elisp.

Leo

Re: moving more cl seq/mapping support into core

[Miles Bader]

Leo <sdl.web@gmail.com> writes:
> Those functions are not there overnight they probably came from people's
> years of experience using and designing lisp. So they will be reinvented
> from time to time for writing any non-trivial elisp packages except now
> we have slightly different versions here and there.
>
> My wish is all of them should be incorporated into elisp.

Some CL functions are so trivial that it's not hard to move them (e.g.,
gensym, though I've somehow gotten the impression it's frowned on these
days), but more tricky are those of obvious usefulness, that have
keyword-heavy interfaces in CL (though I think it's quite common that in
many cases few or none of the keyword argument are actually used very
often).

For instance, `remove-if' is obviously useful (I think it's often one of
those "I use cl.el for that function" functions), not complex, and could
probably be moved into the core in some simplified form.

Suggested function signature:  (remove-if PREDICATE SEQ [TEST]) -- and
of course, change the cl.el version to deal with TEST as well.

-Miles

p.s. Another [kinda related to remove-if] that I've often wanted to use,
but didn't because it wasn't in the core, is `mapcan' -- and nit doesn't
even have any keywords!

-- 
Justice, n. A commodity which in a more or less adulterated condition the
State sells to the citizen as a reward for his allegiance, taxes and personal
service.

RE: moving more cl seq/mapping support into core

[Drew Adams]

> For instance, `remove-if' is obviously useful (I think it's 
> often one of those "I use cl.el for that function" functions), not 
> complex, and could probably be moved into the core in some
> simplified form.
> 
> Suggested function signature:  (remove-if PREDICATE SEQ [TEST]) -- and
> of course, change the cl.el version to deal with TEST as well.

+1 for remove-if(-not)

But I don't know what you meant by TEST.
AFAIK, the CL signature is this:

remove-if PREDICATE SEQ &key :from-end :start :end :count :key

Maybe you were thinking of `remove', which has `:test'?

Re: moving more cl seq/mapping support into core

[Leo]

On 2010-09-25 15:42 +0100, Drew Adams wrote:
> +1 for remove-if(-not)
>
> But I don't know what you meant by TEST.
> AFAIK, the CL signature is this:
>
> remove-if PREDICATE SEQ &key :from-end :start :end :count :key

I think Miles was talking about making a new remove-if that accepts an
optional argument, one that is not second class.

Leo

RE: moving more cl seq/mapping support into core

[Drew Adams]

> > But I don't know what you meant by TEST.
> > AFAIK, the CL signature is this:
> >
> > remove-if PREDICATE SEQ &key :from-end :start :end :count :key
> 
> I think Miles was talking about making a new remove-if that accepts an
> optional argument, one that is not second class.

Sorry, I don't get it.  What is TEST for?  It is typically for a binary
predicate such as `equal' - but we already have PREDICATE here.  And just what
is second class here (if TEST is not added)?

Re: moving more cl seq/mapping support into core

[Leo]

On 2010-09-25 16:23 +0100, Drew Adams wrote:
> Sorry, I don't get it.  What is TEST for?  It is typically for a binary
> predicate such as `equal' - but we already have PREDICATE here.  And just what
> is second class here (if TEST is not added)?

`remove-if' as defined in cl.el can not be used in a package intended to
be included in GNU Emacs due to the use of cl run time functions.

So to move this function into elisp Miles (I think) proposed that we
give up some of its keyword arguments since they mostly are not used and
turned the keyword :test argument into an optional one.

Leo

Re: moving more cl seq/mapping support into core

[David Kastrup]

Leo <sdl.web@gmail.com> writes:

> On 2010-09-25 16:23 +0100, Drew Adams wrote:
>> Sorry, I don't get it.  What is TEST for?  It is typically for a binary
>> predicate such as `equal' - but we already have PREDICATE here.  And just what
>> is second class here (if TEST is not added)?
>
> `remove-if' as defined in cl.el can not be used in a package intended to
> be included in GNU Emacs due to the use of cl run time functions.
>
> So to move this function into elisp Miles (I think) proposed that we
> give up some of its keyword arguments since they mostly are not used and
> turned the keyword :test argument into an optional one.

I found that quite a few use cases can be dealt with by using the idiom
(delq nil ...)
instead.

-- 
David Kastrup

RE: moving more cl seq/mapping support into core

[Drew Adams]

> > Sorry, I don't get it.  What is TEST for?  It is typically 
> > for a binary predicate such as `equal' - but we already have
> > PREDICATE here.  And just what is second class here (if TEST
> > is not added)?
> 
> `remove-if' as defined in cl.el can not be used in a package 
> intended to be included in GNU Emacs due to the use of cl run
> time functions.
> 
> So to move this function into elisp Miles (I think) proposed that we
> give up some of its keyword arguments since they mostly are 
> not used and turned the keyword :test argument into an optional one.

We are not communicating, I think.  At least I am not understanding you.

Turned what :test argument into an optional one?  There is _no_ :test argument
to `remove-if' (in CL).  Do you perhaps mean :key?  Sorry, I don't follow you at
all.  Again, AFAIK, this is the CL sig:

remove-if PREDICATE SEQ &key :from-end :start :end :count :key

RE: moving more cl seq/mapping support into core

[Drew Adams]

> I found that quite a few use cases can be dealt with by using 
> the idiom (delq nil ...) instead.

Agreed.  And there are lots of other ways to do it.
But `remove-if(-not)' is still handy, IMO.

Re: moving more cl seq/mapping support into core

[Leo]

On 2010-09-25 16:55 +0100, Drew Adams wrote:
> remove-if PREDICATE SEQ &key :from-end :start :end :count :key

Oh, looks like I mixed up. You are right.

Leo

Re: moving more cl seq/mapping support into core

[Miles Bader]

On Sat, Sep 25, 2010 at 11:42 PM, Drew Adams <drew.adams@oracle.com> wrote:
>> For instance, `remove-if' is obviously useful (I think it's
>> often one of those "I use cl.el for that function" functions), not
>> complex, and could probably be moved into the core in some
>> simplified form.
>>
>> Suggested function signature:  (remove-if PREDICATE SEQ [TEST]) -- and
>> of course, change the cl.el version to deal with TEST as well.
>
> +1 for remove-if(-not)
>
> But I don't know what you meant by TEST.
> AFAIK, the CL signature is this:
>
> remove-if PREDICATE SEQ &key :from-end :start :end :count :key
>
> Maybe you were thinking of `remove', which has `:test'?

Ah, yes, indeed.  That makes adding remove-if even simpler of course!

-miles

-- 
Cat is power.  Cat is peace.

Re: moving more cl seq/mapping support into core

[Richard Stallman]

    Those functions are not there overnight they probably came from people's
    years of experience using and designing lisp. So they will be reinvented
    from time to time for writing any non-trivial elisp packages except now
    we have slightly different versions here and there.

    My wish is all of them should be incorporated into elisp.

Since they would occupy parts of the address space that users expect
is theirs, we would need to document them all in the Lisp manual.

Aside from that, the keyword-based interfaces of these functions
does not fit the style of Emacs Lisp very well.

Re: moving more cl seq/mapping support into core

[Leo]

On 2010-09-26 11:37 +0100, Richard Stallman wrote:
> Since they would occupy parts of the address space that users expect
> is theirs, we would need to document them all in the Lisp manual.

Thanks. I think we need not to worry the user address space too much.
Those functions are descriptive so it is unlikely someone wants to name
a function say remove-if that does something different. Even if they do
they can still play with the name by adding *, +, ++, x etc as we have
seen in many places. The majority can continue to like what remove-if
does.

> Aside from that, the keyword-based interfaces of these functions does
> not fit the style of Emacs Lisp very well.

But all the major build blocks like define-derived-mode, defcustom,
define-minor-mode, easy-menu-define etc use keyword arguments.

And completing-read, I often want to specify the DEF argument and it is
pretty difficult to remember where it is without checking it out with
C-h f.

Leo

Re: moving more cl seq/mapping support into core

[Miles Bader]

Leo <sdl.web@gmail.com> writes:
>> Aside from that, the keyword-based interfaces of these functions does
>> not fit the style of Emacs Lisp very well.
>
> But all the major build blocks like define-derived-mode, defcustom,
> define-minor-mode, easy-menu-define etc use keyword arguments.

Emacs sometimes uses keywords for complex top-level
nobody-expects-them-to-be-fast functions/macros.  CL uses keywords for
just about every function in the language.  There's a distinct
difference in style, and that should be respected.

-Miles

-- 
Circus, n. A place where horses, ponies and elephants are permitted to see
men, women and children acting the fool.

Re: moving more cl seq/mapping support into core

[Richard Stallman]

    Thanks. I think we need not to worry the user address space too much.

That would be a change for the worse
in our standard of documentation.

    > Aside from that, the keyword-based interfaces of these functions does
    > not fit the style of Emacs Lisp very well.

    But all the major build blocks like define-derived-mode, defcustom,
    define-minor-mode, easy-menu-define etc use keyword arguments.

Those are all rather heavy things by nature, so if the calling
sequence is heavy too, that's ok.
If I added a `remove-if' function I would like it not to be so heavy
to use.

Re: moving more cl seq/mapping support into core

[MON KEY]

On Fri, Sep 24, 2010 at 6:48 PM, MON KEY <monkey@sandpframing.com> wrote:

>
> Are there any others like this that I've missed?
>

Looks like I missed `edmacro-mismatch' .
This is mismatch from cl-seq.el with "mandatory keywords"...

The signatures of the two functions are quite similiar:

 (edmacro-mismatch seq1 seq2 start1 end1 start2 end2)

 (mismatch seq1 seq2 [keyword value]...)

Note, that the _real_ source level signature of `edmacro-mismatch' is:

 (edmacro-mismatch cl-seq1 cl-seq2 cl-start1 cl-end1 cl-start2 cl-end2)

Indeed `edmacro-mismatch's local vars even "borrow" from the cl "namespace" in
that it let binds cl-from-end, cl-test, cl-test-not, cl-key, and cl-from-end.
Curiously (for me anyhow) `edmacro-mismatch's has to let bind over cl-test and
cl-test-not locals b/c like its cl.el counterpart `edmacro-mismatch' even
evaluates `cl-check-match', i.e. it has to go out of its way to nullify the test
symbols.

As such the difference between the two functions is that `edmacro-mismatch'
doesn't evaluate the `cl-parsing-keywords' macro and offers the same basic set
of keywords (lest :test :test-not) they just aren't opaquely lumped into an
&rest as with `mismatch'.

Following is the wordwise diff of the two functions:

***************
*** 1,11 ****
! (defun mismatch (cl-seq1 cl-seq2 &rest cl-keys)
    "Compare SEQ1 with SEQ2, return index of first mismatching element.
  Return nil if the sequences match.  If one sequence is a prefix of the
! other, the return value indicates the end of the shorter sequence.
! \nKeywords supported:  :test :test-not :key :start1 :end1 :start2
:end2 :from-end
! \n(fn SEQ1 SEQ2 [KEYWORD VALUE]...)"
!   (cl-parsing-keywords (:test :test-not :key :from-end
!                               (:start1 0) :end1 (:start2 0) :end2) ()
      (or cl-end1 (setq cl-end1 (length cl-seq1)))
      (or cl-end2 (setq cl-end2 (length cl-seq2)))
      (if cl-from-end
--- 1,9 ----
! (defun edmacro-mismatch (cl-seq1 cl-seq2 cl-start1 cl-end1 cl-start2 cl-end2)
    "Compare SEQ1 with SEQ2, return index of first mismatching element.
  Return nil if the sequences match.  If one sequence is a prefix of the
! other, the return value indicates the end of the shorted sequence.
! \n(fn SEQ1 SEQ2 START1 END1 START2 END2)"
!   (let (cl-test cl-test-not cl-key cl-from-end)
      (or cl-end1 (setq cl-end1 (length cl-seq1)))
      (or cl-end2 (setq cl-end2 (length cl-seq2)))
      (if cl-from-end

--
/s_P\

Re: moving more cl seq/mapping support into core

[Daniel Colascione]

On 9/26/2010 11:27 PM, Richard Stallman wrote:
>     Thanks. I think we need not to worry the user address space too much.
> 
> That would be a change for the worse
> in our standard of documentation.

Every Emacs release adds all sorts of functions that might impinge on
the user namespace, but it's rarely a problem in practice. CL shouldn't
be held to a higher standard.

In practice, many people (require 'cl) almost immediately, and a large
number of packages, even those in GNU Emacs, (eval-when-compile
'(require 'cl)). If cl namespace were actually a problem, we'd have
heard about it long before now.

In short, I really don't buy the argument that CL would cause too much
namespace pollution. The benefits outweigh the tiny risk to the small
number of users who work without CL loaded.

CL really should be dumped along with Emacs proper.

> Those are all rather heavy things by nature, so if the calling
> sequence is heavy too, that's ok.
> If I added a `remove-if' function I would like it not to be so heavy
> to use.

You can use compiler-macros (or an equivalent) to create "fast" calls
when the keywords involved are known beforehand, as they almost always
are. But the current CL functions are fast enough *now*, considering
that they're widely used. If profiling reports that a particular call is
too slow, it can always be replaced with lower-level functionality. But
for the majority of code that isn't on the fast path, having CL
primitives available will greatly simplify implementation.

Re: moving more cl seq/mapping support into core

[Daniel Colascione]

On 9/25/2010 8:30 AM, Leo wrote:
> On 2010-09-25 16:23 +0100, Drew Adams wrote:
>> Sorry, I don't get it.  What is TEST for?  It is typically for a binary
>> predicate such as `equal' - but we already have PREDICATE here.  And just what
>> is second class here (if TEST is not added)?
> 
> `remove-if' as defined in cl.el can not be used in a package intended to
> be included in GNU Emacs due to the use of cl run time functions.
> 
> So to move this function into elisp Miles (I think) proposed that we
> give up some of its keyword arguments since they mostly are not used and
> turned the keyword :test argument into an optional one.

Why bother? A compiler-macro can examine a call to remove-if, see
whether it recognizes the keyword arguments, and if it does, transform
the whole call into something more lightweight. You retain the current
function's expressive power and CL compatibility while losing nothing in
performance.

Re: moving more cl seq/mapping support into core

[Miles Bader]

Daniel Colascione <dan.colascione@gmail.com> writes:
> CL really should be dumped along with Emacs proper.

cl.el is a combination of good useful things that and horrible ugly
things (this isn't a comment on the CL language, btw -- many of the
problems with cl.el are issue with its implementation and consistency
with elisp).

That's one of the big problems with it:  it's a big undifferentiated
blob (to be fair, it was written a long time ago, back when the state
of elisp programming practice was ... less nice).

I interpret the consistent refusal to sanction cl.el to be a simple
resistance to wholesale adoption of ugly things, even if that means
some loss in terms of good things (perhaps with some intent to
pressure people into fixing the problems).

Users don't typically care of course -- they can simply ignore what
they don't like and use the 5 function in cl.el they really find
useful.  But the elisp maintainers need to care.

Of course, there's a solution which would probably make everybody
happy:  adopt the good things only, and fix some of the problems with
the horrible ugly things so that they can be adopted.  But people
advocating cl.el seem to usually stop short of putting any effort into
doing that...

-Miles

-- 
"Suppose we've chosen the wrong god. Every time we go to church we're
just making him madder and madder." -- Homer Simpson

Re: moving more cl seq/mapping support into core

[Richard Stallman]

    Every Emacs release adds all sorts of functions that might impinge on
    the user namespace,

Would you please give some examples for a recent release, so it will
be clear whether you are talking about the same issue?

Most functions in Emacs have name prefixes or suffixes which help
prevent name conflicts; those names don't pose a problem.  We only
rarely add names like `remove-if' which have nothing in the name to
keep them out of the user's way, and we normally document such names
in the manual.  If that practice has changed, I would like to know.

Re: moving more cl seq/mapping support into core

[Richard Stallman]

    > Those are all rather heavy things by nature, so if the calling
    > sequence is heavy too, that's ok.
    > If I added a `remove-if' function I would like it not to be so heavy
    > to use.

    You can use compiler-macros (or an equivalent) to create "fast" calls
    when the keywords involved are known beforehand, as they almost always
    are.

Indeed, one could do that.  However, that doesn't mean that it is
unimportant to choose a good, convenient interface as the one we offer
and document.

Re: moving more cl seq/mapping support into core

[Daniel Colascione]

On 10/1/2010 4:42 AM, Richard Stallman wrote:
>     > Those are all rather heavy things by nature, so if the calling
>     > sequence is heavy too, that's ok.
>     > If I added a `remove-if' function I would like it not to be so heavy
>     > to use.
> 
>     You can use compiler-macros (or an equivalent) to create "fast" calls
>     when the keywords involved are known beforehand, as they almost always
>     are.
> 
> Indeed, one could do that.  However, that doesn't mean that it is
> unimportant to choose a good, convenient interface as the one we offer
> and document.

I understood the primary objection to keyword arguments being about
efficiency. Compiler macros solve that problem. If you instead want to
discuss the convenience of keyword arguments --- well, they're hard to
beat. Functions that take a large number of arguments with various
ad-hoc parameter meanings (I'm looking at you, write-region) are a
nightmare to use, and keyword argument would greatly simplify their use
(which is probably why play-sound takes keyword arguments).

Re: moving more cl seq/mapping support into core

[Daniel Colascione]

On 10/1/2010 4:42 AM, Richard Stallman wrote:
>     Every Emacs release adds all sorts of functions that might impinge on
>     the user namespace,
> 
> Would you please give some examples for a recent release, so it will
> be clear whether you are talking about the same issue?
> 
> Most functions in Emacs have name prefixes or suffixes which help
> prevent name conflicts; those names don't pose a problem.  We only
> rarely add names like `remove-if' which have nothing in the name to
> keep them out of the user's way, and we normally document such names
> in the manual.  If that practice has changed, I would like to know.

Even prefixed functions and symbols can conflict with something the user
has specified; the likelihood is just lower. In general, CL symbols,
even without a namespace qualification, won't cause problems because cl
is *already* so widely-used that anything that would conflict with it
has already been fixed.

Can you provide a real example of something that would break if cl.el
were dumped with Emacs?

Re: moving more cl seq/mapping support into core

[Daniel Colascione]

On 9/30/2010 8:16 PM, Miles Bader wrote:
> Daniel Colascione <dan.colascione@gmail.com> writes:
>> CL really should be dumped along with Emacs proper.
> 
> cl.el is a combination of good useful things that and horrible ugly
> things (this isn't a comment on the CL language, btw -- many of the
> problems with cl.el are issue with its implementation and consistency
> with elisp).

Can you point to a few examples? Parts of cl are ugly (the common use of
make-symbol instead of gensym, for example, makes macroexpansions
difficult to read). But those don't affect functionality or interaction
with other packages.

The other issue is the set of extension functions --- defun*, defmacro*
and so on. If your point is that the functionality offered by these
functions should instead of folded into the regular elisp versions, then
that's a good thing, but news to me.

> Of course, there's a solution which would probably make everybody
> happy:  adopt the good things only, and fix some of the problems with
> the horrible ugly things so that they can be adopted.  But people
> advocating cl.el seem to usually stop short of putting any effort into
> doing that...

What are the good things/ What are the "horrible ugly thing"? What
criteria can you use to distinguish them?

Re: moving more cl seq/mapping support into core

[Chong Yidong]

Daniel Colascione <dan.colascione@gmail.com> writes:

> I understood the primary objection to keyword arguments being about
> efficiency.

No; as RMS and Miles have pointed out, it's also a matter of style.

Re: moving more cl seq/mapping support into core

[MON KEY]

Daniel Colascione wrote:
> What are the good things/ What are the "horrible ugly thing"? What
> criteria can you use to distinguish them?

Maybe the better question is, in anticipation of lexical scoping which
of these distinguishing criteria will no longer be relevant?

Whatever the current/existing rationales may be, surely _some_ will
cease to be relevant by any reasonably sane justification in lieu of
lexically scoped environments.

I would very much like to know now _concretely_ what the existing
criteria are so that these may be revisited as the lexbind integration
moves forward.

cl.el is now at least 13 yrs old and the arguments precluding its
integration remain largely the same as they were when Dave Gillespie
advocated cl.el inclusion in core way back when.  In the interim a
good deal has changed with Emacs but the cl.el runtime ban still
lingers... though the justifications have been allowed to shift.

It is important that those willing to revisit/revise the cl.el
deficiencies have a clear roadmap/guideline as to what is acceptable,
why, and what the sane justifications for these guidelines are.

As it is, I've made a preliminary effort to indicate those cl.el like
features which are duplicated in other diverse emacs core libraries.
It is clear that these features are needed/desired or they wouldn't
exist.  What is not clear is why if these features are desirable must
they be reproduced w/ only minor modification simply to avoid the
appearance of haring the cl namespace.
In particular I would point to `edmacro-mismatch' as an example of
such needless bending over backwards.

--
/s_P\

Re: moving more cl seq/mapping support into core

[David Kastrup]

Daniel Colascione <dan.colascione@gmail.com> writes:

> On 9/30/2010 8:16 PM, Miles Bader wrote:
>> Daniel Colascione <dan.colascione@gmail.com> writes:
>>> CL really should be dumped along with Emacs proper.
>> 
>> cl.el is a combination of good useful things that and horrible ugly
>> things (this isn't a comment on the CL language, btw -- many of the
>> problems with cl.el are issue with its implementation and consistency
>> with elisp).
>
> Can you point to a few examples? Parts of cl are ugly (the common use of
> make-symbol instead of gensym, for example, makes macroexpansions
> difficult to read). But those don't affect functionality or interaction
> with other packages.

That should be possible to solve with lex-bind.

-- 
David Kastrup

Re: moving more cl seq/mapping support into core

[Stefan Monnier]

>> Can you point to a few examples? Parts of cl are ugly (the common use of
>> make-symbol instead of gensym, for example, makes macroexpansions
>> difficult to read).

The use of make-symbol is a fairly recent change, to avoid filling the
obarray with an ever-increasing number of temporary symbols.


        Stefan

Re: moving more cl seq/mapping support into core

[Richard Stallman]

    Whatever the current/existing rationales may be, surely _some_ will
    cease to be relevant by any reasonably sane justification in lieu of
    lexically scoped environments.

I don't think that relates to this issue.
This issue is about global names.

Re: moving more cl seq/mapping support into core

[MON KEY]

On Sun, Oct 3, 2010 at 10:03 PM, Richard Stallman <rms@gnu.org> wrote:
>
> I don't think that relates to this issue.

It can't not relate.

>
> This issue is about global names.
>

Here's the thing about the global names "issue" that I find
particularly irksome:

(let ((tf "./some-temp-file"))
  (with-temp-file tf
    (insert
     "(defun reduce (arg1)\n  arg1)"
     "\n(reduce 'bubba)"))
  (byte-compile-file tf)
  (when (get-buffer "*Compile-Log*")
    (display-buffer "*Compile-Log*"))
  (delete-file tf))

Does your *Compile-Log* buffer have this warning:

,----
|
| In reduce:
| some-temp-file:1:8:Warning: function reduce used to take 2+ arguments, now
|     takes 1
| some-temp-file:3:10:Warning: Function `reduce' from cl package called at
|     runtime
|
`----

If so, how does one justify this second warning as a protection of the
global names?

Either `reduce' is a privileged global name, or it is available to me (the user)
to bind without qualification.

So which is it, do I get to bind `reduce' at my leisure and without spurious CL
related byte-compiler warnings, or is the runtime ban a self-perpetuating and
(oft broken) policy?

Note, the third way isn't much better:

"Our regime of "protecting" the global names and the user from herself can
appear at once authoritarian and sloppy. We apologize for any inconvenience this
might cause but it is for the best of all parties concerned that it be
this way."

--
/s_P\

Re: moving more cl seq/mapping support into core

[Daniel Colascione]

On 10/3/2010 7:03 PM, Richard Stallman wrote:
>     Whatever the current/existing rationales may be, surely _some_ will
>     cease to be relevant by any reasonably sane justification in lieu of
>     lexically scoped environments.
> 
> I don't think that relates to this issue.
> This issue is about global names.
> 

_What_ global names? cl.el is _already_ widely used. What symbols
actually cause a problem _in practice_, _today_?

Re: moving more cl seq/mapping support into core

[Daniel Colascione]

On 10/3/2010 4:15 PM, Stefan Monnier wrote:
>>> Can you point to a few examples? Parts of cl are ugly (the common use of
>>> make-symbol instead of gensym, for example, makes macroexpansions
>>> difficult to read).
> 
> The use of make-symbol is a fairly recent change, to avoid filling the
> obarray with an ever-increasing number of temporary symbols.

But gensymed symbols aren't interned and should be GCed when no longer
referenced. I thought make-symbol was used just to avoid having to cons
up a new string for the symbol name.

Regardless, it'd be nice to be able to switch that off to make some
macro-expansions legible (or maybe I should write a tree walker that
disambiguates identically-named but distinct symbols).

Re: moving more cl seq/mapping support into core

[Richard Stallman]

    _What_ global names? cl.el is _already_ widely used. What symbols
    actually cause a problem _in practice_, _today_?

How would I know?  We're talking about user's own programs here.

The fact that many programmers use cl.el is not conclusive.  Those
programmers won't use cl names for anything else -- others might.

Meanwhile, there is a further issue.  If we move these symbols into
the standard Elisp, we need to document them in the Lisp Reference
manual.  We could certainly do that, with some work.  But it would
also make the Elisp manual bigger and more expensive.

Re: moving more cl seq/mapping support into core

[Helmut Eller]

* Richard Stallman [2010-10-05 09:55] writes:

> Meanwhile, there is a further issue.  If we move these symbols into
> the standard Elisp, we need to document them in the Lisp Reference
> manual.  We could certainly do that, with some work.  But it would
> also make the Elisp manual bigger and more expensive.

There is already a manual in doc/misc/cl.texi.

Helmut

Re: moving more cl seq/mapping support into core

[Ted Zlatanov]

On Tue, 05 Oct 2010 05:55:45 -0400 Richard Stallman <rms@gnu.org> wrote: 

RS> The fact that many programmers use cl.el is not conclusive.  Those
RS> programmers won't use cl names for anything else -- others might.

My experience as a spectator has been that almost all ELisp programmers
are aware of cl.el and don't redefine its functions.

FWIW, I am strongly in favor of making cl.el a first-class citizen.  The
pain of the promotion is outweighed by the benefits IMO.

RS> Meanwhile, there is a further issue.  If we move these symbols into
RS> the standard Elisp, we need to document them in the Lisp Reference
RS> manual.  We could certainly do that, with some work.  But it would
RS> also make the Elisp manual bigger and more expensive.

I hope that's not an impediment.  Maybe it can be an optional extra
volume to the ELisp manual.

Ted

Re: moving more cl seq/mapping support into core

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Tue, 05 Oct 2010 05:55:45 -0400
> Cc: monkey@sandpframing.com, emacs-devel@gnu.org
> 
> Meanwhile, there is a further issue.  If we move these symbols into
> the standard Elisp, we need to document them in the Lisp Reference
> manual.  We could certainly do that, with some work.  But it would
> also make the Elisp manual bigger and more expensive.

That issue can be solved like we did in the Emacs manual: have parts
of it that are only generated in the on-line formats, but not in the
printed copy.

Re: moving more cl seq/mapping support into core

[Eli Zaretskii]

> From: Helmut Eller <eller.helmut@gmail.com>
> Date: Tue, 05 Oct 2010 12:20:45 +0200
> 
> * Richard Stallman [2010-10-05 09:55] writes:
> 
> > Meanwhile, there is a further issue.  If we move these symbols into
> > the standard Elisp, we need to document them in the Lisp Reference
> > manual.  We could certainly do that, with some work.  But it would
> > also make the Elisp manual bigger and more expensive.
> 
> There is already a manual in doc/misc/cl.texi.

Which is not part of the ELisp manual.  Richard's concern is for the
ELisp manual growing too large and too expensive, not with the need to
write the relevant chapters.

Re: moving more cl seq/mapping support into core

[Stefan Monnier]

>>>> Can you point to a few examples? Parts of cl are ugly (the common use of
>>>> make-symbol instead of gensym, for example, makes macroexpansions
>>>> difficult to read).
>> The use of make-symbol is a fairly recent change, to avoid filling the
>> obarray with an ever-increasing number of temporary symbols.
> But gensymed symbols aren't interned and should be GCed when no longer
> referenced. I thought make-symbol was used just to avoid having to cons
> up a new string for the symbol name.

Oh, you're right.  Yes, it's because it's silly to waste time and space
changing the symbol's names.

> Regardless, it'd be nice to be able to switch that off to make some
> macro-expansions legible (or maybe I should write a tree walker that
> disambiguates identically-named but distinct symbols).

The problem is the printed representation, not the use of make-symbol.
The printed representation should use #= and friends to make it clear
which symbols are uninterned and which ones are identical or not.


        Stefan

Re: moving more cl seq/mapping support into core

[MON KEY]

On Mon, Oct 4, 2010 at 1:33 PM, Daniel Colascione
<dan.colascione@gmail.com> wrote:
> On 10/3/2010 7:03 PM, Richard Stallman wrote:
>>     Whatever the current/existing rationales may be, surely _some_ will
>>     cease to be relevant by any reasonably sane justification in lieu of
>>     lexically scoped environments.
>>
>> I don't think that relates to this issue.
>> This issue is about global names.
>>
>
> _What_ global names? cl.el is _already_ widely used. What symbols
> actually cause a problem _in practice_, _today_?
>

Its tautological... CL symbols which name a function exist with global
names and therefor can not be allowed to pollute the global name
space.

As indicated with my earlier example, when `byte-compile-warnings' is
non-nil and '(byte-compile-warning-enabled-p 'cl-functions)' returns
non-nil (per a symbol's membership in `byte-compile-cl-functions'),
then byte-compilation signals a warning that the symbol was called at
runtime (regardless of when/where that symbol was actually defined).

IOW, the CL functions exists in a state of privileged limbo in that
they can not be called at runtime but nor are they allowed to be
redefined at compile time...

IIUC this cycle is a product of the current byte-compilers relatively
flat environment which is what prompted the assertion:

,----
| Whatever the current/existing rationales may be, surely _some_ will
| cease to be relevant by any reasonably sane justification in lieu of
| lexically scoped environments.
`----

The lexbind promotes less flat environments with its `heap environment
vectors' see for example the variables:

 `byte-compile-lexical-environment',
 `byte-compile-current-heap-environment',
 `byte-compile-current-num-closures', `byte-stack-ref',
 `byte-stack-set', `byte-stack-set2', `byte-vec-ref', `byte-vec-set',
 `byte-discardN', `byte-discardN-preserve-tos'

And employs an entirely different compile time expansion of macro
environments e.g. the constant
`byte-compile-initial-macro-environment'. The lexbind byte-compiler
handles these environments and those established by lexical closures
in new ways. This includes modification of the existing
`byte-compile-lambda' with a significantly new re-direction through
`byte-compile-closure', `byte-compile-make-closure',
`byte-compile-make-lambda-lexenv', etc. Indeed, whenever an anonymous
function occurs inside a lexical-binding all compile-time variables
are affected by use of separate 'heapenv' heap environment vectors.

Whatever. I'm sure some of the above is very much in flux in
Stefan/Miles' private branches and likely to change before those
emerge (presumably circa Spring 2011).

The intent of my above referenced query is to learn how much of the
global names argument will remain relevant for those CL symbols (and
their locals) which currently "pollute" the global names but which
(if adapted in lieu of lexbind) might have some not insignificant
portion of this pollution encapsulated with closed over 'lexenv'
environments including any byte-compiler optimizations over the
'lexvars' of these environments.

More specifically, if some aspect(s) of the above are indeed
possible/reasonable and the pollution might be reduced, I'm curious to
learn if those CL functions which accept keywords could be abstracted
in such a way that they might be included in core without constraints
on these necessarily being "keywordless" but instead that there be a
proviso for requisite compile-time analysis capable of determining
whether the function body expects non-keyword arguments or instead a
keyword variant.

This would:

 - Allow backwards compatibility with existing uses of CL functions;

 - Provide an opportunity to revisit certain places where the existing
   cl.el implementation is regarded deficient/sub-optimal;

 - Provide 3rd party developers the opportunity adapt existing code to
   use a 'lighter' non-keyword CL variants where appropriate;

 - Acknowledges Stefan's distaste of maintaining distinct symbol-names
   for the Emacs lisp core function vs the CL counterpart, e.g. push,
   pop, dotimes, dolist, etc.;

  - Ease documentation concerns because portions of the existing CL
    info could be incorporated without requiring that all references
    to the keyword usage be gutted. Instead, documentation for the
    default non-keyword version would be added along with
    clarification of the differences. Given that Emacs installs with
    the cl.info the this should help keep the expense of documentation
    overhead down. (Assumes there aren't lingering arm-waving
    intentions about some vague copyright restriction w/re the cl.info
    manual and the ANSI spec);

It is apropos the possibility that some of the above hypotheticals
might have relevance/traction that I asserted previously:

,----
| It is important that those willing to revisit/revise the cl.el
| deficiencies have a clear roadmap/guideline as to what is acceptable,
| why, and what the sane justifications for these guidelines are.
`----

Right now there are numerous alternative and (functionally divergent)
implementations of the CL functions which often have been written to
not trigger signalling of CL related runtime byte-compiler
warnings. I've pointed out some that are already in core.

Note, none of these appear to have been written in a spirit of:

 "Oh, I wrote this because I don't want all of those CL features."

Indeed, most of the packages which employ cl feature duplication do
rely elsewhere on CL macros, but these don't trigger the runtime
warnings.

So, as an example, one finds that prior to Bzr-98091 of 2009-10-11
eieio.el's `eieio-add-new-slot' called `union' twice, each time inside
deeply nested conditionals. These `union' calls were responsible for
informing the adjustment of class slot attributes pertinent to
custom-group classes. Who knows if anyone is using this stuff (what with
it being custom and all).  Regardless, both of these calls have since
been adapted by virtue of a "manual inlining"... The first such
`union' call involves a tweak of an index into
`class-public-custom-group'. It is triggered when `eieio-add-new-slot'
optional arg DEFAULTOVERRIDE is non-nil. The second also is a similar
tweak triggered in the same manner but instead finds a takes a union
of an index into `class-class-allocation-custom-group'. Now,
presumably user code won't rely on these tweaks occurring all that
frequently as the DEFAULTOVERRIDE arg is the first optional arg in the
`eieio-add-new-slot' lambda-list. Indeed, it follows 11 other
obviously more important parameters e.g.:

 (1- (length '(newc a d doc type cust label custg print
               prot init alloc &optional defaultoverride skipnil)))

and can't possibly be all that important in its priveleged 12th
position given that the _eleven_ preceding args are mandatory...
 "Nah, we don't need no stinkin' keywords."  Or not.

Instance one of `union's demise by duplication:

,---- lisp/emacs-lisp/eieio.el `eieio-add-new-slot' @Bzr-98018
|  (when custg
|    (let ((where-groups
|           (nthcdr num (aref newc class-public-custom-group))))
|      (setcar where-groups
|              (union (car where-groups)
|                     (if (listp custg) custg (list custg))))))
`----

,---- lisp/emacs-lisp/eieio.el `eieio-add-new-slot' @Bzr-101790
| (when custg
|   (let* ((groups
|           (nthcdr num (aref newc class-public-custom-group)))
|          (list1 (car groups))
|          (list2 (if (listp custg) custg (list custg))))
|     (if (< (length list1) (length list2))
|         (setq list1 (prog1 list2 (setq list2 list1))))
|     (dolist (elt list2)
|       (unless (memq elt list1)
|         (push elt list1)))
|     (setcar groups list1)))
`----

Note, the only discernible difference between the chunks above and
those below are the let bindings of the local `groups` var to an index
into `class-class-allocation-custom-group' instead of
`class-public-custom-group', otherwise they are identical.

Instance two of `union's demise by duplication:

,---- lisp/emacs-lisp/eieio.el `eieio-add-new-slot' @Bzr-98018
| (when custg
|   (let ((where-groups
|          (nthcdr num (aref newc class-class-allocation-custom-group))))
|     (setcar where-groups
|             (union (car where-groups)
|                    (if (listp custg) custg (list custg))))))
`----

,---- lisp/emacs-lisp/eieio.el `eieio-add-new-slot' @Bzr-101790
| (when custg
|   (let* ((groups
|           (nthcdr num (aref newc class-class-allocation-custom-group)))
|          (list1 (car groups))
|          (list2 (if (listp custg) custg (list custg))))
|     (if (< (length list1) (length list2))
|         (setq list1 (prog1 list2 (setq list2 list1))))
|     (dolist (elt list2)
|       (unless (memq elt list1)
|         (push elt list1)))
|     (setcar groups list1)))
`----

Now for a game of "one of these things looks alot like the other one":

,---- lisp/emacs-lisp/cl-seq.el `union' @Bzr-101804
|
| (cond ((null cl-list1) cl-list2) ((null cl-list2) cl-list1)
|       ((equal cl-list1 cl-list2) cl-list1)
|       (t
|        (or (>= (length cl-list1) (length cl-list2))
|            (setq cl-list1 (prog1 cl-list2 (setq cl-list2 cl-list1))))
|        (while cl-list2
|          (if (or cl-keys (numberp (car cl-list2)))
|              (setq cl-list1
|                    (apply 'adjoin (car cl-list2) cl-list1 cl-keys))
|            (or (memq (car cl-list2) cl-list1)
|                (push (car cl-list2) cl-list1)))
|          (pop cl-list2))
|        cl-list1))
|
`----

In the absence of a change to the current runtime ban more of the
above will continue to spring up.  Now, no doubt when Chong Yidong
must be bothered to implement this sort of hackery one can expect a
_correct_ hack. Right? Its not like the above isn't mostly a
copy/paste job into an unbelievably deeply chunk of nested
nastiness...

The "problem" though is that many such like workarounds aren't
instrumented courtesy Chong/Stefan/Richard et al, but instead by 3rd
parties whom none-the-less _are_ making attempts to comply with the CL
function runtime ban. It isn't enough for emacs-devels to dismiss
those parties (as they sometimes do) with:

 "Well, either ignore the byte-compiler warnings if they bother
  you so much or don't use the CL features in your library"

This is insults both the 3rd party developer and any party whom
attempts installing the third party library. When byte-compilation of
the 3rd party code signals a bunch of byte-compiler warnings it casts
doubt or creates fear and uncertainty and may make it appear as if the
library is deficient/suspect. All else being equal, the cumulative
effect of these byte-compiler warnings is a form of discriminatory
rankism which privileges the Emacs core libraries which in turn
promotes further entrenchment of the CL runtime ban, e.g.:

 "None of the core libraries require/rely on CL functions which
  pollute the global names in order to get things done."

Indeed, they don't - most likely they just implement it in C instead.

This isn't an option to 3rd party developers. So, to avoid the
_appearance_ of being "unclean" 3rd parties often either:

- copy the CL function definitions verbatim
  (which _is_ itself a colossal pollution of the "global names");

- attempt to (re)implement the portions of the CL feature set they
  would otherwise use were the ban not in place

It is clear that both of the above choices are and have been occurring.

With regards the re-implementation approach, this is where the CL
runtime ban fails to benefit both the community of Emacs users and
3rd party authors in the "real world" because (re)implementing any
subset of the CL feature is:

- hard to accomplish in the absence of runtime usage of the CL
  features, i.e. the act itself lands one in the Turing Tar Pit;

- a waste of energy rebuilding the existent usable, distributed, and
  tested featureset esp. as it at least attempts adherence to a
  formally peer reviewed specification;

- likely won't benefit from the same byte-compiler optimizations as a
  cl-*.el counterpart;

- apt to error in unexpected/unforeseen non-standard ways;

This is grossly unfortunate because it is clear that despite these
obstacles 3rd parties do endeavor re-implementing workarounds to the
CL runtime ban despite Philip Greenspun's "Tenth Rule" leering back at
them.  The gross aspect of this circumstance being that "Greenspun's
Tenth" is most likely a precept well known to the unfortunate
attempting to recapture some semblance of the Common Lisp so denied...

By way of an example we have the following:

,----
| From: 	Stefan Monnier
| Subject: Re: testing framework and package.el
| Date: 	Tue, 28 Sep 2010 00:57:51 +0200
| (URL `http://lists.gnu.org/archive/html/emacs-devel/2010-09/msg01500.html')
|
| IIRC we've agreed to try and install ERT, either on elpa or in Emacs
| itself, and then try and move the few tests we already have
| scattered about to use ERT.  But I haven't heard much about it
| recently
`----

Taking a look at ert.el package on Christian Ohler's git repo:
 (URL `git://github.com/ohler/ert.git')
one finds yet more CL feature duplication attempts. These presumably
queued up for inclusion in Emacs proper in the not so distant future.
Below are the CL "duplicate" features from ert/lisp/emacs-lisp/ert.el

The features `ert--cl-do-remf' and `ert--remprop' are verbatim copies
of `cl-do-remf'. The features `ert--remove-if-not',
`ert--intersection', `ert--set-difference', `ert--set-difference-eq',
and `ert--union' are keywordless re-implementations of CL features
which leverage the `loop' macro to mimic the CL features
`intersection', `set-difference', and `union'.

It should be noted however that (with the exception of `ert--union')
these are not at all faithful duplicates of either the CL features or
their ANSI brethren as they fail to check that their args are
proper-lists/sequences...

Now, I'm given to believe that Ohler is not only an accomplished hack,
but also prob. one well aware of the Common Lisp ANSI spec. So his
floundering on the proper-list/proper-seq thing is prob. indicative of
something.

Does Ohler believe that his CL duplicates are better for their
non-conformance and breaking with both the semantics of both the CL
features and the Common Lisp features? Maybe he finds his duplicates
achieve a desirable performance gain. Or, maybe he just doesn't feel it
worth the trouble to rewrite _all_ of cl*.el.

This said, that ERT is slated as the "blessed" Emacs unit test
framework begs a more important question.

If Ohler implemented his CL feature duplications as a kludge would he
have simply done a (require 'cl) and been done with it were it not for
the CL runtime ban?

And if so, would ERT be better/more correct by so doing?

Whatever. Say hello to the latest iteration of a Lisper doing his best
to suffer a foolish policy.

,---- ert/lisp/emacs-lisp/ert.el (URL `http://github.com/ohler/ert')
|
|
| (defun ert--cl-do-remf (plist tag)
|   "Copy of `cl-do-remf'.  Modify PLIST by removing TAG."
|   (let ((p (cdr plist)))
|     (while (and (cdr p) (not (eq (car (cdr p)) tag)))
|             (setq p (cdr (cdr p))))
|     (and (cdr p) (progn (setcdr p (cdr (cdr (cdr p)))) t))))
|
| (defun ert--remprop (sym tag)
|   "Copy of `cl-remprop'.  Modify SYM's plist by removing TAG."
|   (let ((plist (symbol-plist sym)))
|     (if (and plist (eq tag (car plist)))
| 	(progn (setplist sym (cdr (cdr plist))) t)
|       (ert--cl-do-remf plist tag))))
|
| ;; :NOTE Vector agnostic.
| ;; (ert--remove-if-not 'stringp ["a" "b" c])  ;=> nil
| ;; (ert--remove-if-not 'stringp '("a" "b" c)) ;=> ("a" "b")
| ;; (remove-if-not 'stringp ["a" "b" c])       ;=> ["a" "b"]
| ;; (remove-if-not 'stringp '("a" "b" c))      ;=> ("a" "b")
| (defun ert--remove-if-not (ert-pred ert-list)
|   "A reimplementation of `remove-if-not'.
| ERT-PRED is a predicate, ERT-LIST is the input list."
|   (loop for ert-x in ert-list
|         if (funcall ert-pred ert-x)
|         collect ert-x))
|
| ;; :NOTE `set-difference' w/ Common Lisp's default implicit 'eql test.
| ;; Mostly equivalent to cl-seq.el's `intersection'
| ;;  (intersection '(a b) '(c a) :test 'eql)
| ;; But,  does not check that args are proper lists:
| ;; (ert--intersection '(a . c) '(b   d)) ;=> nil
| ;; (intersection '(a . c) '(b   d)) ;=> (wrong-type-argument listp c)
| (defun ert--intersection (a b)
|   "A reimplementation of `intersection'.  Intersect the sets A and B.
| Elements are compared using `eql'."
|   (loop for x in a
|         if (memql x b)
|         collect x))
|
| ;; :NOTE `set-difference' w/ Common Lisp's default implicit 'eql test.
| ;; Mostly equivalent to cl-seq.el's `set-difference':
| ;;  (set-difference a b :test 'eql)
| ;; But, does not check that args are proper lists:
| ;;  (ert--set-difference '(a . c) '(b   d)) ;=> (a)
| ;;  (set-difference '(a . c) '(b   d) :test 'eql)
| ;;  => (wrong-type-argument listp c)
| (defun ert--set-difference (a b)
|   "A reimplementation of `set-difference'.
| Subtract the set B from the set A.
| Elements are compared using `eql'."
|   (loop for x in a
|         unless (memql x b)
|         collect x))
|
| ;; :NOTE This is roughly cl.el's `set-difference' w/ the implicit
| ;; 'eq test.
| ;; Mostly equivalent to Common Lisp's: (set-difference a b :test 'eq)
| ;; But, does not check that args are proper lists:
| ;; (ert--set-difference-eq '(a . c) '(b   d)) ;=> (a)
| ;; (set-difference '(a . c)  '(b   d)) ;=> (wrong-type-argument listp c)
| (defun ert--set-difference-eq (a b)
|   "A reimplementation of `set-difference'.
| Subtract the set B from the set A.
| Elements are compared using `eq'."
|   (loop for x in a
|         unless (memq x b)
|         collect x))
|
| ;; :NOTE Unlike above doesn't error when args not proper lists
| ;; (ert--union '(a . c) '(b   d)) ;=> (wrong-type-argument listp c)
| (defun ert--union (a b)
|   "A reimplementation of `union'.
| Compute the union of the sets A and B.
| Elements are compared using `eql'."
|   (append a (ert--set-difference b a)))
|
| (eval-and-compile
|   (defvar ert--gensym-counter 0))
|
| ;; :NOTE Does note accept integer for optional arg PREFIX.
| (eval-and-compile
|   (defun ert--gensym (&optional prefix)
|     "Only allows string PREFIX, not compatible with CL."
|     (unless prefix (setq prefix "G"))
|     (make-symbol (format "%s%s"
|                          prefix
|                          (prog1 ert--gensym-counter
|                            (incf ert--gensym-counter))))))
|
| ;; :NOTE Could chang parameter list to ( ... &optional key test)
| (defun* ert--remove* (x list &key key test)
|   "Does not support all the keywords of remove*."
|   (unless key (setq key #'identity))
|   (unless test (setq test #'eql))
|   (loop for y in list
|         unless (funcall test x (funcall key y))
|         collect y))
|
| (defun ert--mismatch (a b)
|   "Return index of first element that differs between A and B.
| Like `mismatch'.  Uses `equal' for comparison."
|   (cond ((or (listp a) (listp b))
|          (ert--mismatch (ert--coerce-to-vector a)
|                         (ert--coerce-to-vector b)))
|         ((> (length a) (length b))
|          (ert--mismatch b a))
|         (t
|          (let ((la (length a))
|                (lb (length b)))
|            (assert (arrayp a) t)
|            (assert (arrayp b) t)
|            (assert (<= la lb) t)
|            (loop for i below la
|                  when (not (equal (aref a i) (aref b i))) return i
|                  finally (return (if (/= la lb)
|                                      la
|                                    (assert (equal a b) t)
|                                    nil)))))))
|
| (defun ert--subseq (seq start &optional end)
|   "Returns a subsequence of SEQ from START to END."
|   (when (char-table-p seq) (error "Not supported"))
|   (let ((vector (substring (ert--coerce-to-vector seq) start end)))
|     (etypecase seq
|       (vector vector)
|       (string (concat vector))
|       (list (append vector nil))
|       (bool-vector (loop with result = (make-bool-vector (length vector) nil)
|                          for i below (length vector) do
|                          (setf (aref result i) (aref vector i))
|                          finally (return result)))
|       (char-table (assert nil)))))
|
| (defun ert--special-operator-p (thing)
|   "Return non-nil if THING is a symbol naming a special operator."
|   (and (symbolp thing)
|        (let ((definition (indirect-function thing t)))
|          (and (subrp definition)
|               (eql (cdr (subr-arity definition)) 'unevalled)))))
|
`----

--
/s_P\

Re: moving more cl seq/mapping support into core

[Richard Stallman]

    Here's the thing about the global names "issue" that I find
    particularly irksome:

    (let ((tf "./some-temp-file"))
      (with-temp-file tf
	(insert
	 "(defun reduce (arg1)\n  arg1)"
	 "\n(reduce 'bubba)"))
      (byte-compile-file tf)
      (when (get-buffer "*Compile-Log*")
	(display-buffer "*Compile-Log*"))
      (delete-file tf))

This writes a file that defines `reduce' globally.

    | some-temp-file:1:8:Warning: function reduce used to take 2+ arguments, now
    |     takes 1
    | some-temp-file:3:10:Warning: Function `reduce' from cl package called at
    |     runtime

I guess you had already loaded cl.

The purpose of this warning is to encourage people not to write their code
to call cl at run time.

I think it was intended to operate in files that load cl at compile
time and don't require cl at run time.  The occurrence of the message
in this case seems like a bug.

Re: moving more cl seq/mapping support into core

[Stephen J. Turnbull]

Richard Stallman writes:

 > The fact that many programmers use cl.el is not conclusive.  Those
 > programmers won't use cl names for anything else -- others might.

XEmacs has been dumping cl for at least a decade, with no complaints I
can recall.  FWIW YMMV

Re: moving more cl seq/mapping support into core

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Richard Stallman writes:
>
>  > The fact that many programmers use cl.el is not conclusive.  Those
>  > programmers won't use cl names for anything else -- others might.
>
> XEmacs has been dumping cl for at least a decade, with no complaints I
> can recall.  FWIW YMMV

My main complaint is that I usually try fixing comprehension problems by
reading the source.

cl's existing DOC strings are often semiobfuscate, requiring a certain
degree of "common Lisp" thinkage in order to make sense of.  The cl code
itself is more or less completely inscrutable.  Partly because of its
nature of macro expansion, partly because of coding techniques/styles
and internals that are neither explained with code comments nor
self-explaining.

It does not tie in nicely with Emacs Lisp and promotes a programming
style that does not map straightforwardly to the Lisp machine and makes
debugging and streamlining haphazard.

My personal gut feeling would be to _first_ give Emacs well-working
lexical binding.  Then more or less reimplement cl from scratch.  If
things like keyword parsing would profit from supporting C functions,
create those functions instead of fiddling with macros all around.  Make
sure that the bytecode has everything to support cl sanely rather than
as a clever afterthought.

-- 
David Kastrup

Re: moving more cl seq/mapping support into core

[Richard Stallman]

    > Meanwhile, there is a further issue.  If we move these symbols into
    > the standard Elisp, we need to document them in the Lisp Reference
    > manual.  We could certainly do that, with some work.  But it would
    > also make the Elisp manual bigger and more expensive.

    There is already a manual in doc/misc/cl.texi.

It would need to be integrated into the Lisp Reference Manual
to make that manual do its job.  And maybe partly rewritten
to make its style fit (I have not checked how much of that
would be needed).

Re: moving more cl seq/mapping support into core

[Richard Stallman]

    I hope that's not an impediment.  Maybe it can be an optional extra
    volume to the ELisp manual.

If they will be used often, putting them in an extra volume
is not a good solution.

Likewise, omitting them from the printed edition is not a solution.

Re: moving more cl seq/mapping support into core

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 557 bytes --]

On 10/6/10 4:41 PM, Richard Stallman wrote:
>     I hope that's not an impediment.  Maybe it can be an optional extra
>     volume to the ELisp manual.
> 
> If they will be used often, putting them in an extra volume
> is not a good solution.
> 
> Likewise, omitting them from the printed edition is not a solution.

If any new feature must be documented, and the documentation cannot
exceed a certain size, then new features cannot be added. Are you
proposing to cap the size of Emacs? Or does this logic only apply to
things named "cl.el"?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: moving more cl seq/mapping support into core

[Ted Zlatanov]

On Wed, 06 Oct 2010 19:41:13 -0400 Richard Stallman <rms@gnu.org> wrote: 

>> Meanwhile, there is a further issue.  If we move these symbols into
>> the standard Elisp, we need to document them in the Lisp Reference
>> manual.  We could certainly do that, with some work.  But it would
>> also make the Elisp manual bigger and more expensive.

RS>     There is already a manual in doc/misc/cl.texi.

RS> It would need to be integrated into the Lisp Reference Manual
RS> to make that manual do its job.  And maybe partly rewritten
RS> to make its style fit (I have not checked how much of that
RS> would be needed).

Many times features are either unexplained or it says to "look at the
examples."  For people who know CL or have experience with cl.el already
it's a pretty good reference but for me the first time was a lot of back
and forth browsing.

In terms of organization it's pretty good.  I found it very readable
when I looked at it.  But if it's integrated within the larger manual,
many functions will need to at least get linked in the relevant sections
(e.g. cl.el sequence functions should be mentioned in the 
"(elisp) Sequences Arrays Vectors" node).

Ted

Re: moving more cl seq/mapping support into core

[Karl Fogel]

Richard Stallman <rms@gnu.org> writes:
>It would need to be integrated into the Lisp Reference Manual
>to make that manual do its job.  And maybe partly rewritten
>to make its style fit (I have not checked how much of that
>would be needed).

Is the size of the reference manual really a concern anymore?  I ordered
a print one years ago; I never use it.  I use exclusively the electronic
copy, and suspect the same is true of most (perhaps even all?) Emacs
Lisp programmers.

Re: moving more cl seq/mapping support into core

[Christian Ohler]

  On 6/10/10 9:59, MON KEY wrote:
> Does Ohler believe that his CL duplicates are better for their
> non-conformance and breaking with both the semantics of both the CL
> features and the Common Lisp features? Maybe he finds his duplicates
> achieve a desirable performance gain. Or, maybe he just doesn't feel it
> worth the trouble to rewrite _all_ of cl*.el.

Because a run-time dependency on cl.el would make it hard for ERT to 
become part of Emacs and triggers distracting warnings, I reimplemented 
the subset of CL that ERT requires.  I don't think this subset is better 
than cl.el, and I haven't checked which implementation is faster.

> If Ohler implemented his CL feature duplications as a kludge would he
> have simply done a (require 'cl) and been done with it were it not for
> the CL runtime ban?

Yes.

> And if so, would ERT be better/more correct by so doing?

Better?  Probably, since I could have spent my time on other features, 
and the definitions in cl.el may well be faster than my reimplementations.

More correct?  I'm not sure.  You are examining the behavior of 
`ert--remove-if-not' etc. for Common Lisp compliance; but there is no 
reason why it would have to be, since these functions are just private 
utilities in ERT.  They only provide the subset of CL functionality that 
ERT requires.  They either document that they only work on lists, or 
signal an error on sequence types that they don't support.

The check for dotted lists that you are looking for is guaranteed in 
Common Lisp only in what Common Lisp calls "safe code".  In safe code, 
Common Lisp also guarantees this check for `loop' and `member'; but 
`loop' and `member' in Emacs Lisp don't have this check.  So, ERT's 
CL-like utility functions behave like unsafe code in this regard, just 
like other Emacs Lisp functions.

If you have a concrete reason to believe that these functions lead to 
bugs in ERT, please let me know.

Reimplementing these CL utility functions (in 
http://github.com/ohler/ert/commit/cdc3dfad4779cb9421a6711f6ef7be522ca8e5b0 
) was actually only part of the work I had to do to avoid a run-time 
dependency on CL.  I also had to implement a special-purpose replacement 
for `typep' in 
http://github.com/ohler/ert/commit/bde37b42d0187356cfd747d5ec245b30690c2fcf 
, and the trickiest part was making ERT's call to `macroexpand' work 
regardless of whether cl.el is loaded, see 
http://github.com/ohler/ert/commit/f481a5c2b38a8ccd06b9ddd9c7243409fb583ea2 
: If cl.el is loaded, ERT needs to pass two arguments to `macroexpand', 
only one otherwise.  The solution is simple, but figuring out that this 
problem exists took me a while.

Christian.

Re: moving more cl seq/mapping support into core

[Miles Bader]

Christian Ohler <ohler+emacs@fastmail.net> writes:
> : If cl.el is loaded, ERT needs to pass two arguments to
> macroexpand', only one otherwise.  The solution is simple, but
> figuring out that this problem exists took me a while.

The normal (non-cl) macroexpand takes the same arguments, so why do
you need to make a distinction?

-Miles

-- 
((lambda (x) (list x x)) (lambda (x) (list x x)))

Re: moving more cl seq/mapping support into core

[Christian Ohler]

  On 8/10/10 11:48, Miles Bader wrote:
> Christian Ohler<ohler+emacs@fastmail.net>  writes:
>> : If cl.el is loaded, ERT needs to pass two arguments to
>> macroexpand', only one otherwise.  The solution is simple, but
>> figuring out that this problem exists took me a while.
> The normal (non-cl) macroexpand takes the same arguments, so why do
> you need to make a distinction?


It takes an environment argument, but I don't know where to get an 
environment value if cl.el is not loaded.  Originally, I defined my 
macros with defmacro* and &environment, but such macros can only be 
expanded when cl.el is loaded, since their definition references 
`cl-macro-environment', which is only bound with cl.el.  So I got rid of 
&environment and instead read `cl-macro-environment' directly if it is 
bound.

It's quite possible that this solution is not quite right or that 
something even simpler works; I don't understand these mechanisms in detail.

Looking at it again, I wonder if the following statement in one of ERT's 
comments is correct:

   ;; If `cl-macroexpand' isn't bound, the code that we're
   ;; compiling doesn't depend on cl and thus doesn't need an
   ;; environment arg for `macroexpand'.

Is it true that we don't need an environment arg?  Or do we have to get 
the byte-compiler's environment and pass it in?

Christian.

Re: moving more cl seq/mapping support into core

[Richard Stallman]

    If any new feature must be documented, and the documentation cannot
    exceed a certain size, then new features cannot be added. Are you
    proposing to cap the size of Emacs? Or does this logic only apply to
    things named "cl.el"?

Obviously neither one.  You are not responding to what I said.
Instead you invented two ways of taking it to an extreme, and attacked
them.  That's mean-spirited as well as unconstructive.

There are many differences between possible new Emacs Lisp features as
regards how much documentation they need and where it is useful to put
that documentation.  Some features are rather specialized and can
perfectly well be left out of the printed manual.  For others the
needed documentation can be short.  CL, however, is complex and would
be meant for widespread use in all sorts of Lisp programs.

Thus, this is a significant issue for CL and might be insignificant
for other extensions.

We might consider installing just part of the CL features
and looking for a concise (but clear) way to document them.

Re: moving more cl seq/mapping support into core

[MON KEY]

On Wed, Oct 6, 2010 at 1:21 AM, Richard Stallman <rms@gnu.org> wrote:
>
> I guess you had already loaded cl.
>

Yes, apparently so. Indeed, the warning does not appear with an "emacs -Q"

> The purpose of this warning is to encourage people not to write their code
> to call cl at run time.
>
> I think it was intended to operate in files that load cl at compile
> time and don't require cl at run time.

OK. But, the CL wasn't called at runtime. I've not seen a similar such
warning when byte-compiling files that redefine other functions
formally globally defined from some other place. AFAIK this only
occurs with the CL's.

>
>  The occurrence of the message in this case seems like a bug.
>

I don't have the impression that this is a bug per se. Its fairly
clear looking at the sources of emacs-lisp/bytecomp.el that this is an
intended (albeit unfortunate) outcome of `byte-compile-cl-warn',
`byte-compile-cl-file-p', `byte-compile-eval'.

What I find troubling is that:

- the warning occurs opaquely and separate from the locus of its
  intended target (it affects the user regardless of whether she is
  the package author);

- the distinction w/re the "global names" is made specific to CL at
  the byte-compiler level.

With regards the latter my impression is that the distinction is made
manifest at such a low-level _because_ it is primarily a
philosophical/political design-decision rather than as a requirement
of the LispMachine.

What is irksome is that the distinction, being at once low-level and
by proxy cumulatively entrenched, both allows and promotes a
conflation of the design-decision aspects of the runtime ban with the
functional requirements of the LispMachine in such a way as to
disincentivise efforts to attempt decoupling the historically abstract
vs. any contemporary practical concerns and intentions around a CL
feature(s).  Whether this outcome has arisen from a conscious
intent/desire to prevent/deter incorporation of Common-Lisp like
features into Emacs Lisp or is the only an inadverdent effect the
result is the same.

--
/s_P\

Re: moving more cl seq/mapping support into core

[Richard Stallman]

    Is the size of the reference manual really a concern anymore? 

Yes -- we sell them and we want to keep selling them.

Re: moving more cl seq/mapping support into core

[Richard Stallman]

    >  The occurrence of the message in this case seems like a bug.
    >

    I don't have the impression that this is a bug per se. Its fairly
    clear looking at the sources of emacs-lisp/bytecomp.el that this is an
    intended (albeit unfortunate) outcome of `byte-compile-cl-warn',

That sounds like a bug to me.