Request: add cl-font-lock to Emacs or Elpa

Request: add cl-font-lock to Emacs or Elpa

[Spenser Truex]

cl-font-lock is a package that does richer syntax highlighting for
Common Lisp. I'm the co-maintainer on it. We are interested in having it
included in Emacs or on Elpa. It is already on Melpa.

https://github.com/cl-font-lock/cl-font-lock

How can we move that along?
-- 
Spenser Truex
spensertruex.com
SFSU Computer Science

Re: Request: add cl-font-lock to Emacs or Elpa

[Eli Zaretskii]

> From: Spenser Truex <lists@spensertruex.com>
> Date: Sat, 14 Mar 2020 23:58:05 -0700
> 
> cl-font-lock is a package that does richer syntax highlighting for
> Common Lisp. I'm the co-maintainer on it. We are interested in having it
> included in Emacs or on Elpa. It is already on Melpa.
> 
> https://github.com/cl-font-lock/cl-font-lock
> 
> How can we move that along?

Is it possible to add this to lisp-mode.el?

Re: Request: add cl-font-lock to Emacs or Elpa

[Stefan Monnier]

> Is it possible to add this to lisp-mode.el?

Indeed, it seems it would make a lot of sense there, tho maybe we don't
want to preload it, so it might make sense to keep it in a separate file
(actually, I think it would also make sense to move some of the
definition of the `lisp-mode` major mode to such a separate file, since
`lisp-mode` (the major-mode) is used only by the a small subset of Emacs
users).

I also wonder how you made those lists?  Have you extracted them from
the CLHS?  If so, it would be good to include (e.g. as comments) the
code&steps that was used.
[ I'm also curious why things like make-method and next-method-p are in
  built-in--symbols rather than in built-in--functions.  ]

Finally, of course, there's the issue of copyright.  I don't see your
name in the copyright.list so it appears you haven't yet signed any
copyright paperwork.  I see Yue Daian as author, but the git log doesn't
mention him.  How does your work relate to his?  Who else contributed?


        Stefan

Re: Request: add cl-font-lock to Emacs or Elpa

[Spenser Truex]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> cl-font-lock is a package that does richer syntax highlighting for
>> Common Lisp. I'm the co-maintainer on it. We are interested in having it
>> included in Emacs or on Elpa. It is already on Melpa.
>>
>> https://github.com/cl-font-lock/cl-font-lock
>>
>> How can we move that along?
>
> Sorry, I saw the (title of the) message you sent a few weeks ago and put
> it into my "todo" but later removed it from my todo because I thought it
> had been handled by someone else (there was some commit soon afterwards
> that seemed related).

Thank you. I hope the package is simple enough to see what it is doing,
so that it can be added without issue.

I think it should be an opt-in feature, not a replacement for the
current syntax highlighting in lisp-mode because many users prefer the
old syntax highlighting. Also, it is not very sophisticated (it is just
regular expressions) and sometimes doesn't highlight right based on the
context.

The other issue with it is that it can't really be unloaded.

So if these issues are too great for it to be added then we will have to
fix them first.

>         Stefan

-- 
Spenser Truex
spensertruex.com
San Francisco, USA

RE: Request: add cl-font-lock to Emacs or Elpa

[Drew Adams]

> I think it should be an opt-in feature, not a replacement for the
> current syntax highlighting in lisp-mode because many users prefer the
> old syntax highlighting. Also, it is not very sophisticated (it is just
> regular expressions) and sometimes doesn't highlight right based on the
> context.

Even without looking at the proposed feature/change,
those statements sound right, to me.

> The other issue with it is that it can't really be unloaded.
> 
> So if these issues are too great for it to be added then we will have to
> fix them first.

I'd also suggest changing the file name from
`cl-font-lock.el' to `font-lock-cl.el'.  This
isn't really part of Common Lisp or its
emulation, and if we get other such code for
other languages then that would naturally use
the same convention: `font-lock-<LANG>.el'.

(No, I don't feel strongly about this.  It's
just something that occurred to me.  Perhaps
there are good counter-arguments.)

Re: Request: add cl-font-lock to Emacs or Elpa

[Spenser]

I’m glad to see no philosophical complaints about this package being added. I will do my best to send updates on development, and I want to make it as elpa-friendly as possible.

> Em 15 de mar de 2020, à(s) 14:01, Drew Adams <drew.adams@oracle.com> escreveu:
> 
>> 
>> I think it should be an opt-in feature, not a replacement for the
>> current syntax highlighting in lisp-mode because many users prefer the
>> old syntax highlighting. Also, it is not very sophisticated (it is just
>> regular expressions) and sometimes doesn't highlight right based on the
>> context.
> 
> Even without looking at the proposed feature/change,
> those statements sound right, to me.
> 
>> The other issue with it is that it can't really be unloaded.
>> 
>> So if these issues are too great for it to be added then we will have to
>> fix them first.
> 
> I'd also suggest changing the file name from
> `cl-font-lock.el' to `font-lock-cl.el'.  This
> isn't really part of Common Lisp or its
> emulation, and if we get other such code for
> other languages then that would naturally use
> the same convention: `font-lock-<LANG>.el'.

We were also asked by Melpa to name change. We are currently doing this. Currently the Github organization and the package have been renamed. Thanks!

> 
> (No, I don't feel strongly about this.  It's
> just something that occurred to me.  Perhaps
> there are good counter-arguments.)

RE: Request: add cl-font-lock to Emacs or Elpa

[Drew Adams]

> > I'd also suggest changing the file name from
> > `cl-font-lock.el' to `font-lock-cl.el'.  This
> > isn't really part of Common Lisp or its
> > emulation, and if we get other such code for
> > other languages then that would naturally use
> > the same convention: `font-lock-<LANG>.el'.
> 
> We were also asked by Melpa to name change. We are currently doing this.
> Currently the Github organization and the package have been renamed. Thanks!

I guess you're saying, by "name change", that the
name will become font-lock-cl.el?

Another reason not to use prefix `cl-', IMO, is
that this is not about emulating Common Lisp
features.  Font-locking is an Emacs feature.

Re: Request: add cl-font-lock to Emacs or Elpa

[Stefan Monnier]

>> I'd also suggest changing the file name from
>> `cl-font-lock.el' to `font-lock-cl.el'.

This is a file that affects `lisp-mode` and not `font-lock` (it's partly
a philosophical choice, but support for "font-lock in mode foo-mode"
traditionally resides in the definition of foo-mode rather than in the
definition of font-lock).

So "cl-<bla>" is preferable to "font-lock-<bla>".

IOW maybe there's a better choice than `cl-font-lock`, but
`font-lock-cl` is definitely not it.


        Stefan

RE: Request: add cl-font-lock to Emacs or Elpa

[Drew Adams]

> >> I'd also suggest changing the file name from
> >> `cl-font-lock.el' to `font-lock-cl.el'.
> 
> This is a file that affects `lisp-mode` and not `font-lock` (it's partly
> a philosophical choice, but support for "font-lock in mode foo-mode"
> traditionally resides in the definition of foo-mode rather than in the
> definition of font-lock).

Ah, yes.  I agree with that.

> So "cl-<bla>" is preferable to "font-lock-<bla>".

Neither looks right to me, now, given your point.

Why isn't this just in lisp-mode.el?  I see your
reply to Eli about that.  In that case, how about
something like lisp-mode-fl.el or lisp-fl.el?

Or if, as you suggested in your reply to Eli, we
moved some of lisp-mode.el to a separate file,
and if it made sense for that moved stuff to be
in the same file as the font-lock stuff, maybe
lisp-extra.el, lisp-misc.el, or something else?

> IOW maybe there's a better choice than `cl-font-lock`, but
> `font-lock-cl` is definitely not it.

Re: Request: add cl-font-lock to Emacs or Elpa

[Stefan Monnier]

> lisp-extra.el, lisp-misc.el, or something else?

In an ideal world, I think the current `lisp/emacs-lisp/lisp-mode.el`
should be called `lisp-misc.el` or some such and we should have a new
`lisp/progmodes/lisp-mode.el` which contains the definition of
`lisp-mode`.

Maybe we can get a bit closer to this ideal by using `lisp/emacs-lisp/lisp.el`
in the role of "lisp-misc.el" (i.e. moving the code that's not specific
to `lisp-mode` to `lisp.el`).


        Stefan

Re: Request: add cl-font-lock to Emacs or Elpa

[YUE Daian]

On 2020-03-17 14:42, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> lisp-extra.el, lisp-misc.el, or something else?
>
> In an ideal world, I think the current `lisp/emacs-lisp/lisp-mode.el`
> should be called `lisp-misc.el` or some such and we should have a new
> `lisp/progmodes/lisp-mode.el` which contains the definition of
> `lisp-mode`.
>
> Maybe we can get a bit closer to this ideal by using `lisp/emacs-lisp/lisp.el`
> in the role of "lisp-misc.el" (i.e. moving the code that's not specific
> to `lisp-mode` to `lisp.el`).
>
>
>         Stefan
>
Hi folks,

Here comes the author of it.

IMHO this should have been part of lisp-mode, since lisp-mode is really
for Common Lisp and this just complements the lacking piece of its font
lock.

Because I was not in this mail list, could anyone please sync with me
the current status?

Are we going to make it into Elpa, or built-in with Emacs?

Re: Request: add cl-font-lock to Emacs or Elpa

[Stefan Monnier]

> IMHO this should have been part of lisp-mode, since lisp-mode is
> really for Common Lisp and this just complements the lacking piece of
> its font lock.

Agreed.

> Because I was not in this mail list, could anyone please sync with me
> the current status?
>
> Are we going to make it into Elpa, or built-in with Emacs?

I think the intention is to add it to Emacs's core.

It will require some tweaks, of course.  But before that there are some
questions which have remained unanswered so far:

On Sun, 15 Mar 202, Stefan Monnier wrote:
> I also wonder how you made those lists?  Have you extracted them from
> the CLHS?  If so, it would be good to include (e.g. as comments) the
> code&steps that was used.
> [ I'm also curious why things like make-method and next-method-p are in
>   built-in--symbols rather than in built-in--functions.  ]
>
> Finally, of course, there's the issue of copyright.  I don't see your
> name in the copyright.list so it appears you haven't yet signed any
> copyright paperwork.  I see Yue Daian as author, but the git log doesn't
> mention him.  How does your work relate to his?  Who else contributed?

The more important ones are in the second paragraph, and the main one
is missing: would the authors be willing to sign the needed
copyright paperwork (AFAIK neither you nor Spenser have signed that
paperwork yet)?


        Stefan

Re: Request: add cl-font-lock to Emacs or Elpa

[YUE Daian]

>> I also wonder how you made those lists?  Have you extracted them from
>> the CLHS?  If so, it would be good to include (e.g. as comments) the
>> code&steps that was used.
>> [ I'm also curious why things like make-method and next-method-p are in
>>   built-in--symbols rather than in built-in--functions.  ]
>>
They are not extracted from CLHS. They are extracted from a SBCL image.
As for the 2 functions...it is because they are "local function/macro",
which was not recognized as "function".

>> Finally, of course, there's the issue of copyright.  I don't see your
>> name in the copyright.list so it appears you haven't yet signed any
>> copyright paperwork.  I see Yue Daian as author, but the git log doesn't
>> mention him.  How does your work relate to his?  Who else contributed?
>
> The more important ones are in the second paragraph, and the main one
> is missing: would the authors be willing to sign the needed
> copyright paperwork (AFAIK neither you nor Spenser have signed that
> paperwork yet)?
>
>
>         Stefan
Spenser suggested me to sign the copyright paperwork.

And I am more than willing to do so.

So...where do we start now? ;-)

Re: Request: add cl-font-lock to Emacs or Elpa

[Stefan Monnier]

>>> I also wonder how you made those lists?  Have you extracted them from
>>> the CLHS?  If so, it would be good to include (e.g. as comments) the
>>> code&steps that was used.
>>> [ I'm also curious why things like make-method and next-method-p are in
>>>   built-in--symbols rather than in built-in--functions.  ]
> They are not extracted from CLHS. They are extracted from a SBCL image.

Works as well ;-)
Do you still have the code you used to extract it, so we can keep it
around (e.g. to update the list at some point, or just as documentation)?

> As for the 2 functions...it is because they are "local function/macro",
> which was not recognized as "function".

Doesn't this apply to `call-next-method` as well, then?

> Spenser suggested me to sign the copyright paperwork.
> And I am more than willing to do so.
> So...where do we start now? ;-)

Sent off-list,


        Stefan

Re: Request: add cl-font-lock to Emacs or Elpa

[YUE Daian]

>>>> I also wonder how you made those lists?  Have you extracted them from
>>>> the CLHS?  If so, it would be good to include (e.g. as comments) the
>>>> code&steps that was used.
>>>> [ I'm also curious why things like make-method and next-method-p are in
>>>>   built-in--symbols rather than in built-in--functions.  ]
>> They are not extracted from CLHS. They are extracted from a SBCL image.
>
> Works as well ;-)
> Do you still have the code you used to extract it, so we can keep it
> around (e.g. to update the list at some point, or just as documentation)?
>
I lost the original source, so I wrote a new one ;-)

I have added it to the README file, here is the snippet:

#+BEGIN_SRC common-lisp
(defvar *functions* nil)
(defvar *symbols* nil)
(defvar *types* nil)

(let ((pack (find-package :common-lisp)))
  (do-all-symbols (sym)
    (cond
      ((not (eql pack (symbol-package sym))) nil)
      ((fboundp sym) (pushnew sym *functions*))
      ((find-class sym nil) (pushnew sym *types*))
      (t (pushnew sym *symbols*)))))
#+END_SRC

Some manual work was done to tune the result.

>> As for the 2 functions...it is because they are "local function/macro",
>> which was not recognized as "function".
>
> Doesn't this apply to `call-next-method` as well, then?
>
Added. Thanks.

Re: Request: add cl-font-lock to Emacs or Elpa

[Spenser Truex]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Since we are trying to get this into lisp-mode.el at this point, I'm
>> going to refrain from further renaming.
>
> Agreed.
>
>> As for the question of my FSF paperwork, I'm currently in a state of
>> limbo. I have submitted my papers, but the president of the institution
>> in my signature is apparently too busy to sign.
>
> Let's hope patience will resolve this problem.

Indeed, with current events it is unlikely to be resolved. Currently I
have waited 9 months. The email is president@sfsu.edu.

>
>> I hope that my limited contribution to the project will permit this to
>> be looked over.
>
> Could be, but I don't know which part you have contributed (I've already
> asked twice what has contributed what but haven't seen any answer to the
> question yet ;-( ), so I can't judge it.
>

I've done a git-blame for myself. Here is the code I added. Before I
contributed the macro the code repeated the pattern it generates.

(defmacro font-lock-cl-add-regexes (fn mode &rest symbol-face)
  "Expand to more than one call to font-lock.
Argument FN is the function used to send off the regex. Commonly
`font-lock-add-keywords' or `font-lock-remove-keywords'. Argument
MODE is the mode where the regexes are sent.
Optional argument SYMBOL-FACE dotted-pair of (regex-var . font-face)."
  `(progn
     ,@(cl-loop for s in symbol-face
                collect
                `(,fn
                  ',mode
                  `((,(regexp-opt ,(car s) 'symbols)
                     . ,(cdr ',s)))))))

(font-lock-cl-add-regexes
 font-lock-add-keywords
 lisp-mode
 (font-lock-cl-built-in--functions . font-lock-function-name-face)
 (font-lock-cl-built-in--variables . font-lock-variable-name-face)
 (font-lock-cl-built-in--types . font-lock-type-face)
 (font-lock-cl-built-in--symbols . font-lock-builtin-face)
 (font-lock-cl--character-names . font-lock-variable-name-face))


-- 
Spenser Truex
spensertruex.com
San Francisco, USA

Re: Request: add cl-font-lock to Emacs or Elpa

[Stefan Monnier]

>> Let's hope patience will resolve this problem.
> Indeed, with current events it is unlikely to be resolved.
> Currently I have waited 9 months. The email is president@sfsu.edu.

9 months??!?

And now that I think about it: there shouldn't be any need for the
university's president to sign anything, AFAIK.  If you need paperwork
from your employer (SFSU), I'm pretty sure there's someone lower in the
"chain of command" with the authority needed to sign that paperwork.

Have you asked assign@gnu.org for help?
Maybe they have some good idea about what to do?

> I've done a git-blame for myself.  Here is the code I added.  Before I
> contributed the macro the code repeated the pattern it generates.

If that's all there is, then it's not a problem: it's small enough not
to need paperwork.


        Stefan

Request: add cl-font-lock to Emacs or Elpa

[Spenser]

>> 
>> Em 15 de mar de 2020, à(s) 02:47, Mattias Engdegård <mattiase@acm.org> escreveu:
>> 
>>>> 15 mars 2020 kl. 07.58 skrev Spenser Truex <lists@spensertruex.com>:
>>> 
>>> cl-font-lock is a package that does richer syntax highlighting for
>>> Common Lisp. I'm the co-maintainer on it. We are interested in having it
>>> included in Emacs or on Elpa. It is already on Melpa.
>>> 
>>> https://github.com/cl-font-lock/cl-font-lock
>> 
>> Did you check that the highlighting actually works for all those identifiers?
>> 
> I have been using it for months to develop CL code, and I think it works quite nicely. I prefer it over the default, and it helps me to notice misspelt symbols that are not highlighted.
> 
> I'm open to tests, and would be interested in adding such a test as suggested to a CI system if it could be automatically done in some way. 
> 
> Do you have a suggestion for how this can be done? If so, can you spell that out and/or submit code for it?
> 
> Much appreciation to you,
> —
> Spenser Truex
> spensertruex.com
> San Francisco State University, Computer Science