bug#23465: 25.0.93; `lambda' after hash-quote not highlighted as keyword

bug#23465: 25.0.93; `lambda' after hash-quote not highlighted as keyword

[Anders Lindgren]

[-- Attachment #1: Type: text/plain, Size: 3052 bytes --]

Hi!

Steps to repeat:

    emacs -Q
    Type: #'(lambda (x))

Here, `lambda' isn't highlighted as a keyword. This is a regression from
Emacs 24.5.

In Emacs 25, keyword highlighting is done using the matcher
function`my-lisp--el-match-keyword' which highlights identifiers that are
1) special forms or 2) macros when is a function call position.

Apparently, `lambda', in this context, is neither.

I think that a minimal impact change would be for
`lisp--el-non-funcall-position-p' to treat '(xxx) as a non-function call
location (like it does today), except when preceded by a #.

    -- Anders


In GNU Emacs 25.0.93.1 (x86_64-w64-mingw32)
 of 2016-04-23 built on KAEL
Windowing system distributor 'Microsoft Corp.', version 6.1.7601
Configured using:
 'configure --prefix=/tmp/emacs --without-imagemagick 'CFLAGS=-O2
 -fomit-frame-pointer -g0''

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND DBUS NOTIFY ACL GNUTLS LIBXML2 ZLIB
TOOLKIT_SCROLL_BARS

Important settings:
  value of $LANG: SVE
  locale-coding-system: cp1252

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
Mark set
Making completion list...

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message dired format-spec rfc822 mml
mml-sec password-cache epg epg-config gnus-util mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail
rfc2047 rfc2045 ietf-drums mm-util help-fns help-mode easymenu
cl-loaddefs pcase cl-lib mail-prsvr mail-utils time-date mule-util
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel dos-w32 ls-lisp disp-table w32-win w32-vars term/common-win
tool-bar dnd fontset image regexp-opt fringe tabulated-list newcomment
elisp-mode lisp-mode prog-mode register page menu-bar rfn-eshadow timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
frame cl-generic cham georgian utf-8-lang misc-lang vietnamese tibetan
thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek romanian
slovak czech european ethiopic indian cyrillic chinese charscript
case-table epa-hook jka-cmpr-hook help simple abbrev minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote w32notify dbusbind w32
multi-tty make-network-process emacs)

Memory information:
((conses 16 89295 12665)
 (symbols 56 19662 0)
 (miscs 48 43 127)
 (strings 32 15944 4609)
 (string-bytes 1 435395)
 (vectors 16 11720)
 (vector-slots 8 422750 4713)
 (floats 8 160 110)
 (intervals 56 261 40)
 (buffers 976 13))

[-- Attachment #2: Type: text/html, Size: 4155 bytes --]

bug#23465: 25.0.93; `lambda' after hash-quote not highlighted as keyword

[Anders Lindgren]

[-- Attachment #1.1: Type: text/plain, Size: 349 bytes --]

Hi!

The attached patch fix the problem. It considers '(xxx) not to be a
function calling position (like before), but #'(xxx) is (unlike before).

Is it the way we should handle this problem? If so, can this go into
emacs-25? (The only use of the modified function, which didn't exist in
Emacs 24, is for highlighting lisp keywords.)

    -- Anders

[-- Attachment #1.2: Type: text/html, Size: 472 bytes --]

[-- Attachment #2: lambda.diff --]
[-- Type: text/plain, Size: 549 bytes --]

diff --git a/lisp/emacs-lisp/lisp-mode.el b/lisp/emacs-lisp/lisp-mode.el
index 315b3d5..70ed40e 100644
--- a/lisp/emacs-lisp/lisp-mode.el
+++ b/lisp/emacs-lisp/lisp-mode.el
@@ -195,7 +195,8 @@ lisp--el-non-funcall-position-p
     (save-excursion
       (ignore-errors
         (goto-char pos)
-        (or (eql (char-before) ?\')
+        (or (and (eql (char-before) ?\')
+                 (not (eq (char-before (- (point) 1) ?#))))
             (let* ((ppss (syntax-ppss))
                    (paren-posns (nth 9 ppss))
                    (parent

bug#23465: 25.0.93; `lambda' after hash-quote not highlighted as keyword

[Anders Lindgren]

[-- Attachment #1: Type: text/plain, Size: 461 bytes --]

On Fri, May 6, 2016 at 8:32 PM, Anders Lindgren <andlind@gmail.com> wrote:

> Hi!
>
> The attached patch fix the problem. It considers '(xxx) not to be a
> function calling position (like before), but #'(xxx) is (unlike before).
>
> Is it the way we should handle this problem? If so, can this go into
> emacs-25? (The only use of the modified function, which didn't exist in
> Emacs 24, is for highlighting lisp keywords.)
>
>
Ping!

John? Eli?

    // Anders

[-- Attachment #2: Type: text/html, Size: 962 bytes --]

bug#23465: 25.0.93; `lambda' after hash-quote not highlighted as keyword

[Eli Zaretskii]

> Date: Tue, 10 May 2016 17:00:28 +0200
> From: Anders Lindgren <andlind@gmail.com>
> 
>  The attached patch fix the problem. It considers '(xxx) not to be a function calling position (like before),
>  but #'(xxx) is (unlike before).
> 
>  Is it the way we should handle this problem? If so, can this go into emacs-25? (The only use of the
>  modified function, which didn't exist in Emacs 24, is for highlighting lisp keywords.)
> 
> Ping!
> 
> John? Eli?

Sorry, I don't know anything about that code, and don't consider
myself an expert on font-lock enough to judge this suggestion.  I hope
someone else will chime in.

bug#23465: 25.0.93; `lambda' after hash-quote not highlighted as keyword

[Dmitry Gutov]

On 05/10/2016 07:25 PM, Eli Zaretskii wrote:

> Sorry, I don't know anything about that code, and don't consider
> myself an expert on font-lock enough to judge this suggestion.  I hope
> someone else will chime in.

FWIW, LGTM.

But should we encourage this use of sharp quotes? There is no need to 
quote lambdas; sharp quote doesn't hurt, but is there a case where it 
would help with a lambda?

As an aside, it would be great if we could get rid of the duplication 
between lisp--el-non-funcall-position-p vs elisp--form-quoted-p, 
elisp--expect-function-p and some bits inside elisp-completion-at-point. 
Or reduce it, at least.

bug#23465: 25.0.93; `lambda' after hash-quote not highlighted as keyword

[Anders Lindgren]

[-- Attachment #1: Type: text/plain, Size: 732 bytes --]

On Wed, May 11, 2016 at 12:09 AM, Dmitry Gutov <dgutov@yandex.ru> wrote:

> On 05/10/2016 07:25 PM, Eli Zaretskii wrote:
>
> Sorry, I don't know anything about that code, and don't consider
>> myself an expert on font-lock enough to judge this suggestion.  I hope
>> someone else will chime in.
>>
>
> FWIW, LGTM.
>
> But should we encourage this use of sharp quotes? There is no need to
> quote lambdas; sharp quote doesn't hurt, but is there a case where it would
> help with a lambda?
>
> As an aside, it would be great if we could get rid of the duplication
> between lisp--el-non-funcall-position-p vs elisp--form-quoted-p,
> elisp--expect-function-p and some bits inside elisp-completion-at-point. Or
> reduce it, at least.
>

[-- Attachment #2: Type: text/html, Size: 1203 bytes --]

bug#23465: 25.0.93; `lambda' after hash-quote not highlighted as keyword

[Anders Lindgren]

[-- Attachment #1: Type: text/plain, Size: 1559 bytes --]

Hi!

Eli:

> Sorry, I don't know anything about that code, and don't consider
> myself an expert on font-lock enough to judge this suggestion.  I hope
> someone else will chime in.

Well, I need to get a "go ahead" from someone to push this to the emacs-25
branch... Anyway, I mailed Stefan asking him to take a look at this.


Dmitry:

> But should we encourage this use of sharp quotes? There is no need to
> quote lambdas; sharp quote doesn't hurt, but is there a case where it would
> help with a lambda?
>

I thought they were the preferred way to write lambda:s. All examples in
the elisp reference guide use #' and there are almost 400 uses of them in
the Emacs source.

Anyway, dropping highlighting support for them isn't the way to go, yet,
even if we would like to deprecate them in the future.


As an aside, it would be great if we could get rid of the duplication
> between lisp--el-non-funcall-position-p vs elisp--form-quoted-p,
> elisp--expect-function-p and some bits inside elisp-completion-at-point. Or
> reduce it, at least.
>

Good points. However, shouldn't this be done in the master branch, not in
emacs-25?

When we're talking about enhancements. Currently, lisp special forms (like
"if" and "and") as well as elisp macros are highlighted using the keyword
face. Shouldn't it be better to use two different faces, or at least give
the user the option to configure this separately? I would imagine that the
real keywords would no longer stand out in lisp source using lots of
macros. (Again, this is not for emacs-25.)

    -- Anders

[-- Attachment #2: Type: text/html, Size: 2623 bytes --]

bug#23465: 25.0.93; `lambda' after hash-quote not highlighted as keyword

[Eli Zaretskii]

> Date: Wed, 11 May 2016 11:34:36 +0200
> From: Anders Lindgren <andlind@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>, 23465@debbugs.gnu.org
> 
> Well, I need to get a "go ahead" from someone to push this to the emacs-25 branch...

I don't see the need to push this to emacs-25.  It doesn't seem to be
a problem that's critical enough.  I think this should be installed on
master, assuming no one objects.

> Anyway, I mailed Stefan asking him to take a look at this.

Thanks.

bug#23465: 25.0.93; `lambda' after hash-quote not highlighted as keyword

[Leo Liu]

On 2016-05-11 12:46 +0300, Eli Zaretskii wrote:
> I don't see the need to push this to emacs-25.  It doesn't seem to be
> a problem that's critical enough.  I think this should be installed on
> master, assuming no one objects.

Not highlighting lambda in those quoted cases isn't wrong because lambda
is no longer a macro but a symbol.

Leo

bug#23465: 25.0.93; `lambda' after hash-quote not highlighted as keyword

[Dmitry Gutov]

On 05/11/2016 12:34 PM, Anders Lindgren wrote:

> I thought they were the preferred way to write lambda:s. All examples in
> the elisp reference guide use #' and there are almost 400 uses of them
> in the Emacs source.

Since #'(lambda is fully equivalent to (lambda, we don't want to write 
the former anymore. The existing uses are probably holdouts from the 
time when the equivalence wasn't true yet.

I see only 4 hits on #'(lambda in our .texi files, and one of them (in 
functions.texi) says that the usages are equivalent and further 
elaborates on that. The others should be changed.

>     As an aside, it would be great if we could get rid of the
>     duplication between lisp--el-non-funcall-position-p vs
>     elisp--form-quoted-p, elisp--expect-function-p and some bits inside
>     elisp-completion-at-point. Or reduce it, at least.
>
>
> Good points. However, shouldn't this be done in the master branch, not
> in emacs-25?

Yep.

> When we're talking about enhancements. Currently, lisp special forms
> (like "if" and "and") as well as elisp macros are highlighted using the
> keyword face. Shouldn't it be better to use two different faces, or at
> least give the user the option to configure this separately?

Let's not give the users a choice, it's not exactly a meaningful one.

We should rather have a standard which faces to use where that's kept 
consistent across major modes.

Alas, our choice is not easy here, since `font-lock-builtin-face' is 
currently used for keyword symbols.

 > I would
 > imagine that the real keywords would no longer stand out in lisp
 > source using lots of macros. (Again, this is not for emacs-25.)

What's a keyword in Lisp? I'm not sure you can draw a definite line, 
especially since we could have a primitive defined in C in one version 
of Emacs, and moved to Lisp in the next version.

bug#23465: 25.0.93; `lambda' after hash-quote not highlighted as keyword

[Stefan Monnier]

>> I don't see the need to push this to emacs-25.  It doesn't seem to be
>> a problem that's critical enough.  I think this should be installed on
>> master, assuming no one objects.
> Not highlighting lambda in those quoted cases isn't wrong because lambda
> is no longer a macro but a symbol.

I tend to agree.  It'd be fine to highlight them, but it's
not necessary.

The form #'(lambda ...) is not deprecated nor do I expect it to become
deprecated any time soon, but most people like myself prefer to simply
use (lambda ...) instead.

In #'(lambda ..) the `lambda' symbol is definitely not a macro, but is
a kind of special form, yet one that's yet different from progn/if/...

So it's OK to highlight it, but it's also perfectly OK not to.

In my buffers it looks like λ anyway, so it really doesn't matter.


        Stefan


PS: If we want to refine highlighting of #' then we could also highlight
    #'<symbol> differently depending on whether <symbol> is a function,
    a macro/specialform (probably highlighted as some sort of error) or
    none of those.

bug#23465: 25.0.93; `lambda' after hash-quote not highlighted as keyword

[Stefan Monnier]

> Since #'(lambda is fully equivalent to (lambda, we don't want to write the
> former anymore. The existing uses are probably holdouts from the time when
> the equivalence wasn't true yet.

I think there was something else at play (maybe the efficiency impact of
macro-expanding `lambda'? I don't know): the equivalence has been true
for *many* years already (the `lambda' macro was added sometime in
Emacs-18 already, IIRC (and no, I don't remember it from when it
appeared because I wasn't an Emacs user yet back then, but I traced it
back the RCS history at some point)).

> Let's not give the users a choice, it's not exactly a meaningful one.

Agreed.  In most cases whether something is a macro or a special form is
a detail of implementation (typically driven by efficiency or
simplicity).

> We should rather have a standard which faces to use where that's kept
> consistent across major modes.
> Alas, our choice is not easy here, since `font-lock-builtin-face' is
> currently used for keyword symbols.

Not sure what you mean.  To the extent that the macro symbols form
syntactic constructs, they correspond to the "reserved keywords" that we
usually highlight in font-lock-keyword-face in other languages, so
I think in this respect, we're pretty much consistent.


        Stefan

bug#23465: 25.0.93; `lambda' after hash-quote not highlighted as keyword

[Dmitry Gutov]

On 05/11/2016 03:32 PM, Stefan Monnier wrote:

> Not sure what you mean.  To the extent that the macro symbols form
> syntactic constructs, they correspond to the "reserved keywords" that we
> usually highlight in font-lock-keyword-face in other languages, so
> I think in this respect, we're pretty much consistent.

I gave an example of a problem in trying to differentiate between the 
special forms and the rest of the macros: we'd have to use 
font-lock-builtin-face, and it's already taken.

But continuing to highlight them both with font-lock-keyword-face is 
just fine by me.

bug#23465: 25.0.93; `lambda' after hash-quote not highlighted as keyword

[Anders Lindgren]

[-- Attachment #1: Type: text/plain, Size: 1041 bytes --]

Hi!


> > Let's not give the users a choice, it's not exactly a meaningful one.
>
> Agreed.  In most cases whether something is a macro or a special form is
> a detail of implementation (typically driven by efficiency or
> simplicity).
>

I, partially, agree.

Emacs has for a long time highlighted "keywords" (taken with a grain of
salt) like "progn", "if", "when", "and", "or" etc. In other words, the core
functions used when writing elisp. I agree in that it should not matter
whether they are implemented in C or using a macro.

However, now Emacs also highlights *all* macros when they are used. In some
cases, like in "define-lex-block-type-analyzer", the keyword face is used
so much that the visual cues it normally provides are lost. In this case,
it would be better to use a different face for the user macros. One
candidate would be font-lock-preprocessor-face.


Anyway, we're striding away from the subject here. I'll commit the fix for
the "lambda" problem on the master branch and leave it broken on emacs-25.

    -- Anders

[-- Attachment #2: Type: text/html, Size: 1586 bytes --]

bug#23465: 25.0.93; `lambda' after hash-quote not highlighted as keyword

[Stefan Monnier]

> Emacs has for a long time highlighted "keywords" (taken with a grain of
> salt) like "progn", "if", "when", "and", "or" etc.  In other words, the core
> functions used when writing elisp.

This is obviously a question of opinion, but the way I look at it, it
was a manually-managed approximation of "the set of macros/specialforms".
So yes, it included (usually) less than the new code, but it included
a good bit more than "the core" (for my own opinion of what is "core",
clearly).

> However, now Emacs also highlights *all* macros when they are used. In some
> cases, like in "define-lex-block-type-analyzer", the keyword face is used
> so much that the visual cues it normally provides are lost.

Hmm... I don't understand the problem you're referring to.

I looked at the uses of define-lex-block-type-analyzer I could
find in Emacs's trunk and they look OK to me.  I also looked at the
definition of define-lex-block-type-analyzer and its highlighting also
looked fine to me.

> In this case, it would be better to use a different face for the user
> macros.  One candidate would be font-lock-preprocessor-face.

I think you expect something different, indeed.  Personally I don't care
who/where the macro is defined.  I only care about the fact that the
evaluation rules will likely not be the same as a function call.

IOW I do find some of the new highlighting distracting, but only for
those macros which behave just like functions (i.e. typically macros
which would better be defined with defsubst or define-inline).

> Anyway, we're striding away from the subject here. I'll commit the fix for
> the "lambda" problem on the master branch and leave it broken on emacs-25.

Not sure I like this fix: it causes the `if` of #'(if bar baz) to be
highlighted even though it's not a special form nor a macro in
that position.


        Stefan

bug#23465: 25.0.93; `lambda' after hash-quote not highlighted as keyword

[Anders Lindgren]

[-- Attachment #1: Type: text/plain, Size: 1415 bytes --]

Hi!

> I looked at the uses of define-lex-block-type-analyzer I could
> find in Emacs's trunk and they look OK to me.  I also looked at the
> definition of define-lex-block-type-analyzer and its highlighting also
> looked fine to me.

It was the definition I was referring to. (I just picked it as an example,
as it uses many macros.)

Anyway, I'll drop the idea (or implement it as a stand-alone package, if I
ever get around to it) as it apparently didn't resonate well here.


> Anyway, we're striding away from the subject here. I'll commit the fix for
> > the "lambda" problem on the master branch and leave it broken on
> emacs-25.
>
> Not sure I like this fix: it causes the `if` of #'(if bar baz) to be
> highlighted even though it's not a special form nor a macro in
> that position.
>

It is already highlighted in that context, today, even without my patch.
("if" is a special form whereas "lambda" isn't.)

Also, I've looked through a lot of elisp code, and the only two uses of
hash-quote I found were #'symbol and #'(lambda (...) ...).

Besides, I don't mind having keywords highlighted in quoted or hash-quoted
expressions. In fact, in elisp, code is often written in a quoted context,
e.g. in font-lock keywords, where highlighting of special forms still makes
sense.

If you have a better way to handle this (e.g. making "lambda" a special
form), feel free to suggest another patch.

    -- Anders

[-- Attachment #2: Type: text/html, Size: 2535 bytes --]

bug#23465: 25.0.93; `lambda' after hash-quote not highlighted as keyword

[Leo Liu]

On 2016-05-11 17:02 +0200, Anders Lindgren wrote:
> It was the definition I was referring to. (I just picked it as an example,
> as it uses many macros.)
>
> Anyway, I'll drop the idea (or implement it as a stand-alone package, if I
> ever get around to it) as it apparently didn't resonate well here.

People forget that in emacs 24.x many aren't font-locked, for example,
and, or etc. Now too many are. It's a bit annoying at first but I seem
to get used to ignoring them.

Leo

bug#23465: 25.0.93; `lambda' after hash-quote not highlighted as keyword

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

>> Well, I need to get a "go ahead" from someone to push this to the emacs-25
>> branch...

> I don't see the need to push this to emacs-25. It doesn't seem to be a
> problem that's critical enough. I think this should be installed on master,
> assuming no one objects.

It's not critical and we've locked down emacs-25, so please push to master.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

bug#23465: 25.0.93; `lambda' after hash-quote not highlighted as keyword

[Noam Postavsky]

Anders Lindgren <andlind@gmail.com> writes:

> Anyway, we're striding away from the subject here. I'll commit the
> fix for the "lambda" problem on the master branch and leave it broken
> on emacs-25.

Was this ever pushed?  Seems like not.  It's by now already too late
for emacs-26, but could still go to master I think.

bug#23465: 25.0.93; `lambda' after hash-quote not highlighted as keyword

[Anders Lindgren]

[-- Attachment #1: Type: text/plain, Size: 595 bytes --]

Hi!

No, it was never pushed. However, this has resolved itself since `lambda`
now is a macro, so it's highlighted using the normal macro mechanism.

    -- Anders

On Mon, Jan 29, 2018 at 12:31 AM, Noam Postavsky <
npostavs@users.sourceforge.net> wrote:

> Anders Lindgren <andlind@gmail.com> writes:
>
> > Anyway, we're striding away from the subject here. I'll commit the
> > fix for the "lambda" problem on the master branch and leave it broken
> > on emacs-25.
>
> Was this ever pushed?  Seems like not.  It's by now already too late
> for emacs-26, but could still go to master I think.
>

[-- Attachment #2: Type: text/html, Size: 1077 bytes --]

bug#23465: 25.0.93; `lambda' after hash-quote not highlighted as keyword

[Noam Postavsky]

Anders Lindgren <andlind@gmail.com> writes:

> No, it was never pushed. However, this has resolved itself since
> `lambda` now is a macro, so it's highlighted using the normal macro
> mechanism.

Not when it's hash quoted though, which I thought was the point of this
bug.

bug#23465: 25.0.93; `lambda' after hash-quote not highlighted as keyword

[Anders Lindgren]

[-- Attachment #1: Type: text/plain, Size: 473 bytes --]

Ahh, right. I mixed it up with when `lambda':s weren't highlighted at all...

    -- Anders


On Mon, Jan 29, 2018 at 1:25 PM, Noam Postavsky <
npostavs@users.sourceforge.net> wrote:

> Anders Lindgren <andlind@gmail.com> writes:
>
> > No, it was never pushed. However, this has resolved itself since
> > `lambda` now is a macro, so it's highlighted using the normal macro
> > mechanism.
>
> Not when it's hash quoted though, which I thought was the point of this
> bug.
>

[-- Attachment #2: Type: text/html, Size: 933 bytes --]

bug#23465: 25.0.93; `lambda' after hash-quote not highlighted as keyword

[Lars Ingebrigtsen]

Anders Lindgren <andlind@gmail.com> writes:

> The attached patch fix the problem. It considers '(xxx) not to be a function calling
> position (like before), but #'(xxx) is (unlike before).

[...]

> -        (or (eql (char-before) ?\')
> +        (or (and (eql (char-before) ?\')
> +                 (not (eq (char-before (- (point) 1) ?#))))

It seemed like everybody in the thread agreed that this was the correct
solution, so I applied it to Emacs 28 (after fixing it slightly).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no