From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: JD Smith Newsgroups: gmane.emacs.bugs Subject: bug#71345: Feature: unleash font-lock's secret weapon; handle Qfontified = non-nil Date: Tue, 4 Jun 2024 18:41:01 -0400 Message-ID: <2D33113E-2F9F-44BD-8CB9-5BC8C0AED5AC@gmail.com> References: <8A929E16-AF10-4D2B-AD71-AEAD4435F016@gmail.com> <1F2B8726-7594-494F-AB9D-08C48B7BCC43@gmail.com> <798B70AF-69BD-479E-992E-5CE9B4924820@gmail.com> Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.500.171.1.1\)) Content-Type: multipart/alternative; boundary="Apple-Mail=_FBE09669-58EE-4D37-9844-F962BF85DC88" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="11929"; mail-complaints-to="usenet@ciao.gmane.io" Cc: 71345@debbugs.gnu.org To: Stefan Monnier Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Wed Jun 05 00:43:10 2024 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1sEcs1-0002uB-Pd for geb-bug-gnu-emacs@m.gmane-mx.org; Wed, 05 Jun 2024 00:43:09 +0200 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sEcri-0002yt-T6; Tue, 04 Jun 2024 18:42:50 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sEcrh-0002yC-B2 for bug-gnu-emacs@gnu.org; Tue, 04 Jun 2024 18:42:49 -0400 Original-Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1sEcrh-0003Zz-3L for bug-gnu-emacs@gnu.org; Tue, 04 Jun 2024 18:42:49 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1sEcru-0005d8-AZ for bug-gnu-emacs@gnu.org; Tue, 04 Jun 2024 18:43:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: JD Smith Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Tue, 04 Jun 2024 22:43:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 71345 X-GNU-PR-Package: emacs Original-Received: via spool by 71345-submit@debbugs.gnu.org id=B71345.171754094821541 (code B ref 71345); Tue, 04 Jun 2024 22:43:02 +0000 Original-Received: (at 71345) by debbugs.gnu.org; 4 Jun 2024 22:42:28 +0000 Original-Received: from localhost ([127.0.0.1]:58028 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sEcrL-0005bM-RD for submit@debbugs.gnu.org; Tue, 04 Jun 2024 18:42:28 -0400 Original-Received: from mail-il1-f178.google.com ([209.85.166.178]:54474) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sEcrJ-0005ay-LY for 71345@debbugs.gnu.org; Tue, 04 Jun 2024 18:42:26 -0400 Original-Received: by mail-il1-f178.google.com with SMTP id e9e14a558f8ab-374a817a13bso6546725ab.0 for <71345@debbugs.gnu.org>; Tue, 04 Jun 2024 15:42:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1717540866; x=1718145666; darn=debbugs.gnu.org; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date:message-id:reply-to; bh=X0xIb9vAhgjOClAXRKprXREwSP0kdyZi+TI2M1YQKOo=; b=cEZDgCfK7HgdRi1F1LCJAWBKGvyCermf09OLHRHTTGo6cRk3GuX3jOBuSsRhfEzyFo OBWFLbc9oCzg1uxxtq6v0UoMEhus2/s6KBnZ5IN/LYR0Ebt3Sf4zwC3myXqdgufvdgal sYK4nK4hYrdQ0MKbZrTwxRe03HSQWuYMna4WXdsL+dAkbN+Hl+enWPBicfd9/ge6Jxat 4yN5FUfLYUqsid5Q3WRjrDTn/UCxx8sPS5Y0wV8xO93qHG9oTPY6acOOeBuJFAnCRTsA 4FqBshBPwmiHrN2ylAOjjDvE4oxn5BVXOmgY/5eC4LOKFu7eIM16/Nd/lCOtA2IpMYAE czrA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717540866; x=1718145666; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=X0xIb9vAhgjOClAXRKprXREwSP0kdyZi+TI2M1YQKOo=; b=TyJ7/rY9wyZ5hPIuQ/kXfQ8BU3dQCipfxzzjDbgN0JOTQZOQW/gKez7WIThWx34n87 V3SX4s2zG8tdRM8c7/hXVbaR4fWg7wzqw5NEobAbQLM02kLgsFVT5BJB7rGa6XNIJkRc lnJBzvpJNbzrc/LSoUyNy1jzFdqtDZrDVREbbOBvBoa0q4a61CQinvSnLzAss3l+VrCc ZGfJkFs13KGzbob+9xq9PlUXqQZ75iwiGDhaM6hp2zf6lU3j0ZAhCcWeB4D/t0fpTnHK wTm9Sz2en+6w6EGcjEIl4AY8vI0pLb76VCcb44Vw7QrBwgjArGOx4sO3QU2Ge5KfvOhA brkQ== X-Gm-Message-State: AOJu0YzlNWKUrwLzIV3642onsByr7enGMXiP3pYV1Ta17SWx3JYJZ6/M wYvaH2xsT0wD0CQ6+oFXe8YdbZlDYR9yAoxIj1es6VP6eklk7fyewxrRdA== X-Google-Smtp-Source: AGHT+IFkO03AM82wbCwkLwFAXfVI2VmIfxzzlQk3UfGWu99h27hk38BjzvmLXVCL3o2r816fsu/xEg== X-Received: by 2002:a05:6e02:1685:b0:374:aba6:5102 with SMTP id e9e14a558f8ab-374b1ef300cmr7477515ab.9.1717540866037; Tue, 04 Jun 2024 15:41:06 -0700 (PDT) Original-Received: from smtpclient.apple (cm-24-53-187-34.buckeyecom.net. [24.53.187.34]) by smtp.gmail.com with ESMTPSA id e9e14a558f8ab-3748a2059d0sm23897325ab.50.2024.06.04.15.41.05 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 04 Jun 2024 15:41:05 -0700 (PDT) In-Reply-To: X-Mailer: Apple Mail (2.3774.500.171.1.1) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:286565 Archived-At: --Apple-Mail=_FBE09669-58EE-4D37-9844-F962BF85DC88 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Jun 4, 2024, at 5:52=E2=80=AFPM, Stefan Monnier = wrote: > Also, I think it's worth distinguishing the API from its = implementation. Fair point. =20 > or on the contrary if it's careful to do something like >=20 > (jit-lock-flush (min my-pd-old-beg (my-pd-new-beg-estimate)) > (max my-pd-old-end (my-pd-new-end-estimate)) > #'my-pd-backend) I don't need to estimate, I have treesitter to tell me precisely! >> Agreed that could be an issue. In practice keyword-based = fontification can >> lead to these same sorts of conflicts for non trivial FACE forms too. >> So backends would need to ensure the changes they are making in the = buffer >> are interoperable with the other likely backends (in particular = font-lock). >=20 > Because a given buffer can have several (window-)points, > position-dependent highlighting will ideally want to be added via > (window-specific) overlays rather than text-properties. In general, yes. In my case having the scope be per-buffer not per = window makes the most sense in terms of the functionality. >> This means other backends would still need to add to >> font-lock-extra-managed-props any unusual properties they will apply >> (or do the equivalent on their own during unfontify). >=20 > No: `font-lock-extra-managed-props` is a variable that belongs to > font-lock, not jit-lock. `font-lock` is *one* backend of jit-lock. I tend to associate font-lock and after-change in my head, but that's = backwards. Presumably after-change would clobber fontified and whatever = additional properties jit-lock uses to track its backends. So font-lock = has no special status. Perhaps I tend to think of it as special because = it touches so many locations in the buffer. > Other backends will have to use something else (and should use other > text-properties or should use overlays (or make sure font-lock is not > used), otherwise you'll get into conflicts with font-lock). Sometimes those are unavoidable, if you need to add some 'face on top of = what font-lock did. > If we want to allow backends that are not independent (e.g. your PD > highlighting that has to run after font-lock), then it make the API = more > complex, since `jit-lock-flush` on the font-lock backend would have to > know to also flush the PD backend I think it will be limiting to insist that all backends must be fully = orthogonal (i.e. can apply in any arbitrary order), so some sort of = priority or ordering system seems important. That said, mine only has = to run over font-lock because of multi-line strings/comments, and the = inability to set an independent 'face property. It's actually too bad font-lock doesn't itself use font-lock-face for = much, despite what the docs say [1]. Since it sets 'face directly, it = overrides any other property alias for 'face. If it instead used = 'font-lock-face (or left that to modes, and used some higher-ranked = 'font-lock-priority-face), other backends could use their own = 'alternate-face, and the order in char-property-alias-alist would set = the priority. Then it would be much easier to make face-using backends = more orthogonal. Regarding the priority of backends, while looking through jit-lock, I = was reminded of the completion-at-point-function's API, and the ability = to claim :exclusive access, halting calls to further functions on the = list. That's a strong power, but if used wisely, could be effective. > I think the code would loop over chunks of text where the property is > `eq` (e.g. using `next-single-property-change`). Then within each = such > chunk of text it would iterate over either all backends (and skip > those mentioned in the `already-fontified` text-property) or over the > `pending` property backends. That seems easy enough. But those may be small regions which are inefficient to work on. This = as you say is an implementation detail so probably not important yet. JD [1] =46rom `Search-based Fontification':=20 If it is =E2=80=98prepend=E2=80=99, the face specified by FACESPEC is = added to the beginning of the =E2=80=98font-lock-face=E2=80=99 property. = If it is =E2=80=98append=E2=80=99, the face is added to the end of the = =E2=80=98font-lock-face=E2=80=99 property. But I believe all font-lock keywords actually just set the 'face = property.= --Apple-Mail=_FBE09669-58EE-4D37-9844-F962BF85DC88 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8

On Jun 4, 2024, at 5:52=E2=80=AFPM, Stefan Monnier = <monnier@iro.umontreal.ca> = wrote:

Also, I think it's worth distinguishing the API = from its = implementation.

Fair = point.  

or on the = contrary if it's careful to do something like

=    (jit-lock-flush (min my-pd-old-beg = (my-pd-new-beg-estimate))
=             &n= bsp;      (max my-pd-old-end = (my-pd-new-end-estimate))
=             &n= bsp;      #'my-pd-backend)

I don't need to estimate, I have = treesitter to tell me precisely!

Agreed that could be = an issue.  In practice keyword-based fontification can
lead to = these same sorts of conflicts for non trivial FACE forms too.
So = backends would need to ensure the changes they are making in the = buffer
are interoperable with the other likely backends (in = particular font-lock).

Because a given buffer can = have several (window-)points,
position-dependent highlighting will = ideally want to be added via
(window-specific) overlays rather than = text-properties.

In general, = yes.  In my case having the scope be per-buffer not per window = makes the most sense in terms of the = functionality.

This means other = backends would still need to add to
font-lock-extra-managed-props any = unusual properties they will apply
(or do the equivalent on their own = during unfontify).

No: = `font-lock-extra-managed-props` is a variable that belongs = to
font-lock, not jit-lock.  `font-lock` is *one* backend of = jit-lock.

I tend to = associate font-lock and after-change in my head, but that's backwards. =  Presumably after-change would clobber fontified and whatever = additional properties jit-lock uses to track its backends.  So = font-lock has no special status.  Perhaps I tend to think of it as = special because it touches so many locations in the = buffer.

Other backends will = have to use something else (and should use other
text-properties or = should use overlays (or make sure font-lock is not
used), otherwise = you'll get into conflicts with = font-lock).

Sometimes = those are unavoidable, if you need to add some 'face on top of what = font-lock did.

If = we want to allow backends that are not independent (e.g. your = PD
highlighting that has to run after font-lock), then it make the = API more
complex, since `jit-lock-flush` on the font-lock backend = would have to
know to also flush the PD = backend

I think it will = be limiting to insist that all backends must be fully orthogonal (i.e. = can apply in any arbitrary order), so some sort of priority or ordering = system seems important.  That said, mine only has to run over = font-lock because of multi-line strings/comments, and the inability to = set an independent 'face property.

It's actually too bad font-lock doesn't itself use = font-lock-face for much, despite what the docs say [1].  Since it = sets 'face directly, it overrides any other property alias for 'face. =  If it instead used 'font-lock-face (or left that to modes, and = used some higher-ranked 'font-lock-priority-face), other backends could = use their own 'alternate-face, and the order = in char-property-alias-alist would set the priority. =  Then it would be much easier to make face-using backends more = orthogonal.

Regarding the priority of backends, while looking through = jit-lock, I was reminded of the completion-at-point-function's API, and = the ability to claim :exclusive access, halting calls to further = functions on the list.  That's a strong power, but if used wisely, = could be effective.

I think = the code would loop over chunks of text where the property is
`eq` = (e.g. using `next-single-property-change`).  Then within each = such
chunk of text it would iterate over either all backends (and = skip
those mentioned in the `already-fontified` text-property) or = over the
`pending` property backends.  That seems easy = enough.

But those may be = small regions which are inefficient to work on.  This as you say is = an implementation detail so probably not important = yet.

JD

[1] =46rom= `Search-based Fontification': 
If it is =E2=80=98prepend=E2=80=99, = the face specified by FACESPEC is added to the beginning of the = =E2=80=98font-lock-face=E2=80=99 property.  If it is =E2=80=98append=E2= =80=99, the face is added to the end of the =E2=80=98font-lock-face=E2=80=99= property.
But I believe all font-lock = keywords actually just set the 'face = property.= --Apple-Mail=_FBE09669-58EE-4D37-9844-F962BF85DC88--