bug#67390: 28; shorthands-font-lock-shorthands assumes shorthand uses same separator

[Joseph Turner via "Bug reports for GNU Emacs, the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>]

João Távora <joaotavora@gmail.com> writes:

> On Mon, Nov 27, 2023, 12:12 Eli Zaretskii <eliz@gnu.org> wrote:
>
>  > From: João Távora <joaotavora@gmail.com>
>  > Date: Sun, 26 Nov 2023 22:02:01 +0000
>  > Cc: Jonas Bernoulli <jonas@bernoul.li>, 67390@debbugs.gnu.org,
>  >       Adam Porter <adam@alphapapa.net>
>  >
>  > On Sun, Nov 26, 2023 at 8:38 PM Joseph Turner <joseph@ushin.org> wrote:
>  > >
>  > > João Távora <joaotavora@gmail.com> writes:
>  > >
>  > > > On Sat, Nov 25, 2023 at 10:43 PM Joseph Turner <joseph@ushin.org> wrote:
>  >
>  > > > So, benchmarking it will have to be, I'm afraid, because AFAIK
>  > > > font-locking is a very performance sensitive area of Emacs.
>  > >
>  > > Yes.  I would like to learn how to do this!
>  >
>  > I'm CCing Eli.
>  >
>  > In the past, ISTR, Eli suggested to benchmark such things by visiting a
>  > very large file in its beginning, then scrolling down by holding
>  > the down arrow or PgDn for some fixed time period, like 30 seconds.
>  > The  Emacs that scrolls the farthest is the most performant.  Not
>  > entirely fail-proof (other processes may interfere, etc), but not
>  > bad either.
>
>  I still recommend this method.  Something like the below:
>
>    (defun scroll-up-benchmark ()
>      (interactive)
>      (let ((oldgc gcs-done)
>            (oldtime (float-time)))
>        (condition-case nil (while t (scroll-up) (redisplay))
>          (error (message "GCs: %d Elapsed time: %f seconds"
>                          (- gcs-done oldgc) (- (float-time) oldtime))))))
>
>  Evaluate the above, and the invoke it at the beginning of a large
>  file.  Then compare the timings with different font-lock arrangements.
>
>  A variant is to scroll by N lines, not by pages.  Just change the
>  above to call scroll-up with the argument of N, for example 1 (or any
>  other number, if you want).
>
> Joseph can you try these variations? They're slightly more exact. Also show at least one of the large lisp files or tell me how to generate
> one. If you still don't find any significant slowdown, I think we can merge your patch.

I'm happy to try Eli's variation if you don't beat me to it ;)

My large lisp file consisted of copy-pasting with a kbd macro the body of
https://git.sr.ht/~ushin/hyperdrive.el/tree/master/item/hyperdrive-lib.el
until the file reached ~50K lines -- well over the limit I expected to
reach on my machine.

One potential issue with the patch is that multiple shorthand prefixes
might match, while assoc will return the first matching cons pair
read-symbol-shorthand.

For example in hyperdrive-lib.el, we use the following shorthands in
order to display "//" instead of "/-" as the prefix separator for
private symbols, like "h//format-entry" instead of "h/-format-entry":

;; read-symbol-shorthands: (
;;   ("he//" . "hyperdrive-entry--")
;;   ("he/"  . "hyperdrive-entry-")
;;   ("h//"  . "hyperdrive--")
;;   ("h/"   . "hyperdrive-"))

However, if we rearrange the values like:

;; read-symbol-shorthands: (
;;   ("he/"  . "hyperdrive-entry-")
;;   ("he//" . "hyperdrive-entry--")
;;   ("h/"   . "hyperdrive-")
;;   ("h//"  . "hyperdrive--"))

then shorthands doesn't add fontification for either "h//" or "he//".
(This surprised me - I was expecting to see just the "h/" and "he/"
highlit)

However, I'm starting to wonder whether this is a case of user error.
Should we avoid overlapping shorthand prefixes?

Thank you!!

Joseph