Re: [PATCH] add compiled regexp primitive lisp object

[Eli Zaretskii <eliz@gnu.org>]

> Date: Tue, 06 Aug 2024 15:15:31 +0000
> From: Danny McClanahan <dmcc2@hypnicjerk.ai>
> Cc: "emacs-devel@gnu.org" <emacs-devel@gnu.org>
> 
> Upon checking out how the `translate' table is used, I actually think it would
> be possible and even reasonable to avoid keeping track of it at all after
> compiling the regexp, if the regexp is instead compiled into a form that relies
> on character sets instead of translating input chars (this is how the rust regex
> crate handles its equivalent of `case-fold-search'). This would also remove
> another blocker to enabling SIMD searching in some ways, although patterns with
> case- or char-folding also tend to foil SIMD search techniques (maybe something
> clever can be done about this).

Are you aware that in Emacs case-conversion uses the buffer-local
case-tables if they are defined?  This means that the case-table in
effect when the regex was compiled and when it is executed could be
different, and your character set trick will not do what the user
expects.
> > > As above, I think the existing `regexp_cache' work has actually done a great job
> > > at nailing down what invalidates a regexp, so I think we can extend that
> > > framework to ensure compiled regexps have all of the configuration set at
> > > compile time to ensure intuitive behavior.
> >
> > Indeed. Again, my preference is to pretend the world is UTF-8, because
> > charset interactions make my head hurt, and declare that a compiled
> > regexp simply matches or does not match a given array of bytes (plus a
> > marker position and BOL/EOL flags, but you get the idea), and that
> > changing the flags results in a new and different compiled regexp.
> 
> I have actually already created a rust crate for emacs multibyte en/decoding
> (currently very spare: https://docs.rs/emacs-multibyte) for use with the regexp
> compiler I'm implementing in rust and hoping to introduce to emacs as an
> optional dependency (https://github.com/cosmicexplorer/emacs-regexp). On the
> face of it, I'm under the impression that multibyte encoding still produces
> a deterministic representation for any string of bytes not mappable to UTF-8, as
> those characters are just stored in the high bits not used by UTF-8 (I am really
> unfamiliar with charsets and char-tables though). However, this is indeed what
> most regex engines I am aware of do in order to support UTF-8 and still just
> operate on an array of bytes, although encoding unicode-aware character classes
> still requires dropping down to a char-by-char loop instead of fancy SIMD, which
> is why many offer the ability to turn off unicode-aware character classes
> (although I think we can do better for performance of non-ASCII users than
> this, perhaps by enabling the compilation of character classes to a specific
> language/unicode subset to enable further optimization).

I don't understand this part at all.  As long as you are dealing with
Emacs buffers and strings, there's only one "encoding": the internal
multibyte representation of characters Emacs uses.  It is a superset
of UTF-8, and you need never care about anything else.  (Well, there's
unibyte strings, but that can be addressed later.)