Hello emacs-devel, I have been having a wonderful time playing around with explicit regex compilation and have been able to produce a robust benchmark! I was spending a lot of time getting more familiar with the codebase, so my next task will be reading/responding to Pip Cet's wonderfully helpful feedback on my prior message. I am posting this here even though it is in an unpolished state because this demonstrates a backwards-compatible matching API without having to reimplement very much search logic. If you apply the attached patch with 'git am', you should be able to play with the following: (1) '(make-regexp "asdf")' => this produces a Lisp_Regexp pseudovector struct which can be provided to any search method in `search.c` instead of a string. (2) '(make-match-data ...)' => this produces a Lisp_Match pseudovector struct which wraps the existing 'struct re_registers'. - This is more of an implementation detail and lisp users actually don't need to be aware of this. The benchmark below does not touch match data explicitly at all, demonstrating a very similar end-user API. - For correctness, explicitly separating out the match result is extremely useful. Most of the work in this diff is replacing the implicit thread-local variables like 'search_regs' and 'searchbufs' with a 'struct regexp_match_info' parameter that writes match data to a provided match object instead of a global. (3) Despite introducing a couple new pseudovec types, we are actually able to avoid modifying 'struct re_registers' or 'struct re_pattern_buffer' at all! - The 'struct regexp_match_info' type was thrown together haphazardly to thread match data through search.c, but it's very simple and allows us to avoid introducing any really complex changes to regex-emacs.c. (4) I have extended the performance benchmarking for overlays and generalized it into 'test/manual/perf/perf.el', making it extremely simple to add further benchmarks. This part is great! I really do plan to revise this thanks to the wonderfully helpful feedback I've received (thanks!! ^_^), but I believe everyone should be able to reproduce the following benchmark results without issue. The gist of the following is that 'no-compile' tests are performed by providing a string directly to either `string-match' or `re-search-forward', whereas 'compiled' tests use a precompiled Lisp_Regexp with `make-regexp'. For example, consider: > perf-match/no-match-data/buffer/compiled/long-patterns/many-patterns/alternation 0.79 > perf-match/no-match-data/string/compiled/long-patterns/many-patterns/alternation 0.81 > perf-match/no-match-data/buffer/no-compile/long-patterns/many-patterns/alternation 2.17 > perf-match/no-match-data/string/no-compile/long-patterns/many-patterns/alternation 2.17 This test generates 30 separate regexp patterns which look like `"\\|"', where '' and '' are randomly-generated 40-character ASCII strings. It then executes either `re-search-forward' or `string-match' on a randomly-generated buffer or string about ~40k bytes long (also ASCII strings). Note that I intentionally do *not* count compilation time of each precompiled regexp into the overall runtime. However, it is nice to see that buffer and string searching take a similar amount of time in both cases. I have not verified this with profiling yet, but I chose the number 30 because it's larger than the default cache size for 'searchbufs' (20), and should therefore induce a lot of recompilation with the current string regexp cache. > ; make -C test/manual/regexp perf > make: Entering directory '/home/cosmicexplorer/tools/emacs/rex-build/test/manual/regexp' > cp -v ../../../../test/manual/regexp/regexp-perf.el regexp-perf.el > '../../../../test/manual/regexp/regexp-perf.el' -> 'regexp-perf.el' > ../../../src/emacs -Q -l ./perf.el -l ./regexp-perf.el -f perf-run-batch > > perf-match/match-data/string/legacy 0.03 > > perf-match/no-match-data/buffer/compiled/long-patterns/few-patterns/alternation 0.11 > > perf-match/no-match-data/buffer/compiled/long-patterns/few-patterns/grouped 0.05 > > perf-match/no-match-data/buffer/compiled/long-patterns/many-patterns/alternation 0.79 > > perf-match/no-match-data/buffer/compiled/long-patterns/many-patterns/grouped 0.41 > > perf-match/no-match-data/buffer/compiled/short-patterns/few-patterns/alternation 0.00 > > perf-match/no-match-data/buffer/compiled/short-patterns/few-patterns/grouped 0.05 > > perf-match/no-match-data/buffer/compiled/short-patterns/many-patterns/alternation 0.01 > > perf-match/no-match-data/buffer/compiled/short-patterns/many-patterns/grouped 0.42 > > perf-match/no-match-data/buffer/no-compile/long-patterns/few-patterns/alternation 0.33 > > perf-match/no-match-data/buffer/no-compile/long-patterns/few-patterns/grouped 0.22 > > perf-match/no-match-data/buffer/no-compile/long-patterns/many-patterns/alternation 2.17 > > perf-match/no-match-data/buffer/no-compile/long-patterns/many-patterns/grouped 1.53 > > perf-match/no-match-data/buffer/no-compile/short-patterns/few-patterns/alternation 0.00 > > perf-match/no-match-data/buffer/no-compile/short-patterns/few-patterns/grouped 0.21 > > perf-match/no-match-data/buffer/no-compile/short-patterns/many-patterns/alternation 0.02 > > perf-match/no-match-data/buffer/no-compile/short-patterns/many-patterns/grouped 1.49 > > perf-match/no-match-data/string/compiled/long-patterns/few-patterns/alternation 0.11 > > perf-match/no-match-data/string/compiled/long-patterns/few-patterns/grouped 0.06 > > perf-match/no-match-data/string/compiled/long-patterns/many-patterns/alternation 0.81 > > perf-match/no-match-data/string/compiled/long-patterns/many-patterns/grouped 0.42 > > perf-match/no-match-data/string/compiled/short-patterns/few-patterns/alternation 0.00 > > perf-match/no-match-data/string/compiled/short-patterns/few-patterns/grouped 0.06 > > perf-match/no-match-data/string/compiled/short-patterns/many-patterns/alternation 0.01 > > perf-match/no-match-data/string/compiled/short-patterns/many-patterns/grouped 0.42 > > perf-match/no-match-data/string/no-compile/long-patterns/few-patterns/alternation 0.33 > > perf-match/no-match-data/string/no-compile/long-patterns/few-patterns/grouped 0.22 > > perf-match/no-match-data/string/no-compile/long-patterns/many-patterns/alternation 2.17 > > perf-match/no-match-data/string/no-compile/long-patterns/many-patterns/grouped 1.56 > > perf-match/no-match-data/string/no-compile/short-patterns/few-patterns/alternation 0.00 > > perf-match/no-match-data/string/no-compile/short-patterns/few-patterns/grouped 0.22 > > perf-match/no-match-data/string/no-compile/short-patterns/many-patterns/alternation 0.02 > > perf-match/no-match-data/string/no-compile/short-patterns/many-patterns/grouped 1.49 You'll note that there are some commented-out DEFUNs in regex-emacs.c around extracting match data from the Lisp_Match object. I spent too much time prematurely trying to optimize that part: it turns out `match-data' and `set-match-data' are actually quite competitive in performance, even though they write match data into a list instead of a preallocated vector. We do have a greater opportunity to e.g. preallocate match registers if we move towards more AOT compilation of regexps, but it's probably more important to focus on a coherent end-user interface, since regular elisp appears to be quite performant. I agree that this interface of pre-compiling regexps in the first place induces a lot of subtle behavior that's worth thinking more about (as Pip has identified). Currently, these Lisp_Regexp objects simply take all of the dynamic variables used to compile a `struct re_pattern_buffer` upon construction. The fields of Lisp_Regexp are largely intended to mimic the behavior of the 'searchbufs' cache, although they're not actually used whatsoever (the only fields that are actually used for Lisp_Regexp are `struct re_pattern_buffer *buffer' and 'Lisp_Object default_match_target'. Thanks for reading! I'll be looking to minimize the size of this change (probably removing a lot of the DEFUNs over match data) and incorporating prior review feedback. I will also be looking to add new benchmark results. I have been surprised by how little I needed to modify search/matching logic, and I'm hopeful this can be introduced for a performance improvement without breaking too much lisp code at all! Thanks, Danny