Re: Scan of regexps in Emacs (March 17)

["Mattias Engdegård" <mattiase@acm.org>]

[-- Attachment #1: Type: text/plain, Size: 4250 bytes --]

20 mars 2019 kl. 23.01 skrev Paul Eggert <eggert@cs.ucla.edu>:
> 
> On 3/19/19 7:20 PM, Stefan Monnier wrote:
>> I wonder why the doc doesn't just say that `-` should be the last
>> character and not mention the other possibilities which just make the
>> rule unnecessarily complex.

Agreed, that is what the 'how to write regexps' part of the docs should say. But don't we also need a precise description of exactly how they are interpreted by the engine? Otherwise, a user cannot read and understand existing code. (Unless he or she uses xr!) Perhaps there needs to be a separate 'gritty details' section.

> * The doc already says that regular expressions like "*foo" and "+foo"
> are problematic (they're confusing, and POSIX says the behavior is
> undefined) and should be avoided. REs like "[a-m-z]" and "[!-[:alpha:]]"
> and "[[:alpha:]-~]" are problematic in the same way and also should be
> avoided.

I'm with Stefan here; `-' should go last. Anything else is a gritty detail.

> * The doc doesn't clearly say when the Emacs range behavior is an
> extension to POSIX; saying this will help people know better when they
> can export Emacs regular expressions to other programs.

Documenting differences from POSIX regexps is useful. Do you prefer having those differences being spread out, or all concentrated into one section?

These days, a user may be more familiar with the various PCRE dialects than traditional or extended POSIX. Should that be taken into account?

> * The doc is confused (and there's a comment about this) about what
> happens when one end of a range is unibyte and the other is multibyte. I
> added something saying that if one bound is a raw 8-bit byte then the
> other should be a unibyte character (either ASCII, or a raw 8-bit byte).
> I don't see any good way to specify the behavior when one bound is a raw
> 8-bit byte and the other bound is a multibyte character, in such a way
> that it's a natural extension of the documented behavior, so the
> documentation now recommends against that.

The terminology is a bit confusing. Is 'raw 8-bit byte' included in 'unibyte'? Is \x7f ever a raw 8-bit byte?
I agree that [å-\xff], say, should be invalid but I've never seen such constructs.

> * We might as well go ahead and say that [b-a] matches nothing, as
> enough code (ab)uses regexps in that way, and there is value in having a
> simple regular expression that always fails to match. However, I expect
> that we should say that users should avoid wilder examples like [~-!] so
> that the trawler can catch them as typos.

It already does, and some bugs were found that way. As a special case, it no longer complains about z-a because that is unlikely to be an accident and occurs in some code on purpose.

I'm not sure it's a good idea to document reversed ranges as a recommended way to match any or no character (although the description of the semantics would belong in a 'gritty details' section), and only to use [Y-X] where Y=X+1. More about that in a separate post.

> These new recommendations ("should"s in the attached patch) will give
> the trawler license to diagnose questionable REs like "[a-m-z]",
> "[!-[:alpha:]]", "[~-!]", and (my favorite) "[\u00FF-\xFF]". There is no
> change to actual Emacs behavior.

As an experiment, I added detection of 'chained' ranges like [a-m-z] to xr and found a handful in both Emacs and GNU ELPA, but none of them carried a freeload of bugs. Keeping that check didn't seem worthwhile; the regexps may be a bit odd-looking, but aren't wrong.

[!-[:alpha:]] is already detected since xr parses it correctly and will complain about the duplication of ':'. The reverse, [[:digit:]-z], is seen occasionally but again does not seem to be a serious bug proxy.

Much as I would like to outlaw ranges where a typical programmer has to consult an ASCII table to understand what's included, they just seem too common, with too many false positives, to merit inclusion in xr.
Nevertheless I had a quick look and extracted a few that might merit attention; see attachment.

Similarly, a rule finding [X-Y] where Y=X+1 found one or two questionable cases in a sea of false positives (also in the attachment).

[-- Attachment #2: possibly-broken-regexps.log --]
[-- Type: application/octet-stream, Size: 992 bytes --]

/Users/mattias/emacs/lisp/vc/diff-mode.el:2215:19: In call to re-search-forward: Unintuitive range `+-<' (pos 3)
  "\n[!+-<>]\\(-- [0-9]+\\(,[0-9]+\\)? ----\n\\( .*\n\\)*[+]\\)?"
   ....^
/Users/mattias/emacs/lisp/speedbar.el:2852:42: In call to re-search-forward: Unintuitive range `+-?' (pos 21)
  "^\\([0-9]+\\):\\s-*[[<][+-?][]>] "
   ........................^
/Users/mattias/emacs/lisp/speedbar.el:2903:42: In call to re-search-forward: Unintuitive range `+-?' (pos 19)
  "^\\([0-9]+\\):\\s-*\\[[+-?]\\] "
   .......................^
/Users/mattias/emacs/lisp/woman.el:3514:26: In call to looking-at: Unintuitive range `+-/' (pos 1)
  "[+-/*%]"
   .^
/Users/mattias/emacs/lisp/net/webjump.el:345:39: In call to string-match: Two-character range `.-/' (pos 8)
  "[a-zA-Z_.-/]"
   ........^
/Users/mattias/emacs/lisp/align.el:386:3: In align-rules-list (perl-assignment): Two-character range `*-+' (pos 6)
  "[^=!^&*-+<>/| \t\n]\\(\\s-*\\)=[~>]?\\(\\s-*\\)\\([^>= \t\n]\\|$\\)"
   ......^