using non-Emacs regexp syntax

using non-Emacs regexp syntax

[Paul Pogonyshev]

Hi,

Is there a function to convert non-Emacs regexps (e.g. "ab(c+|d)" to
Emacs regexps (example to "ab\(c+\|d\)")?

If there is none, are you interested in adding such functions?  (Of
course, not now, but after the release.)  I assume it is not worth it
to implement in C, so a Lisp implementation is in order?

(Abstract task is like this: be able to read regexps from an (XML)
file, which should be readable not only by Emacs; since Emacs syntax
is not widespread, regexps would use a different syntax.)

Paul

Re: using non-Emacs regexp syntax

[Richard Stallman]

    Is there a function to convert non-Emacs regexps (e.g. "ab(c+|d)" to
    Emacs regexps (example to "ab\(c+\|d\)")?

The first form appears to be an "extended regexp" or egrep-style regexp.
The second appears to be a "basic regexp" or grep-style regexp.

This conversion feature in Lisp would be useful to add after the release.

RE: using non-Emacs regexp syntax

[Drew Adams]

>     Is there a function to convert non-Emacs regexps (e.g. "ab(c+|d)" to
>     Emacs regexps (example to "ab\(c+\|d\)")?
>
> The first form appears to be an "extended regexp" or egrep-style regexp.
> The second appears to be a "basic regexp" or grep-style regexp.
>
> This conversion feature in Lisp would be useful to add after the release.

Very glad to hear that.

I'm hoping there will also be support for toggling the newline sensitivity
of dot. This means a "doc-matches-newline" mode (aka "single-line" mode)
where `.' will also match newline. Please see the thread "short regexp to
match any character?" from 2006/03/04 and 03/11.

Re: using non-Emacs regexp syntax

[David Kastrup]

"Drew Adams" <drew.adams@oracle.com> writes:

>>     Is there a function to convert non-Emacs regexps (e.g. "ab(c+|d)" to
>>     Emacs regexps (example to "ab\(c+\|d\)")?
>>
>> The first form appears to be an "extended regexp" or egrep-style regexp.
>> The second appears to be a "basic regexp" or grep-style regexp.
>>
>> This conversion feature in Lisp would be useful to add after the release.
>
> Very glad to hear that.
>
> I'm hoping there will also be support for toggling the newline sensitivity
> of dot. This means a "doc-matches-newline" mode (aka "single-line" mode)
> where `.' will also match newline. Please see the thread "short regexp to
> match any character?" from 2006/03/04 and 03/11.

I don't know any other matcher where dot matches a newline.  Quite
more relevant would be inverse character ranges like [^A-Z] that do
_not_ match newline by default.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: using non-Emacs regexp syntax

[Paul Pogonyshev]

David Kastrup wrote:
> "Drew Adams" <drew.adams@oracle.com> writes:
> 
> >>     Is there a function to convert non-Emacs regexps (e.g. "ab(c+|d)" to
> >>     Emacs regexps (example to "ab\(c+\|d\)")?
> >>
> >> The first form appears to be an "extended regexp" or egrep-style regexp.
> >> The second appears to be a "basic regexp" or grep-style regexp.
> >>
> >> This conversion feature in Lisp would be useful to add after the release.
> >
> > Very glad to hear that.
> >
> > I'm hoping there will also be support for toggling the newline sensitivity
> > of dot. This means a "doc-matches-newline" mode (aka "single-line" mode)
> > where `.' will also match newline. Please see the thread "short regexp to
> > match any character?" from 2006/03/04 and 03/11.
> 
> I don't know any other matcher where dot matches a newline.  Quite
> more relevant would be inverse character ranges like [^A-Z] that do
> _not_ match newline by default.

As far as I remember, Perl regexp syntax has a flag to match or not match
newline by default.  Emacs could adopt a similar flag facility, or use native
flag variables (like `case-fold-search', just with a different meaning.)

Paul

Re: using non-Emacs regexp syntax

[Jari Aalto]

Paul Pogonyshev <pogonyshev@gmx.net> writes:

> David Kastrup wrote:
> > "Drew Adams" <drew.adams@oracle.com> writes:
> > 
> > >>     Is there a function to convert non-Emacs regexps (e.g. "ab(c+|d)" to
> > >>     Emacs regexps (example to "ab\(c+\|d\)")?
> > >>
> > >> The first form appears to be an "extended regexp" or egrep-style regexp.
> > >> The second appears to be a "basic regexp" or grep-style regexp.
> > >>
> > >> This conversion feature in Lisp would be useful to add after the release.
> > >
> > > Very glad to hear that.
> > >
> > > I'm hoping there will also be support for toggling the newline sensitivity
> > > of dot. This means a "doc-matches-newline" mode (aka "single-line" mode)
> > > where `.' will also match newline. Please see the thread "short regexp to
> > > match any character?" from 2006/03/04 and 03/11.
> > 
> > I don't know any other matcher where dot matches a newline.  Quite
> > more relevant would be inverse character ranges like [^A-Z] that do
> > _not_ match newline by default.
> 
> As far as I remember, Perl regexp syntax has a flag to match or not match
> newline by default.  

Yes, it. It goes like this:

    "This\nLine"

    /.*/        "This"
    /.*/s       Make dot to match (m)ultiline: "This\nLine"

Common modifiers used are:

    i           Case (i)nsensitive match.
    g           (g)lobal match; all occurrances until last one
    m           (m)ultiline achors. The ^ and $ match in between the
                lines, not just at the beginning or end of string. 
    s           Change semantics of "." to also match \r or \n.
    x           e(x)tended. Treat all white space in regexp
                non-significant. Makes it possible to wrire readable
                regular expressions /Like  \s  This/
    
Jari

RE: using non-Emacs regexp syntax

[Drew Adams]

> > I don't know any other matcher where dot matches a newline.  Quite
> > more relevant would be inverse character ranges like [^A-Z] that do
> > _not_ match newline by default.
>
> As far as I remember, Perl regexp syntax has a flag to
> match or not match newline by default.

Exactly. This (perl's "single-line" mode) was mentioned in the thread I
cited, as well as the reason (use) for such a feature for Emacs.

> Emacs could adopt a similar flag facility,
> or use native flag variables (like `case-fold-search',
> just with a different meaning.)

Juri proposed a simple implementation: 'setting a new variable
`search-dot-regexp' to "\\(.\\|[\\n]\\)".'

Please see the (3-message) thread. This would be very handy for interactive
use of regexps, IMO, especially with a toggle command (bound to, say,
`C-.').

Re: using non-Emacs regexp syntax

[Stefan Monnier]

> Juri proposed a simple implementation: 'setting a new variable
> `search-dot-regexp' to "\\(.\\|[\\n]\\)".'

There's already a much more efficient implementation.  See regex.h:

   /* If this bit is set, then . matches newline.
      If not set, then it doesn't.  */
   #define RE_DOT_NEWLINE (RE_CONTEXT_INVALID_OPS << 1)


-- Stefan

RE: using non-Emacs regexp syntax

[Drew Adams]

> > Juri proposed a simple implementation: 'setting a new variable
> > `search-dot-regexp' to "\\(.\\|[\\n]\\)".'
>
> There's already a much more efficient implementation.  See regex.h:
>
>    /* If this bit is set, then . matches newline.
>       If not set, then it doesn't.  */
>    #define RE_DOT_NEWLINE (RE_CONTEXT_INVALID_OPS << 1)

Fine, as far as implementation goes. What's still needed then is Lisp access
(Lisp variable) and making a toggle command for the variable.

Re: using non-Emacs regexp syntax

[Stuart D. Herring]

> I don't know any other matcher where dot matches a newline.  Quite
> more relevant would be inverse character ranges like [^A-Z] that do
> _not_ match newline by default.

Not to compare the editors, or even really to inform the decision, but
just for complete information: dot always matches a newline in sed(1),
although often sed is matching against strings that contain no newlines,
since it takes a line at a time without the newline.  However, if newlines
are added to the pattern space, . matches them.

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or
too sparse, it is because mass-energy conversion has occurred during
shipping.

Re: using non-Emacs regexp syntax

[Paul Pogonyshev]

Richard Stallman wrote:
>     Is there a function to convert non-Emacs regexps (e.g. "ab(c+|d)" to
>     Emacs regexps (example to "ab\(c+\|d\)")?
> 
> The first form appears to be an "extended regexp" or egrep-style regexp.
> The second appears to be a "basic regexp" or grep-style regexp.
> 
> This conversion feature in Lisp would be useful to add after the release.

If you don't mind, I'll work on it now.  Changes can be added to whatever
.el file in the distribution later.

Also, is there sense in supporting conversion to and from several formats?
E.g. some require that plus operator is escaped, while everything else is
not.  E.g. something like this:

	(convert-regexp :sed :emacs some-regexp)
			FROM   TO   PATTERN-STRING

Of course, it will add more complexity, but it shouldn't be much of a
problem for users of this function and implementing it in Lisp should still
be not hard.

Paul

Re: using non-Emacs regexp syntax

[Stuart D. Herring]

[-- Attachment #1: Type: text/plain, Size: 2007 bytes --]

> If you don't mind, I'll work on it now.  Changes can be added to whatever
> .el file in the distribution later.
>
> Also, is there sense in supporting conversion to and from several formats?
> E.g. some require that plus operator is escaped, while everything else is
> not.  E.g. something like this:
>
> 	(convert-regexp :sed :emacs some-regexp)
> 			FROM   TO   PATTERN-STRING
>
> Of course, it will add more complexity, but it shouldn't be much of a
problem for users of this function and implementing it in Lisp should
still
> be not hard.

I've already started on this sort of thing, writing a converter just
between the two formats supported by GNU grep.  (These are
"GNU-extended-basic-RE" and "extended-RE with backreferences".)  As it
happens, that conversion can be done with one function because the formats
are so similar.  I had planned to go on to the more general case, but for
now I'll just provide what I have for comment and/or use.  (I have papers,
so any use is fine.)  If, Paul, you'd like, we can collaborate on this, or
one of us of your choice can go on with it.

For reference/goal purposes, I've been looking at the (somewhat outdated)
Mastering Regular Expressions and it describes these syntaxes:
1.  vi
2. (modern) grep
3. egrep
4. sed
5. lex
6. old awk
7. new awk(s) (don't know how different they really are from each other or
from old awk)
8. Emacs
9. Perl (obviously we can only convert a subset of Perl's syntax...)
10. Tcl
11. a Tcl library called Expect (although I don't know if/why it has a
different syntax from Tcl itself)
12. Python (complicated by the old regex and the new re packages, and how
the former had a variable syntax)

Hope it's helpful,
Davis

PS - I originally wrote this using some convenience macros of mine.  It
seems to work after I standardized it, but that's probably why if it
doesn't.

-- 
This product is sold by volume, not by mass.  If it appears too dense or
too sparse, it is because mass-energy conversion has occurred during
shipping.

[-- Attachment #2: convert-re.el --]
[-- Type: application/octet-stream, Size: 1396 bytes --]

;; Remember the exceedingly-basic regexes as used by sed(1)... might need to
;; support them too, although converting into them can be a pain.  Obviously,
;; in general you can't have just one function.

(defun convert-regexp (re)
	"Convert the regexp RE from basic to extended format or back."
	(let ((chars (string-to-list re)) ret backslash)
		(while chars
			(let ((curchar (car chars)))
				(cond
				 ((eq curchar ?\\)
					(unless (setq backslash (not backslash))
						(push ?\\ ret) (push ?\\ ret)))
				 ((eq curchar ?\[)
					(if backslash (progn (push ?\\ ret) (push ?\[ ret))
						;; Otherwise, it's a character class:
						(push ?\[ ret)
						(setq chars (cdr chars))
						(let ((level 1) (first 0))
							(while (and chars (> level 0))
								(let ((clch (car chars)))
									(push clch ret)
									(cond
									 ((eq clch ?\[) (incf level))
									 ((eq clch ?\]) (unless first (decf level)))
									 ((eq clch ?^) (if first (setq first t)))))
								(setq first (and first (unless (numberp first) 0)))
								(unless (zerop level) (setq chars (cdr chars)))))))
				 ((memq curchar (string-to-list "?+()|{}"))
					(unless backslash (push ?\\ ret))
					(push (car chars) ret))
				 (t (if backslash (push ?\\ ret)) (push (car chars) ret))))
			(setq backslash (and backslash (unless (numberp backslash) 0))
						chars (cdr chars)))
		(concat (nreverse ret))))

[-- Attachment #3: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: using non-Emacs regexp syntax

[Paul Pogonyshev]

Stuart D. Herring wrote:
> > If you don't mind, I'll work on it now.  Changes can be added to whatever
> > .el file in the distribution later.
> >
> > Also, is there sense in supporting conversion to and from several formats?
> > E.g. some require that plus operator is escaped, while everything else is
> > not.  E.g. something like this:
> >
> > 	(convert-regexp :sed :emacs some-regexp)
> > 			FROM   TO   PATTERN-STRING
> >
> > Of course, it will add more complexity, but it shouldn't be much of a
> > problem for users of this function and implementing it in Lisp should
> > still
> > be not hard.
> 
> I've already started on this sort of thing, writing a converter just
> between the two formats supported by GNU grep.  (These are
> "GNU-extended-basic-RE" and "extended-RE with backreferences".)  As it
> happens, that conversion can be done with one function because the formats
> are so similar.  I had planned to go on to the more general case, but for
> now I'll just provide what I have for comment and/or use.  (I have papers,
> so any use is fine.)  If, Paul, you'd like, we can collaborate on this, or
> one of us of your choice can go on with it.
> 
> [...]

I will happily pass this to you if you wish.  I planned a more generic
implementation which can be briefly described as this:

* Each implemented format provides a table of associations
  construct-name -> construct-generator (some constructs,  like []
  character class, will require a parameter.)  In the simplest form,
  construct-generator can be just a fixed string, which will suffice in
  most cases.

* Each format also provides a parser that splits a regexp into a list
  of construct-name.

* Entry function (or a helper for it) combines together a table for
  output format and a parser for input format.  The result is a regexp
  in output format.

Maybe it is too slow, though.  However, given that Emacs lived happily
without this sort of function, it can hardly be too slow.  But maybe
you can come up with a simpler solution.

(One more thing: it probably makes sense to add conversion function
for replacement strings too.  E.g. some formats require $N, some
(like Emacs) use \N for referencing the matched group.)

Paul

Re: using non-Emacs regexp syntax

[Juri Linkov]

> * Each implemented format provides a table of associations
>   construct-name -> construct-generator (some constructs,  like []
>   character class, will require a parameter.)  In the simplest form,
>   construct-generator can be just a fixed string, which will suffice in
>   most cases.
>
> * Each format also provides a parser that splits a regexp into a list
>   of construct-name.
>
> * Entry function (or a helper for it) combines together a table for
>   output format and a parser for input format.  The result is a regexp
>   in output format.

A good implementation would parse a regexp string to the sregex
Lisp-like syntax (qv emacs-lisp/sregex.el) which would allow doing
such things as (sregexq (a-new-function-to-parse-Perl-regexp PERL-REGEXP))
to convert a non-Emacs regexp to the Emacs one.

PS: I would much prefer to use this feature in query-replace-regexp
with an option to use extended regexps instead of basic regexps in its
interactive arguments.

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: using non-Emacs regexp syntax

[Stefan Monnier]

> I've already started on this sort of thing, writing a converter just
> between the two formats supported by GNU grep.  (These are

BTW, if the output of your function is only ever passed to Emacs, then it
may be worth it to instead provide ways to access from elisp the full
functionality of the underlying features of the regexp.c code (which was
originally not specific to Emacs and has flags to support various syntax
options, including whether { ( | and friends should be backslashed or not.
See regexp.h).
Of course, maybe both would be useful, depending on the application.


        Stefan