Re: Raw strings (experimental patches inside)

Re: Raw strings (experimental patches inside)

[Dmitry Gutov]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:
 >> If insults won't persuade you, here are some real benefits.
 >> raw-strings are a huge convenience when writing.[1]  They correspond
 >> to the way you enter a regexp to isearch, and to the documentation.
 >
 > And here we're back at regexps.  I already agreed that they're
 > convenient for regexps, but pointed out that a better solution would
 > be to fix the regexp syntax so it doesn't backslash-escape every
 > special character.

This would be a backward-incompatible change to the regexp engine,
wouldn't it?

--Dmitry

Re: Raw strings (experimental patches inside)

[Stephen J. Turnbull]

Dmitry Gutov writes:

 > This would be a backward-incompatible change to the regexp engine,
 > wouldn't it?

Not to the matching engine, but to the regexp compiler.

It would be reasonable easy to implement backward-compatibly:  if a
string's first character has a non-nil "punc-are-operators" property,
then all punctuation would be interpreted as regex operators *unless*
they are escaped.

Neither the property name nor the API are likely to be optimal, but it
would work.  You could also do it with some kind of variable.

Re: Raw strings (experimental patches inside)

[Dmitry Gutov]

On 11.08.2012 11:49, Stephen J. Turnbull wrote:
>   > This would be a backward-incompatible change to the regexp engine,
>   > wouldn't it?
>
> Not to the matching engine, but to the regexp compiler.
>
> It would be reasonable easy to implement backward-compatibly:  if a
> string's first character has a non-nil "punc-are-operators" property,
> then all punctuation would be interpreted as regex operators *unless*
> they are escaped.

At which point would the string obtain this text property? If we're 
discussing the case of using it in Elisp source code, this would need to 
be done in the reader, no? Which brings us back to "special syntax".

To be clear, I'm in favor of special syntax for regexps (raw strings or 
otherwise), and I was trying to argue that adding special syntax might 
be easier than work around the backward compatibility problem.

> Neither the property name nor the API are likely to be optimal, but it
> would work.  You could also do it with some kind of variable.

I suppose it would work if the new variable is used similarly to 
`lexical-binding'.

Re: Raw strings (experimental patches inside)

[Andreas Schwab]

Dmitry Gutov <dgutov@yandex.ru> writes:

> At which point would the string obtain this text property? If we're
> discussing the case of using it in Elisp source code, this would need to
> be done in the reader, no? Which brings us back to "special syntax".

We have already a read syntax for text properties (though not particular
human writable).

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Raw strings (experimental patches inside)

[Dmitry Gutov]

On 11.08.2012 21:57, Andreas Schwab wrote:
> Dmitry Gutov <dgutov@yandex.ru> writes:
>
>> At which point would the string obtain this text property? If we're
>> discussing the case of using it in Elisp source code, this would need to
>> be done in the reader, no? Which brings us back to "special syntax".
>
> We have already a read syntax for text properties (though not particular
> human writable).

Yes, I don't think anyone would consider #("(\\w+)|123z" 0 1 (foo t)) a 
good substitute for /(\w+)|123z/.

Re: Raw strings (experimental patches inside)

[Stephen J. Turnbull]

Dmitry Gutov writes:

 > At which point would the string obtain this text property? If we're 
 > discussing the case of using it in Elisp source code, this would need to 
 > be done in the reader, no? Which brings us back to "special
 > syntax".

Nothing "needs" to be done in the Lisp reader.  Certainly it would be
more convenient that way, and I recommend it.

Re: Raw strings (experimental patches inside)

[Stefan Monnier]

> This would be a backward-incompatible change to the regexp engine,
> wouldn't it?

Clearly, we'd need to make such a change in a way that doesn't break all
the existing Elisp code.


        Stefan

Raw strings (experimental patches inside)

[Aurélien Aptel]

[-- Attachment #1: Type: text/plain, Size: 864 bytes --]

Hi all,

When I type a regex I'm always annoyed by the amount of escaping I have to do.
I've always wished Emacs Lisp had raw strings ie. a syntax to type
litteral text without interpretation.

I've made 2 patch for the reader (src/lread.c). There are proofs of
concepts, they should work on correct input but don't expect much.

raw-string-python.diff use a syntax similar to python:

$ ./emacs -Q -batch --eval '(message #r"""ha"\nha""")'
ha"\nha

raw-string-sed.diff use a syntax similar to sed or perl quotes. You
can chose any delimiter.

$ ./emacs -Q -batch --eval '(message #r,ha"\nha,)'
ha"\nha
$ ./emacs -Q -batch --eval '(message #r~ha"\nha~)'
ha"\nha

You get the idea.

Although the reader works, this breaks several things. C-x C-e doesn't
work well, sexp navigation is broken, etc. There is work to do to make
the rest of emacs aware of raw strings.

[-- Attachment #2: raw-string-python.diff --]
[-- Type: application/octet-stream, Size: 2819 bytes --]

=== modified file 'src/lread.c'
--- src/lread.c	2012-08-01 20:51:44 +0000
+++ src/lread.c	2012-08-03 01:45:01 +0000
@@ -2384,6 +2384,84 @@
 
     case '#':
       c = READCHAR;
+
+      /* raw string with """ delimiters
+         #r"""foo bar""" */
+      if (c == 'r')
+        {
+          int old;
+          int ndelim = 0;
+          char *p = read_buffer;
+          char *end = read_buffer + read_buffer_size;
+          register int ch;
+          /* Nonzero if we saw an escape sequence specifying
+             a multibyte character.  */
+          int force_multibyte = 0;
+          /* Nonzero if we saw an escape sequence specifying
+             a single-byte character.  */
+          int force_singlebyte = 0;
+          int cancel = 0;
+          ptrdiff_t nchars = 0;
+
+          /* read 3 first delim */
+          if (READCHAR != '"' || READCHAR != '"' || READCHAR != '"')
+            invalid_syntax ("#r\"\"\"...\"\"\"");
+
+          while (1)
+            {
+              ch = READCHAR;
+              if (ch < 0)
+                break;
+
+              if (ch == '"')
+                {
+                  ndelim++;
+                  if (ndelim == 3)
+                    break;
+                }
+              else
+                {
+                  ndelim = 0;
+                }
+
+              if (end - p < MAX_MULTIBYTE_LENGTH)
+                {
+                  ptrdiff_t offset = p - read_buffer;
+                  if (min (PTRDIFF_MAX, SIZE_MAX) / 2 < read_buffer_size)
+                    memory_full (SIZE_MAX);
+                  read_buffer = xrealloc (read_buffer, read_buffer_size * 2);
+                  read_buffer_size *= 2;
+                  p = read_buffer + offset;
+                  end = read_buffer + read_buffer_size;
+                }
+
+              p += CHAR_STRING (ch, (unsigned char *) p);
+              if (CHAR_BYTE8_P (ch))
+                force_singlebyte = 1;
+              else if (! ASCII_CHAR_P (ch))
+                force_multibyte = 1;
+
+              nchars++;
+            }
+
+          /* last " was not added, only remove 2 */
+          p -= 2;
+          nchars -= 2;
+
+          if (! force_multibyte && force_singlebyte)
+            {
+              /* READ_BUFFER contains raw 8-bit bytes and no multibyte
+                 forms.  Convert it to unibyte.  */
+              nchars = str_as_unibyte ((unsigned char *) read_buffer,
+                                       p - read_buffer);
+              p = read_buffer + nchars;
+            }
+
+          return make_specified_string (read_buffer, nchars, p - read_buffer,
+                                        (force_multibyte
+                                         || (p - read_buffer != nchars)));
+        }
+
       if (c == 's')
 	{
 	  c = READCHAR;


[-- Attachment #3: raw-string-sed.diff --]
[-- Type: application/octet-stream, Size: 2306 bytes --]

=== modified file 'src/lread.c'
--- src/lread.c	2012-08-01 20:51:44 +0000
+++ src/lread.c	2012-08-03 00:27:01 +0000
@@ -2384,6 +2384,60 @@
 
     case '#':
       c = READCHAR;
+
+      /* raw string with custom delimiter 
+         #r(foo) #r,foo, etc */
+      if (c == 'r')
+        {
+          int delimiter = READCHAR;
+          char *p = read_buffer;
+          char *end = read_buffer + read_buffer_size;
+          register int ch;
+          /* Nonzero if we saw an escape sequence specifying
+             a multibyte character.  */
+          int force_multibyte = 0;
+          /* Nonzero if we saw an escape sequence specifying
+             a single-byte character.  */
+          int force_singlebyte = 0;
+          int cancel = 0;
+          ptrdiff_t nchars = 0;
+
+          while ((ch = READCHAR) >= 0
+                 && ch != delimiter)
+            {
+              if (end - p < MAX_MULTIBYTE_LENGTH)
+                {
+                  ptrdiff_t offset = p - read_buffer;
+                  if (min (PTRDIFF_MAX, SIZE_MAX) / 2 < read_buffer_size)
+                    memory_full (SIZE_MAX);
+                  read_buffer = xrealloc (read_buffer, read_buffer_size * 2);
+                  read_buffer_size *= 2;
+                  p = read_buffer + offset;
+                  end = read_buffer + read_buffer_size;
+                }
+              
+              p += CHAR_STRING (ch, (unsigned char *) p);
+              if (CHAR_BYTE8_P (ch))
+                force_singlebyte = 1;
+              else if (! ASCII_CHAR_P (ch))
+                force_multibyte = 1;
+              
+              nchars++;
+            }
+          if (! force_multibyte && force_singlebyte)
+            {
+              /* READ_BUFFER contains raw 8-bit bytes and no multibyte
+                 forms.  Convert it to unibyte.  */
+              nchars = str_as_unibyte ((unsigned char *) read_buffer,
+                                       p - read_buffer);
+              p = read_buffer + nchars;
+            }
+
+          return make_specified_string (read_buffer, nchars, p - read_buffer,
+                                        (force_multibyte
+                                         || (p - read_buffer != nchars)));
+        }
+
       if (c == 's')
 	{
 	  c = READCHAR;

Re: Raw strings (experimental patches inside)

[Pascal J. Bourguignon]

Aurélien Aptel <aurelien.aptel+emacs@gmail.com> writes:

> Hi all,
>
> When I type a regex I'm always annoyed by the amount of escaping I have to do.
> I've always wished Emacs Lisp had raw strings ie. a syntax to type
> litteral text without interpretation.
>
> I've made 2 patch for the reader (src/lread.c). There are proofs of
> concepts, they should work on correct input but don't expect much.
>
> raw-string-python.diff use a syntax similar to python:

Why go to python, when Common Lisp has a perfectly nice reader macro
system to do that kind of things?

-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
A bad day in () is better than a good day in {}.

Re: Raw strings (experimental patches inside)

[Aurélien Aptel]

On Fri, Aug 3, 2012 at 11:45 AM, Pascal J. Bourguignon
<pjb@informatimago.com> wrote:
> Why go to python, when Common Lisp has a perfectly nice reader macro
> system to do that kind of things?

This is simple a proof of concept. If someone is interested by porting
the CL reader macro to emacs I'm also fine with it but it's a lot more
work.

I've made a little article on the Emacs Lisp reader while I was doing
this. It's available here:
http://definitelyaplug.b0.cx/post/A-look-at-Emacs-Lisp-reader

Re: Raw strings (experimental patches inside)

[Pascal J. Bourguignon]

Aurélien Aptel <aurelien.aptel+emacs@gmail.com> writes:

> On Fri, Aug 3, 2012 at 11:45 AM, Pascal J. Bourguignon
> <pjb@informatimago.com> wrote:
>> Why go to python, when Common Lisp has a perfectly nice reader macro
>> system to do that kind of things?
>
> This is simple a proof of concept. If someone is interested by porting
> the CL reader macro to emacs I'm also fine with it but it's a lot more
> work.

I have a CL reader, written in CL.  One could port it to emacs-24 easily
enough I'd say.  It's AGPL3, I wouldn't mind assigning the copyright of
a derived work of it to the FSF for inclusion in emacs.


> I've made a little article on the Emacs Lisp reader while I was doing
> this. It's available here:
> http://definitelyaplug.b0.cx/post/A-look-at-Emacs-Lisp-reader

Thanks.
-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
A bad day in () is better than a good day in {}.

Re: Raw strings (experimental patches inside)

[Aurélien Aptel]

On Sat, Aug 4, 2012 at 9:41 PM, Pascal J. Bourguignon
<pjb@informatimago.com> wrote:
> Aurélien Aptel <aurelien.aptel+emacs@gmail.com> writes:
> I have a CL reader, written in CL.  One could port it to emacs-24 easily
> enough I'd say.  It's AGPL3, I wouldn't mind assigning the copyright of
> a derived work of it to the FSF for inclusion in emacs.

That's nice! Could you put it somewhere online to let curious people
and potential porters see it?

Re: Raw strings (experimental patches inside)

[Pascal J. Bourguignon]

Aurélien Aptel <aurelien.aptel+emacs@gmail.com> writes:

> On Sat, Aug 4, 2012 at 9:41 PM, Pascal J. Bourguignon
> <pjb@informatimago.com> wrote:
>> Aurélien Aptel <aurelien.aptel+emacs@gmail.com> writes:
>> I have a CL reader, written in CL.  One could port it to emacs-24 easily
>> enough I'd say.  It's AGPL3, I wouldn't mind assigning the copyright of
>> a derived work of it to the FSF for inclusion in emacs.
>
> That's nice! Could you put it somewhere online to let curious people
> and potential porters see it?

Of course, it's so obvious to me, I forgot to mention its url:
https://gitorious.org/com-informatimago/com-informatimago/blobs/master/common-lisp/lisp-reader/reader.lisp

-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
A bad day in () is better than a good day in {}.

Re: Raw strings (experimental patches inside)

[Lars Brinkhoff]

Pascal J. Bourguignon wrote:
> Aurélien Aptel wrote:
>> Pascal J. Bourguignon wrote:
>>> Why go to python, when Common Lisp has a perfectly nice reader
>>> macro system to do that kind of things?
>> This is simple a proof of concept. If someone is interested by
>> porting the CL reader macro to emacs I'm also fine with it but it's
>> a lot more work.
> I have a CL reader, written in CL.

Here's one written in Emacs Lisp:

https://github.com/larsbrinkhoff/emacs-cl/blob/master/src/cl-reader.el

(Part of a CL implementation in elisp.)

Re: Raw strings (experimental patches inside)

[Stefan Monnier]

> I have a CL reader, written in CL.  One could port it to emacs-24 easily
> enough I'd say.  It's AGPL3, I wouldn't mind assigning the copyright of
> a derived work of it to the FSF for inclusion in emacs.

I think the problem is to get emacs-lisp-mode to understand it.


        Stefan

Re: Raw strings (experimental patches inside)

[Pascal J. Bourguignon]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I have a CL reader, written in CL.  One could port it to emacs-24 easily
>> enough I'd say.  It's AGPL3, I wouldn't mind assigning the copyright of
>> a derived work of it to the FSF for inclusion in emacs.
>
> I think the problem is to get emacs-lisp-mode to understand it.

Indeed, CL reader macros are a problem for editors.  But the only practical
solution involves having the CL reader (and thus the whole CL language)
implemented in the editor.  Therefore including emacs-cl would indeed be
more indicated (bar a rewrite of emacs in Common Lisp).


In general, you need to be able to read the reader macro to know where
it starts and where it ends.  Reading is recursive, and reading a reader
macro may involve reading other embedded reader macros, thus you can
obtain a tree of ranges.

For indenting, we need in general more information, but this could be
provided by a table mapping reader macros and indenting rules.

To give a taste of the problem, here is an example of the kind of Common
Lisp code I'm working with now:

    (with-handle (viewh view)
      [viewh setNeedsDisplayInRect:(nsrect #@(10 10) 
                                           #@(200 100))])

Here, we have:
- one standard CL reader macro for #\(
- one non standardreader macro for #\[
- one non standard dispatching reader macro for #\# #\@.

The syntax of the text inside [] is complex, similar to Objective-C, and
we would like to have Objective-C-like font-locking and indenting, but
the parts in parentheses in this reader macro are actually read by the
standard CL reader macro for #\(, and here we want the normal Common
Lisp font-locking and indenting.

The dispatching reader macro for #\# #\@ here is a usual example of CL
reader macros, where the syntax following it is read with the standard
CL lisp reader (contrarily to what's read by the #\[ reader macro).

That information however can only be learned by executing the reader
macro and taking note of what reader macros are run.  There are
difficulties such as the fact that reader macros may seek in the stream
and read some parts twice, but a conforming reader macro should not have
side effects preventing its repeated use on the same text, so the editor
itself can do that.


-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
A bad day in () is better than a good day in {}.

Re: Raw strings (experimental patches inside)

[Stefan Monnier]

>> I think the problem is to get emacs-lisp-mode to understand it.
> Indeed, CL reader macros are a problem for editors.
[...examples of complex situations...]

I do not think the benefits justify such complexity at this point.


        Stefan

Re: Raw strings (experimental patches inside)

[Pascal J. Bourguignon]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>>> I think the problem is to get emacs-lisp-mode to understand it.
>> Indeed, CL reader macros are a problem for editors.
> [...examples of complex situations...]
>
> I do not think the benefits justify such complexity at this point.

Specifically, it would only target one user class of emacs: the Common
Lisp programmers.  The problem is that if CL programmers eventually fork
to their own clone of emacs, a lot of them won't be programming emacs
lisp anymore.  This would not necessarily be a better situation.

Moreover, notice that it hints to a general solution to the multi-mode
problem.

-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
A bad day in () is better than a good day in {}.

Re: Raw strings (experimental patches inside)

[Stefan Monnier]

> When I type a regex I'm always annoyed by the amount of escaping
> I have to do.  I've always wished Emacs Lisp had raw strings
> ie. a syntax to type litteral text without interpretation.

AFAIK, the only real use-case is indeed regexp, and for them a better
solution would be to not escape the parentheses at all.

> I've made 2 patch for the reader (src/lread.c). There are proofs of
> concepts, they should work on correct input but don't expect much.

Adapting the reader is only one part, since you also need to adapt the
elisp-mode syntax-tables accordingly (and check that the edebug reader
doesn't need matching changes).

As mentioned above, I'm not too excited by the idea of raw strings and
would prefer solving the underlying root cause of dissatisfaction.
Actually, we already have such a thing with `rx', but admittedly,
a strings representation of regexps tends to be more concise, so maybe
we also need some other trick like maybe a (re-escape "(?:foo)") which
would return "\\(?:foo\\)".


        Stefan

Re: Raw strings (experimental patches inside)

[Ivan Andrus]

On Aug 4, 2012, at 12:43 AM, Stefan Monnier wrote:

>> When I type a regex I'm always annoyed by the amount of escaping
>> I have to do.  I've always wished Emacs Lisp had raw strings
>> ie. a syntax to type litteral text without interpretation.
> 
> AFAIK, the only real use-case is indeed regexp, and for them a better
> solution would be to not escape the parentheses at all.

That doesn't handle the horror that is searching for backslashes...

>> I've made 2 patch for the reader (src/lread.c). There are proofs of
>> concepts, they should work on correct input but don't expect much.
> 
> Adapting the reader is only one part, since you also need to adapt the
> elisp-mode syntax-tables accordingly (and check that the edebug reader
> doesn't need matching changes).
> 
> As mentioned above, I'm not too excited by the idea of raw strings and
> would prefer solving the underlying root cause of dissatisfaction.
> Actually, we already have such a thing with `rx', but admittedly,
> a strings representation of regexps tends to be more concise, so maybe
> we also need some other trick like maybe a (re-escape "(?:foo)") which
> would return "\\(?:foo\\)".

It would also be nice to have a function which takes a string representation and converts it to an `rx' representation, which I couldn't find in rx when I looked.  I would like to convert some old libraries to use rx, but I'd rather not have to retype them all with the possibility of making mistakes etc.  

I think pcre2el [1] has such a thing, though I haven't tried it yet.  Is that on your radar for things to potentially include in Emacs?

-Ivan

[1] https://github.com/joddie/pcre2el

Re: Raw strings (experimental patches inside)

[Stefan Monnier]

>>> When I type a regex I'm always annoyed by the amount of escaping
>>> I have to do.  I've always wished Emacs Lisp had raw strings
>>> ie. a syntax to type litteral text without interpretation.
>> AFAIK, the only real use-case is indeed regexp, and for them a better
>> solution would be to not escape the parentheses at all.
> That doesn't handle the horror that is searching for backslashes...

That is not a problem significant enough to warrant the introduction of
something like raw-strings.

> It would also be nice to have a function which takes a string representation
> and converts it to an `rx' representation, which I couldn't find in rx when
> I looked.

Indeed, we don't have that right now.
My lex.el includes something pretty close, but I'd rather not include
lex.el into Emacs because it's not clean enough.  OTOH maybe I could add
it to GNU ELPA.


        Stefan

Re: Raw strings (experimental patches inside)

[Aurélien Aptel]

> That is not a problem significant enough to warrant the introduction of
> something like raw-strings.

When I showed my patch on IRC it was well received. We should wait for
more feedbacks from users.

Re: Raw strings (experimental patches inside)

[Stefan Monnier]

>> That is not a problem significant enough to warrant the introduction of
>> something like raw-strings.
> When I showed my patch on IRC it was well received. We should wait for
> more feedbacks from users.

I'd guess that those IRC members don't spend their time maintaining
Emacs ;-)


        Stefan

Re: Raw strings (experimental patches inside)

[Vr Rm]

Hi,

Sorry for jumping in on this thread so late!  But this is the biggest, 
by far, usability problem I have with Emacs.  I don't really have a good 
idea of the amount of effort raw strings would be (in my eyes, pretty 
much all all Emacs changes seem to have been accomplish by 
super-humans.)  But the difficulty in using Emacs regular expressions 
shouldn't be underestimated, especially for those of us on the dim side 
of the superhuman spectrum.

Several years ago I finally bit the bullet and decided that I was going 
to figure out how to find and replace buffer carriage returns in a file 
I was editing even if I had to instrument the sum total of Emacs' elisp 
code and single step through it all defun by defun. It took me an 
ungodly amount of time, I think over an hour, before I finally managed 
to find the solution, I know that it had something to do with c-j.  And 
when I blogged about it, it was  and still is by far the most trafficked 
item I have ever posted.  And now that the blog is long defunct, I still 
don't know how to search for tabs, carriage returns or use Emacs regular 
expressions, meaning that I still, to this day -- in 2012!, have to use 
grep and awk to do do regular expression editing operations that were 
standard issue for editors in the 80s.  I do love Emacs and it would 
take vastly more than that to dissuade me from using it for pretty much 
all my editing.  And of course the capability for these things is there, 
I just don't either have the natural aptitude or the inclination to 
learn them.  (though I have tried)

Still, I'm certainly not alone in my inability.  Regular expression 
handling continues I would say is the biggest Bugbear for many people 
learning Emacs or using it to it's full capabilities.  And it's 
obviously a core piece of editor functionality.


On 08/04/2012 04:47 PM, Stefan Monnier wrote:
>>>> When I type a regex I'm always annoyed by the amount of escaping
>>>> I have to do.  I've always wished Emacs Lisp had raw strings
>>>> ie. a syntax to type litteral text without interpretation.
>>> AFAIK, the only real use-case is indeed regexp, and for them a better
>>> solution would be to not escape the parentheses at all.
>> That doesn't handle the horror that is searching for backslashes...
>
> That is not a problem significant enough to warrant the introduction of
> something like raw-strings.
>
>> It would also be nice to have a function which takes a string representation
>> and converts it to an `rx' representation, which I couldn't find in rx when
>> I looked.
>
> Indeed, we don't have that right now.
> My lex.el includes something pretty close, but I'd rather not include
> lex.el into Emacs because it's not clean enough.  OTOH maybe I could add
> it to GNU ELPA.
>
>
>          Stefan
>
>

Re: Raw strings (experimental patches inside)

[Stephen J. Turnbull]

Vr Rm writes:

 > But the difficulty in using Emacs regular expressions shouldn't be
 > underestimated, especially for those of us on the dim side of the
 > superhuman spectrum.

Nothing personal, but if you can't even search for tabs (C-s C-q TAB
will do), of course you're going to find regexps difficult.  It's hard
to see how they can be simplified -- as used in all languages I know,
regexps are extremely compact notation.  Such notation is going to be
non-trivial if you can't remember to use C-q to insert command
characters literally, and I don't think raw strings are going to help.
(A sane assignment of character classes such that punctuation is
always an operator, as in Perl or Python, would be much more mnemonic
help, I think.)

You're going to have to appeal from a somewhat more proficient level;
Emacsen are designed to be used with a vocabulary of editing gestures
that is only about one order of magnitude smaller than the English
language.

That said,

 > > That is not a problem significant enough to warrant the
 > > introduction of something like raw-strings.

Oh, come on, Stefan.  That's just elitism.

If insults won't persuade you, here are some real benefits.
raw-strings are a huge convenience when writing.[1]  They correspond to
the way you enter a regexp to isearch, and to the documentation.  They
are a big win for readability, both because of the greatly increased
S/N in long complex regexps, and because they reduce the number of
line-breaks needed for medium-complexity regexps in search functions
and the like.

The costs are about as small as they get.  Raw strings are extremely
easy to implement directly in the reader (OK, you'll probably
duplicate some of the code in regular string lexing, but really...),
if you use #r"" as XEmacsen do they can't break working code, and the
implementation is highly localized and therefore subject to
refactoring if later you decide to do it terms of a generic reader
macro rather than C code in the lexer.

And if that doesn't work, how about "S?XEmacs has them and Emacs
doesn't, nyaaah, nyaaah, nyaaah!"?

At least think one more time about it.  And maybe about a syntax for
PCRE regexps.  (Why a syntax?  Because wrapping them in a function is
more unnecessary verbosity, as would be duplicating all regexp-using
functions with pcre- versions.)

Footnotes: 
[1]  One could bind ?\\ to (lambda () (interactive) (insert "\\")),
but that would require a mode switch on entry and exit which is a
PITA itself, and not as visible to the author as a raw string syntax.

Re: Raw strings (experimental patches inside)

[Stefan Monnier]

>> > That is not a problem significant enough to warrant the
>> > introduction of something like raw-strings.
> Oh, come on, Stefan.  That's just elitism.

No.  My immediate response was simply based on the fact that searching
for backslashes is not something people do often.  Of course, if I had
taken the time to think about it some more, I'd have pointed out that
raw strings don't help with interactive use of Emacs but only with
writing Elisp code, so "searching for backslashes" is even less of
a justification for introducing raw strings.

> If insults won't persuade you, here are some real benefits.
> raw-strings are a huge convenience when writing.[1]  They correspond to
> the way you enter a regexp to isearch, and to the documentation.  They

And here we're back at regexps.  I already agreed that they're
convenient for regexps, but pointed out that a better solution would be
to fix the regexp syntax so it doesn't backslash-escape every
special character.
That would help not just Elisp coders but Emacs users as well and would
make Elisp regexps even more readable than raw strings can ever hope to
make them.


        Stefan

Re: Raw strings (experimental patches inside)

[Stephen J. Turnbull]

Stefan Monnier writes:

 > And here we're back at regexps.  I already agreed that they're
 > convenient for regexps, but pointed out that a better solution would be
 > to fix the regexp syntax so it doesn't backslash-escape every
 > special character.

Yeah, I know -- I pointed that out too.  That's still not a substitute
for raw strings in code.  You still need to *double*-backslash-escape,
backslash-escaping is insufficient.  It's only "better" if you have
rawstring phobia.

 > That would help not just Elisp coders but Emacs users as well and
 > would make Elisp regexps even more readable than raw strings can
 > ever hope to make them.

True, but both is better.

Re: Raw strings (experimental patches inside)

[Stefan Monnier]

>> And here we're back at regexps.  I already agreed that they're
>> convenient for regexps, but pointed out that a better solution would be
>> to fix the regexp syntax so it doesn't backslash-escape every
>> special character.
> Yeah, I know -- I pointed that out too.  That's still not a substitute
> for raw strings in code.

Why not?

> You still need to *double*-backslash-escape, backslash-escaping is
> insufficient.

If the special chars don't need to be backslash escaped, then you don't
need to double escape either, obviously.

> It's only "better" if you have rawstring phobia.

I don't have rawstring phobia.  I just think it's a workaround which
makes Elisp's syntax more complex without fixing the real problem.

>> That would help not just Elisp coders but Emacs users as well and
>> would make Elisp regexps even more readable than raw strings can
>> ever hope to make them.
> True, but both is better.

I still haven't heard of a good argument why you'd still need
raw-strings if the regexp syntax was made not to need backslashes for
most special chars.
Could it make the syntax slightly better?  Yes.  Would the difference
be significant?  I doubt it.


        Stefan

Re: Raw strings (experimental patches inside)

[Stephen J. Turnbull]

Stefan Monnier writes:

 > > You still need to *double*-backslash-escape, backslash-escaping is
 > > insufficient.
 > 
 > If the special chars don't need to be backslash escaped, then you don't
 > need to double escape either, obviously.

Of course they need to be backslash-escaped, fairly frequently.  I
often search for things like '(foo' and '?\' in Lisp, '{' and '}' in
Python or C, '*' in ReST, etc, because I want to find something in a
particular syntactic context.  There's a reason why basic REs
backslashed some of the basic operators, it wasn't just an epidemic of
early-onset Alzheimer's at Bell Labs.  They were wrong to put weight
on that reason, of course, but they realized their mistake a lot
faster than we Emacs developers did. :-(

It's true that a saner regexp syntax would reduce the need, but only
rawstrings can help with stuff like the regexp needed to find an
end-of-row in LaTeX arrays: "\\\\\\\\".  Those of us with a bit of
astigmatism can't even be sure that's 8 backslashes without careful
counting.

 > I don't have rawstring phobia.  I just think it's a workaround which
 > makes Elisp's syntax more complex without fixing the real problem.

Off-by-one error!  The point is that there are *two* problems here.

I don't know if it's a good idea, but you could even fix both with one
syntax.  Ie, when you read a rawstring, the resulting string is
automatically given a "if-you-use-me-as-a-regex-use-pcre-syntax-please"
property.

 > Could it make the syntax slightly better?  Yes.  Would the difference
 > be significant?  I doubt it.

Works-for-me is not a good way to design an Emacs for other people.

BTW, I was -0.5 on rawstrings when they were introduced in XEmacs.  I
was wrong then, and I had better reasons (ie, Emacs compatibility) for
being negative.

Re: Raw strings (experimental patches inside)

[Dmitri Paduchikh]

Stefan Monnier:

>>> And here we're back at regexps.  I already agreed that they're
>>> convenient for regexps, but pointed out that a better solution would be
>>> to fix the regexp syntax so it doesn't backslash-escape every
>>> special character.
>> Yeah, I know -- I pointed that out too.  That's still not a substitute
>> for raw strings in code.

SM> Why not?

>> You still need to *double*-backslash-escape, backslash-escaping is
>> insufficient.

SM> If the special chars don't need to be backslash escaped, then you don't
SM> need to double escape either, obviously.

I like the Lua way of solving this problem. They use different
characters for string and regexp escaping (\ and % correspondingly).
Choosing the same character for both purposes really creates a huge
opportunity for confusion, IMO.

$ lua
Lua 5.2.1  Copyright (C) 1994-2012 Lua.org, PUC-Rio
> =string.find('-xyz-', '(%w+)')
2	4	xyz
> 

But they also do not escape parentheses and have raw strings in the form
of `[[...]]`.

-- 
With best regards,
Dmitri Paduchikh

Re: Raw strings (experimental patches inside)

[Stephen J. Turnbull]

Dmitri Paduchikh writes:

 > I like the Lua way of solving this problem.

So write a Lua-macs.  This is not a good idea for mainline Emacs, for
backward compatibility reasons.

 > They use different characters for string and regexp escaping (\ and
 > % correspondingly).

Definitely % is inappropriate, since it's used for string formatting,
and I've seen many applications of regexps that use regexps built with
`format'.  Talk about huge opportunities for confusion!

Re: Raw strings (experimental patches inside)

[Vr Rm]

On 08/09/2012 10:08 PM, Stephen J. Turnbull wrote:
> Vr Rm writes:
>
>   > But the difficulty in using Emacs regular expressions shouldn't be
>   > underestimated, especially for those of us on the dim side of the
>   > superhuman spectrum.
>
> Nothing personal, but if you can't even search for tabs (C-s C-q TAB
> will do), of course you're going to find regexps difficult.  It's hard
> to see how they can be simplified -- as used in all languages I know,
> regexps are extremely compact notation.  Such notation is going to be
> non-trivial if you can't remember to use C-q to insert command
> characters literally, and I don't think raw strings are going to help.
> (A sane assignment of character classes such that punctuation is
> always an operator, as in Perl or Python, would be much more mnemonic
> help, I think.)

I do totally agree that I'm appealing from the lower 50%, maybe even the 
top 90%, of the population of programming proficiency.  For myself, I 
wish it weren't true but alas it is.  (But as a profession does, as a 
middle aged man -- for which there's nothing else I'm qualified to do, 
nor would anyone have the slightest interest hiring me to learn -- keep 
me from going homeless and hungry and from becoming a burden to society.)

However, I have been using Emacs for 20 years, since 1992 when it was 
installed in our university's brand new Sparc stations, on a somewhat 
continual basis. (And for the last 5 years intensely for more or less 
all my programming tasks.)  In addition, I use regular expressions, if 
not ever day, quite often in a variety of languages, Javascript, Groovy, 
Python and Clojure being my favourite implementations.  And, while they 
all have their own idiosyncrasy I can move between them pretty easily. 
Yet every time I try to use Emacs regular expressions, either on the 
command line or in elisp, for anything more complex than "ca+dr", I'm 
thwarted.

As for the TAB, yes I completely agree, Emacs has a fully complete 
regular expression system.  Yet, I always fail at using it.  I thought 
I'd try it again just to see if perhaps things have improved or maybe I 
was miss remembering but if I want to search for:

/.*\)$/

i.e. a parenthesis at the end of a line.  I can't do it!  Same is true 
for a search find all tabs not at the beginning of the line and not 
preceded by a tab:

/^[^\t]+\t/

It's entirely possible that the above are actually close.  Or that the 
Emacs versions are even more efficient and compact. But I can't figure 
them out without spending a bunch of time reading through info, and 
then, by the next time I return to them, I will have forgotten the Emacs 
specific quirks.  (Or can only remember the quirks of other languages, 
if you prefer.)

Again, I'm fully aware that with respect to Emacs proficiency (and 
probably programming proficiency in general) I'm far below most 
subscribers to this list and certainly the contributors.

And I'm not really arguing this feature for myself, but rather, on 
humanitarian grounds if you will :) ...  as a single data point of a 
population that would dramatically benefit if they could actually use 
Emacs regular expressions.

Re: Raw strings (experimental patches inside)

[Davis Herring]

> /.*\)$/

This would work in Perl (among other places); in Emacs you just drop the
\.  That is, type

C-M-s ) $

(You needn't include the .* unless you want to match the whole line.)
In Elisp, type

")$"

as the regexp.  (No //.)

> /^[^\t]+\t/

Keyboard: C-M-s ^ [ ^ TAB ] + TAB
Elisp: "^[^\t]+\t" (exactly what you wrote, without the // delimiters.

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or
too sparse, it is because mass-energy conversion has occurred during
shipping.

Re: Raw strings (experimental patches inside)

[Stephen J. Turnbull]

Vr Rm writes:

 > /.*\)$/
 > 
 > i.e. a parenthesis at the end of a line.  I can't do it!

What you have is a syntax error interactively.  It should work in a
program (the backslash is a no-op in a string).  /)$/ is good enough
and works in both interactive regexp search and in a program (unless
you need to match the whole line, in which case /^.*)$/ is what you
want (AFAIK Emacs doesn't guarantee longest match even today; I think
you can omit the "^" here, but I'd need to read the source to be
sure).

 > [I'm a member of a] population that would dramatically benefit if
 > they could actually use Emacs regular expressions.

Sure, but your blockage is that you aren't using Emacs REs, you're
trying to use some other kind of RE.

I think you might be helped a tiny bit by rawstrings, but I have to
admit that Stefan is right in your case: the problem is not the
proliferation of backslashes, it's that Emacs has a unique and
comparatively awkward syntax for regular expressions.  Rawstrings
might make it a little bit easier for you to adapt, though.