[PATCH] let octave-mode understand single-quoted strings

[PATCH] let octave-mode understand single-quoted strings

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 3365 bytes --]

This lets octave-mode recognize single-quoted string syntax. It doesn't
handle newlines in strings, but hopefully those are rare.

[octave-mod.diff]
--- octave-mod.el.orig 2010-05-30 15:41:40.000000000 -0400
+++ octave-mod.el 2010-05-30 16:56:10.000000000 -0400
@@ -163,7 +163,24 @@
 The string `function' and its name are given by the first and third
 parenthetical grouping.")

-(defvar octave-font-lock-keywords
+(defconst octave-string-prefix "\\(?:^\\|[^]})a-zA-Z0-9_.']\\)"
+  "Regexp used to represent the character before the string char '.
+The ' character has restrictions on what starts a string which is needed
+when attempting to understand the current context.
+
+Contains no matching groups.")
+
+(defconst octave-whole-string-regexp
+  (concat octave-string-prefix "\\('\\)\\(?:''\\|[^']\\)*\\('\\)" )
+  "Regular expression that matches a whole single-quoted string.
+In Matlab/Octave single-quoted strings, `'' is escaped by using
+it twice.")
+
+(defconst octave-font-lock-syntactic-keywords
+  ;; Fontify regular expressions
+  `((,octave-whole-string-regexp (1 "|") (2 "|"))))
+
+(defconst octave-font-lock-keywords
   (list
    ;; Fontify all builtin keywords.
    (cons (concat "\\<\\("
@@ -299,6 +316,8 @@
     (modify-syntax-entry ?\\ "\\" table)
     (modify-syntax-entry ?\' "."  table)
     (modify-syntax-entry ?\` "w"  table)
+    ;; single-quoted strings are handled separately because of
+    ;; ambiguity with the transpose operator
     (modify-syntax-entry ?\" "\"" table)
     (modify-syntax-entry ?. "w"   table)
     (modify-syntax-entry ?_ "w"   table)
@@ -531,13 +550,36 @@
   (make-local-variable 'normal-auto-fill-function)
   (setq normal-auto-fill-function 'octave-auto-fill)

-  (make-local-variable 'font-lock-defaults)
-  (setq font-lock-defaults '(octave-font-lock-keywords nil nil))
+  (set (make-local-variable 'parse-sexp-ignore-comments) t)
+  (set (make-local-variable 'parse-sexp-lookup-properties) t)

+  (make-local-variable 'font-lock-defaults)
+  (setq font-lock-defaults
+        '(octave-font-lock-keywords ; keywords
+          nil                       ; keywords-only
+          nil                       ; case-fold
+          nil                       ; syntax-alist
+          nil                       ; syntax-begin
+          (font-lock-syntactic-keywords
+           . octave-font-lock-syntactic-keywords)))
+
   (make-local-variable 'imenu-generic-expression)
   (setq imenu-generic-expression octave-mode-imenu-generic-expression
         imenu-case-fold-search nil)

+  ;; Important to fontify the whole buffer syntactically! If we don't,
+  ;; then we might have strings literals that aren't marked
+  ;; as strings, which will screw up parse-partial-sexp, scan-lists,
+  ;; etc. and and produce maddening "unbalanced parenthesis" errors.
+  ;; When we attempt to find the error and scroll to the portion of
+  ;; the buffer containing the problem, JIT-lock will apply the
+  ;; correct syntax to the regular expresion literal and the problem
+  ;; will mysteriously disappear.
+  (font-lock-set-defaults)
+
+  (let (font-lock-keywords) ; leaves syntactic keywords intact
+    (font-lock-fontify-buffer))
+
   (octave-add-octave-menu)
   (octave-initialize-completions)
   (run-mode-hooks 'octave-mode-hook))


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [PATCH] let octave-mode understand single-quoted strings

[Stefan Monnier]

> This lets octave-mode recognize single-quoted string syntax. It doesn't
> handle newlines in strings, but hopefully those are rare.

I just bumped into this old email that I must have missed back then.
In the mean time I installed a different patch which tries to do the
same thing.

Sorry about this waste of effort.  I'm not familiar with Octave, so your
code might be more correct in some cases, OTOH my code handles
multi-line strings.  Do you think we should try and get the best of both
worlds, or should we replace my code with yours, or should we just stick
with my code?


        Stefan

Re: [PATCH] let octave-mode understand single-quoted strings

[Jordi Gutiérrez Hermoso]

Hi, I worked with Daniel on this patch. I'm a frequent Octave user and
so far casual contributor.

On 31 August 2010 07:32, Stefan Monnier <monnier@iro.umontreal.ca>
wrote:
> I'm not familiar with Octave, so your code might be more correct in
> some cases, OTOH my code handles multi-line strings. Do you think we
> should try and get the best of both worlds, or should we replace my
> code with yours, or should we just stick with my code?

Daniel's code is correct in several cases I tried, so I would
recommend it. I don't know what you may have done with multiline
strings, but they seem to work just as well. Of course it would be
nice if both features worked.

Octave mode still requires a lot of work in other directions. Octave
syntax has been enriched somewhat since the mode was first written.
With time I may be able to provide these necessary patches myself.

- Jordi G. H.

Re: [PATCH] let octave-mode understand single-quoted strings

[Stefan Monnier]

>> I'm not familiar with Octave, so your code might be more correct in
>> some cases, OTOH my code handles multi-line strings. Do you think we
>> should try and get the best of both worlds, or should we replace my
>> code with yours, or should we just stick with my code?

> Daniel's code is correct in several cases I tried, so I would
> recommend it.  I don't know what you may have done with multiline
> strings, but they seem to work just as well.

IIUC Daniel's code, his multiline strings will only work if font-lock
happens to fontify the whole string at a time.  If it only needs to
update a single line (e.g. when you edit the first or last line of
a multiline single-quoted string), it will forget that it's a string (I
think it may even lead to nasty misfontifications where the rest of the
text gets fontified as a string).

> Of course it would be nice if both features worked.

That's definitely doable.  Tho if multiline strings are very rare and
are discouraged/deprecated, it might not be worth the trouble.

> Octave mode still requires a lot of work in other directions. Octave
> syntax has been enriched somewhat since the mode was first written.
> With time I may be able to provide these necessary patches myself.

Please check the new code in Emacs's trunk (uses the new SMIE package to
do the navigation and indentation) and report any missing feature (with
patch if you can: hopefully the new code makes it easier).


        Stefan

Re: [PATCH] let octave-mode understand single-quoted strings

[Jordi Gutiérrez Hermoso]

On 1 September 2010 01:46, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> That's definitely doable.  Tho if multiline strings are very rare and
> are discouraged/deprecated, it might not be worth the trouble.

They certainly are rare in the code I've seen. What prompted you to
improve support for them?

Re: [PATCH] let octave-mode understand single-quoted strings

[Stefan Monnier]

>> That's definitely doable.  Tho if multiline strings are very rare and
>> are discouraged/deprecated, it might not be worth the trouble.
> They certainly are rare in the code I've seen. What prompted you to
> improve support for them?

Nothing at all.  I just implemented the single-quote string support based
on code I previously wrote for perl-mode, where it's used for
string-like thingies that often span multiple lines.  I.e. it came "for
free".


        Stefan

Re: [PATCH] let octave-mode understand single-quoted strings

[Jordi Gutiérrez Hermoso]

2010/9/2 Stefan Monnier <monnier@iro.umontreal.ca>:
> I just implemented the single-quote string support based
> on code I previously wrote for perl-mode, where it's used for
> string-like thingies that often span multiple lines.  I.e. it came "for
> free".

The thing is that Matlab has a quirk where a' means the transpose of a
and 'a' means the string containing a, and Octave inherits this
syntax. That's the trickiest thing about single-quote strings in
Octave (and why Octave sources encourage using double-quote strings
all the time).

I still don't get why this is related to multiline issues, though.
Where is your patch?

Re: [PATCH] let octave-mode understand single-quoted strings

[Stefan Monnier]

>> I just implemented the single-quote string support based
>> on code I previously wrote for perl-mode, where it's used for
>> string-like thingies that often span multiple lines.  I.e. it came "for
>> free".
> The thing is that Matlab has a quirk where a' means the transpose of a
> and 'a' means the string containing a, and Octave inherits this
> syntax. That's the trickiest thing about single-quote strings in
> Octave (and why Octave sources encourage using double-quote strings
> all the time).

Right.

> I still don't get why this is related to multiline issues, though.

It's not directly related.

> Where is your patch?

In Emacs's trunk.  See below.
BTW, where can I find a definition of the syntax of single-quoted
strings (i.e. how does Octave distinguish a transpose from a single-quote
starting a string)?
The Octave docs I have don't say anything about it, AFAICT.


        Stefan


=== modified file 'lisp/progmodes/octave-mod.el'
--- lisp/progmodes/octave-mod.el	2010-08-12 14:44:16 +0000
+++ lisp/progmodes/octave-mod.el	2010-08-17 15:49:30 +0000
@@ -181,6 +179,30 @@
 	 '(3 font-lock-function-name-face nil t)))
   "Additional Octave expressions to highlight.")
 
+(defvar octave-font-lock-syntactic-keywords
+  ;; Try to distinguish the string-quotes from the transpose-quotes.
+  '(("[[({,; ]\\('\\)" (1 "\"'"))
+    (octave-font-lock-close-quotes)))
+
+(defun octave-font-lock-close-quotes (limit)
+  "Fix the syntax-table of the closing quotes of single-quote strings."
+  ;; Freely inspired from perl-font-lock-special-syntactic-constructs.
+  (let ((state (syntax-ppss)))
+    (while (< (point) limit)
+      (cond
+       ((eq (nth 3 state) ?\')
+        ;; A '..' string.
+        (save-excursion
+          (when (and (or (looking-at "\\('\\)")
+                         (re-search-forward "[^\\]\\(?:\\\\\\\\\\)*\\('\\)"
+                                            nil t))
+                     (not (eobp)))
+            (put-text-property (match-beginning 1) (match-end 1)
+                               'syntax-table (string-to-syntax "\"'"))))))
+
+      (setq state (parse-partial-sexp (point) limit nil nil state
+				      'syntax-table)))))
+
 (defcustom inferior-octave-buffer "*Inferior Octave*"
   "Name of buffer for running an inferior Octave process."
   :type 'string

Re: [PATCH] let octave-mode understand single-quoted strings

[John W. Eaton]

On  3-Sep-2010, Stefan Monnier wrote:

| >> I just implemented the single-quote string support based
| >> on code I previously wrote for perl-mode, where it's used for
| >> string-like thingies that often span multiple lines.  I.e. it came "for
| >> free".

In Octave and Matlab, character strings can't span multiple lines.

| BTW, where can I find a definition of the syntax of single-quoted
| strings (i.e. how does Octave distinguish a transpose from a single-quote
| starting a string)?
| The Octave docs I have don't say anything about it, AFAICT.

It's messy.

A single-quote character is recognized as a transpose operator if it

* Follows a ), ], or } character

    ( .. )'     => tranpose expression
    [ .. ]'     => tranpose matrix expression
    { .. }'     => tranpose cell array expression

* Follows a literal double-quoted string, or a literal single-quoted
  string if there is a space between the transpose operator and the
  final single-quote character that delimits the string.  For example

    "string"'   => transpose the character array 'string'
    'string' '  => transpose the character array 'string'
    'string''   => unterminated string constant
    'a''b'      => 1x3 character array containing the characters a'b

* Follows an identifier (variable or function name) unless the
  identifier could be a "command".  For example

    pwd '       => unterminated string constant
    (pwd)'      => transpose the output from the pwd function
    dir 'foo'   => list contents of the directory foo

* Follows an expression that references a structure element:

    a.b'        => return the transpose of a.b

  The Octave manual has a more complete description of how
  "command-style" parsing works (yes, this is ugly; it was implemented
  because it is required for compatibility with Matlab).

Does that help?

jwe

Re: [PATCH] let octave-mode understand single-quoted strings

[Eric M. Ludlam]

The matlab mode that I support can be found here:

http://matlab-emacs.cvs.sourceforge.net/viewvc/matlab-emacs/matlab-emacs/

It has had many contributors so it couldn't be a part of emacs, but the 
string & comment font locking I wrote wholly myself as far as I 
remember.  If you would like to use it for octave I'm sure that could be 
arranged.

If not, the basic idea behind it is to use font-lock's functional 
matching stuff.  Look to these functions:

matlab-font-lock-string-match-normal
matlab-font-lock-string-match-unterminated
matlab-font-lock-comment-match

The strings and comments have to be done together, so you end up having 
to use these functions in a particular order, checking to see what faces 
are left behind while looking at comment chars to make sure it isn't in 
a string.

Eric

On 10/05/2010 04:28 AM, John W. Eaton wrote:
> On  3-Sep-2010, Stefan Monnier wrote:
>
> |>>  I just implemented the single-quote string support based
> |>>  on code I previously wrote for perl-mode, where it's used for
> |>>  string-like thingies that often span multiple lines.  I.e. it came "for
> |>>  free".
>
> In Octave and Matlab, character strings can't span multiple lines.
>
> | BTW, where can I find a definition of the syntax of single-quoted
> | strings (i.e. how does Octave distinguish a transpose from a single-quote
> | starting a string)?
> | The Octave docs I have don't say anything about it, AFAICT.
>
> It's messy.
>
> A single-quote character is recognized as a transpose operator if it
>
> * Follows a ), ], or } character
>
>      ( .. )'     =>  tranpose expression
>      [ .. ]'     =>  tranpose matrix expression
>      { .. }'     =>  tranpose cell array expression
>
> * Follows a literal double-quoted string, or a literal single-quoted
>    string if there is a space between the transpose operator and the
>    final single-quote character that delimits the string.  For example
>
>      "string"'   =>  transpose the character array 'string'
>      'string' '  =>  transpose the character array 'string'
>      'string''   =>  unterminated string constant
>      'a''b'      =>  1x3 character array containing the characters a'b
>
> * Follows an identifier (variable or function name) unless the
>    identifier could be a "command".  For example
>
>      pwd '       =>  unterminated string constant
>      (pwd)'      =>  transpose the output from the pwd function
>      dir 'foo'   =>  list contents of the directory foo
>
> * Follows an expression that references a structure element:
>
>      a.b'        =>  return the transpose of a.b
>
>    The Octave manual has a more complete description of how
>    "command-style" parsing works (yes, this is ugly; it was implemented
>    because it is required for compatibility with Matlab).
>
> Does that help?
>
> jwe
>
>

Re: [PATCH] let octave-mode understand single-quoted strings

[Stefan Monnier]

>>>>> John W. Eaton <jwe@octave.org> writes:
> A single-quote character is recognized as a transpose operator if it
[...]
> Does that help?

Yes, it does, thank you.

> The Octave manual has a more complete description of how
> "command-style" parsing works (yes, this is ugly; it was implemented
> because it is required for compatibility with Matlab).

Could you give me a pointer to it?  I haven't found it.

>>>>> "Eric" == Eric M Ludlam <eric@siege-engine.com> writes:
> If not, the basic idea behind it is to use font-lock's functional matching
> stuff.  Look to these functions:

> matlab-font-lock-string-match-normal
> matlab-font-lock-string-match-unterminated
> matlab-font-lock-comment-match

Thanks.  octave-mod.el uses syntax-table text-properties instead (since
it has the advantage of making buffer navigation functions work
correctly as well), so it's necessarily a bit different.

> The strings and comments have to be done together, so you end up having to
> use these functions in a particular order, checking to see what faces are
> left behind while looking at comment chars to make sure it isn't in
> a string.

FWIW, I've appended the code I use now in octave-mod.el, since I think
it's a good bit simpler.  It uses the new syntax-propertize feature, but
you could use font-lock-syntactic-keywords instead.


        Stefan


(defun octave-syntax-propertize-function (start end)
  (goto-char start)
  (octave-syntax-propertize-sqs end)
  (funcall (syntax-propertize-rules
            ;; Try to distinguish the string-quotes from the transpose-quotes.
            ("[[({,; ]\\('\\)"
             (1 (prog1 "\"'" (octave-syntax-propertize-sqs end)))))
           (point) end))

(defun octave-syntax-propertize-sqs (end)
  "Propertize the content/end of single-quote strings."
  (when (eq (nth 3 (syntax-ppss)) ?\')
    ;; A '..' string.
    (when (re-search-forward
           "\\(?:\\=\\|[^']\\)\\(?:''\\)*\\('\\)\\($\\|[^']\\)" end 'move)
      (goto-char (match-beginning 2))
            (when (eq (char-before (match-beginning 1)) ?\\)
              ;; Backslash cannot escape a single quote.
              (put-text-property (1- (match-beginning 1)) (match-beginning 1)
                                 'syntax-table (string-to-syntax ".")))
            (put-text-property (match-beginning 1) (match-end 1)
                         'syntax-table (string-to-syntax "\"'")))))

Re: [PATCH] let octave-mode understand single-quoted strings

[Jordi Gutiérrez Hermoso]

On 7 October 2010 03:41, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>>>>>> John W. Eaton <jwe@octave.org> writes:
>> A single-quote character is recognized as a transpose operator if it
> [...]
>> Does that help?
>
> Yes, it does, thank you.
>
>> The Octave manual has a more complete description of how
>> "command-style" parsing works (yes, this is ugly; it was implemented
>> because it is required for compatibility with Matlab).
>
> Could you give me a pointer to it?  I haven't found it.
>
>>>>>> "Eric" == Eric M Ludlam <eric@siege-engine.com> writes:
>> If not, the basic idea behind it is to use font-lock's functional matching
>> stuff.  Look to these functions:
>
>> matlab-font-lock-string-match-normal
>> matlab-font-lock-string-match-unterminated
>> matlab-font-lock-comment-match
>
> Thanks.  octave-mod.el uses syntax-table text-properties instead (since
> it has the advantage of making buffer navigation functions work
> correctly as well), so it's necessarily a bit different.
>
>> The strings and comments have to be done together, so you end up having to
>> use these functions in a particular order, checking to see what faces are
>> left behind while looking at comment chars to make sure it isn't in
>> a string.
>
> FWIW, I've appended the code I use now in octave-mod.el, since I think
> it's a good bit simpler.  It uses the new syntax-propertize feature, but
> you could use font-lock-syntactic-keywords instead.
>
>
>        Stefan
>
>
> (defun octave-syntax-propertize-function (start end)
>  (goto-char start)
>  (octave-syntax-propertize-sqs end)
>  (funcall (syntax-propertize-rules
>            ;; Try to distinguish the string-quotes from the transpose-quotes.
>            ("[[({,; ]\\('\\)"
>             (1 (prog1 "\"'" (octave-syntax-propertize-sqs end)))))
>           (point) end))
>
> (defun octave-syntax-propertize-sqs (end)
>  "Propertize the content/end of single-quote strings."
>  (when (eq (nth 3 (syntax-ppss)) ?\')
>    ;; A '..' string.
>    (when (re-search-forward
>           "\\(?:\\=\\|[^']\\)\\(?:''\\)*\\('\\)\\($\\|[^']\\)" end 'move)
>      (goto-char (match-beginning 2))
>            (when (eq (char-before (match-beginning 1)) ?\\)
>              ;; Backslash cannot escape a single quote.
>              (put-text-property (1- (match-beginning 1)) (match-beginning 1)
>                                 'syntax-table (string-to-syntax ".")))
>            (put-text-property (match-beginning 1) (match-end 1)
>                         'syntax-table (string-to-syntax "\"'")))))
>

Ping!

It's been almost a year. And still no single-quote strings in Octave
mode. I have forgotten what the issues were. Can this code be pushed
now to the bzr repo?

- Jordi G. H.

Re: [PATCH] let octave-mode understand single-quoted strings

[Stefan Monnier]

> It's been almost a year. And still no single-quote strings in Octave
> mode. I have forgotten what the issues were. Can this code be pushed
> now to the bzr repo?

Which version of octave-mode are you using?
The one that comes with Emacs-24 does have support for single quoted strings.


        Stefan