Need help fixing comint fontification for python.

Need help fixing comint fontification for python.

[Fabian Ezequiel Gallina]

In python.el there's an option called `python-shell-enable-font-lock'
that enables fontification in the comint buffer. It works pretty good
except for one thing, if the shell returns some output with unbalanced
quotes then all the following text after the
`comint-last-prompt-overlay' will be fontified as a string.

What would be the best way remove the syntax class for all the output?
I tried with set-text-properties to font-lock-unfontify-buffer with no
luck.


Thanks in advance,
Fabián.

Re: Need help fixing comint fontification for python.

[Stefan Monnier]

> What would be the best way remove the syntax class for all the output?

Use a syntax-propertize-function.

Look at the `field' property and wherever its value is `output', place
an appropriate syntax-table property.
Or alternatively, use a "safe" syntax-table in the buffer, but add
Python's syntax-table as a property whereas the `field' property is nil.


        Stefan

Re: Need help fixing comint fontification for python.

[Michael Mauger]

On Sun, 07 Oct 2012 10:37:28 -0400, Stefan Monnier said:  > > What would be the best way remove the syntax class for all the output?
> 
> Use a syntax-propertize-function.
> 
> Look at the `field' property and wherever its value is `output', place
> an appropriate syntax-table property.
> Or alternatively, use a "safe" syntax-table in the buffer, but add
> Python's syntax-table as a property whereas the `field' property is nil.
> 
> 
>         Stefan I've tried this and it doesn't appear that the field text property has
been applied to the output text when the syntax-propertize-function
fires. Once the display is complete, the field property is present but
in the hook function it isn't present.  Am I missing something? At the start of the syntax-propertize-function on the START location: ------------------------------------------------------------------------ position: 12182 of 13643 (89%), column: 4 character: [ (displayed as [) (codepoint 91, #o133, #x5b) preferred charset: iso-8859-1 (Latin-1 (ISO/IEC 8859-1))
code point in charset: 0x5B syntax: (]	which means: open, matches ] category: .:Base, a:ASCII, l:Latin, r:Roman to input: type "C-x 8 RET HEX-CODEPOINT" or "C-x 8 RET NAME" buffer code: #x5B file code: #x5B (encoded by coding system iso-latin-1-unix) display: by this font (glyph code) xft:-unknown-DejaVu Sans Mono-normal-normal-normal-*-13-*-*-*-m-0-iso10646-1 (#x3E) Character code properties: customize what to show name: LEFT SQUARE BRACKET old-name: OPENING SQUARE BRACKET general-category: Ps (Punctuation, Open) decomposition: (91) ('[') There is an overlay here: From 12182 to 12182 face                 hl-line window               #<window 44 on *Python*> There are text properties here: charset              iso-8859-1 fontified            nil ------------------------------------------------------------------------ At the same location, in the comint buffer ------------------------------------------------------------------------ position: 12182 of
 13643 (89%), column: 0 character: [ (displayed as [) (codepoint 91, #o133, #x5b) preferred charset: iso-8859-1 (Latin-1 (ISO/IEC 8859-1))
code point in charset: 0x5B syntax: (]	which means: open, matches ] category: .:Base, a:ASCII, l:Latin, r:Roman to input: type "C-x 8 RET HEX-CODEPOINT" or "C-x 8 RET NAME" buffer code: #x5B file code: #x5B (encoded by coding system iso-latin-1-unix) display: by this font (glyph code) xft:-unknown-DejaVu Sans Mono-normal-normal-normal-*-13-*-*-*-m-0-iso10646-1 (#x3E) Character code properties: customize what to show name: LEFT SQUARE BRACKET old-name: OPENING SQUARE BRACKET general-category: Ps (Punctuation, Open) decomposition: (91) ('[') There is an overlay here: From 12182 to 12183 face                 show-paren-match priority             1000 There are text properties here: charset              iso-8859-1 field                output fontified            t front-sticky         (field inhibit-line-move-field-capture) inhibit-line-move-field-capture t rear-nonsticky       t [back] ------------------------------------------------------------------------
Here's what I did:

  (defun my-comint-output-propertize (start end)
    (let (output-start output-end)
      (while (< start end)
        (describe-char start)
        (setq output-start (text-property-any start end 'field 'output))
        (if output-start    ;; <-- Never satisfied
            (progn 
              (setq output-end 
                    (min 
                     (next-single-char-property-change output-start 'field nil end)
                     (next-single-char-property-change output-start 'font-lock-face nil end)))
              (unless (eq (get-char-property output-start 'font-lock-face) 
                          'comint-highlight-prompt)
                (put-text-property output-start output-end 'font-lock-face 'italic))
              (setq start output-end))
          (setq start end)))))

  (add-hook 'comint-mode-hook 
            (lambda () (setq syntax-propertize-function 'my-comint-output-propertize)))


The output above is from the `describe-char' call; the
`text-property-any' call is never detecting the `field' property being
set to `output'.  The second output is from me positioning the caret and
hitting C-u C-x =.

Obviously, dope slap me if appropriate, I can't learn
otherwise. Be gentle. 

-- Michael

Re: Need help fixing comint fontification for python.

[Stefan Monnier]

[ Your message was very weirdly wrapped, making it difficult to read,
  hopefully it was just some transient problem.  ]

> I've tried this and it doesn't appear that the field text property has
> been applied to the output text when the syntax-propertize-function
> fires.

Hmm.. the `field' property is given the `output' value from
comint-output-filter right after running comint-output-filter-functions,
so if syntax-propertize doesn't see it, that's presumably because
something in comint-output-filter-functions runs syntax-propertize
(maybe via syntax-ppss).

Makes me realize that a better approach might be to add
a comint-output-filter-function that adds the syntax-table property
between comint-last-output-start and (process-mark process).

Of course, if the comint buffer also has a syntax-propertize-function,
then the two will tend to fight each other.


        Stefan