html, css, and js modes working together

html, css, and js modes working together

[Tom Tromey]

I tried to send this last night but it bounced for some reason.

This patch changes the html, css, and js modes to work together a bit.
With this, the contents of a <style> element are syntax-highlighted and
indented according to css-mode rules, and the contents of a <script>
element are syntax-highlighted and indented according to js-mode rules.
I'd appreciate comments on this approach.

This required small changes in SMIE and a minor unrelated change to
break the now-cyclic sgml/css-mode dependency.  I chose to break the
sgml/js cyclic dependency in a different way; neither one seems really
great.  Maybe moving html-mode to its own file would help.

* I didn't read SMIE deeply enough to see if the indentation parts
  relied heavily on the forward/backward-a-token rules.  If so then the
  new macro will need some additional work.

* I used the existing (but apparently unused) prog-indentation-context
  variable in this.  I was initially skeptical of using a global
  variable but it did turn out to be pretty handy for SMIE.

* This work doesn't address the need for per-region font locking at all.
  That would be nice to have.  I'd like to hear if there are plans for
  how to do this.

* Likewise there are other things provided by a major mode that aren't
  handled here, for example imenu, add-log-current-defun-function,
  comment-*, electric indent keys, ... probably more that I am not
  thinking of or that css-mode isn't using.  It might also be nice if
  dir-locals.el settings for css-mode and js-mode affected html-mode as
  well.

* Not sure but maybe I also need to define
  syntax-propertize-extend-region-functions now?

Tom

diff --git a/lisp/emacs-lisp/smie.el b/lisp/emacs-lisp/smie.el
index 4d02b75..d793eca 100644
--- a/lisp/emacs-lisp/smie.el
+++ b/lisp/emacs-lisp/smie.el
@@ -123,6 +123,8 @@
 
 (eval-when-compile (require 'cl-lib))
 
+(require 'prog-mode)
+
 (defgroup smie nil
   "Simple Minded Indentation Engine."
   :group 'languages)
@@ -1455,7 +1457,7 @@ smie-indent-bob
   ;; Start the file at column 0.
   (save-excursion
     (forward-comment (- (point)))
-    (if (bobp) 0)))
+    (if (bobp) (prog-first-column))))
 
 (defun smie-indent-close ()
   ;; Align close paren with opening paren.
@@ -1838,6 +1840,16 @@ smie-auto-fill
         (funcall do-auto-fill)))))
 
 
+(defmacro with-smie-rules (spec &rest body)
+  "Temporarily set up SMIE indentation and evaluate BODY.
+SPEC is of the form (GRAMMAR RULES-FUNCTION); see `smie-setup'.
+BODY is evaluated with the relevant SMIE variables temporarily bound."
+  (declare (indent 1))
+  `(let ((smie-grammar ,(car spec))
+         (smie-rules-function ,(cadr spec))
+         (indent-line-function #'smie-indent-line))
+     ,@body))
+
 (defun smie-setup (grammar rules-function &rest keywords)
   "Setup SMIE navigation and indentation.
 GRAMMAR is a grammar table generated by `smie-prec2->grammar'.
diff --git a/lisp/progmodes/js.el b/lisp/progmodes/js.el
index 74dd4ad..f3d90a9 100644
--- a/lisp/progmodes/js.el
+++ b/lisp/progmodes/js.el
@@ -53,6 +53,7 @@
 (require 'moz nil t)
 (require 'json nil t)
 (require 'sgml-mode)
+(require 'prog-mode)
 
 (eval-when-compile
   (require 'cl-lib)
@@ -2102,7 +2103,7 @@ js--proper-indentation
 
           ((js--continued-expression-p)
            (+ js-indent-level js-expr-indent-offset))
-          (t 0))))
+          (t (prog-first-column)))))
 
 ;;; JSX Indentation
 
diff --git a/lisp/textmodes/css-mode.el b/lisp/textmodes/css-mode.el
index 19f74da..14bacdc 100644
--- a/lisp/textmodes/css-mode.el
+++ b/lisp/textmodes/css-mode.el
@@ -33,7 +33,6 @@
 ;;; Code:
 
 (require 'seq)
-(require 'sgml-mode)
 (require 'smie)
 (require 'eww)
 
@@ -869,10 +868,6 @@ css--complete-property-value
                 (append '("inherit" "initial" "unset")
                         (css--property-values property))))))))
 
-(defvar css--html-tags (mapcar #'car html-tag-alist)
-  "List of HTML tags.
-Used to provide completion of HTML tags in selectors.")
-
 (defvar css--nested-selectors-allowed nil
   "Non-nil if nested selectors are allowed in the current mode.")
 (make-variable-buffer-local 'css--nested-selectors-allowed)
@@ -900,6 +895,8 @@ css--foreign-completions
         (funcall (symbol-value extractor))))
     (buffer-list))))
 
+(declare-function html-memoized-tag-list "sgml-mode")
+
 (defun css--complete-selector ()
   "Complete part of a CSS selector at point."
   (when (or (= (nth 0 (syntax-ppss)) 0) css--nested-selectors-allowed)
@@ -916,7 +913,7 @@ css--complete-selector
                 (css--foreign-completions 'css-class-list-function))
                ((eq start-char ?#)
                 (css--foreign-completions 'css-id-list-function))
-               (t css--html-tags))))))))))
+               (t (html-memoized-tag-list)))))))))))
 
 (defun css-completion-at-point ()
   "Complete current symbol at point.
diff --git a/lisp/textmodes/sgml-mode.el b/lisp/textmodes/sgml-mode.el
index e148b06..676ebcf 100644
--- a/lisp/textmodes/sgml-mode.el
+++ b/lisp/textmodes/sgml-mode.el
@@ -32,7 +32,9 @@
 
 ;;; Code:
 
+(require 'css-mode)
 (require 'dom)
+(require 'prog-mode)
 (require 'seq)
 (require 'subr-x)
 (eval-when-compile
@@ -341,19 +343,23 @@ sgml-font-lock-keywords-2
 (defvar sgml-font-lock-keywords sgml-font-lock-keywords-1
   "Rules for highlighting SGML code.  See also `sgml-tag-face-alist'.")
 
+(eval-when-compile
+  (defconst sgml-syntax-propertize-rules
+    (syntax-propertize-precompile-rules
+     ;; Use the `b' style of comments to avoid interference with the -- ... --
+     ;; comments recognized when `sgml-specials' includes ?-.
+     ;; FIXME: beware of <!--> blabla <!--> !!
+     ("\\(<\\)!--" (1 "< b"))
+     ("--[ \t\n]*\\(>\\)" (1 "> b"))
+     ;; Double quotes outside of tags should not introduce strings.
+     ;; Be careful to call `syntax-ppss' on a position before the one we're
+     ;; going to change, so as not to need to flush the data we just computed.
+     ("\"" (0 (if (prog1 (zerop (car (syntax-ppss (match-beginning 0))))
+                    (goto-char (match-end 0)))
+                  (string-to-syntax ".")))))))
+
 (defconst sgml-syntax-propertize-function
-  (syntax-propertize-rules
-   ;; Use the `b' style of comments to avoid interference with the -- ... --
-   ;; comments recognized when `sgml-specials' includes ?-.
-  ;; FIXME: beware of <!--> blabla <!--> !!
-   ("\\(<\\)!--" (1 "< b"))
-    ("--[ \t\n]*\\(>\\)" (1 "> b"))
-    ;; Double quotes outside of tags should not introduce strings.
-    ;; Be careful to call `syntax-ppss' on a position before the one we're
-    ;; going to change, so as not to need to flush the data we just computed.
-    ("\"" (0 (if (prog1 (zerop (car (syntax-ppss (match-beginning 0))))
-                   (goto-char (match-end 0)))
-           (string-to-syntax ".")))))
+  (syntax-propertize-rules sgml-syntax-propertize-rules)
   "Syntactic keywords for `sgml-mode'.")
 
 ;; internal
@@ -2024,6 +2030,13 @@ html-tag-alist
       ("wbr" t)))
   "Value of `sgml-tag-alist' for HTML mode.")
 
+(defvar html-tag-list (mapcar #'car html-tag-alist)
+  "List of HTML tags.")
+
+;;;###autoload
+(defun html-memoized-tag-list ()
+  html-tag-list)
+
 (defvar html-tag-help
   `(,@sgml-tag-help
     ("a" . "Anchor of point or link elsewhere")
@@ -2233,6 +2246,68 @@ html-current-buffer-ids
         ids))))
 
 \f
+;; js.el is loaded when entering html-mode.
+(defvar js-mode-syntax-table)
+(declare-function js-indent-line "js")
+(declare-function js-syntax-propertize "js")
+
+(defconst html-syntax-propertize-function
+  (syntax-propertize-rules
+   ("<style.*?>\\(\\(\n\\|.\\)*?\\)</style>"
+    (1
+     (prog1 css-mode-syntax-table
+       (let ((start (nth 2 (match-data)))
+             (end (nth 3 (match-data))))
+       (funcall css-syntax-propertize-function start end)
+       (add-text-properties start end '(syntax-multiline t))))))
+   ("<script.*?>\\(\\(\n\\|.\\)*?\\)</script>"
+    (1
+     (prog1 js-mode-syntax-table
+       (let ((start (nth 2 (match-data)))
+             (end (nth 3 (match-data))))
+         (js-syntax-propertize start end)
+         (add-text-properties start end '(syntax-multiline t))))))
+   sgml-syntax-propertize-rules)
+  "Syntactic keywords for `html-mode'.")
+
+(defun html-indent-line ()
+  "Indent the current line as HTML."
+  (interactive)
+  (let* ((context (save-excursion (car (last (sgml-get-context)))))
+         (tag (when (and context
+                         (eq (sgml-tag-type context) 'open)
+                         (> (point) (sgml-tag-start context)))
+                (sgml-tag-name context))))
+    (cond
+     ((equal tag "style")
+      ;; CSS.
+      (save-restriction
+        (let ((base-indent (save-excursion
+                             (goto-char (sgml-tag-end context))
+                             (sgml-calculate-indent))))
+          (narrow-to-region (sgml-tag-end context) (point-max))
+          (let ((prog-indentation-context (list base-indent
+                                                (cons (point-min) nil)
+                                                nil)))
+            (with-smie-rules (css-smie-grammar #'css-smie-rules)
+              (smie-indent-line))))))
+     ((equal tag "script")
+      ;; Javascript.
+      (save-restriction
+        (let ((base-indent (save-excursion
+                             (goto-char (sgml-tag-end context))
+                             (sgml-calculate-indent))))
+          (narrow-to-region (sgml-tag-end context) (point-max))
+          (let ((prog-indentation-context (list base-indent
+                                                (cons (point-min) nil)
+                                                nil)))
+            (js-indent-line)))))
+     (t
+      ;; HTML.
+      (sgml-indent-line)))))
+
+\f
+
 ;;;###autoload
 (define-derived-mode html-mode sgml-mode '(sgml-xml-mode "XHTML" "HTML")
   "Major mode based on SGML mode for editing HTML documents.
@@ -2270,6 +2345,8 @@ html-mode
    (eval-after-load \"sgml-mode\" \\='(aset sgml-char-names ?\\=' nil))
 
 \\{html-mode-map}"
+  ;; A hack to avoid a circular dependency.
+  (require 'js)
   (setq-local sgml-display-text html-display-text)
   (setq-local sgml-tag-face-alist html-tag-face-alist)
   (setq-local sgml-tag-alist html-tag-alist)
@@ -2281,6 +2358,11 @@ html-mode
 	      (lambda () (char-before (match-end 0))))
   (setq-local add-log-current-defun-function #'html-current-defun-name)
   (setq-local sentence-end-base "[.?!][]\"'”)}]*\\(<[^>]*>\\)*")
+  (setq-local indent-line-function #'html-indent-line)
+
+  (setq-local syntax-propertize-function html-syntax-propertize-function)
+  (add-hook 'syntax-propertize-extend-region-functions
+            #'syntax-propertize-multiline 'append 'local)
 
   (when (fboundp 'libxml-parse-html-region)
     (defvar css-class-list-function)

Re: html, css, and js modes working together

[Clément Pit-Claudel]

On 2017-01-31 15:34, Tom Tromey wrote:
> This patch changes the html, css, and js modes to work together a bit.
> With this, the contents of a <style> element are syntax-highlighted and
> indented according to css-mode rules, and the contents of a <script>
> element are syntax-highlighted and indented according to js-mode rules.
> I'd appreciate comments on this approach.

Hi Tom,

I haven't tried the patch, but I can offer a few comments:

I'm a bit wary of rules like the following:

    "<script.*?>\\(\\(\n\\|.\\)*?\\)</script>"

Won't this pick up <script> tags in comments?  And won't matching large <script> tags going to be costly?  An alternative would be to find the opening tag, check the context that it appears in, and if it is indeed a proper <script> tag search forward to find the closing tag.

I also don't think adding #'syntax-propertize-multiline to 'syntax-propertize-extend-region-functions is enough to ensure proper syntax-propertization (this could miss large scripts — larger than a screenful, couldn't it?).

Finally: how much of a performance impact does this have? Recomputing syntax properties for entire <script> and <style> tags for every edit sounds a bit costly.

Cheers,
Clément.

Re: html, css, and js modes working together

[Tom Tromey]

>>>>> "Clément" == Clément Pit-Claudel <cpitclaudel@gmail.com> writes:

Clément> I haven't tried the patch, but I can offer a few comments:

Thank you.

Clément> I'm a bit wary of rules like the following:
Clément>     "<script.*?>\\(\\(\n\\|.\\)*?\\)</script>"

Clément> Won't this pick up <script> tags in comments?

I tried this just now and it seemed to work ok.


I don't have answers to your other questions but it seems that Stefan's
reply covers much of the same ground, so I'm going to try the things he
suggested and report back whenever I have a new patch.

Tom

Re: html, css, and js modes working together

[Tom Tromey]

>>>>> "Tom" == Tom Tromey <tom@tromey.com> writes:

Clément> I'm a bit wary of rules like the following:
Clément> "<script.*?>\\(\\(\n\\|.\\)*?\\)</script>"
Clément> Won't this pick up <script> tags in comments?

Tom> I tried this just now and it seemed to work ok.

Well, I was wrong about that... it worked at the time somehow but with
my latest code it is wrong.  Time to write those test cases I guess.
Maybe this can be made to work by having the syntax-propertize function
notice when it's already in an HTML comment.

Tom

Re: html, css, and js modes working together

[Stefan Monnier]

> * I didn't read SMIE deeply enough to see if the indentation parts
>   relied heavily on the forward/backward-a-token rules.  If so then the
>   new macro will need some additional work.

In most major modes, the tokenizer needs to be adjusted (IOW, the
default values of smie-*ward-token-functions only exists so it's easier
to get a new major mode half-working): a quick "grep '(smie-setup'
**/*.el" indicates that only ps-mode relies on the default tokenizer.

So yes, it's important for smie-with-rules to set the
for/backward-token-function.

> -    (if (bobp) 0)))
> +    (if (bobp) (prog-first-column))))

Hmm.. if we want to obey prog-indentation-context,
don't we want something like

    (if (and prog-indentation-context
             (<= (point) (caadr prog-indentation-context)))
        (car prog-indentation-context)
      0)

> +(defmacro with-smie-rules (spec &rest body)
> +  "Temporarily set up SMIE indentation and evaluate BODY.
> +SPEC is of the form (GRAMMAR RULES-FUNCTION); see `smie-setup'.
> +BODY is evaluated with the relevant SMIE variables temporarily bound."
> +  (declare (indent 1))
> +  `(let ((smie-grammar ,(car spec))
> +         (smie-rules-function ,(cadr spec))
> +         (indent-line-function #'smie-indent-line))
> +     ,@body))

I think it's likely that the implementation of this macro will change
over time, even if it's external appearance doesn't.  So I think we
should make it expand to a code that calls an smie function that does
the actual work.  E.g.

    (defmacro smie-with-rules (spec &rest body)
      (smie-funcall-with-rules (list ,@spec) (lambda () . ,body)))
    (defun smie-funcall-with-rules (spec fun)
      (let ((smie-grammar (car spec))
            (smie-rules-function (cadr spec))
            (indent-line-function #'smie-indent-line))
        (funcall fun)))

Then again, maybe we want something even more generic that's not
specific to SMIE (see below).

> +          (t (prog-first-column)))))

I suspect here as well, we should obey (caadr prog-indentation-context).

> +(defconst html-syntax-propertize-function
> +  (syntax-propertize-rules
> +   ("<style.*?>\\(\\(\n\\|.\\)*?\\)</style>"
> +    (1
> +     (prog1 css-mode-syntax-table
> +       (let ((start (nth 2 (match-data)))
> +             (end (nth 3 (match-data))))
> +       (funcall css-syntax-propertize-function start end)
> +       (add-text-properties start end '(syntax-multiline t))))))
> +   ("<script.*?>\\(\\(\n\\|.\\)*?\\)</script>"
> +    (1
> +     (prog1 js-mode-syntax-table
> +       (let ((start (nth 2 (match-data)))
> +             (end (nth 3 (match-data))))
> +         (js-syntax-propertize start end)
> +         (add-text-properties start end '(syntax-multiline t))))))
> +   sgml-syntax-propertize-rules)
> +  "Syntactic keywords for `html-mode'.")

As pointed out by Clément, these regexps are a bad idea.
Better do something like:

    (defun html-syntax-propertize (start end)
      (goto-char start)
      (when (get-text-property (point) 'html-submode)
        (html--syntax-propertize-submode
         (get-text-property (point) 'html-submode)
         end))
      (funcall
       (syntax-propertize-rules
        ("<style.*?>"
         (0 (ignore (goto-char (match-end 0))
                    (html--syntax-propertize-submode 'css-mode end))))
        ("<script.*?>"
         (0 (ignore (goto-char (match-end 0))
                    (html--syntax-propertize-submode 'js-mode end)))))
       (point) end))


> +    (cond
> +     ((equal tag "style")
> +      ;; CSS.
> +      (save-restriction
> +        (let ((base-indent (save-excursion
> +                             (goto-char (sgml-tag-end context))
> +                             (sgml-calculate-indent))))
> +          (narrow-to-region (sgml-tag-end context) (point-max))
> +          (let ((prog-indentation-context (list base-indent
> +                                                (cons (point-min) nil)
> +                                                nil)))
> +            (with-smie-rules (css-smie-grammar #'css-smie-rules)
> +              (smie-indent-line))))))
> +     ((equal tag "script")
> +      ;; Javascript.
> +      (save-restriction
> +        (let ((base-indent (save-excursion
> +                             (goto-char (sgml-tag-end context))
> +                             (sgml-calculate-indent))))
> +          (narrow-to-region (sgml-tag-end context) (point-max))
> +          (let ((prog-indentation-context (list base-indent
> +                                                (cons (point-min) nil)
> +                                                nil)))
> +            (js-indent-line)))))
> +     (t
> +      ;; HTML.
> +      (sgml-indent-line)))))

How 'bout instead doing something like:

    (defun html--get-mode-vars (mode)
      (with-temp-buffer
        (funcall mode)
        (buffer-local-variables)))

    (defconst html--css-vars (html--get-mode-vars 'css-mode))
    (defconst html--js-vars (html--get-mode-vars js-mode))

    (defun html--funcall-with-mode-var (vars fun)
      (cl-progv (mapcar #'car vars) (mapcar #'cdr vars) (funcall fun)))

and then do

    ((member tag '("style" "script"))
     (let* ((start (sgml-tag-end context))
            (base-indent (save-excursion
                           (goto-char start)
                           (sgml-calculate-indent)))
            (prog-indentation-context (list base-indent
                                               (cons start nil)
                                               nil))
            (vars (if (equal tag "style") html--css-vars html--js-vars)))
       (cl-progv (mapcar #'car vars) (mapcar #'cdr vars)
         (indent-according-to-mode))))

I'm sure the above will break miserably, because you'll need to tweak
the list of vars that really matter (most likely some of the
buffer-local-variables should not be in html--*-vars), but such an
approach should also help break the cyclic dependencies.


        Stefan

Re: html, css, and js modes working together

[Stefan Monnier]

Stefan> As pointed out by Clément, these regexps are a bad idea.
Stefan> Better do something like:
> This seems reasonable but I don't understand where to check for the end
> of the element.  Don't I still need a search forward in the
> html-syntax-propertize function?  Or in your example were you supposing
> that this would be done by "html--syntax-propertize-submode"?

Yes, html--syntax-propertize-submode would search for the end of the
sub-region (only up to `end`, of course), mark the found area with
a text-property, and then call the sub-mode's syntax-propertize-function
on that area.

Stefan> (defun html--get-mode-vars (mode)
Stefan> (with-temp-buffer
Stefan> (funcall mode)
Stefan> (buffer-local-variables)))

> Tricky!  I looked at the buffer-local-variables here though and there is
> a lot of stuff that I think either isn't relevant or is potentially
> harmful.  So I think for the time being I will stick to a more explicit
> approach.

Yes, you need to filter out the elements you don't want, or pick the few
elements you do want.  But I think you're better off manually
maintaining a list of relevant variables, than manually maintaining not
only which variables are relevant but also the values used by each of the
sub-modes (which in turns causes circular dependencies between the
packages).


        Stefan

Re: html, css, and js modes working together

[Dmitry Gutov]

Hi Tom,

On 31.01.2017 22:34, Tom Tromey wrote:
> I tried to send this last night but it bounced for some reason.
> 
> This patch changes the html, css, and js modes to work together a bit.
> With this, the contents of a <style> element are syntax-highlighted and
> indented according to css-mode rules, and the contents of a <script>
> element are syntax-highlighted and indented according to js-mode rules.
> I'd appreciate comments on this approach.

I'm glad you are interested in the problem of mixed modes, but 
implementing it inside html-mode is likely to make life more difficult 
for the existing mixed-mode frameworks because now html-mode's 
indentation code and font-lock rules are that much more complex.

So this is great as an experiment, and maybe a direction toward creating 
a built-in mixed-mode solution, but why not call the result 
html-and-stuff-mode in the meantime?

One might argue that we wouldn't need mmm-mode for the HTML-CSS-and-JS 
combination, but even aside from backward compatibility considerations, 
the third-party frameworks will continue to be useful for the edge cases 
they handle, such as solving some of the problems you mentioned in your 
next-to-last item.

> * I used the existing (but apparently unused) prog-indentation-context
>    variable in this.

So far it's only used in python-mode, I think.

> I was initially skeptical of using a global
>    variable but it did turn out to be pretty handy for SMIE.

A global variable is usually easy to use, if you remember to do so. I'd 
like to replace it with a different indent-line-function variable and 
calling convention, but that's a separate discussion.

> * This work doesn't address the need for per-region font locking at all.

Not sure what you mean by that. But I've applied the patch, and I don't 
see any JS or CSS specific highlighting. Indentation kinda works, though.

Couple more things to think about:

- Both css-mode and js-mode call syntax-ppss in their indentation code. 
Luckily, their syntax tables are fairly compatible. But 
sgml-syntax-propertize-rules calls syntax-ppss as well.

That might lead to some problems with the cache. Maybe it's not a 
problem as long as () and {} have the same syntax classes in html-mode.

- Try this example:

<html>
   <script>
     var a = 4;

     alert(a);
   </script>
</html>


It indents fine. Now try replacing "4" with "4 < 5" and reindenting the 
"alert" line. It jumps to the right.

Re: html, css, and js modes working together

[Tom Tromey]

>>>>> "Dmitry" == Dmitry Gutov <dgutov@yandex.ru> writes:

Just a quick reply to a couple of points.

Dmitry> I'm glad you are interested in the problem of mixed modes, but
Dmitry> implementing it inside html-mode is likely to make life more difficult
Dmitry> for the existing mixed-mode frameworks because now html-mode's
Dmitry> indentation code and font-lock rules are that much more complex.

I'm not sure what you're referring to here.  Do some of the existing
mixed mode things piggyback on the existing html mode?

I guess my view is that it is always ok to make things in-tree work
better with each other, even at the expense of some code that is
out-of-tree and presumably relying on implementation details to do its
work.

Though I don't actually know any details... the one such mode I've used,
web-mode, reimplements everything on its own.

Dmitry> So far it's only used in python-mode, I think.

grep shows no users in-tree, so maybe this never went in?  Or is this
one of the various out-of-tree python modes?

>> * This work doesn't address the need for per-region font locking at all.

Dmitry> Not sure what you mean by that. But I've applied the patch, and I
Dmitry> don't see any JS or CSS specific highlighting. Indentation kinda
Dmitry> works, though.

Yeah, what I mean is that font-locking does not occur, but it would be
good to have.  I couldn't find anything saying how this might be solved.
I haven't come up with any really good ideas myself.  Maybe font-lock
could also look at text properties to decide what keywords to use?

Dmitry> - Both css-mode and js-mode call syntax-ppss in their indentation
Dmitry> code. Luckily, their syntax tables are fairly compatible. But
Dmitry> sgml-syntax-propertize-rules calls syntax-ppss as well.

I thought that a syntax-table property on the characters would make
syntax-ppss do the "right" thing; namely, notice where the syntax table
changes and change its parsing method accordingly.  This seems to be
what is implemented in syntax.c when parse-sexp-lookup-properties is
set... but I didn't dig through that code in detail so I am probably
misunderstanding somehow.

I didn't try your example yet.  Thanks for providing that, that's very
helpful.

Tom

Re: html, css, and js modes working together

[Dmitry Gutov]

On 06.02.2017 05:26, Tom Tromey wrote:

> I'm not sure what you're referring to here.  Do some of the existing
> mixed mode things piggyback on the existing html mode?

Both https://github.com/purcell/mmm-mode/ and 
https://github.com/vspinu/polymode/ do.

> I guess my view is that it is always ok to make things in-tree work
> better with each other, even at the expense of some code that is
> out-of-tree and presumably relying on implementation details to do its
> work.

Still, defining a new major mode instead of directly reusing an existing 
one shouldn't take a lot of effort. Especially while the result is 
functional but not ideal.

And breaking packages that worked fine for many years is not nice, 
especially when it can be avoided.

> Dmitry> So far it's only used in python-mode, I think.
> 
> grep shows no users in-tree, so maybe this never went in?  Or is this
> one of the various out-of-tree python modes?

lisp/progmodes/python.el uses prog-first-column and prog-widen, both of 
which refer to prog-indentation-context.

> Yeah, what I mean is that font-locking does not occur, but it would be
> good to have.  I couldn't find anything saying how this might be solved.
> I haven't come up with any really good ideas myself.  Maybe font-lock
> could also look at text properties to decide what keywords to use?

Here's one way to do it: 
https://github.com/purcell/mmm-mode/blob/master/mmm-region.el#L768-L787

> Dmitry> - Both css-mode and js-mode call syntax-ppss in their indentation
> Dmitry> code. Luckily, their syntax tables are fairly compatible. But
> Dmitry> sgml-syntax-propertize-rules calls syntax-ppss as well.
> 
> I thought that a syntax-table property on the characters would make
> syntax-ppss do the "right" thing; namely, notice where the syntax table
> changes and change its parsing method accordingly.

That seems orthogonal to the possibility of breaking the cache after 
calling syntax-ppss while a different syntax table is in effect.

The characters with the syntax-table text property set should be the 
same in both "world views", but they are usually a minority.

> I didn't try your example yet.  Thanks for providing that, that's very
> helpful.

No problem. BTW, I've worked around it with a hack like this: 
https://github.com/purcell/mmm-mode/blob/master/mmm-erb.el#L93-L98

Re: html, css, and js modes working together

[Clément Pit-Claudel]

On 2017-02-05 22:46, Dmitry Gutov wrote:
> 
>> Yeah, what I mean is that font-locking does not occur, but it would be
>> good to have.  I couldn't find anything saying how this might be solved.
>> I haven't come up with any really good ideas myself.  Maybe font-lock
>> could also look at text properties to decide what keywords to use?
> 
> Here's one way to do it: https://github.com/purcell/mmm-mode/blob/master/mmm-region.el#L768-L787

Org-mode does it pretty well, too (see org-src-fontify-natively). For smaller snippets, see https://github.com/cpitclaudel/indirect-font-lock

Re: html, css, and js modes working together

[Stefan Monnier]

>> I'm not sure what you're referring to here.  Do some of the existing
>> mixed mode things piggyback on the existing html mode?
> Both https://github.com/purcell/mmm-mode/ and
> https://github.com/vspinu/polymode/ do.

BTW: any chance to see some of those in elpa.git some day?

>> I guess my view is that it is always ok to make things in-tree work
>> better with each other, even at the expense of some code that is
>> out-of-tree and presumably relying on implementation details to do its
>> work.

What Dmitry is saying is that it's OK to define an ad-hoc multi-mode for
html+css+js (after all, they're all part of the HTML standard, oh and we
should also add SVG in there), but it should be structured as a separate
mode on top of html-mode, css-mode, and js-mode.

> That seems orthogonal to the possibility of breaking the cache after calling
> syntax-ppss while a different syntax table is in effect.

BTW, the better solution might be to extend syntax.el to provide some
hooks for that.  Not sure what's the best functionality to export, but
maybe a good solution is a way to manage several caches.  Or maybe we
can just have syntax-propertize-function place ad-hoc cache entries
(via some new syntax-ppss-add-to-cache function) at every
major-mode boundary.


        Stefan

Re: html, css, and js modes working together

[Tom Tromey]

>>>>> "Stefan" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> I guess my view is that it is always ok to make things in-tree work
>>> better with each other, even at the expense of some code that is
>>> out-of-tree and presumably relying on implementation details to do its
>>> work.

Stefan> What Dmitry is saying is that it's OK to define an ad-hoc multi-mode for
Stefan> html+css+js (after all, they're all part of the HTML standard, oh and we
Stefan> should also add SVG in there), but it should be structured as a separate
Stefan> mode on top of html-mode, css-mode, and js-mode.

I think there are two things going on here.

One is the structure of the code.  I think it's fine to structure things
so that outside code can continue to reuse miscellaneous bits of the
various modes.  I mean, I think it's a pain since there's nothing
in-tree indicating what can or cannot be done; but assuming someone can
review patches, then it'll be ok.

The other thing is what the user sees.  I think it would be bad to tell
users that to get full html support, including support for embedded js
and css, they must do anything at all -- the default mode ought to
support this.

It's not clear to me if you mean you disagree with the second thing.

Tom

Re: html, css, and js modes working together

[Dmitry Gutov]

On 06.02.2017 22:25, Tom Tromey wrote:

> The other thing is what the user sees.  I think it would be bad to tell
> users that to get full html support, including support for embedded js
> and css, they must do anything at all -- the default mode ought to
> support this.

To be clear, I don't have anything against having html-and-stuff-mode in 
auto-mode-alist. It just has to become functional first.

Re: html, css, and js modes working together

[Stefan Monnier]

>> The other thing is what the user sees.  I think it would be bad to tell
>> users that to get full html support, including support for embedded js
>> and css, they must do anything at all -- the default mode ought to
>> support this.

> To be clear, I don't have anything against having html-and-stuff-mode in
> auto-mode-alist. It just has to become functional first.

And even once it's fully functional, I think it's better to have two
separate modes: one for "HTML-core" and "HTML-with-all-the-sub-modes".

I don't really care which of the two gets to be called `html-mode` and
which is called `html-<something>-mode`, OTOH.


        Stefan

Re: html, css, and js modes working together

[Lennart Borgman]

[-- Attachment #1: Type: text/plain, Size: 2040 bytes --]

On Mon, Feb 6, 2017 at 3:17 PM, Stefan Monnier <monnier@iro.umontreal.ca>
wrote:

> >> I'm not sure what you're referring to here.  Do some of the existing
> >> mixed mode things piggyback on the existing html mode?
> > Both https://github.com/purcell/mmm-mode/ and
> > https://github.com/vspinu/polymode/ do.
>
> BTW: any chance to see some of those in elpa.git some day?
>
> >> I guess my view is that it is always ok to make things in-tree work
> >> better with each other, even at the expense of some code that is
> >> out-of-tree and presumably relying on implementation details to do its
> >> work.
>
> What Dmitry is saying is that it's OK to define an ad-hoc multi-mode for
> html+css+js (after all, they're all part of the HTML standard, oh and we
> should also add SVG in there), but it should be structured as a separate
> mode on top of html-mode, css-mode, and js-mode.
>
> > That seems orthogonal to the possibility of breaking the cache after
> calling
> > syntax-ppss while a different syntax table is in effect.
>
> BTW, the better solution might be to extend syntax.el to provide some
> hooks for that.  Not sure what's the best functionality to export, but
> maybe a good solution is a way to manage several caches.  Or maybe we
> can just have syntax-propertize-function place ad-hoc cache entries
> (via some new syntax-ppss-add-to-cache function) at every
> major-mode boundary.
>
>
>         Stefan
>
>
> Excuse me for not taking part here, but I simply do not have time now. As
you probably know I tried to do these things in MuMaMo (part of nXhtml).
Unfortunately I found that it can not be done without some rather deep
changes to Emacs.

The problem is that basic functions which are used in simple language
parsing in Emacs can't easily be restricted to a region. At least not as
far as I can see. If I am right these basic functions (in C) must be
enhanced to allow this. If that is done then MuMaMo can do the work. (With
some fixes now I guess for changes at the elisp level which I have not had
time to do.)

[-- Attachment #2: Type: text/html, Size: 3542 bytes --]

Re: html, css, and js modes working together

[Tom Tromey]

>>>>> "Dmitry" == Dmitry Gutov <dgutov@yandex.ru> writes:

Dmitry> - Try this example:
Dmitry> <html>
Dmitry>   <script>
Dmitry>     var a = 4;
Dmitry>     alert(a);
Dmitry>   </script>
Dmitry> </html>

Dmitry> It indents fine. Now try replacing "4" with "4 < 5" and reindenting
Dmitry> the "alert" line. It jumps to the right.

I debugged this tonight.

The problem here is that sgml-parse-tag-backward looks for "<" or ">"
characters, but doesn't consider the syntax.  The appended patch fixes
this test case.

My hope is that the html-syntax-propertize-function -- maybe not the one
I wrote but one that's been fixed according to the various comments in
this thread -- should suffice to fix all such problems in principle.
Something like this problem in sgml-parse-tag-backward doesn't
invalidate the scheme; this is just a buglet.  What do you think?

Tom

diff --git a/lisp/textmodes/sgml-mode.el b/lisp/textmodes/sgml-mode.el
index a2f132c..5e0a407 100644
--- a/lisp/textmodes/sgml-mode.el
+++ b/lisp/textmodes/sgml-mode.el
@@ -1290,13 +1290,24 @@ sgml-tag-text-p
       (let ((pps (parse-partial-sexp start end 2)))
 	(and (= (nth 0 pps) 0))))))
 
+(defun sgml--find-<>-backward (limit)
+  "Search backward for a '<' or '>' character.
+The character must have open or close syntax.
+Returns t if found, nil otherwise."
+  (catch 'found
+    (while (re-search-backward "[<>]" limit 'move)
+      ;; If this character has "open" or "close" syntax, then we've
+      ;; found the one we want.
+      (when (memq (syntax-class (syntax-after (point))) '(4 5))
+        (throw 'found t)))))
+
 (defun sgml-parse-tag-backward (&optional limit)
   "Parse an SGML tag backward, and return information about the tag.
 Assume that parsing starts from within a textual context.
 Leave point at the beginning of the tag."
   (catch 'found
     (let (tag-type tag-start tag-end name)
-      (or (re-search-backward "[<>]" limit 'move)
+      (or (sgml--find-<>-backward limit)
 	  (error "No tag found"))
       (when (eq (char-after) ?<)
 	;; Oops!! Looks like we were not in a textual context after all!.

Re: html, css, and js modes working together

[Dmitry Gutov]

On 07.02.2017 05:40, Tom Tromey wrote:

> Dmitry> It indents fine. Now try replacing "4" with "4 < 5" and reindenting
> Dmitry> the "alert" line. It jumps to the right.
> 
> I debugged this tonight.
> 
> The problem here is that sgml-parse-tag-backward looks for "<" or ">"
> characters, but doesn't consider the syntax.  The appended patch fixes
> this test case.

It should work. But maybe another thing to try is avoid using 
sgml-get-context as the dispatch function for html-indent-line.

After all, html-syntax-propertize-function already knows how to find the 
limits of the hunks. The method may change, but hopefully you'd be able 
to reuse it. It could also leave a text property (or some overlays) 
which the indentation function can look up.

> My hope is that the html-syntax-propertize-function -- maybe not the one
> I wrote but one that's been fixed according to the various comments in
> this thread -- should suffice to fix all such problems in principle.

I think so. IME adding syntax-table property on such `<` and `>` works 
well enough. And the main issue to look out for is indentation of HTML 
tags following the tags with such rogue < and > inside.

> Something like this problem in sgml-parse-tag-backward doesn't
> invalidate the scheme; this is just a buglet.  What do you think?

I agree, at least as far as this specific combination of modes in 
concerned. Things get complicated when we try to combine arbitrary modes.

Re: html, css, and js modes working together

[Tom Tromey]

>>>>> "Tom" == Tom Tromey <tom@tromey.com> writes:

Tom> This patch changes the html, css, and js modes to work together a bit.

Here's the second version of this patch.  I'd appreciate comments once
again.

A big thanks to everyone for their help so far.

This version addresses most of the review comments.  It also cleans up
the implementation quite a bit (IMO anyway) and adds a couple new
features: a mode-line highlighter (e.g., it says "HTML+JS" in a <script>
element), and it uses the sub-mode's keymap when point is in a sub-mode
region.

I didn't implement Stefan's suggestion for capturing local variables.  I
did put a special indentation wrapper into css-mode.el to make this area
a bit cleaner.  But, I may still implement his idea; not sure yet.

It turns out there are a bunch of other variables that would be nice to
capture and set when point is in a sub-mode region.  Here I'm thinking
of comment-*, electric characters, font-lock-keywords ... so one
solution that suggests itself is to pull these settings out of the
define-derived-mode invocations and into something that can be reused
from mhtml-mode.

I called the new mode "mhtml-mode"; "m" for "multi".  Naming isn't
always my forte, I'd appreciate suggestions.  I still don't really
understand why a separate new mode is desirable, but I caved to it, and
it does at least solve the circular dependency problem.

One thing I noticed while digging around is that at least align.el looks
specifically for 'html-mode.  This spot (or spots, I didn't look for
more) should be changed to use derived-mode-p.  It's on my to-do list...

A few other to-do items:

* Tests

* Font-lock, as discussed.

* In a sub-mode, disable flyspell, or perhaps rather enable
  flyspell-prog-mode instead.

* ... which brings up the funny issue that mhtml-mode is both a text-
  and a prog-mode and arguably should derive from both.

* imenu and which-func support

* comment-* variables should change depending on the current region

* Electric characters should change depending on region

* ... your feature here?

FWIW I am not trying to tackle multiple major modes in full generality.
I just want to be able to edit mochitests in Firefox with the built-in
Emacs modes.

It's not clear how much of the above is really a requirement.  My
feeling is that improvements can go in even though the result doesn't
implement every possible multi-major-mode feature.

Tom> * Not sure but maybe I also need to define
Tom>   syntax-propertize-extend-region-functions now?

I looked into this and I think the default of
syntax-propertize-wholelines is sufficient.

Stefan> Hmm.. if we want to obey prog-indentation-context,
Stefan> don't we want something like
[...]

In the end I think not.  I think the region parts of that variable are
solely for prog-widen; it's up to the caller instead to simply not set
the variable if this behavior isn't needed.

Tom

diff --git a/lisp/emacs-lisp/smie.el b/lisp/emacs-lisp/smie.el
index 4d02b75..3cb70b5 100644
--- a/lisp/emacs-lisp/smie.el
+++ b/lisp/emacs-lisp/smie.el
@@ -123,6 +123,8 @@
 
 (eval-when-compile (require 'cl-lib))
 
+(require 'prog-mode)
+
 (defgroup smie nil
   "Simple Minded Indentation Engine."
   :group 'languages)
@@ -1455,7 +1457,7 @@ smie-indent-bob
   ;; Start the file at column 0.
   (save-excursion
     (forward-comment (- (point)))
-    (if (bobp) 0)))
+    (if (bobp) (prog-first-column))))
 
 (defun smie-indent-close ()
   ;; Align close paren with opening paren.
@@ -1838,17 +1840,25 @@ smie-auto-fill
         (funcall do-auto-fill)))))
 
 
-(defun smie-setup (grammar rules-function &rest keywords)
-  "Setup SMIE navigation and indentation.
-GRAMMAR is a grammar table generated by `smie-prec2->grammar'.
-RULES-FUNCTION is a set of indentation rules for use on `smie-rules-function'.
-KEYWORDS are additional arguments, which can use the following keywords:
-- :forward-token FUN
-- :backward-token FUN"
+(defmacro smie-with-rules (spec &rest body)
+  "Temporarily set up SMIE indentation and evaluate BODY.
+SPEC is of the form (GRAMMAR RULES-FUNCTION &rest KEYWORDS); see `smie-setup'.
+BODY is evaluated with the relevant SMIE variables temporarily bound."
+  (declare (indent 1))
+  `(smie-funcall-with-rules (list ,@spec) (lambda () . ,body)))
+
+(defun smie-funcall-with-rules (spec fun)
+  (let ((smie-rules-function smie-rules-function)
+        (smie-grammar smie-grammar)
+        (forward-sexp-function forward-sexp-function)
+        (smie-forward-token-function smie-forward-token-function)
+        (smie-backward-token-function smie-backward-token-function))
+    (smie--basic-setup (car spec) (cadr spec) (cddr spec))
+    (funcall fun)))
+
+(defun smie--basic-setup (grammar rules-function keywords)
   (setq-local smie-rules-function rules-function)
   (setq-local smie-grammar grammar)
-  (setq-local indent-line-function #'smie-indent-line)
-  (add-function :around (local 'normal-auto-fill-function) #'smie-auto-fill)
   (setq-local forward-sexp-function #'smie-forward-sexp-command)
   (while keywords
     (let ((k (pop keywords))
@@ -1858,7 +1868,18 @@ smie-setup
          (set (make-local-variable 'smie-forward-token-function) v))
         (`:backward-token
          (set (make-local-variable 'smie-backward-token-function) v))
-        (_ (message "smie-setup: ignoring unknown keyword %s" k)))))
+        (_ (message "smie-setup: ignoring unknown keyword %s" k))))))
+
+(defun smie-setup (grammar rules-function &rest keywords)
+  "Setup SMIE navigation and indentation.
+GRAMMAR is a grammar table generated by `smie-prec2->grammar'.
+RULES-FUNCTION is a set of indentation rules for use on `smie-rules-function'.
+KEYWORDS are additional arguments, which can use the following keywords:
+- :forward-token FUN
+- :backward-token FUN"
+  (smie--basic-setup grammar rules-function keywords)
+  (setq-local indent-line-function #'smie-indent-line)
+  (add-function :around (local 'normal-auto-fill-function) #'smie-auto-fill)
   (let ((ca (cdr (assq :smie-closer-alist grammar))))
     (when ca
       (setq-local smie-closer-alist ca)
diff --git a/lisp/files.el b/lisp/files.el
index b7d1048..77c1e41 100644
--- a/lisp/files.el
+++ b/lisp/files.el
@@ -2422,7 +2422,7 @@ auto-mode-alist
    (lambda (elt)
      (cons (purecopy (car elt)) (cdr elt)))
    `(;; do this first, so that .html.pl is Polish html, not Perl
-     ("\\.[sx]?html?\\(\\.[a-zA-Z_]+\\)?\\'" . html-mode)
+     ("\\.[sx]?html?\\(\\.[a-zA-Z_]+\\)?\\'" . mhtml-mode)
      ("\\.svgz?\\'" . image-mode)
      ("\\.svgz?\\'" . xml-mode)
      ("\\.x[bp]m\\'" . image-mode)
@@ -2784,8 +2784,8 @@ magic-fallback-mode-alist
 		comment-re "*"
 		"\\(?:!DOCTYPE[ \t\r\n]+[^>]*>[ \t\r\n]*<[ \t\r\n]*" comment-re "*\\)?"
 		"[Hh][Tt][Mm][Ll]"))
-     . html-mode)
-    ("<!DOCTYPE[ \t\r\n]+[Hh][Tt][Mm][Ll]" . html-mode)
+     . mhtml-mode)
+    ("<!DOCTYPE[ \t\r\n]+[Hh][Tt][Mm][Ll]" . mhtml-mode)
     ;; These two must come after html, because they are more general:
     ("<\\?xml " . xml-mode)
     (,(let* ((incomment-re "\\(?:[^-]\\|-[^-]\\)")
diff --git a/lisp/progmodes/js.el b/lisp/progmodes/js.el
index e42e014..aab5bd6 100644
--- a/lisp/progmodes/js.el
+++ b/lisp/progmodes/js.el
@@ -53,6 +53,7 @@
 (require 'moz nil t)
 (require 'json nil t)
 (require 'sgml-mode)
+(require 'prog-mode)
 
 (eval-when-compile
   (require 'cl-lib)
@@ -2102,7 +2103,7 @@ js--proper-indentation
 
           ((js--continued-expression-p)
            (+ js-indent-level js-expr-indent-offset))
-          (t 0))))
+          (t (prog-first-column)))))
 
 ;;; JSX Indentation
 
diff --git a/lisp/textmodes/css-mode.el b/lisp/textmodes/css-mode.el
index 19746c6..d837756 100644
--- a/lisp/textmodes/css-mode.el
+++ b/lisp/textmodes/css-mode.el
@@ -970,6 +970,13 @@ css-completion-at-point
                   (list sel-beg sel-end))
               ,(completion-table-merge prop-table sel-table)))))))
 
+(defun css-advertized-indent-line ()
+  "A wrapper for `smie-indent-line' that first installs the SMIE rules."
+  (smie-with-rules (css-smie-grammar #'css-smie-rules
+                                     :forward-token #'css-smie--forward-token
+                                     :backward-token #'css-smie--backward-token)
+    (smie-indent-line)))
+
 ;;;###autoload
 (define-derived-mode css-mode prog-mode "CSS"
   "Major mode to edit Cascading Style Sheets."
diff --git a/lisp/textmodes/mhtml-mode.el b/lisp/textmodes/mhtml-mode.el
new file mode 100644
index 0000000..0eaea56
--- /dev/null
+++ b/lisp/textmodes/mhtml-mode.el
@@ -0,0 +1,143 @@
+;;; mhtml-mode.el --- HTML editing mode that handles CSS and JS -*- lexical-binding:t -*-
+
+;; Copyright (C) 2017 Free Software Foundation, Inc.
+
+;; Keywords: wp, hypermedia, comm, languages
+
+;; This file is part of GNU Emacs.
+
+;; GNU Emacs is free software: you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+
+;; GNU Emacs is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.
+
+;;; Code:
+
+(eval-and-compile
+  (require 'sgml-mode)
+  (require 'smie))
+(require 'js)
+(require 'css-mode)
+(require 'prog-mode)
+
+(cl-defstruct mhtml--submode
+  ;; Name of this mode.
+  name
+  ;; HTML end tag.
+  end-tag
+  ;; Syntax table.
+  syntax-table
+  ;; Propertize function.
+  propertize
+  ;; Indentation function.
+  indenter
+  ;; Keymap.
+  keymap)
+
+(defconst mhtml--css-submode
+  (make-mhtml--submode :name "CSS"
+                       :end-tag "</style>"
+                       :syntax-table css-mode-syntax-table
+                       :propertize css-syntax-propertize-function
+                       :indenter #'css-advertized-indent-line
+                       :keymap css-mode-map))
+
+(defconst mhtml--js-submode
+  (make-mhtml--submode :name "JS"
+                       :end-tag "</script>"
+                       :syntax-table js-mode-syntax-table
+                       :propertize #'js-syntax-propertize
+                       :indenter #'js-indent-line
+                       :keymap js-mode-map))
+
+(defun mhtml--submode-lighter ()
+  "Mode-line lighter indicating the current submode."
+  (let ((submode (get-text-property (point) 'mhtml-submode)))
+    (if submode
+        (mhtml--submode-name submode)
+      "")))
+
+(defun mhtml--syntax-propertize-submode (submode end)
+  (save-excursion
+    (when (search-forward (mhtml--submode-end-tag submode) end t)
+      (setq end (match-beginning 0))))
+  (set-text-properties (point) end
+                       (list 'mhtml-submode submode
+                             'syntax-table (mhtml--submode-syntax-table submode)
+                             ;; We want local-map here so that we act
+                             ;; more like the sub-mode and don't
+                             ;; override minor mode maps.
+                             'local-map (mhtml--submode-keymap submode)
+                             'cursor-sensor-functions
+                             (list (lambda (_window _old-point _action)
+                                     (force-mode-line-update)))))
+  (funcall (mhtml--submode-propertize submode) (point) end)
+  (goto-char end))
+
+(defun mhtml-syntax-propertize (start end)
+  (goto-char start)
+  (when (get-text-property (point) 'mhtml-submode)
+    (mhtml--syntax-propertize-submode (get-text-property (point) 'mhtml-submode)
+                                      end))
+  (funcall
+   (syntax-propertize-rules
+    ("<style.*?>"
+     (0 (ignore
+         (goto-char (match-end 0))
+         (mhtml--syntax-propertize-submode mhtml--css-submode end))))
+    ("<script.*?>"
+     (0 (ignore
+         (goto-char (match-end 0))
+         (mhtml--syntax-propertize-submode mhtml--js-submode end))))
+    sgml-syntax-propertize-rules)
+   ;; Make sure to handle the situation where
+   ;; mhtml--syntax-propertize-submode moved point.
+   (point) end))
+
+(defun mhtml-indent-line ()
+  "Indent the current line as HTML, JS, or CSS, according to its context."
+  (interactive)
+  (let ((submode (save-excursion
+                   (back-to-indentation)
+                   (get-text-property (point) 'mhtml-submode))))
+    (if submode
+        (save-restriction
+          (let* ((region-start (previous-single-property-change (point)
+                                                                'mhtml-submode))
+                 (base-indent (save-excursion
+                                (goto-char region-start)
+                                (sgml-calculate-indent))))
+            (narrow-to-region region-start (point-max))
+            (let ((prog-indentation-context (list base-indent
+                                                  (cons (point-min) nil)
+                                                  nil)))
+              (funcall (mhtml--submode-indenter submode)))))
+      ;; HTML.
+      (sgml-indent-line))))
+
+;;;###autoload
+(define-derived-mode mhtml-mode html-mode
+  '((sgml-xml-mode "XHTML+" "HTML+") (:eval (mhtml--submode-lighter)))
+  "Major mode based on `html-mode', but works with embedded JS and CSS.
+
+Code inside a <script> element is indented using the rules from
+`js-mode'; and code inside a <style> element is indented using
+the rules from `css-mode'."
+  (cursor-sensor-mode)
+  (setq-local indent-line-function #'mhtml-indent-line)
+  (setq-local parse-sexp-lookup-properties t)
+  (setq-local syntax-propertize-function #'mhtml-syntax-propertize)
+  (add-hook 'syntax-propertize-extend-region-functions
+            #'syntax-propertize-multiline 'append 'local))
+
+(provide 'mhtml-mode)
+
+;;; mhtml-mode.el ends here
diff --git a/lisp/textmodes/sgml-mode.el b/lisp/textmodes/sgml-mode.el
index e148b06..8ad7cfb 100644
--- a/lisp/textmodes/sgml-mode.el
+++ b/lisp/textmodes/sgml-mode.el
@@ -341,19 +341,23 @@ sgml-font-lock-keywords-2
 (defvar sgml-font-lock-keywords sgml-font-lock-keywords-1
   "Rules for highlighting SGML code.  See also `sgml-tag-face-alist'.")
 
+(eval-and-compile
+  (defconst sgml-syntax-propertize-rules
+    (syntax-propertize-precompile-rules
+     ;; Use the `b' style of comments to avoid interference with the -- ... --
+     ;; comments recognized when `sgml-specials' includes ?-.
+     ;; FIXME: beware of <!--> blabla <!--> !!
+     ("\\(<\\)!--" (1 "< b"))
+     ("--[ \t\n]*\\(>\\)" (1 "> b"))
+     ;; Double quotes outside of tags should not introduce strings.
+     ;; Be careful to call `syntax-ppss' on a position before the one we're
+     ;; going to change, so as not to need to flush the data we just computed.
+     ("\"" (0 (if (prog1 (zerop (car (syntax-ppss (match-beginning 0))))
+                    (goto-char (match-end 0)))
+                  (string-to-syntax ".")))))))
+
 (defconst sgml-syntax-propertize-function
-  (syntax-propertize-rules
-   ;; Use the `b' style of comments to avoid interference with the -- ... --
-   ;; comments recognized when `sgml-specials' includes ?-.
-  ;; FIXME: beware of <!--> blabla <!--> !!
-   ("\\(<\\)!--" (1 "< b"))
-    ("--[ \t\n]*\\(>\\)" (1 "> b"))
-    ;; Double quotes outside of tags should not introduce strings.
-    ;; Be careful to call `syntax-ppss' on a position before the one we're
-    ;; going to change, so as not to need to flush the data we just computed.
-    ("\"" (0 (if (prog1 (zerop (car (syntax-ppss (match-beginning 0))))
-                   (goto-char (match-end 0)))
-           (string-to-syntax ".")))))
+  (syntax-propertize-rules sgml-syntax-propertize-rules)
   "Syntactic keywords for `sgml-mode'.")
 
 ;; internal
@@ -1284,13 +1288,24 @@ sgml-tag-text-p
       (let ((pps (parse-partial-sexp start end 2)))
 	(and (= (nth 0 pps) 0))))))
 
+(defun sgml--find-<>-backward (limit)
+  "Search backward for a '<' or '>' character.
+The character must have open or close syntax.
+Returns t if found, nil otherwise."
+  (catch 'found
+    (while (re-search-backward "[<>]" limit 'move)
+      ;; If this character has "open" or "close" syntax, then we've
+      ;; found the one we want.
+      (when (memq (syntax-class (syntax-after (point))) '(4 5))
+        (throw 'found t)))))
+
 (defun sgml-parse-tag-backward (&optional limit)
   "Parse an SGML tag backward, and return information about the tag.
 Assume that parsing starts from within a textual context.
 Leave point at the beginning of the tag."
   (catch 'found
     (let (tag-type tag-start tag-end name)
-      (or (re-search-backward "[<>]" limit 'move)
+      (or (sgml--find-<>-backward limit)
 	  (error "No tag found"))
       (when (eq (char-after) ?<)
 	;; Oops!! Looks like we were not in a textual context after all!.

Re: html, css, and js modes working together

[Stefan Monnier]

Regarding the mode's name.  You could rename html-mode to html-only-mode
(or html-basic-mode?) and then use html-mode for that new mode.
Ideally, you could even make html-only-mode a parameter so that your new
html-mode could be used with other "basic" html editing modes such as
nxml-mode or psgml-mode.

> @@ -1455,7 +1457,7 @@ smie-indent-bob
>    ;; Start the file at column 0.
>    (save-excursion
>      (forward-comment (- (point)))
> -    (if (bobp) 0)))
> +    (if (bobp) (prog-first-column))))
 
This looks good.

> +(defmacro smie-with-rules (spec &rest body)
> +  "Temporarily set up SMIE indentation and evaluate BODY.
> +SPEC is of the form (GRAMMAR RULES-FUNCTION &rest KEYWORDS); see `smie-setup'.
> +BODY is evaluated with the relevant SMIE variables temporarily bound."
> +  (declare (indent 1))
> +  `(smie-funcall-with-rules (list ,@spec) (lambda () . ,body)))
> +
> +(defun smie-funcall-with-rules (spec fun)
> +  (let ((smie-rules-function smie-rules-function)
> +        (smie-grammar smie-grammar)
> +        (forward-sexp-function forward-sexp-function)
> +        (smie-forward-token-function smie-forward-token-function)
> +        (smie-backward-token-function smie-backward-token-function))
> +    (smie--basic-setup (car spec) (cadr spec) (cddr spec))
> +    (funcall fun)))

The use of buffer-local-variables+progv should make this unnecessary.

> -     ("\\.[sx]?html?\\(\\.[a-zA-Z_]+\\)?\\'" . html-mode)
> +     ("\\.[sx]?html?\\(\\.[a-zA-Z_]+\\)?\\'" . mhtml-mode)

FWIW, I think that html-mode should not belong to any html mode package
in particular (maybe it should use `define-alternatives`).


        Stefan

Re: html, css, and js modes working together

[Tom Tromey]

>>>>> "Stefan" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

Stefan> [ smie-with-rules ]
Stefan> The use of buffer-local-variables+progv should make this unnecessary.

I implemented the buffer-local-variables stuff last night, so I will try
removing this code soon.

>> -     ("\\.[sx]?html?\\(\\.[a-zA-Z_]+\\)?\\'" . html-mode)
>> +     ("\\.[sx]?html?\\(\\.[a-zA-Z_]+\\)?\\'" . mhtml-mode)

Stefan> FWIW, I think that html-mode should not belong to any html mode package
Stefan> in particular (maybe it should use `define-alternatives`).

Do you mean that neither the new nor old modes should be called
'html-mode'?  I don't understand what benefit this would provide.

Tom

Re: html, css, and js modes working together

[Stefan Monnier]

Stefan> FWIW, I think that html-mode should not belong to any html mode package
Stefan> in particular (maybe it should use `define-alternatives`).

> Do you mean that neither the new nor old modes should be called
> 'html-mode'?

I think I'm leaning that way, yes.

> I don't understand what benefit this would provide.

To keep `html-mode` as the dispatch function for the various options
(there's web-mode, mumamo's xhtml-mode, psgaml-mode, nxml-mode, ...).


        Stefan

Re: html, css, and js modes working together

[Tom Tromey]

>>>>> "Stefan" == Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

Stefan> To keep `html-mode` as the dispatch function for the various options
Stefan> (there's web-mode, mumamo's xhtml-mode, psgaml-mode, nxml-mode, ...).

The way I see it is that Emacs should come with something reasonable out
of the box.  Disabling a feature because it might cause difficulty for
some other mode that isn't in-tree is both a disservice to users (the
default Emacs is missing a useful feature) and a maintenance problem
(who knows what else is out there).

Also, concretely speaking, these modes will have an easy time adapting.
In particular the new mode sets 4 local variables:

  (setq-local indent-line-function #'mhtml-indent-line)
  (setq-local parse-sexp-lookup-properties t)
  (setq-local syntax-propertize-function #'mhtml-syntax-propertize)
  (setq-local font-lock-fontify-region-function
              #'mhtml--submode-fontify-region)

... so all some other deriving mode would have to do is reset these (and
maybe not even parse-sexp-lookup-properties, since it's harmless).

As for hooking in to auto-mode-alist or whatever, there are already ok
approaches to this, I think.  web-mode or the like can either prepend an
entry to auto-mode-alist, or even fset html-mode to point to their mode
function.  (web-mode in particular is funny because it tries to do
everything; so I suppose in theory if html-mode must be subjected to
alternatives, so should js-, css-, ... whatever web-mode adds next.)

Tom

Re: html, css, and js modes working together

[Stefan Monnier]

> The way I see it is that Emacs should come with something reasonable out
> of the box.

Agreed: html-mode should default to be an alias for "the most popular
html mode that comes with Emacs".  That doesn't mean that the canonical
name of that mod should be `html-mode`.


        Stefan

Re: html, css, and js modes working together

[Dmitry Gutov]

On 11.02.2017 22:20, Stefan Monnier wrote:
>> The way I see it is that Emacs should come with something reasonable out
>> of the box.
> 
> Agreed: html-mode should default to be an alias for "the most popular
> html mode that comes with Emacs".  That doesn't mean that the canonical
> name of that mod should be `html-mode`.

Why not call it selected-html-mode or something? Isn't the most 
important thing to just have it in auto-mode-alist for *.html files out 
of the box? Also, if the function name doesn't end with "-mode", people 
won't try to derive from it, which is good because derived-mode-p 
doesn't handle aliases.

Not a huge concern, but it would make life a bit easier for mmm-mode and 
polymode if html-mode remainted the same as html-basic-mode.

Re: html, css, and js modes working together

[Dmitry Gutov]

On 11.02.2017 19:46, Tom Tromey wrote:

> Stefan> To keep `html-mode` as the dispatch function for the various options
> Stefan> (there's web-mode, mumamo's xhtml-mode, psgaml-mode, nxml-mode, ...).
> 
> The way I see it is that Emacs should come with something reasonable out
> of the box.  Disabling a feature because it might cause difficulty for
> some other mode that isn't in-tree is both a disservice to users (the
> default Emacs is missing a useful feature) and a maintenance problem
> (who knows what else is out there).

The default Emacs will have this feature if html-rich-mode is associated 
with *.html in auto-mode-alist.

> Also, concretely speaking, these modes will have an easy time adapting.
> In particular the new mode sets 4 local variables:
> 
>    (setq-local indent-line-function #'mhtml-indent-line)
>    (setq-local parse-sexp-lookup-properties t)
>    (setq-local syntax-propertize-function #'mhtml-syntax-propertize)
>    (setq-local font-lock-fontify-region-function
>                #'mhtml--submode-fontify-region)
> 
> ... so all some other deriving mode would have to do is reset these (and
> maybe not even parse-sexp-lookup-properties, since it's harmless).

How does your buffer-locals approach work?

In mmm-mode, you first call a major mode to get its values of 
indent-line-function and so on. Without them being available after 
calling html-mode, we'd have to hardcode an annoying amount of stuff, 
and become less resilient against renames.

Re: html, css, and js modes working together

[Tom Tromey]

>>>>> "Dmitry" == Dmitry Gutov <dgutov@yandex.ru> writes:

Dmitry> How does your buffer-locals approach work?

I've appended the current code.

It makes a temporary buffer for each submode and captures a bunch of
locals.  There are two kinds of capture: those that are applied when
entering or leaving a region, and those that are only let-bound in
certain situations.

Dmitry> In mmm-mode, you first call a major mode to get its values of
Dmitry> indent-line-function and so on. Without them being available after
Dmitry> calling html-mode, we'd have to hardcode an annoying amount of stuff,
Dmitry> and become less resilient against renames.

Yeah, this is the same; it was Stefan's suggestion upthread.

Maybe all of this should just be done by pulling mmm-mode into Emacs?

Tom

;;; mhtml-mode.el --- HTML editing mode that handles CSS and JS -*- lexical-binding:t -*-

;; Copyright (C) 2017 Free Software Foundation, Inc.

;; Keywords: wp, hypermedia, comm, languages

;; This file is part of GNU Emacs.

;; GNU Emacs is free software: you can redistribute it and/or modify
;; it under the terms of the GNU General Public License as published by
;; the Free Software Foundation, either version 3 of the License, or
;; (at your option) any later version.

;; GNU Emacs is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;; GNU General Public License for more details.

;; You should have received a copy of the GNU General Public License
;; along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.

;;; Code:

(eval-and-compile
  (require 'sgml-mode))
(require 'js)
(require 'css-mode)
(require 'prog-mode)
(require 'font-lock)

(cl-defstruct mhtml--submode
  ;; Name of this submode.
  name
  ;; HTML end tag.
  end-tag
  ;; Syntax table.
  syntax-table
  ;; Propertize function.
  propertize
  ;; Keymap.
  keymap
  ;; Captured locals that are set when entering a region.
  crucial-captured-locals
  ;; Other captured local variables; these are not set when entering a
  ;; region but let-bound during certain operations, e.g.,
  ;; indentation.
  captured-locals)

(defconst mhtml--crucial-variable-prefix
  (regexp-opt '("comment-" "uncomment-" "electric-indent-"
                "smie-" "forward-sexp-function"))
  "Regexp matching the prefix of \"crucial\" buffer-locals we want to capture.")

(defconst mhtml--variable-prefix
  (regexp-opt '("font-lock-" "indent-line-function" "major-mode"))
  "Regexp matching the prefix of buffer-locals we want to capture.")

(defun mhtml--construct-submode (mode &rest args)
  "A wrapper for make-mhtml--submode that computes the buffer-local variables."
  (let ((captured-locals nil)
        (crucial-captured-locals nil)
        (submode (apply #'make-mhtml--submode args)))
    (with-temp-buffer
      (funcall mode)
      ;; Make sure font lock is all set up.
      (font-lock-set-defaults)
      (dolist (iter (buffer-local-variables))
        (when (string-match mhtml--crucial-variable-prefix
                            (symbol-name (car iter)))
          (push iter crucial-captured-locals))
        (when (string-match mhtml--variable-prefix (symbol-name (car iter)))
          (push iter captured-locals)))
      (setf (mhtml--submode-crucial-captured-locals submode)
            crucial-captured-locals)
      (setf (mhtml--submode-captured-locals submode) captured-locals))
    submode))

(defun mhtml--mark-buffer-locals (submode)
  (dolist (iter (mhtml--submode-captured-locals submode))
    (make-local-variable (car iter))))

(defvar-local mhtml--crucial-variables nil
  "List of all crucial variable symbols.")

(defun mhtml--mark-crucial-buffer-locals (submode)
  (dolist (iter (mhtml--submode-crucial-captured-locals submode))
    (make-local-variable (car iter))
    (push (car iter) mhtml--crucial-variables)))

(defconst mhtml--css-submode
  (mhtml--construct-submode 'css-mode
                            :name "CSS"
                            :end-tag "</style>"
                            :syntax-table css-mode-syntax-table
                            :propertize css-syntax-propertize-function
                            :keymap css-mode-map))

(defconst mhtml--js-submode
  (mhtml--construct-submode 'js-mode
                            :name "JS"
                            :end-tag "</script>"
                            :syntax-table js-mode-syntax-table
                            :propertize #'js-syntax-propertize
                            :keymap js-mode-map))

(defmacro mhtml--with-locals (submode &rest body)
  (declare (indent 1))
  `(cl-progv
       (when submode (mapcar #'car (mhtml--submode-captured-locals submode)))
       (when submode (mapcar #'cdr (mhtml--submode-captured-locals submode)))
     (cl-progv
         (when submode (mapcar #'car (mhtml--submode-crucial-captured-locals
                                      submode)))
         (when submode (mapcar #'cdr (mhtml--submode-crucial-captured-locals
                                      submode)))
       ,@body)))

(defun mhtml--submode-lighter ()
  "Mode-line lighter indicating the current submode."
  (let ((submode (get-text-property (point) 'mhtml-submode)))
    (if submode
        (mhtml--submode-name submode)
      "")))

(defun mhtml--submode-fontify-one-region (submode beg end &optional loudly)
  (if submode
      (mhtml--with-locals submode
        (save-restriction
          (narrow-to-region beg end)
          (font-lock-set-defaults)
          (font-lock-default-fontify-region (point-min) (point-max) loudly)))
    (font-lock-set-defaults)
    (font-lock-default-fontify-region beg end loudly)))

(defun mhtml--submode-fontify-region (beg end loudly)
  (while (< beg end)
    (let ((submode (get-text-property beg 'mhtml-submode))
          (this-end (next-single-property-change beg 'mhtml-submode
                                                 nil end)))
      (mhtml--submode-fontify-one-region submode beg this-end loudly)
      (setq beg this-end))))

(defvar-local mhtml--last-submode nil
  "Record the last visited submode, so the cursor-sensor function
can function properly.")

(defvar-local mhtml--stashed-crucial-variables nil
  "Alist of stashed values of the crucial variables.")

(defun mhtml--stash-crucial-variables ()
  (setq mhtml--stashed-crucial-variables
        (mapcar (lambda (sym)
                  (cons sym (buffer-local-value sym (current-buffer))))
                mhtml--crucial-variables)))

(defun mhtml--map-in-crucial-variables (alist)
  (dolist (item alist)
    (set (car item) (cdr item))))

(defun mhtml--cursor-sensor (_window pos _action)
  (let ((submode (get-text-property (point) 'mhtml-submode)))
    ;; If we're entering a submode, and the previous submode was nil,
    ;; then stash the current values first.  This lets the user at
    ;; least modify some values directly.
    (when (and submode (not mhtml--last-submode))
      (mhtml--stash-crucial-variables))
    (mhtml--map-in-crucial-variables
     (if submode
         (mhtml--submode-crucial-captured-locals submode)
       mhtml--stashed-crucial-variables))
    (setq mhtml--last-submode submode)
    (force-mode-line-update)))

(defun mhtml--syntax-propertize-submode (submode end)
  (save-excursion
    (when (search-forward (mhtml--submode-end-tag submode) end t)
      (setq end (match-beginning 0))))
  (set-text-properties (point) end
                       (list 'mhtml-submode submode
                             'syntax-table (mhtml--submode-syntax-table submode)
                             ;; We want local-map here so that we act
                             ;; more like the sub-mode and don't
                             ;; override minor mode maps.
                             'local-map (mhtml--submode-keymap submode)
                             'cursor-sensor-functions
                             '(mhtml--cursor-sensor)))
  (funcall (mhtml--submode-propertize submode) (point) end)
  (goto-char end))

(defun mhtml-syntax-propertize (start end)
  (goto-char start)
  (when (get-text-property (point) 'mhtml-submode)
    (mhtml--syntax-propertize-submode (get-text-property (point) 'mhtml-submode)
                                      end))
  (funcall
   (syntax-propertize-rules
    ("<style.*?>"
     (0 (ignore
         (goto-char (match-end 0))
         (mhtml--syntax-propertize-submode mhtml--css-submode end))))
    ("<script.*?>"
     (0 (ignore
         (goto-char (match-end 0))
         (mhtml--syntax-propertize-submode mhtml--js-submode end))))
    sgml-syntax-propertize-rules)
   ;; Make sure to handle the situation where
   ;; mhtml--syntax-propertize-submode moved point.
   (point) end))

(defun mhtml-indent-line ()
  "Indent the current line as HTML, JS, or CSS, according to its context."
  (interactive)
  (let ((submode (save-excursion
                   (back-to-indentation)
                   (get-text-property (point) 'mhtml-submode))))
    (if submode
        (save-restriction
          (let* ((region-start (previous-single-property-change (point)
                                                                'mhtml-submode))
                 (base-indent (save-excursion
                                (goto-char region-start)
                                (sgml-calculate-indent))))
            (narrow-to-region region-start (point-max))
            (let ((prog-indentation-context (list base-indent
                                                  (cons (point-min) nil)
                                                  nil)))
              (mhtml--with-locals submode
                ;; indent-line-function was rebound by
                ;; mhtml--with-locals.
                (funcall indent-line-function)))))
      ;; HTML.
      (sgml-indent-line))))

;;;###autoload
(define-derived-mode mhtml-mode html-mode
  '((sgml-xml-mode "XHTML+" "HTML+") (:eval (mhtml--submode-lighter)))
  "Major mode based on `html-mode', but works with embedded JS and CSS.

Code inside a <script> element is indented using the rules from
`js-mode'; and code inside a <style> element is indented using
the rules from `css-mode'."
  (cursor-sensor-mode)
  (setq-local indent-line-function #'mhtml-indent-line)
  (setq-local parse-sexp-lookup-properties t)
  (setq-local syntax-propertize-function #'mhtml-syntax-propertize)
  (setq-local font-lock-fontify-region-function
              #'mhtml--submode-fontify-region)

  ;; Make any captured variables buffer-local.
  (mhtml--mark-buffer-locals mhtml--css-submode)
  (mhtml--mark-buffer-locals mhtml--js-submode)

  (mhtml--mark-crucial-buffer-locals mhtml--css-submode)
  (mhtml--mark-crucial-buffer-locals mhtml--js-submode)
  (setq mhtml--crucial-variables (delete-dups mhtml--crucial-variables))

  ;: Hack
  (js--update-quick-match-re))

(provide 'mhtml-mode)

;;; mhtml-mode.el ends here

Re: html, css, and js modes working together

[Dmitry Gutov]

On 12.02.2017 18:52, Tom Tromey wrote:
 > I've appended the current code.

Thanks.

> Maybe all of this should just be done by pulling mmm-mode into Emacs?

Hmm, now that I'm looking at the commit stats in there, there are two 
main developers we only need to contact for copyright assignments, or 
maybe just one if Alan Thomas Shutko <ats@acm.org> is the same person as 
Alan Shutko <ats@springies.com>.

And both of them were responsive 4 years ago when we moved the 
repository. Not sure why I assumed this can't be done, previously.

To answer the actual question, using mmm-mode would not help this 
approach immediately, since it's a minor mode, not a tool for 
constructing derived modes, but the work to change it into the latter 
should be straightforward (though probably not effortless) if we make 
such decision.

It might be good for ideas and/or stealing code either way.

Re: html, css, and js modes working together

[Dmitry Gutov]

On 12.02.2017 18:52, Tom Tromey wrote:

> I've appended the current code.

I'm getting "Symbol’s value as variable is void: 
sgml-syntax-propertize-rules". Does it still require parts of the 
previous patch?

Re: html, css, and js modes working together

[Tom Tromey]

>>>>> "Dmitry" == Dmitry Gutov <dgutov@yandex.ru> writes:

>> I've appended the current code.

Dmitry> I'm getting "Symbol’s value as variable is void:
Dmitry> sgml-syntax-propertize-rules". Does it still require parts of the
Dmitry> previous patch?

Yeah, sorry, I only appended the new mode's file since I thought you
only wanted it to see what it was doing.

Here's my current patch, now with pre- and post-command hooks and
flyspell integration.  Font lock mostly works but has some oddities, and
I still didn't fix the html comment/syntax bug that Clément found.

This is way smaller than I thought it would end up being, basically I
think because syntax-propertizing is a nice tool.

Tom

diff --git a/lisp/emacs-lisp/smie.el b/lisp/emacs-lisp/smie.el
index 4d02b75..7baccbc 100644
--- a/lisp/emacs-lisp/smie.el
+++ b/lisp/emacs-lisp/smie.el
@@ -123,6 +123,8 @@
 
 (eval-when-compile (require 'cl-lib))
 
+(require 'prog-mode)
+
 (defgroup smie nil
   "Simple Minded Indentation Engine."
   :group 'languages)
@@ -1455,7 +1457,7 @@ smie-indent-bob
   ;; Start the file at column 0.
   (save-excursion
     (forward-comment (- (point)))
-    (if (bobp) 0)))
+    (if (bobp) (prog-first-column))))
 
 (defun smie-indent-close ()
   ;; Align close paren with opening paren.
diff --git a/lisp/files.el b/lisp/files.el
index b7d1048..77c1e41 100644
--- a/lisp/files.el
+++ b/lisp/files.el
@@ -2422,7 +2422,7 @@ auto-mode-alist
    (lambda (elt)
      (cons (purecopy (car elt)) (cdr elt)))
    `(;; do this first, so that .html.pl is Polish html, not Perl
-     ("\\.[sx]?html?\\(\\.[a-zA-Z_]+\\)?\\'" . html-mode)
+     ("\\.[sx]?html?\\(\\.[a-zA-Z_]+\\)?\\'" . mhtml-mode)
      ("\\.svgz?\\'" . image-mode)
      ("\\.svgz?\\'" . xml-mode)
      ("\\.x[bp]m\\'" . image-mode)
@@ -2784,8 +2784,8 @@ magic-fallback-mode-alist
 		comment-re "*"
 		"\\(?:!DOCTYPE[ \t\r\n]+[^>]*>[ \t\r\n]*<[ \t\r\n]*" comment-re "*\\)?"
 		"[Hh][Tt][Mm][Ll]"))
-     . html-mode)
-    ("<!DOCTYPE[ \t\r\n]+[Hh][Tt][Mm][Ll]" . html-mode)
+     . mhtml-mode)
+    ("<!DOCTYPE[ \t\r\n]+[Hh][Tt][Mm][Ll]" . mhtml-mode)
     ;; These two must come after html, because they are more general:
     ("<\\?xml " . xml-mode)
     (,(let* ((incomment-re "\\(?:[^-]\\|-[^-]\\)")
diff --git a/lisp/progmodes/js.el b/lisp/progmodes/js.el
index b42b2bc..05bf635 100644
--- a/lisp/progmodes/js.el
+++ b/lisp/progmodes/js.el
@@ -53,6 +53,7 @@
 (require 'moz nil t)
 (require 'json nil t)
 (require 'sgml-mode)
+(require 'prog-mode)
 
 (eval-when-compile
   (require 'cl-lib)
@@ -2109,7 +2110,7 @@ js--proper-indentation
 
           ((js--continued-expression-p)
            (+ js-indent-level js-expr-indent-offset))
-          (t 0))))
+          (t (prog-first-column)))))
 
 ;;; JSX Indentation
 
diff --git a/lisp/textmodes/mhtml-mode.el b/lisp/textmodes/mhtml-mode.el
new file mode 100644
index 0000000..a647060
--- /dev/null
+++ b/lisp/textmodes/mhtml-mode.el
@@ -0,0 +1,274 @@
+;;; mhtml-mode.el --- HTML editing mode that handles CSS and JS -*- lexical-binding:t -*-
+
+;; Copyright (C) 2017 Free Software Foundation, Inc.
+
+;; Keywords: wp, hypermedia, comm, languages
+
+;; This file is part of GNU Emacs.
+
+;; GNU Emacs is free software: you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+
+;; GNU Emacs is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.
+
+;;; Code:
+
+(eval-and-compile
+  (require 'sgml-mode))
+(require 'js)
+(require 'css-mode)
+(require 'prog-mode)
+(require 'font-lock)
+
+(cl-defstruct mhtml--submode
+  ;; Name of this submode.
+  name
+  ;; HTML end tag.
+  end-tag
+  ;; Syntax table.
+  syntax-table
+  ;; Propertize function.
+  propertize
+  ;; Keymap.
+  keymap
+  ;; Captured locals that are set when entering a region.
+  crucial-captured-locals
+  ;; Other captured local variables; these are not set when entering a
+  ;; region but let-bound during certain operations, e.g.,
+  ;; indentation.
+  captured-locals)
+
+(defconst mhtml--crucial-variable-prefix
+  (regexp-opt '("comment-" "uncomment-" "electric-indent-"
+                "smie-" "forward-sexp-function"))
+  "Regexp matching the prefix of \"crucial\" buffer-locals we want to capture.")
+
+(defconst mhtml--variable-prefix
+  (regexp-opt '("font-lock-" "indent-line-function" "major-mode"))
+  "Regexp matching the prefix of buffer-locals we want to capture.")
+
+(defun mhtml--construct-submode (mode &rest args)
+  "A wrapper for make-mhtml--submode that computes the buffer-local variables."
+  (let ((captured-locals nil)
+        (crucial-captured-locals nil)
+        (submode (apply #'make-mhtml--submode args)))
+    (with-temp-buffer
+      (funcall mode)
+      ;; Make sure font lock is all set up.
+      (font-lock-set-defaults)
+      (dolist (iter (buffer-local-variables))
+        (when (string-match mhtml--crucial-variable-prefix
+                            (symbol-name (car iter)))
+          (push iter crucial-captured-locals))
+        (when (string-match mhtml--variable-prefix (symbol-name (car iter)))
+          (push iter captured-locals)))
+      (setf (mhtml--submode-crucial-captured-locals submode)
+            crucial-captured-locals)
+      (setf (mhtml--submode-captured-locals submode) captured-locals))
+    submode))
+
+(defun mhtml--mark-buffer-locals (submode)
+  (dolist (iter (mhtml--submode-captured-locals submode))
+    (make-local-variable (car iter))))
+
+(defvar-local mhtml--crucial-variables nil
+  "List of all crucial variable symbols.")
+
+(defun mhtml--mark-crucial-buffer-locals (submode)
+  (dolist (iter (mhtml--submode-crucial-captured-locals submode))
+    (make-local-variable (car iter))
+    (push (car iter) mhtml--crucial-variables)))
+
+(defconst mhtml--css-submode
+  (mhtml--construct-submode 'css-mode
+                            :name "CSS"
+                            :end-tag "</style>"
+                            :syntax-table css-mode-syntax-table
+                            :propertize css-syntax-propertize-function
+                            :keymap css-mode-map))
+
+(defconst mhtml--js-submode
+  (mhtml--construct-submode 'js-mode
+                            :name "JS"
+                            :end-tag "</script>"
+                            :syntax-table js-mode-syntax-table
+                            :propertize #'js-syntax-propertize
+                            :keymap js-mode-map))
+
+(defmacro mhtml--with-locals (submode &rest body)
+  (declare (indent 1))
+  `(cl-progv
+       (when submode (mapcar #'car (mhtml--submode-captured-locals submode)))
+       (when submode (mapcar #'cdr (mhtml--submode-captured-locals submode)))
+     (cl-progv
+         (when submode (mapcar #'car (mhtml--submode-crucial-captured-locals
+                                      submode)))
+         (when submode (mapcar #'cdr (mhtml--submode-crucial-captured-locals
+                                      submode)))
+       ,@body)))
+
+(defun mhtml--submode-lighter ()
+  "Mode-line lighter indicating the current submode."
+  (let ((submode (get-text-property (point) 'mhtml-submode)))
+    (if submode
+        (mhtml--submode-name submode)
+      "")))
+
+(defun mhtml--submode-fontify-one-region (submode beg end &optional loudly)
+  (if submode
+      (mhtml--with-locals submode
+        (save-restriction
+          (narrow-to-region beg end)
+          (font-lock-set-defaults)
+          (font-lock-default-fontify-region (point-min) (point-max) loudly)))
+    (font-lock-set-defaults)
+    (font-lock-default-fontify-region beg end loudly)))
+
+(defun mhtml--submode-fontify-region (beg end loudly)
+  (while (< beg end)
+    (let ((submode (get-text-property beg 'mhtml-submode))
+          (this-end (next-single-property-change beg 'mhtml-submode
+                                                 nil end)))
+      (mhtml--submode-fontify-one-region submode beg this-end loudly)
+      (setq beg this-end))))
+
+(defvar-local mhtml--last-submode nil
+  "Record the last visited submode, so the cursor-sensor function
+can function properly.")
+
+(defvar-local mhtml--stashed-crucial-variables nil
+  "Alist of stashed values of the crucial variables.")
+
+(defun mhtml--stash-crucial-variables ()
+  (setq mhtml--stashed-crucial-variables
+        (mapcar (lambda (sym)
+                  (cons sym (buffer-local-value sym (current-buffer))))
+                mhtml--crucial-variables)))
+
+(defun mhtml--map-in-crucial-variables (alist)
+  (dolist (item alist)
+    (set (car item) (cdr item))))
+
+(defun mhtml--pre-command ()
+  (let ((submode (get-text-property (point) 'mhtml-submode)))
+    (unless (eq submode mhtml--last-submode)
+      ;; If we're entering a submode, and the previous submode was
+      ;; nil, then stash the current values first.  This lets the user
+      ;; at least modify some values directly.  FIXME maybe always
+      ;; stash into the current mode?
+      (when (and submode (not mhtml--last-submode))
+        (mhtml--stash-crucial-variables))
+      (mhtml--map-in-crucial-variables
+       (if submode
+           (mhtml--submode-crucial-captured-locals submode)
+         mhtml--stashed-crucial-variables))
+      (setq mhtml--last-submode submode))))
+
+(defun mhtml--syntax-propertize-submode (submode end)
+  (save-excursion
+    (when (search-forward (mhtml--submode-end-tag submode) end t)
+      (setq end (match-beginning 0))))
+  (set-text-properties (point) end
+                       (list 'mhtml-submode submode
+                             'syntax-table (mhtml--submode-syntax-table submode)
+                             ;; We want local-map here so that we act
+                             ;; more like the sub-mode and don't
+                             ;; override minor mode maps.
+                             'local-map (mhtml--submode-keymap submode)))
+  (funcall (mhtml--submode-propertize submode) (point) end)
+  (goto-char end))
+
+(defun mhtml-syntax-propertize (start end)
+  (goto-char start)
+  (when (get-text-property (point) 'mhtml-submode)
+    (mhtml--syntax-propertize-submode (get-text-property (point) 'mhtml-submode)
+                                      end))
+  (funcall
+   (syntax-propertize-rules
+    ("<style.*?>"
+     (0 (ignore
+         (goto-char (match-end 0))
+         (mhtml--syntax-propertize-submode mhtml--css-submode end))))
+    ("<script.*?>"
+     (0 (ignore
+         (goto-char (match-end 0))
+         (mhtml--syntax-propertize-submode mhtml--js-submode end))))
+    sgml-syntax-propertize-rules)
+   ;; Make sure to handle the situation where
+   ;; mhtml--syntax-propertize-submode moved point.
+   (point) end))
+
+(defun mhtml-indent-line ()
+  "Indent the current line as HTML, JS, or CSS, according to its context."
+  (interactive)
+  (let ((submode (save-excursion
+                   (back-to-indentation)
+                   (get-text-property (point) 'mhtml-submode))))
+    (if submode
+        (save-restriction
+          (let* ((region-start (previous-single-property-change (point)
+                                                                'mhtml-submode))
+                 (base-indent (save-excursion
+                                (goto-char region-start)
+                                (sgml-calculate-indent))))
+            (narrow-to-region region-start (point-max))
+            (let ((prog-indentation-context (list base-indent
+                                                  (cons (point-min) nil)
+                                                  nil)))
+              (mhtml--with-locals submode
+                ;; indent-line-function was rebound by
+                ;; mhtml--with-locals.
+                (funcall indent-line-function)))))
+      ;; HTML.
+      (sgml-indent-line))))
+
+(defun mhtml--flyspell-check-word ()
+  (let ((submode (get-text-property (point) 'mhtml-submode)))
+    (if submode
+        (flyspell-generic-progmode-verify)
+      t)))
+
+;;;###autoload
+(define-derived-mode mhtml-mode html-mode
+  '((sgml-xml-mode "XHTML+" "HTML+") (:eval (mhtml--submode-lighter)))
+  "Major mode based on `html-mode', but works with embedded JS and CSS.
+
+Code inside a <script> element is indented using the rules from
+`js-mode'; and code inside a <style> element is indented using
+the rules from `css-mode'."
+  (cursor-sensor-mode)
+  (setq-local indent-line-function #'mhtml-indent-line)
+  (setq-local parse-sexp-lookup-properties t)
+  (setq-local syntax-propertize-function #'mhtml-syntax-propertize)
+  (setq-local font-lock-fontify-region-function
+              #'mhtml--submode-fontify-region)
+
+  ;; Attach this to both pre- and post- hooks just in case it ever
+  ;; changes a key binding that might be accessed from the menu bar.
+  (add-hook 'pre-command-hook #'mhtml--pre-command nil t)
+  (add-hook 'post-command-hook #'mhtml--pre-command nil t)
+
+  ;; Make any captured variables buffer-local.
+  (mhtml--mark-buffer-locals mhtml--css-submode)
+  (mhtml--mark-buffer-locals mhtml--js-submode)
+
+  (mhtml--mark-crucial-buffer-locals mhtml--css-submode)
+  (mhtml--mark-crucial-buffer-locals mhtml--js-submode)
+  (setq mhtml--crucial-variables (delete-dups mhtml--crucial-variables))
+
+  ;: Hack
+  (js--update-quick-match-re))
+
+(put 'mhtml-mode 'flyspell-mode-predicate #'mhtml--flyspell-check-word)
+
+(provide 'mhtml-mode)
+
+;;; mhtml-mode.el ends here
diff --git a/lisp/textmodes/sgml-mode.el b/lisp/textmodes/sgml-mode.el
index e148b06..8ad7cfb 100644
--- a/lisp/textmodes/sgml-mode.el
+++ b/lisp/textmodes/sgml-mode.el
@@ -341,19 +341,23 @@ sgml-font-lock-keywords-2
 (defvar sgml-font-lock-keywords sgml-font-lock-keywords-1
   "Rules for highlighting SGML code.  See also `sgml-tag-face-alist'.")
 
+(eval-and-compile
+  (defconst sgml-syntax-propertize-rules
+    (syntax-propertize-precompile-rules
+     ;; Use the `b' style of comments to avoid interference with the -- ... --
+     ;; comments recognized when `sgml-specials' includes ?-.
+     ;; FIXME: beware of <!--> blabla <!--> !!
+     ("\\(<\\)!--" (1 "< b"))
+     ("--[ \t\n]*\\(>\\)" (1 "> b"))
+     ;; Double quotes outside of tags should not introduce strings.
+     ;; Be careful to call `syntax-ppss' on a position before the one we're
+     ;; going to change, so as not to need to flush the data we just computed.
+     ("\"" (0 (if (prog1 (zerop (car (syntax-ppss (match-beginning 0))))
+                    (goto-char (match-end 0)))
+                  (string-to-syntax ".")))))))
+
 (defconst sgml-syntax-propertize-function
-  (syntax-propertize-rules
-   ;; Use the `b' style of comments to avoid interference with the -- ... --
-   ;; comments recognized when `sgml-specials' includes ?-.
-  ;; FIXME: beware of <!--> blabla <!--> !!
-   ("\\(<\\)!--" (1 "< b"))
-    ("--[ \t\n]*\\(>\\)" (1 "> b"))
-    ;; Double quotes outside of tags should not introduce strings.
-    ;; Be careful to call `syntax-ppss' on a position before the one we're
-    ;; going to change, so as not to need to flush the data we just computed.
-    ("\"" (0 (if (prog1 (zerop (car (syntax-ppss (match-beginning 0))))
-                   (goto-char (match-end 0)))
-           (string-to-syntax ".")))))
+  (syntax-propertize-rules sgml-syntax-propertize-rules)
   "Syntactic keywords for `sgml-mode'.")
 
 ;; internal
@@ -1284,13 +1288,24 @@ sgml-tag-text-p
       (let ((pps (parse-partial-sexp start end 2)))
 	(and (= (nth 0 pps) 0))))))
 
+(defun sgml--find-<>-backward (limit)
+  "Search backward for a '<' or '>' character.
+The character must have open or close syntax.
+Returns t if found, nil otherwise."
+  (catch 'found
+    (while (re-search-backward "[<>]" limit 'move)
+      ;; If this character has "open" or "close" syntax, then we've
+      ;; found the one we want.
+      (when (memq (syntax-class (syntax-after (point))) '(4 5))
+        (throw 'found t)))))
+
 (defun sgml-parse-tag-backward (&optional limit)
   "Parse an SGML tag backward, and return information about the tag.
 Assume that parsing starts from within a textual context.
 Leave point at the beginning of the tag."
   (catch 'found
     (let (tag-type tag-start tag-end name)
-      (or (re-search-backward "[<>]" limit 'move)
+      (or (sgml--find-<>-backward limit)
 	  (error "No tag found"))
       (when (eq (char-after) ?<)
 	;; Oops!! Looks like we were not in a textual context after all!.

Re: html, css, and js modes working together

[Dmitry Gutov]

On 13.02.2017 04:48, Tom Tromey wrote:

> Here's my current patch, now with pre- and post-command hooks and
> flyspell integration.  Font lock mostly works but has some oddities, and
> I still didn't fix the html comment/syntax bug that Clément found.

Thanks, the patch looks good on a high level.

I do see the oddities, and it seems the wonkiness is more related to 
setting up submode regions, because when the font-lock acts up, 
indentation can fail on that line as well.

Here's a broken example, for you reference. From the outset, the closing 
"style" tag and both "script" tags (though not their contents) are 
unfontified. By adding/deleting the closing ">" in the closing 
"</script>" tag, I can break the fontification of the remainder of the 
file, and indentation gets broken as well.

I'm not sure, but maybe the missing thing is the removal of the 
'mhtml-submode' text property near the beginning of mhtml-syntax-propertize.

<!DOCTYPE html>
<html>
   <head>
     <title>test</title>
     <style type="text/css">
       h1 {
         font-family: 'Spinnaker', sans-serif;
       }
     </style>
     <script>
       function() {
         return 25;
       }
     </script>
   </head>
   <body>
   </body>
</html>

Re: html, css, and js modes working together

[Tom Tromey]

>>>>> "Dmitry" == Dmitry Gutov <dgutov@yandex.ru> writes:

Dmitry> I do see the oddities, and it seems the wonkiness is more related to
Dmitry> setting up submode regions, because when the font-lock acts up,
Dmitry> indentation can fail on that line as well.

I spent some time debugging this and I fixed up the fontification here.
I also fixed the bug Clément found, where sub-mode fontification was
being applied even in HTML comments.  I've played with it quite a bit
and used it on real code here, and it's more robust than before.

Dmitry> Here's a broken example, for you reference.

Thanks.  This one works now.


At this point I think I'm nearly ready to start checking things in.
The current state is:

* I have a bunch of small patches to change various things to use
  derived-mode-p rather than eq.  This is needed to deal with having the
  new mode not be html-mode.  I don't really want to test most of these
  (like changes in viper or cedet, about which I know very little), so I
  thought I'd send them for review.  The changes are all very small.

* Most basic functionality works, like fontification, indentation,
  flyspell, and comment-*.

* Imenu and some other things don't work, but I think it's better to
  move forward with what we have than to fix every feature beforehand.
  I wonder if you agree.

* I need to write some tests.  I think this plus documentation is the
  last bit I need to do.

* One open question is whether the new mode should run prog-mode-hook.
  It is sort of both a prog mode and a text mode.

Tom

Re: html, css, and js modes working together

[Dmitry Gutov]

Hi Tom,

On 19.03.2017 19:30, Tom Tromey wrote:

> At this point I think I'm nearly ready to start checking things in.
> The current state is:
> 
> * I have a bunch of small patches to change various things to use
>    derived-mode-p rather than eq.  This is needed to deal with having the
>    new mode not be html-mode.  I don't really want to test most of these
>    (like changes in viper or cedet, about which I know very little), so I
>    thought I'd send them for review.  The changes are all very small.
> 
> * Most basic functionality works, like fontification, indentation,
>    flyspell, and comment-*.
> 
> * Imenu and some other things don't work, but I think it's better to
>    move forward with what we have than to fix every feature beforehand.
>    I wonder if you agree.
> 
> * I need to write some tests.  I think this plus documentation is the
>    last bit I need to do.

Sounds good!

> * One open question is whether the new mode should run prog-mode-hook.
>    It is sort of both a prog mode and a text mode.

IIRC Stefan opined that all programming-related modes should derive from 
prog-mode. On the other hand, it would be weird if this mode does, but 
html-mode does not.

Re: html, css, and js modes working together

[Tom Tromey]

>>>>> "Dmitry" == Dmitry Gutov <dgutov@yandex.ru> writes:

Tom> At this point I think I'm nearly ready to start checking things in.
...
Tom> * I need to write some tests.  I think this plus documentation is the
Tom> last bit I need to do.

Dmitry> Sounds good!

Ok, I've pushed my branch to feature/mthml-mode.
I would appreciate reviews.

The first few patches on the branch are just minor changes to use
derived-mode-p in various spots.  I found these spots by grepping for
"html-mode".

The next few patches are just small preparatory patches for the main
change.  I understood Stefan as already saying these are ok, but another
go-around won't hurt.

Finally there is the patch to introduce the new mode, and then the patch
to enable it by default.  I split everything up to make it a bit simpler
to review and discuss.

thanks,
Tom

Re: html, css, and js modes working together

[Stefan Monnier]

> Ok, I've pushed my branch to feature/mthml-mode.
> I would appreciate reviews.

LGTM (haven't really looked at mhtml-mode, tho, only looked at the
changes to other files).


        Stefan

Re: html, css, and js modes working together

[Toon Claes]

Tom Tromey <tom@tromey.com> writes:

> Ok, I've pushed my branch to feature/mthml-mode.
> I would appreciate reviews.

Pretty cool! Looks very good.

I've found one small issue:
Yasnippet doesn't update to the correct set of snippets when inside js
or css.

I haven't dug into it, so I don't know if this is a yasnippet issue,
or a mthml issue.


-- Toon

Re: html, css, and js modes working together

[Noam Postavsky]

On Fri, Mar 24, 2017 at 8:08 AM, Toon Claes <toon@iotcl.com> wrote:
>
> Yasnippet doesn't update to the correct set of snippets when inside js
> or css.

Yasnippet chooses snippet tables based on the value of `major-mode'.

Re: html, css, and js modes working together

[Tom Tromey]

>>>>> "Noam" == Noam Postavsky <npostavs@users.sourceforge.net> writes:

Noam> On Fri, Mar 24, 2017 at 8:08 AM, Toon Claes <toon@iotcl.com> wrote:
>> 
>> Yasnippet doesn't update to the correct set of snippets when inside js
>> or css.

Noam> Yasnippet chooses snippet tables based on the value of `major-mode'.

I wonder what would break if I had the sub-modes save and restore major-mode.

Tom

Re: html, css, and js modes working together

[Tom Tromey]

Tom> Ok, I've pushed my branch to feature/mthml-mode.
Tom> I would appreciate reviews.

I'm going to push this in shortly.

Please make sure to CC me on any problems you encounter.
I'll fix them as soon as I'm able.

Tom

Re: html, css, and js modes working together

[Leo Liu]

On 2017-04-05 16:01 -0600, Tom Tromey wrote:
> I'm going to push this in shortly.
>
> Please make sure to CC me on any problems you encounter.
> I'll fix them as soon as I'm able.

Does electric-pair-mode work with such buffers with multiple syntax
tables in Emacs 26? I am experiencing an annoying bug in Emacs 25.1
(https://debbugs.gnu.org/26327) where electric-pair-mode keeps using the
default html syntax table and inserting unwanted pairs.

Leo

Re: html, css, and js modes working together

[Dmitry Gutov]

On 06.04.2017 01:01, Tom Tromey wrote:
> Tom> Ok, I've pushed my branch to feature/mthml-mode.
> Tom> I would appreciate reviews.
> 
> I'm going to push this in shortly.

Thank you. Sorry for taking too long to take a look at them, but we've 
discussed most of the contents before.

There's not much I can really add, except the superficial: please 
capitalize the commit message summaries. But that ship has sailed for 
this set of patches.

Let's be on the lookout for regressions, then.

Re: html, css, and js modes working together

[Tom Tromey]

Tom> I implemented the buffer-local-variables stuff last night, so I
Tom> will try removing this code soon.

This cleaned things up nicely; and in particular only minimal changes
are needed to the smie, js, and css modes now.

I am still having a problem that I could use help with.  My mode is
using cursor-sensor-functions to detect when point enters or leaves a
sub-mode's region.  When this happens the mode resets a bunch of
buffer-local variables.  For example, comment-start is modified
depending on the current region.

I discovered, though, that this doesn't do the right thing if I show an
mhtml-mode buffer in multiple windows.  For example if I enter a <style>
element in one window, and a <script> element in another window, then
switching between windows doesn't call cursor-sensor-functions, leaving
comment-start (et al) with the wrong values.

It seems to me that switching windows should call
cursor-sensor-functions.  What do you think?

Alternatively, what are my options?

I considered not remapping variables like this, and instead rebinding
the various comment-* functions to wrappers that would let-bind the
variables.  The difficulty here is that it's difficult to be sure all
the functions are wrapped, especially as Emacs evolves.

I could maybe listen to buffer-list-update-hook and do the same
remapping there?  Is there a more specific hook that could be used?

Tom

Re: html, css, and js modes working together

[Stefan Monnier]

> It seems to me that switching windows should call
> cursor-sensor-functions.  What do you think?

That's not the intention behind cursor-sensor-functions: as the name
implies, this functionality is bound to the notion of "cursor".
When you switch window, cursor don't move.

> Alternatively, what are my options?

pre/post-command-hook?


        Stefan

Re: html, css, and js modes working together

[Tom Tromey]

>>>>> "Stefan" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> It seems to me that switching windows should call
>> cursor-sensor-functions.  What do you think?

Stefan> That's not the intention behind cursor-sensor-functions: as the name
Stefan> implies, this functionality is bound to the notion of "cursor".
Stefan> When you switch window, cursor don't move.

I find this pretty surprising.

I wanted to understand what the "cursor" was -- it's not an Emacs term I
knew -- so I looked, and the elisp manual says:

   As far as the user is concerned, point is where the cursor is, and
when the user switches to another buffer, the cursor jumps to the
position of point in that buffer.


So by that definition at least, it should be called.

Or, I suppose, "cursor" should be redefined.  Though what would that way
be?  My mental model was something like "point in the selected window".
What else would make sense?

Tom

Re: html, css, and js modes working together

[Stefan Monnier]

>    As far as the user is concerned, point is where the cursor is, and
> when the user switches to another buffer, the cursor jumps to the
> position of point in that buffer.

Maybe you've configured your Emacs to do that, but by default, every
window gets its own cursor, AFAIK.  The selected window's cursor looks
different than the others, but the others do exist.

In any case, by cursor I mostly mean window-point (modulo the fact
that the cursor only takes a meaning when the window gets rendered).


        Stefan

Re: html, css, and js modes working together

[Tom Tromey]

>>>>> "Stefan" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> As far as the user is concerned, point is where the cursor is, and
>> when the user switches to another buffer, the cursor jumps to the
>> position of point in that buffer.

Stefan> Maybe you've configured your Emacs to do that, but by default, every
Stefan> window gets its own cursor, AFAIK.  The selected window's cursor looks
Stefan> different than the others, but the others do exist.

Sorry, maybe it wasn't clear, but the quoted paragraph above comes from
the elisp manual.  (info "(elisp) Window Point")

Stefan> In any case, by cursor I mostly mean window-point (modulo the fact
Stefan> that the cursor only takes a meaning when the window gets rendered).

That's what that section of the manual is about too.

Tom

Re: html, css, and js modes working together

[Stefan Monnier]

> Sorry, maybe it wasn't clear, but the quoted paragraph above comes from
> the elisp manual.  (info "(elisp) Window Point")
[...]
> That's what that section of the manual is about too.

Hmm... sorry, I obviously spend too much time with too many windows and
tend to forget that the "usual" way to use Emacs is a single window, and
`switch-to-buffer` whereas for me it means "go over to the other window
that shows the other buffer".

But to my defense: you tricked me.  You started with:

>> It seems to me that switching windows should call
>> cursor-sensor-functions.  What do you think?

and then ended with:

>>    As far as the user is concerned, point is where the cursor is, and
>> when the user switches to another buffer, the cursor jumps to the
>> position of point in that buffer.

Notice how in the first case you switch windows and in the second you
switch buffers?  Very different cases (and yes, cursor-sensor-functions
should probably be called in some cases when switching buffer in
a window).


        Stefan

Re: html, css, and js modes working together

[Tom Tromey]

>>>>> "Stefan" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

Stefan> Notice how in the first case you switch windows and in the second you
Stefan> switch buffers?  Very different cases (and yes, cursor-sensor-functions
Stefan> should probably be called in some cases when switching buffer in
Stefan> a window).

Hah, actually, no, I didn't notice this.
Thanks for pointing that out, I agree that this doesn't support my case.

But at the same time, I still don't know what a cursor is; or really how
cursor-sensor-functions can be useful given this limitation.  I'll look
for some other approach though.  I had actually considered
post-command-hook, which you suggested, but I was under the impression
that this is very expensive and to be avoided.

Tom

Re: html, css, and js modes working together

[martin rudalics]

 >     As far as the user is concerned, point is where the cursor is, and
 > when the user switches to another buffer, the cursor jumps to the
 > position of point in that buffer.

Take this with a grain of salt.  The cursor is an attempt to visualize
the position in a buffer the user most likely wants to see marked as the
center of her current or future editing or viewing activity.  Sometimes
Emacs fails in that attempt.  I think we should rewrite that sentence.

martin

Re: html, css, and js modes working together

[Eli Zaretskii]

> Date: Sun, 12 Feb 2017 12:14:43 +0100
> From: martin rudalics <rudalics@gmx.at>
> Cc: emacs-devel@gnu.org
> 
>  >     As far as the user is concerned, point is where the cursor is, and
>  > when the user switches to another buffer, the cursor jumps to the
>  > position of point in that buffer.
> 
> Take this with a grain of salt.  The cursor is an attempt to visualize
> the position in a buffer the user most likely wants to see marked as the
> center of her current or future editing or viewing activity.  Sometimes
> Emacs fails in that attempt.

Indeed.  The "cursor" in that node means that rectangular block which
marks where point is; Emacs moves it to the coordinates that best fit
the location of point when point moves or the buffer displayed in a
window changes.  "The cursor jumps" wants to say that Emacs draws the
cursor at those coordinates (or thereabouts).

> I think we should rewrite that sentence.

Done.

> But at the same time, I still don't know what a cursor is; or really how
> cursor-sensor-functions can be useful given this limitation.

In this sense, the "cursor" part of cursor-sensor-functions is a
misnomer; it should have used "point".  However, given that this was a
replacement for point-entered and point-left properties, I can
understand why that name was chosen.

Re: html, css, and js modes working together

[Tom Tromey]

>>>>> "Eli" == Eli Zaretskii <eliz@gnu.org> writes:

>> But at the same time, I still don't know what a cursor is; or really how
>> cursor-sensor-functions can be useful given this limitation.

Eli> In this sense, the "cursor" part of cursor-sensor-functions is a
Eli> misnomer; it should have used "point".  However, given that this was a
Eli> replacement for point-entered and point-left properties, I can
Eli> understand why that name was chosen.

If that's the case, then it seems to me that cursor-sensor-functions
should be run in the scenario I described -- because in that scenario,
point does change.

Tom

Re: html, css, and js modes working together

[Stefan Monnier]

> If that's the case, then it seems to me that cursor-sensor-functions
> should be run in the scenario I described -- because in that scenario,
> point does change.

I think what you want is to run commands in a context that depends on
point.  In that case I think that pre-command-hook sounds like the
better option (and no, it's definitely not particularly expensive: it's
run less often than the cursor-sensor--detect used internally by
cursor-sensor-mode).

As to whether cursor-sensor-functions would be more or less useful if it
were defined to react on movement of the "buffer cursor" rather than the
"window cursor".  You might be right.  I truly don't know.  Many uses of
such "position sensitivity" should ideally be "per window", IMO, which
is why I implemented it this way, but at the same time, many users of
that feature then go on and change buffer state (often with not
much other choice) rather than window state.

E.g. table.el uses cursor-sensor-functions to detect when we're inside
a table and update the mode-line lighter accordingly.  But that
currently breaks when we have several windows showing the same buffer
because its lighter's state is define per-buffer rather than per-window.


        Stefan

Re: html, css, and js modes working together

[Tom Tromey]

>>>>> "Stefan" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

Stefan> I think what you want is to run commands in a context that depends on
Stefan> point.  In that case I think that pre-command-hook sounds like the
Stefan> better option (and no, it's definitely not particularly expensive: it's
Stefan> run less often than the cursor-sensor--detect used internally by
Stefan> cursor-sensor-mode).

Ok, thanks.  I will do that.

Stefan> As to whether cursor-sensor-functions would be more or less useful if it
Stefan> were defined to react on movement of the "buffer cursor" rather than the
Stefan> "window cursor".  You might be right.  I truly don't know.

It occurred to me just today that maybe my use of
cursor-sensor-functions would be incoherent when using multiple
terminals anyhow -- since which one would win?

Tom

Re: html, css, and js modes working together

[Eli Zaretskii]

> From: Tom Tromey <tom@tromey.com>
> Cc: martin rudalics <rudalics@gmx.at>,  tom@tromey.com,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Sun, 12 Feb 2017 09:32:56 -0700
> 
> >>>>> "Eli" == Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> But at the same time, I still don't know what a cursor is; or really how
> >> cursor-sensor-functions can be useful given this limitation.
> 
> Eli> In this sense, the "cursor" part of cursor-sensor-functions is a
> Eli> misnomer; it should have used "point".  However, given that this was a
> Eli> replacement for point-entered and point-left properties, I can
> Eli> understand why that name was chosen.
> 
> If that's the case, then it seems to me that cursor-sensor-functions
> should be run in the scenario I described -- because in that scenario,
> point does change.

Not sure what scenario you refer to, but if it's switching buffers in
a window, then no, point doesn't move in that scenario.

Re: html, css, and js modes working together

[Stefan Monnier]

> In this sense, the "cursor" part of cursor-sensor-functions is a
> misnomer; it should have used "point".  However, given that this was a
> replacement for point-entered and point-left properties, I can
> understand why that name was chosen.

Yes, I used the term "cursor" to insist on the fact that it relates to
the movement of point as seen by the user (i.e. via its rendition on
screen) as opposed to the previous behavior of
intangible/point-entered/point-left which also cared about temporary
point movements internal to the implementation of commands.



        Stefan

Please ack

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Could you please respond if you get this message?
I've been trying to reach you for weeks about another topic
and got no response.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.