bug#36358: Indentation of not matched braces in latex-mode

[Sebastian Urban <mrsebastianurban@gmail.com>]

So, I tried to write something that could work better:
1st method: noindent, unless indent; basically current approach
             reversed: indent, unless noindent;
2nd method: no text before AND after - indent, otherwise don't.

Although, I'm not sure, what to do with things like:
    Some paragraph...

    {\small Quisque ullamcorper placerat ipsum.  Cras nibh.  Morbi
      vel justo vitae lacus tincidunt ultrices.  Lorem ipsum dolor sit
      amet, consectetuer adipiscing elit.}

    Other paragraph...
if it's not inside of a paragraph, should it be indented?

BTW I did not test them thoroughly...

=======================================
1. Check if command should be indented.
=======================================

By default it won't indent commands, unless they are in
latex-indent-commands.  Commands like section or TAlign, will have to
be added to that var.  This could be done for the most common
commands, like mentioned "section", by default.  This, basically,
reverses current approach, and I think that it'll be easier (less
commands typed into var) to tell what should be indented, than what
shouldn't.  But I didn't use LaTeX extensively, so I may be wrong.

--8<---------------cut here---------------start------------->8---
--- old/tex-mode.el	2020-06-26 18:34:05.000000000 +0200
+++ new/tex-mode.el	2020-07-09 11:22:15.246268900 +0200
@@ -2802,8 +2802,8 @@
    :group 'tex-file
    :version "27.1")

-(defcustom latex-noindent-commands '("emph" "footnote")
-  "Commands for which `tex-indent-basic' should not be used."
+(defcustom latex-indent-commands '("section")
+  "Commands for which `tex-indent-basic' should be used."
    :type '(repeat string)
    :safe (lambda (x) (not (memq nil (mapcar #'stringp x))))
    :group 'tex-file
@@ -2915,17 +2915,20 @@
  	     ;; We're the first element after a hanging brace.
  	     (goto-char up-list-pos)
  	     (+ (if (if (eq (char-after) ?\{)
-                        (save-excursion
-                          (skip-chars-backward " \t")
-                          (let ((end (point)))
-                            (skip-chars-backward "a-zA-Z")
-                            (and (eq (char-before) ?\\)
-                                 (member (buffer-substring (point) end)
-                                         latex-noindent-commands))))
+                        (unless (save-excursion
+				  (skip-chars-backward " \t")
+				  (if (eq (char-before) ?\])
+				      (backward-list))
+				  (let ((end (point)))
+                                    (skip-chars-backward "a-zA-Z")
+                                    (and (eq (char-before) ?\\)
+					 (member (buffer-substring (point) end)
+						 latex-indent-commands))))
+                          t)
                        (and (looking-at "\\\\begin *{\\([^\n}]+\\)")
-			 (member (match-string 1)
-				 latex-noindent-environments)))
-		    0 tex-indent-basic)
+                           (member (match-string 1)
+                                   latex-noindent-environments)))
+                    0 tex-indent-basic)
  		indent (latex-find-indent 'virtual))))
  	  ;; We're now at the "beginning" of a line.
  	  ((not (and (not virtual) (eq (char-after) ?\\)))
--8<---------------cut here---------------end--------------->8---


=======================================================
2. Check if command/declaration(?) is within paragraph.
=======================================================

This checks if there is nothing but "beginning of the line" (ignoring
whitespaces) before command or "{" AND nothing but "end of the line"
(ignoring whitespaces) after "}" - if "yes" then indent, otherwise
don't.

The good thing about this, is that we don't need
latex-noindent-commands.  The bad thing is, that constructs like:
    prefix: \TAlign{
      foo\\
      bar
    }
won't work, because of the "prefix:" at BOL - it'll think it's
a paragraph.

If, "not-in-paragraph" declarations, like:
    {\delaration Integer tempus convallis augue.  Etiam facilisis.
    Nunc elementum fermentum wisi.  Integer tempus convallis augue.}
should NOT be indented (like in example above), then:
- uncomment: (skip-chars-backward " \t");
- delete: (eq (char-after) ?\{) AND "or" from first condition of "and".

--8<---------------cut here---------------start------------->8---
--- old/tex-mode.el	2020-06-26 18:34:05.000000000 +0200
+++ new/tex-mode.el	2020-07-09 12:42:14.067428300 +0200
@@ -2802,13 +2802,6 @@
    :group 'tex-file
    :version "27.1")

-(defcustom latex-noindent-commands '("emph" "footnote")
-  "Commands for which `tex-indent-basic' should not be used."
-  :type '(repeat string)
-  :safe (lambda (x) (not (memq nil (mapcar #'stringp x))))
-  :group 'tex-file
-  :version "27.1")
-
  (defvar tex-latex-indent-syntax-table
    (let ((st (make-syntax-table tex-mode-syntax-table)))
      (modify-syntax-entry ?$ "." st)
@@ -2915,17 +2908,22 @@
  	     ;; We're the first element after a hanging brace.
  	     (goto-char up-list-pos)
  	     (+ (if (if (eq (char-after) ?\{)
-                        (save-excursion
-                          (skip-chars-backward " \t")
-                          (let ((end (point)))
-                            (skip-chars-backward "a-zA-Z")
-                            (and (eq (char-before) ?\\)
-                                 (member (buffer-substring (point) end)
-                                         latex-noindent-commands))))
+			(not (and (save-excursion
+				    ;; (skip-chars-backward " \t")
+				    (cond ((eq (char-before) ?\]) (backward-list)))
+				    (skip-chars-backward "a-zA-Z")
+				    (cond ((or (eq (char-before) ?\\)
+					       (eq (char-after) ?\{))
+					   (skip-chars-backward " \t\\\\")
+					   (bolp))))
+				  (save-excursion
+				    (forward-list)
+				    (skip-chars-forward " \t")
+				    (eolp))))
                        (and (looking-at "\\\\begin *{\\([^\n}]+\\)")
-			 (member (match-string 1)
-				 latex-noindent-environments)))
-		    0 tex-indent-basic)
+                           (member (match-string 1)
+                                   latex-noindent-environments)))
+                    0 tex-indent-basic)
  		indent (latex-find-indent 'virtual))))
  	  ;; We're now at the "beginning" of a line.
  	  ((not (and (not virtual) (eq (char-after) ?\\)))
--8<---------------cut here---------------end--------------->8---


=============================
? Indentation in itemize env.
=============================

I'm not sure how it suppose to look, but when in example below
    \begin{itemize}#
      \item foo*
      \item bar
    \end{itemize}
the cursor is where "#" is, and I press <RET> - cursor moves to the
next line and is auto-indented by the value of tex-indent-basic.  But
when the cursor is at the position of "*", and I press <TAB>, the
indentation is gone (zero).  The question is, should \item(s) be
indented or not?  If they should, maybe code below will help.

--8<---------------cut here---------------start------------->8---
--- old/tex-mode.el	2020-06-26 18:34:05.000000000 +0200
+++ new/tex-mode.el	2020-07-09 11:04:11.583564000 +0200
@@ -2880,7 +2880,13 @@
         ;; Default (maybe an argument)
         (let ((pos (point))
  	     ;; Outdent \item if necessary.
-	     (indent (if (looking-at tex-indent-item-re) (- tex-indent-item) 0))
+	     (indent (if (looking-at tex-indent-item-re)
+			 (if (save-excursion
+			       (forward-line -1)
+			       (beginning-of-line)
+			       (looking-at "\\\\begin *{\\([^\n}]+\\)"))
+			     0 (- tex-indent-item))
+		       0))
  	     up-list-pos)
  	 ;; Find the previous point which determines our current indentation.
  	 (condition-case err
--8<---------------cut here---------------end--------------->8---


Anyway, I hope it'll help... somehow.

S. U.