Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

>     Replace prog-widen with consolidating widen calls

So, IIUC the idea is that the multi-major-mode framework will *have* to
use narrowing to indicate to the sub-modes the boundaries of the current
chunk, right?


        Stefan


> ---
>  doc/lispref/text.texi       | 45 +++++----------------------------------------
>  etc/NEWS                    |  8 ++++----
>  lisp/indent.el              | 12 +++++++++---
>  lisp/progmodes/prog-mode.el | 28 ++--------------------------
>  lisp/progmodes/python.el    | 34 ++++++++--------------------------
>  lisp/progmodes/ruby-mode.el |  1 -
>  lisp/vc/add-log.el          |  4 +++-
>  7 files changed, 31 insertions(+), 101 deletions(-)

> diff --git a/doc/lispref/text.texi b/doc/lispref/text.texi
> index 1e19f75..5a63e5b 100644
> --- a/doc/lispref/text.texi
> +++ b/doc/lispref/text.texi
> @@ -2407,42 +2407,17 @@ such multi-mode indentation.
>  This variable, when non-@code{nil}, holds the indentation context for
>  the sub-mode's indentation engine provided by the superior major mode.
>  The value should be a list of the form @code{(@var{first-column}
> -@w{(@var{start} . @var{end})} @code{prev-chunk})}.  The members of the
> -list have the following meaning:
> +@code{...rest})}.  The members of the list have the following meaning:
 
>  @table @var
>  @item first-column
>  The column to be used for top-level constructs.  This replaces the
>  default value of the top-level column used by the sub-mode, usually
>  zero.
> -@item start
> -@itemx end
> -The region of the code chunk to be indented by the sub-mode.  The
> -value of @var{end} can be @code{nil}, which stands for the value of
> -@code{point-max}.
> -@item prev-chunk
> -If this is non-@code{nil}, it should provide the sub-mode's
> -indentation engine with a virtual context of the code chunk.  Valid
> -values include:
> -
> -@itemize @minus
> -@item
> -A string whose contents is the text the sub-mode's indentation engine
> -should consider to precede the code chunk.  The sub-mode's indentation
> -engine can add text properties to that string, to be reused in
> -repeated calls with the same string, thus using it as a cache.  An
> -example where this is useful is code chunks that need to be indented
> -as function bodies, but lack the function's preamble---the string
> -could then include that missing preamble.
> -@item
> -A function.  It is expected to be called with the start position of
> -the current chunk, and should return a cons cell
> -@w{@code{(@var{prev-start} . @var{prev-end})}} that specifies the
> -region of the previous code chunk, or @code{nil} if there is no previous
> -chunk.  This is useful in literate-programming sources, where code is
> -split into chunks, and correct indentation needs to access previous
> -chunks.
> -@end itemize
> +@item rest
> +Reserved for future use.  The intent is to provide the sub-mode's
> +indentation engine with a virtual context of the code chunk.  Working
> +patches that use this in a significant fashion are welcome.
>  @end table
>  @end defvar
 
> @@ -2457,16 +2432,6 @@ function's value is the column number to use for top-level constructs.
>  When no superior mode is in effect, this function returns zero.
>  @end defun
 
> -@defun prog-widen
> -Call this function instead of @code{widen} to remove any restrictions
> -imposed by the mode's indentation engine and restore the restrictions
> -recorded in @code{prog-indentation-context}.  This prevents the
> -indentation engine of a sub-mode from inadvertently operating on text
> -outside of the chunk it was supposed to indent, and preserves the
> -restriction imposed by the superior mode.  When no superior mode is in
> -effect, this function just calls @code{widen}.
> -@end defun
> -
 
>  @node Region Indent
>  @subsection Indenting an Entire Region
> diff --git a/etc/NEWS b/etc/NEWS
> index 4ccf468..6e1561d 100644
> --- a/etc/NEWS
> +++ b/etc/NEWS
> @@ -1108,10 +1108,10 @@ programming languages in the same buffer, like literate programming
>  environments or ANTLR programs with embedded Python code.
 
>  A major mode can provide indentation context for a sub-mode through
> -the 'prog-indentation-context' variable.  To support this, modes that
> -provide indentation should use 'prog-widen' instead of 'widen' and
> -'prog-first-column' instead of a literal zero.  See the node
> -"(elisp) Mode-Specific Indent" in the ELisp manual for more details.
> +the 'prog-indentation-context' variable.  To support this, modes
> +should use 'prog-first-column' instead of a literal zero and never
> +call 'widen' in their indentation functions.  See the node "(elisp)
> +Mode-Specific Indent" in the ELisp manual for more details.
 
>  ** ERC
 
> diff --git a/lisp/indent.el b/lisp/indent.el
> index d5ba0bd..ccf0e99 100644
> --- a/lisp/indent.el
> +++ b/lisp/indent.el
> @@ -69,6 +69,8 @@ variable is `indent-relative' or `indent-relative-maybe', handle
>  it specially (since those functions are used for tabbing); in
>  that case, indent by aligning to the previous non-blank line."
>    (interactive)
> +  (save-restriction
> +    (widen)
>    (syntax-propertize (line-end-position))
>    (if (memq indent-line-function
>  	    '(indent-relative indent-relative-maybe))
> @@ -84,7 +86,7 @@ that case, indent by aligning to the previous non-blank line."
>  	    (indent-line-to column)
>  	  (save-excursion (indent-line-to column))))
>      ;; The normal case.
> -    (funcall indent-line-function)))
> +    (funcall indent-line-function))))
 
>  (defun indent--default-inside-comment ()
>    (unless (or (> (current-column) (current-indentation))
> @@ -144,7 +146,9 @@ prefix argument is ignored."
>            (indent--default-inside-comment)
>            (when (or (<= (current-column) (current-indentation))
>                      (not (eq tab-always-indent 'complete)))
> -            (funcall (default-value 'indent-line-function))))
> +            (save-restriction
> +              (widen)
> +              (funcall (default-value 'indent-line-function)))))
 
>        (cond
>         ;; If the text was already indented right, try completion.
> @@ -538,7 +542,9 @@ column to indent to; if it is nil, use one of the three methods above."
>  	  (forward-line 1)))))
>     ;; Use indent-region-function is available.
>     (indent-region-function
> -    (funcall indent-region-function start end))
> +    (save-restriction
> +      (widen)
> +      (funcall indent-region-function start end)))
>     ;; Else, use a default implementation that calls indent-line-function on
>     ;; each line.
>     (t (indent-region-line-by-line start end)))
> diff --git a/lisp/progmodes/prog-mode.el b/lisp/progmodes/prog-mode.el
> index f727e45..b36d242 100644
> --- a/lisp/progmodes/prog-mode.el
> +++ b/lisp/progmodes/prog-mode.el
> @@ -64,37 +64,13 @@ mode, it should bind this variable to non-nil around the call.
 
>  The non-nil value should be a list of the form:
 
> -   (FIRST-COLUMN (START . END) PREVIOUS-CHUNKS)
> +   (FIRST-COLUMN ...REST)
 
>  FIRST-COLUMN is the column the indentation engine of the sub-mode
>  should use for top-level language constructs inside the code
>  chunk (instead of 0).
 
> -START and END specify the region of the code chunk.  END can be
> -nil, which stands for the value of `point-max'.  The function
> -`prog-widen' uses this to restore restrictions imposed by the
> -sub-mode's indentation engine.
> -
> -PREVIOUS-CHUNKS, if non-nil, provides the indentation engine of
> -the sub-mode with the virtual context of the code chunk.  Valid
> -values are:
> -
> - - A string containing text which the indentation engine can
> -   consider as standing in front of the code chunk.  To cache the
> -   string's calculated syntactic information for repeated calls
> -   with the same string, the sub-mode can add text-properties to
> -   the string.
> -
> -   A typical use case is for grammars with code chunks which are
> -   to be indented like function bodies -- the string would contain
> -   the corresponding function preamble.
> -
> - - A function, to be called with the start position of the current
> -   chunk.  It should return either the region of the previous chunk
> -   as (PREV-START . PREV-END), or nil if there is no previous chunk.
> -
> -   A typical use case are literate programming sources -- the
> -   function would successively return the previous code chunks.")
> +REST is currently unused, but can be defined in future versions.")
 
>  (defun prog-indent-sexp (&optional defun)
>    "Indent the expression after point.
> diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
> index d4226e5..104889a 100644
> --- a/lisp/progmodes/python.el
> +++ b/lisp/progmodes/python.el
> @@ -287,10 +287,6 @@
>  ;;; 24.x Compat
>  \f
 
> -(unless (fboundp 'prog-widen)
> -  (defun prog-widen ()
> -    (widen)))
> -
>  (unless (fboundp 'prog-first-column)
>    (defun prog-first-column ()
>      0))
> @@ -785,7 +781,7 @@ work on `python-indent-calculate-indentation' instead."
>    (interactive)
>    (save-excursion
>      (save-restriction
> -      (prog-widen)
> +      (widen)
>        (goto-char (point-min))
>        (let ((block-end))
>          (while (and (not block-end)
> @@ -883,8 +879,6 @@ keyword
>  :at-dedenter-block-start
>   - Point is on a line starting a dedenter block.
>   - START is the position where the dedenter block starts."
> -  (save-restriction
> -    (prog-widen)
>      (let ((ppss (save-excursion
>                    (beginning-of-line)
>                    (syntax-ppss))))
> @@ -1022,7 +1016,7 @@ keyword
>                        (looking-at (python-rx block-ender)))
>                      :after-block-end)
>                     (t :after-line))
> -             (point)))))))))
> +             (point))))))))
 
>  (defun python-indent--calculate-indentation ()
>    "Internal implementation of `python-indent-calculate-indentation'.
> @@ -1030,8 +1024,6 @@ May return an integer for the maximum possible indentation at
>  current context or a list of integers.  The latter case is only
>  happening for :at-dedenter-block-start context since the
>  possibilities can be narrowed to specific indentation points."
> -  (save-restriction
> -    (prog-widen)
>      (save-excursion
>        (pcase (python-indent-context)
>          (`(:no-indent . ,_) (prog-first-column)) ; usually 0
> @@ -1081,7 +1073,7 @@ possibilities can be narrowed to specific indentation points."
>          (`(,(or :inside-paren-newline-start-from-block) . ,start)
>           ;; Add two indentation levels to make the suite stand out.
>           (goto-char start)
> -         (+ (current-indentation) (* python-indent-offset 2)))))))
> +         (+ (current-indentation) (* python-indent-offset 2))))))
 
>  (defun python-indent--calculate-levels (indentation)
>    "Calculate levels list given INDENTATION.
> @@ -4589,8 +4581,6 @@ To this:
>  Optional argument INCLUDE-TYPE indicates to include the type of the defun.
>  This function can be used as the value of `add-log-current-defun-function'
>  since it returns nil if point is not inside a defun."
> -  (save-restriction
> -    (prog-widen)
>      (save-excursion
>        (end-of-line 1)
>        (let ((names)
> @@ -4644,7 +4634,7 @@ since it returns nil if point is not inside a defun."
>              (and (= (current-indentation) 0) (throw 'exit t))))
>          (and names
>               (concat (and type (format "%s " type))
> -                     (mapconcat 'identity names ".")))))))
> +                     (mapconcat 'identity names "."))))))
 
>  (defun python-info-current-symbol (&optional replace-self)
>    "Return current symbol using dotty syntax.
> @@ -4788,12 +4778,10 @@ likely an invalid python file."
>    "Message the first line of the block the current statement closes."
>    (let ((point (python-info-dedenter-opening-block-position)))
>      (when point
> -      (save-restriction
> -        (prog-widen)
>          (message "Closes %s" (save-excursion
>                                 (goto-char point)
>                                 (buffer-substring
> -                                (point) (line-end-position))))))))
> +                                (point) (line-end-position)))))))
 
>  (defun python-info-dedenter-statement-p ()
>    "Return point if current statement is a dedenter.
> @@ -4809,8 +4797,6 @@ statement."
>    "Return non-nil if current line ends with backslash.
>  With optional argument LINE-NUMBER, check that line instead."
>    (save-excursion
> -    (save-restriction
> -      (prog-widen)
>        (when line-number
>          (python-util-goto-line line-number))
>        (while (and (not (eobp))
> @@ -4819,14 +4805,12 @@ With optional argument LINE-NUMBER, check that line instead."
>                    (not (equal (char-before (point)) ?\\)))
>          (forward-line 1))
>        (when (equal (char-before) ?\\)
> -        (point-marker)))))
> +        (point-marker))))
 
>  (defun python-info-beginning-of-backslash (&optional line-number)
>    "Return the point where the backslashed line start.
>  Optional argument LINE-NUMBER forces the line number to check against."
>    (save-excursion
> -    (save-restriction
> -      (prog-widen)
>        (when line-number
>          (python-util-goto-line line-number))
>        (when (python-info-line-ends-backslash-p)
> @@ -4835,15 +4819,13 @@ Optional argument LINE-NUMBER forces the line number to check against."
>                   (python-syntax-context 'paren))
>            (forward-line -1))
>          (back-to-indentation)
> -        (point-marker)))))
> +        (point-marker))))
 
>  (defun python-info-continuation-line-p ()
>    "Check if current line is continuation of another.
>  When current line is continuation of another return the point
>  where the continued line ends."
>    (save-excursion
> -    (save-restriction
> -      (prog-widen)
>        (let* ((context-type (progn
>                               (back-to-indentation)
>                               (python-syntax-context-type)))
> @@ -4869,7 +4851,7 @@ where the continued line ends."
>                 (python-util-forward-comment -1)
>                 (when (and (equal (1- line-start) (line-number-at-pos))
>                            (python-info-line-ends-backslash-p))
> -                 (point-marker))))))))
> +                 (point-marker)))))))
 
>  (defun python-info-block-continuation-line-p ()
>    "Return non-nil if current line is a continuation of a block."
> diff --git a/lisp/progmodes/ruby-mode.el b/lisp/progmodes/ruby-mode.el
> index dc1b0f8..7c10114 100644
> --- a/lisp/progmodes/ruby-mode.el
> +++ b/lisp/progmodes/ruby-mode.el
> @@ -1364,7 +1364,6 @@ delimiter."
>                                               "\\)\\>")))
>                      (eq (ruby-deep-indent-paren-p t) 'space)
>                      (not (bobp)))
> -                   (widen)
>                     (goto-char (or begin parse-start))
>                     (skip-syntax-forward " ")
>                     (current-column))
> diff --git a/lisp/vc/add-log.el b/lisp/vc/add-log.el
> index 392147b..9da8abc 100644
> --- a/lisp/vc/add-log.el
> +++ b/lisp/vc/add-log.el
> @@ -1133,6 +1133,8 @@ identifiers followed by `:' or `='.  See variables
>  Has a preference of looking backwards."
>    (condition-case nil
>        (save-excursion
> +        (save-restriction
> +          (widen)
>  	(if add-log-current-defun-function
>  	    (funcall add-log-current-defun-function)
>  	  ;; If all else fails, try heuristics
> @@ -1147,7 +1149,7 @@ Has a preference of looking backwards."
>  	      (when (string-match "\\([^ \t\n\r\f].*[^ \t\n\r\f]\\)"
>  				  result)
>  		(setq result (match-string-no-properties 1 result)))
> -	      result))))
> +	      result)))))
>      (error nil)))
 
>  (defvar change-log-get-method-definition-md)

> _______________________________________________
> Emacs-diffs mailing list
> Emacs-diffs@gnu.org
> https://lists.gnu.org/mailman/listinfo/emacs-diffs

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 11/30/17 1:53 AM, Stefan Monnier wrote:
>>      Replace prog-widen with consolidating widen calls
> 
> So, IIUC the idea is that the multi-major-mode framework will *have* to
> use narrowing to indicate to the sub-modes the boundaries of the current
> chunk, right?

Yes. Just like it would *have* to bind prog-indentation-context to a 
complex-ish value in the previous proposal.

I can't imagine a case where it would be inconvenient to do so.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 11/30/17 8:59 AM, Dmitry Gutov wrote:
> On 11/30/17 1:53 AM, Stefan Monnier wrote:
>>>      Replace prog-widen with consolidating widen calls
>>
>> So, IIUC the idea is that the multi-major-mode framework will *have* to
>> use narrowing to indicate to the sub-modes the boundaries of the current
>> chunk, right?
> 
> Yes. Just like it would *have* to bind prog-indentation-context to a 
> complex-ish value in the previous proposal.

And while the difference isn't night and day, allow me to elaborate why 
I think this is a move in a good direction:

- Major mode authors have been forgetting to widen in all the cases they 
actually have to. Or they would do that in indent-line-function, but 
forget in beginning-of-defun-function, etc. Now, they are *mandated* to 
obey the current narrowing (deleting code is easy, and some won't have 
delete anything at all). The indentation code might fail to work inside 
narrowing (like all of CC Mode, probably), but that's an orthogonal issue.

- Multi-mode packages have been using narrowing for this purpose for 
years, so they won't have to change much.

And, it's kind of funny, mhtml-indent-line which we've reviewed and 
added recently, uses narrow-to-region too. :-) While disregarding the 
second element of prog-indentation-context (which it set to the value 
"widen all you want"). Which works well enough because SMIE doesn't 
widen, and I've removed the `widen' call from js-mode indentation code 
3.5 years ago.

I'm adding Tom and Vitalie to Cc. Maybe they have something to add.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Tom Tromey]

>>>>> "Dmitry" == Dmitry Gutov <dgutov@yandex.ru> writes:

Dmitry> And, it's kind of funny, mhtml-indent-line which we've reviewed and
Dmitry> added recently, uses narrow-to-region too. :-) While disregarding the
Dmitry> second element of prog-indentation-context (which it set to the value
Dmitry> "widen all you want"). Which works well enough because SMIE doesn't
Dmitry> widen, and I've removed the `widen' call from js-mode indentation code
Dmitry> 3.5 years ago.

Yes, IIRC mhtml-mode narrows so that a sub-mode won't try to examine
parts of the buffer that have a different syntax.  Whether you consider
this ugly or not is perhaps related to your stance on the whole widening
issue.  My view, though, is that there is a problem here intrinsic to
multi-modes: something has to tell the sub-modes about the relevant
boundaries in some way; so if the narrowing/widening stuff is changed,
something new will have to be invented here.

Tom

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 11/30/17 3:31 PM, Tom Tromey wrote:
> Yes, IIRC mhtml-mode narrows so that a sub-mode won't try to examine
> parts of the buffer that have a different syntax.  Whether you consider
> this ugly or not is perhaps related to your stance on the whole widening
> issue.  My view, though, is that there is a problem here intrinsic to
> multi-modes: something has to tell the sub-modes about the relevant
> boundaries in some way;

Err, no. Ugly or not is beside the point (I personally prefer this 
approach, please also take a look at the branch referenced in this 
thread subject).

Thing is, if you're using prog-indentation-context as it's now 
implemented, you're "supposed to" indicate the narrowing bounds via 
setting the second element of that list, instead of calling 
narrow-to-region directly. Of course, that also requires support from 
the major modes in question.

 > so if the narrowing/widening stuff is changed,
 > something new will have to be invented here.

See above. The "something new" has a few rough edges, so this discussion 
is about revisiting those choices.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Alan Mackenzie]

Hello, Dmitry.

On Thu, Nov 30, 2017 at 10:58:47 +0000, Dmitry Gutov wrote:
> On 11/30/17 8:59 AM, Dmitry Gutov wrote:
> > On 11/30/17 1:53 AM, Stefan Monnier wrote:
> >>>      Replace prog-widen with consolidating widen calls

> >> So, IIUC the idea is that the multi-major-mode framework will *have* to
> >> use narrowing to indicate to the sub-modes the boundaries of the current
> >> chunk, right?

Using narrowing for marking the bounds of a sub-mode is a bad thing,
since it is likely to cause contention with other uses of narrowing.
There are better ways.

> > Yes. Just like it would *have* to bind prog-indentation-context to a 
> > complex-ish value in the previous proposal.

> And while the difference isn't night and day, allow me to elaborate why 
> I think this is a move in a good direction:

> - Major mode authors have been forgetting to widen in all the cases they 
> actually have to. Or they would do that in indent-line-function, but 
> forget in beginning-of-defun-function, etc. Now, they are *mandated* to 
> obey the current narrowing

It's not clear what is meant here, but mandating maintainers of major
modes to use narrowing in a particular way is at best controversial, and
probably will render many major modes non-functional.

> (deleting code is easy, and some won't have delete anything at all).
> The indentation code might fail to work inside narrowing (like all of
> CC Mode, probably), but that's an orthogonal issue.

Narrowing belongs to users and major modes.  I am lacking the context of
your post, which appears to start deep down in some mail thread which I
don't have, so I can't follow the details of your proposal.  But I would
appreciate you confirming that these proposals won't place any
constraints whatsoever on how users and major modes can use narrowing.

You can also understand me being a bit concerned at the reference to CC
Mode.  ;-)

> - Multi-mode packages have been using narrowing for this purpose for 
> years, so they won't have to change much.

They have used narrowing because that is the only tool they have had.

> And, it's kind of funny, mhtml-indent-line which we've reviewed and 
> added recently, uses narrow-to-region too. :-) While disregarding the 
> second element of prog-indentation-context (which it set to the value 
> "widen all you want"). Which works well enough because SMIE doesn't 
> widen, and I've removed the `widen' call from js-mode indentation code 
> 3.5 years ago.

> I'm adding Tom and Vitalie to Cc. Maybe they have something to add.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

Hi Alan,

On 11/30/17 9:46 PM, Alan Mackenzie wrote:

> Using narrowing for marking the bounds of a sub-mode is a bad thing,
> since it is likely to cause contention with other uses of narrowing.

Please give an example.

> It's not clear what is meant here, 

To rephrase, don't widen in indent-line-function or 
beginning-of-defun-function.

> but mandating maintainers of major
> modes to use narrowing in a particular way is at best controversial, and
> probably will render many major modes non-functional.

I don't see a reason why. Even more, it should be compatible with all 
uses of narrowing that I know about.

> Narrowing belongs to users and major modes. 

It can serve all.

> I am lacking the context of
> your post, which appears to start deep down in some mail thread which I
> don't have, so I can't follow the details of your proposal.  But I would
> appreciate you confirming that these proposals won't place any
> constraints whatsoever on how users and major modes can use narrowing.

Please see 
http://lists.gnu.org/archive/html/emacs-devel/2017-11/msg00691.html and 
http://lists.gnu.org/archive/html/emacs-devel/2016-12/msg00796.html.

> You can also understand me being a bit concerned at the reference to CC
> Mode.  ;-)

You shouldn't be: CC Mode can just ignore this new rule, as long as it's 
too hard to support embedding it in multi-mode buffers.

>> - Multi-mode packages have been using narrowing for this purpose for
>> years, so they won't have to change much.
> 
> They have used narrowing because that is the only tool they have had.

The only other tool we have is also narrowing, semantically.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Alan Mackenzie]

Hello, Dmitry.

Just to be clear, in the following, by "narrowing" I mean all operations
of narrowing and widening regarded generically.

On Thu, Nov 30, 2017 at 23:09:00 +0000, Dmitry Gutov wrote:
> Hi Alan,

> On 11/30/17 9:46 PM, Alan Mackenzie wrote:

> > Using narrowing for marking the bounds of a sub-mode is a bad thing,
> > since it is likely to cause contention with other uses of narrowing.

> Please give an example.

A low level function which as an essential part of its functionality
(for example, to make sure point-min isn't within a comment or string)
widens.

> > It's not clear what is meant here, 

> To rephrase, don't widen in indent-line-function or 
> beginning-of-defun-function.

This is an intolerable restriction.  The low level function mentioned
above cannot, should not, must not know whether it's being called
(indirectly) from indent-line-function or b-o-d-function.  It will have
to widen in all cases or none.  Therefore there will be failures whilst
being called either from one of the two noted functions or from
elsewhere.

> > but mandating maintainers of major
> > modes to use narrowing in a particular way is at best controversial, and
> > probably will render many major modes non-functional.

> I don't see a reason why. Even more, it should be compatible with all 
> uses of narrowing that I know about.

See above.

> > Narrowing belongs to users and major modes. 

> It can serve all.

Indeed it can, and it must.  A super-mode thus may not "reserve"
narrowing for its own purposes to the exclusion of other uses.

[ .... ]

> > You can also understand me being a bit concerned at the reference to CC
> > Mode.  ;-)

> You shouldn't be: CC Mode can just ignore this new rule, as long as it's 
> too hard to support embedding it in multi-mode buffers.

The multi-mode mechanism should be designed to be usable with any major
mode.  There's nothing particularly hard about supporting CC Mode in a
well designed multi-mode scheme.

> >> - Multi-mode packages have been using narrowing for this purpose for
> >> years, so they won't have to change much.

> > They have used narrowing because that is the only tool they have had.

> The only other tool we have is also narrowing, semantically.

We need better tools.  I have already proposed and offered to implement
such tools.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

>> To rephrase, don't widen in indent-line-function or
>> beginning-of-defun-function.
> This is an intolerable restriction.  The low level function mentioned
> above cannot, should not, must not know whether it's being called
> (indirectly) from indent-line-function or b-o-d-function.

It's a change.  So, to make it work right, some code will need to adapt.
As mentioned by Dmitry, this shouldn't introduce breakage in general:
the code will only need to adapt *if* it wants to work with
multi-major-mode solutions.  For the normal use case, you can still
widen/narrow to your heart's content and nobody will be the wiser.

There's nothing special here: any multi-major-mode solution will need to
define some convention/protocol that major need to follow in order to
work well.

Now if you worry that (after the code is adapted) the result is code
that is too complex/brittle, time will tell, but I get the impression
that it won't be too bad, likely no worse than any other solution.

The way your above situation would be handled is to say "the low level
function mentioned above assumes that the buffer has already been
properly widened".  This way, it doesn't need to know whether it's being
called (indirectly) from indent-line-function or b-o-d-function.

But some of its callers may need to be changed to do the widening, and
indeed, in order to know how far to widen, they'll need the help from
the multi-major-mode package.

So maybe we need something like:

    (defvar prog-widen-function #'widen)
    (defun prog-widen () (funcall prog-widen-function))

[ And maybe for the same reason, we need a function (rather than
  a variable) that returns the `first-column`, so we can get the info
  anywhere/anytime rather than only during indentation.  ]


        Stefan

RE: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Drew Adams]

> > > Narrowing belongs to users and major modes.
> > It can serve all.
> 
> Indeed it can, and it must.  A super-mode thus may not "reserve"
> narrowing for its own purposes to the exclusion of other uses.

+1

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

> Indeed it can, and it must.  A super-mode thus may not "reserve"
> narrowing for its own purposes to the exclusion of other uses.

99% of the multi-major-modes out there use narrowing for that purpose,
and in practice it doesn't restrict other uses of narrowing.

I've opposed the use of narrowing for that in the past, but the evidence
against that opposition is overwhelming.  Let's do that and move on.

It's not perfect, but we've already looked much too long for a less
imperfect solution.


        Stefan

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Alan Mackenzie]

Hello, Stefan.

On Fri, Dec 01, 2017 at 11:26:08 -0500, Stefan Monnier wrote:
> >> To rephrase, don't widen in indent-line-function or
> >> beginning-of-defun-function.
> > This is an intolerable restriction.  The low level function mentioned
> > above cannot, should not, must not know whether it's being called
> > (indirectly) from indent-line-function or b-o-d-function.

> It's a change.  So, to make it work right, some code will need to adapt.
> As mentioned by Dmitry, this shouldn't introduce breakage in general:

That has yet to be shown.

> the code will only need to adapt *if* it wants to work with
> multi-major-mode solutions.  For the normal use case, you can still
> widen/narrow to your heart's content and nobody will be the wiser.

If we're going to have a multi-major-mode solution (and I think we
should), let's at least get it right.  I.e., it should work with _any_
major mode without grotesque restrictions on that mode.

I would very much like to have AWK Mode working inside a
shell-script-mode buffer.  Why not?  The MMM solution which appears to
be being proposed won't achieve this.

I would very much like the option of handling C buffers as MMM buffers,
where macros would be handled by a (slightly) different mode from normal
code.  I'm not sure whether this is a good or practicable idea, but it
would be nice to have the option.

> There's nothing special here: any multi-major-mode solution will need to
> define some convention/protocol that major need to follow in order to
> work well.

No.  Our MMM solution should work without imposing restrictions on the
major modes.  All the awkwardness should be dealt with by the MMM.  This
is possible.

> Now if you worry that (after the code is adapted) the result is code
> that is too complex/brittle, time will tell, but I get the impression
> that it won't be too bad, likely no worse than any other solution.

As I keep saying, we should adapt a solution that doesn't place manacles
on hackers.

> The way your above situation would be handled is to say "the low level
> function mentioned above assumes that the buffer has already been
> properly widened".  This way, it doesn't need to know whether it's being
> called (indirectly) from indent-line-function or b-o-d-function.

Then it ceases to be a general low level function and becomes one that
can only be called in certain circumstances.  This effectively transmits
high level state to the low levels, and isn't good software engineering.

> But some of its callers may need to be changed to do the widening, and
> indeed, in order to know how far to widen, they'll need the help from
> the multi-major-mode package.

This is intolerable.  It will lead to twisted code.

Why should a major mode need to know "how far to widen"?  This is only
due to the proposal to misuse narrowing to control regions with
different syntax-tables, I think.  If these regions are controlled
without using narrowing, this and many other problems cease to exist.

> So maybe we need something like:

>     (defvar prog-widen-function #'widen)
>     (defun prog-widen () (funcall prog-widen-function))

No, we need

    (widen)

like we've had for many decades.

> [ And maybe for the same reason, we need a function (rather than
>   a variable) that returns the `first-column`, so we can get the info
>   anywhere/anytime rather than only during indentation.  ]


>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/1/17 4:26 PM, Stefan Monnier wrote:
> The way your above situation would be handled is to say "the low level
> function mentioned above assumes that the buffer has already been
> properly widened".  This way, it doesn't need to know whether it's being
> called (indirectly) from indent-line-function or b-o-d-function.

That one way, I'd call it "clean". But the change might take quite a bit 
of effort, depending on the size of the code base.

> But some of its callers may need to be changed to do the widening, and
> indeed, in order to know how far to widen, they'll need the help from
> the multi-major-mode package.

That would be the "dirty" way.

> So maybe we need something like:
> 
>      (defvar prog-widen-function #'widen)
>      (defun prog-widen () (funcall prog-widen-function))

I don't know if we need that in the common API, but individual packages 
can define such shims if they find that convenient.

E.g.

(defvar cc-mode-inside-indent-line nil)

(defun cc-mode-maybe-widen ()
   ;; You can also throw in an Emacs version check here,
   ;; for good measure.
   (unless cc-mode-inside-indent-line (widen)))

And then the "low-level primitives" can call cc-mode-maybe-widen.

> [ And maybe for the same reason, we need a function (rather than
>    a variable) that returns the `first-column`, so we can get the info
>    anywhere/anytime rather than only during indentation.  ]

Please elaborate.

How does using a function instead of a variable change the duration when 
prog-first-column is set?

And when else will we need this value?

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/1/17 3:49 PM, Alan Mackenzie wrote:

>> Please give an example.
> 
> A low level function which as an essential part of its functionality
> (for example, to make sure point-min isn't within a comment or string)
> widens.

See Stefan and I's responses. Including an example of a shim that 
wouldn't require you to change your code too much.

But first and foremost, CC Mode probably doesn't need to worry about 
complying with this guideline (yet). In my experience, it doesn't work 
well in multi-mode context for reasons other than it calling widen. So 
worrying about this change in protocol might be premature.

>> To rephrase, don't widen in indent-line-function or
>> beginning-of-defun-function.
> 
> This is an intolerable restriction.  The low level function mentioned
> above cannot, should not, must not know whether it's being called
> (indirectly) from indent-line-function or b-o-d-function.  It will have
> to widen in all cases or none.  Therefore there will be failures whilst
> being called either from one of the two noted functions or from
> elsewhere.

See that shim in the other email.

> Indeed it can, and it must.  A super-mode thus may not "reserve"
> narrowing for its own purposes to the exclusion of other uses.

Right.

> The multi-mode mechanism should be designed to be usable with any major
> mode.  There's nothing particularly hard about supporting CC Mode in a
> well designed multi-mode scheme.

It's harder than supporting Ruby, CSS and JS modes, that I can tell you 
for sure. For one thing, the extra caches CC Mode uses have given me 
multiple "argument out of bounds" errors when used with narrowing.

So these are my takeaways:

1) The "low-level primitives" in CC Mode do not widen consistently. Some 
of them miss that responsibility (or else I wouldn't get those kind of 
errors). That tells me that moving (widen) call to some higher level is 
generally a good idea, but you can disagree.

2) The other errors should be solvable, but they require a separate 
investigation, which nobody has found the time and energy to do so far, 
over many years. So maybe it's not so necessary.

3) The solution to the other problems is most likely orthogonal to the 
current discussion. Allowing multi-mode packages to use narrowing 
certainly shouldn't hurt.

> We need better tools.  I have already proposed and offered to implement
> such tools.

It was much more complex, nowhere near even a proof of concept, and 
doesn't seem to move anywhere, so far.

The change we're discussing is much smaller and pretty much implemented.

And again, there's no reason why they couldn't coexist.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

>> (defvar prog-widen-function #'widen)
>> (defun prog-widen () (funcall prog-widen-function))
> I don't know if we need that in the common API, but individual packages can
> define such shims if they find that convenient.

If we don't add such a thing in the common API, how could individual
packages know the boundary of the current chunk without adding
MMM-specific hacks?

> (defvar cc-mode-inside-indent-line nil)
> (defun cc-mode-maybe-widen ()
>   ;; You can also throw in an Emacs version check here,
>   ;; for good measure.
>   (unless cc-mode-inside-indent-line (widen)))
> And then the "low-level primitives" can call cc-mode-maybe-widen.

But just `widen` will do thew wrong thing (it'll widen too much) when
we're inside an MMM chunk.

>> [ And maybe for the same reason, we need a function (rather than
>> a variable) that returns the `first-column`, so we can get the info
>> anywhere/anytime rather than only during indentation.  ]
> Please elaborate.
> How does using a function instead of a variable change the duration when
> prog-first-column is set?

Having it as a function means you don't have to compute the value of
prog-first-column every time to "enter" a chunk, but instead we only
compute it if/when the submode decides it needs to know what is the
value of prog-first-column.

> And when else will we need this value?

Here are some examples I can think of:
- auto-fill-mode may like to compute the hypothetical indentation that
  would result from inserting a newline somewhere (and do that before
  touching the buffer).
- some package may like to highlight lines which aren't currently
  indented right (so, it won't call indent-according-to-mode, but
  it will need to compute the "desired" indentation).
I'm sure there can be other cases.


        Stefan

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/1/17 7:51 PM, Stefan Monnier wrote:
>>> (defvar prog-widen-function #'widen)
>>> (defun prog-widen () (funcall prog-widen-function))
>> I don't know if we need that in the common API, but individual packages can
>> define such shims if they find that convenient.
> 
> If we don't add such a thing in the common API, how could individual
> packages know the boundary of the current chunk without adding
> MMM-specific hacks?

The idea for multi-mode packages has always been to make do without 
making the major modes actually aware of hunk boundaries.

And the snippet of code won't make them aware either, since the chunk 
boundaries are conveyed via narrowing only in particular dynamic 
contexts, such as when indent-line-functions is called.

Adding other means too might make sense, but that's expanding the scope 
of the current proposal. I'd rather make it small and try to get it into 
Emacs 26.

>> (defvar cc-mode-inside-indent-line nil)
>> (defun cc-mode-maybe-widen ()
>>    ;; You can also throw in an Emacs version check here,
>>    ;; for good measure.
>>    (unless cc-mode-inside-indent-line (widen)))
>> And then the "low-level primitives" can call cc-mode-maybe-widen.
> 
> But just `widen` will do thew wrong thing (it'll widen too much) when
> we're inside an MMM chunk.

Yes, so CC Mode primitives all should call cc-mode-maybe-widen. Having 
them call prog-widen vs widen is very much the same difference.

> Having it as a function means you don't have to compute the value of
> prog-first-column every time to "enter" a chunk, but instead we only
> compute it if/when the submode decides it needs to know what is the
> value of prog-first-column.

You probably mean adding a variable prog-first-column-function, then?

Or will we expect multi-mode packages to simply advise the 
prog-first-column function? I'm fine with that.

>> And when else will we need this value?
> 
> Here are some examples I can think of:
> - auto-fill-mode may like to compute the hypothetical indentation that
>    would result from inserting a newline somewhere (and do that before
>    touching the buffer).

Won't it need to call indent-line-function to find out how much the next 
line should be indented?

> - some package may like to highlight lines which aren't currently
>    indented right (so, it won't call indent-according-to-mode, but
>    it will need to compute the "desired" indentation).

This example will *definitely* need to call indent-line-function.

And both of them should be solved by exchanging indent-line-function for 
(non-mutating) line-intentation-function. But that's a change for 
another day.

> I'm sure there can be other cases.

I'm sure there can be. A complete proposal to let the modes know chunk 
boundaries, etc, has yet to be finalized, however. And just having 
font-lock and indentation work reliably in multi-mode contexts will be a 
significant win.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Vitalie Spinu]

Please correct me if I am wrong but the current patch requires all modes to use
`prog-widen` and reserve `widen` to the interactive use, right? If so, this is
the opposite of the older idea of adding `hard-limits` for widening which only
multi-modes should use.

I have proposed a patch for `set-widen-limits` 1.5 years ago which still lives
in scratch/hard-narrow branch. It turned to be a pretty straightforward feat and
worked without requiring any changes of the excising elisp code.

The lengthy discussion started here:

  https://lists.gnu.org/archive/html/emacs-devel/2016-04/msg00629.html

Somewhere along the way we decided in favor of adding an extra `hard` argument
to `narrow` instead of having a new `set-widen-limits` function. That turned to
be impossible as it required backward-incompatible changes to the byte-code. I
run out of steam by then and the whole idea died :(

If adding a new function is back on the table then I think `set-widen-limits` is
a better approach than `prog-widen`. It doesn't impose new restrictions on the
existing code and it avoids introducing new types of narrowing or widening.

Unless something changed in the meanwhile, the changes to the C code to make
that work were minimal. If there is interest I can get it another shot.

  Vitalie

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/1/17 9:27 PM, Vitalie Spinu wrote:
> 
> Please correct me if I am wrong but the current patch requires all modes to use
> `prog-widen` and reserve `widen` to the interactive use, right? If so, this is
> the opposite of the older idea of adding `hard-limits` for widening which only
> multi-modes should use.

Quite the opposite: the patch under discussion, in scratch/widen-less, 
removes prog-widen altogether.

> I have proposed a patch for `set-widen-limits` 1.5 years ago which still lives
> in scratch/hard-narrow branch. It turned to be a pretty straightforward feat and
> worked without requiring any changes of the excising elisp code.
> 
> The lengthy discussion started here:
> 
>    https://lists.gnu.org/archive/html/emacs-devel/2016-04/msg00629.html
> 
> Somewhere along the way we decided in favor of adding an extra `hard` argument
> to `narrow` instead of having a new `set-widen-limits` function. That turned to
> be impossible as it required backward-incompatible changes to the byte-code. I
> run out of steam by then and the whole idea died :(

If that was the only difficulty, having set-widen-limits a separate 
function would have been a better choice back then (I wasn't aware, FWIW).

The new proposal is a bit simpler, OTOH.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Alan Mackenzie]

Hello, Dmitry.

On Fri, Dec 01, 2017 at 19:13:27 +0000, Dmitry Gutov wrote:
> On 12/1/17 3:49 PM, Alan Mackenzie wrote:

> >> Please give an example.

> > A low level function which as an essential part of its functionality
> > (for example, to make sure point-min isn't within a comment or string)
> > widens.

> See Stefan and I's responses. Including an example of a shim that 
> wouldn't require you to change your code too much.

A shim?  Yuck!  My idea is an MMM that doesn't require any such shims.
Major modes should not be "polluted" to adapt to one particular MMM.  It
solidifies that MMM, making it difficult to change to a better one.

> But first and foremost, CC Mode probably doesn't need to worry about 
> complying with this guideline (yet). In my experience, it doesn't work 
> well in multi-mode context for reasons other than it calling widen. So 
> worrying about this change in protocol might be premature.

You mean, it would be better to wait until this protocol is firmly
established before arguing against it?  I'm guessing that the main
reason CC Mode doesn't work well in MMM is the contention between CC
Mode's narrowing and MMM's narrowing.  (This is the example you asked
for earlier on.)

> >> To rephrase, don't widen in indent-line-function or
> >> beginning-of-defun-function.

> > This is an intolerable restriction.  The low level function mentioned
> > above cannot, should not, must not know whether it's being called
> > (indirectly) from indent-line-function or b-o-d-function.  It will have
> > to widen in all cases or none.  Therefore there will be failures whilst
> > being called either from one of the two noted functions or from
> > elsewhere.

> See that shim in the other email.

The shim doesn't help.  It's still horribly ugly.

> > Indeed it can, and it must.  A super-mode thus may not "reserve"
> > narrowing for its own purposes to the exclusion of other uses.

> Right.

Glad we agree upon this.  However that contradicts your proposal that
anything a beginning-of-defun-function calls can't use narrowing and
widening.

> > The multi-mode mechanism should be designed to be usable with any major
> > mode.  There's nothing particularly hard about supporting CC Mode in a
> > well designed multi-mode scheme.

> It's harder than supporting Ruby, CSS and JS modes, that I can tell you 
> for sure. For one thing, the extra caches CC Mode uses have given me 
> multiple "argument out of bounds" errors when used with narrowing.

There's your example for contention between major modes' narrowing and
MMM's narrowing.  If you didn't narrow, CC Mode would behave properly.

> So these are my takeaways:

> 1) The "low-level primitives" in CC Mode do not widen consistently.

They widen when they need to, which is not always.

> Some of them miss that responsibility (or else I wouldn't get those
> kind of errors).

It is the multiple-mode code that is responsible for not getting these
errors.  At least it should be.

> That tells me that moving (widen) call to some higher level is
> generally a good idea, but you can disagree.

> 2) The other errors should be solvable, but they require a separate 
> investigation, which nobody has found the time and energy to do so far, 
> over many years. So maybe it's not so necessary.

I would like AWK Mode to work inside shell-script-mode, since short AWK
scripts are so often embedded in scripts.

> 3) The solution to the other problems is most likely orthogonal to the 
> current discussion. Allowing multi-mode packages to use narrowing 
> certainly shouldn't hurt.

Of course not.  It's prohibiting major modes from using
narrowing/widening that is objectionable.

> > We need better tools.  I have already proposed and offered to implement
> > such tools.

> It was much more complex, nowhere near even a proof of concept, and 
> doesn't seem to move anywhere, so far.

Investing the time and energy on an implementation was too risky.  In
practice, Stefan has veto power over what goes into the syntax area, and
I've been at the sharp end of that veto once too often (the rejection of
my "comment-cache" code which solved the problem of open parens at
column zero in comments).  In the last exchange about the "syntactic
islands" concept, Stefan showed little enthusiasm for it: there was no
"OK, implement it and we'll see what we can do with it", no
encouragement whatsoever.  There was no discussion of major topics, such
as the possibilities it would open up, or any plans to use it.  Mainly,
the talk was about minor aspects of the proposal, at one point
degenerating into a squabble about the use of the word "island".

I'm still prepared to implement the "islands" proposal, which would
allow a better MMM mode than one based on narrowing, but ONLY if I get
some assurance from John, Eli, or even Stefan himself beforehand that
the implementation is, in principle, going to be incorporated into
Emacs.  That the only thing which would stop that would be solid
technical reasons.

> The change we're discussing is much smaller and pretty much implemented.

But it has negative implications.  It will only work with major modes
specially adapted for it, and it imposes severe restrictions on those
major modes.  I think it needs those modes to be "polluted" with MMM
code.

> And again, there's no reason why they couldn't coexist.

The "pollution" mentioned above would make it more difficult.

If the new code you're proposing is incorporated into Emacs, it will
acquire some momentum, making it difficult to introduce a better
solution.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Alan Mackenzie]

Hello, Stefan.

On Fri, Dec 01, 2017 at 13:03:05 -0500, Stefan Monnier wrote:
> > Indeed it can, and it must.  A super-mode thus may not "reserve"
> > narrowing for its own purposes to the exclusion of other uses.

> 99% of the multi-major-modes out there use narrowing for that purpose,
> and in practice it doesn't restrict other uses of narrowing.

If that were the case, there would be no restrictions on a major mode
widening and narrowing.  From what I understand about the new proposal,
your last paragraph is false.

By the way, what about that other 1% of multiple-major-modes?  How do
they work?

> I've opposed the use of narrowing for that in the past, but the evidence
> against that opposition is overwhelming.  Let's do that and move on.

No.  The evidence is that these modes can only work with specially
adapted major modes, and possibly that MMM code pollutes those major
modes.

> It's not perfect, but we've already looked much too long for a less
> imperfect solution.

I don't think that's the case.  My "syntactic islands" proposal was
never given proper consideration.  That was in last February.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/1/17 10:35 PM, Alan Mackenzie wrote:

> A shim?  Yuck!  My idea is an MMM that doesn't require any such shims.M

Ideas have the tendency to be more ideal than their eventual 
implementations.

> Major modes should not be "polluted" to adapt to one particular MMM.  It
> solidifies that MMM, making it difficult to change to a better one.

It's not "one particular MMM". All existing MMM packages use narrowing.

And it's not just to adapt. Calling `widen' in fewer places just makes 
sense from engineering point of view.

> You mean, it would be better to wait until this protocol is firmly
> established before arguing against it? 

The proposal isn't going to hurt you in any significant way. That's what 
I mean.

> The shim doesn't help.  It's still horribly ugly.

Calling widen from dozens of functions in a package is ugly. A shim just 
helps you avoid rearchitecting it.

>>> Indeed it can, and it must.  A super-mode thus may not "reserve"
>>> narrowing for its own purposes to the exclusion of other uses.
> 
>> Right.
> 
> Glad we agree upon this.  However that contradicts your proposal that
> anything a beginning-of-defun-function calls can't use narrowing and
> widening.

Nope.

> There's your example for contention between major modes' narrowing and
> MMM's narrowing. 

Not sure what you mean by that. It's already apparent that some major 
modes will have to change to adapt to this proposal. Isn't it?

> If you didn't narrow, CC Mode would behave properly.

Without narrowing it gives us wrong syntax highlighting and indentation.

>> So these are my takeaways:
> 
>> 1) The "low-level primitives" in CC Mode do not widen consistently.
> 
> They widen when they need to, which is not always.

I don't think I'd get this error if they did.

But hmm, I think this error was in fontification code, actually. 
mmm-mode, of course, narrows to each individual hunk before using the 
corresponding mode's font-lock-keywords.

>> 2) The other errors should be solvable, but they require a separate
>> investigation, which nobody has found the time and energy to do so far,
>> over many years. So maybe it's not so necessary.
> 
> I would like AWK Mode to work inside shell-script-mode, since short AWK
> scripts are so often embedded in scripts.

When we're talking about short snippets, sometimes it's easier to use 
e.g. js-mode instead. That's what I've been advising for C code, too.

But if complex syntax in AWK happens often, and CC Mode has good support 
for it, maybe supporting it is a worthy endeavor.

>> 3) The solution to the other problems is most likely orthogonal to the
>> current discussion. Allowing multi-mode packages to use narrowing
>> certainly shouldn't hurt.
> 
> Of course not.  It's prohibiting major modes from using
> narrowing/widening that is objectionable.

Sometimes code has to follow certain protocols. That's nothing new.

> Investing the time and energy on an implementation was too risky.  In
> practice, Stefan has veto power over what goes into the syntax area, and
> I've been at the sharp end of that veto once too often (the rejection of
> my "comment-cache" code which solved the problem of open parens at
> column zero in comments).  In the last exchange about the "syntactic
> islands" concept, Stefan showed little enthusiasm for it: there was no
> "OK, implement it and we'll see what we can do with it", no
> encouragement whatsoever.  There was no discussion of major topics, such
> as the possibilities it would open up, or any plans to use it.  Mainly,
> the talk was about minor aspects of the proposal, at one point
> degenerating into a squabble about the use of the word "island".

No discussion, really? I've pointed out the major difficulties I could 
think of, which is normally the area that should get the most attention 
during the discussion of a technical proposal.

The fact that few people seemed interested... well, that happens. 
There's no use complaining that a proposal is unpopular. I've had a few 
over the years myself. Please feel free to push for it, of course.

> But it has negative implications.  It will only work with major modes
> specially adapted for it, and it imposes severe restrictions on those
> major modes.  I think it needs those modes to be "polluted" with MMM
> code.

We already have several major modes adhering to it, and for them the 
change was trivial. As far as major modes are concerned, web programming 
is the most important area, so CC Mode support is less urgent anyway.

> If the new code you're proposing is incorporated into Emacs, it will
> acquire some momentum, making it difficult to introduce a better
> solution.

Like mentioned, for most major mode the changes are trivial, if at all 
needed.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

>>>> (defvar prog-widen-function #'widen)
>>>> (defun prog-widen () (funcall prog-widen-function))
> The idea for multi-mode packages has always been to make do without making
> the major modes actually aware of hunk boundaries.
[...]
> And the snippet of code won't make them aware either, since the chunk
> boundaries are conveyed via narrowing only in particular dynamic contexts,
> such as when indent-line-functions is called.

But widening/narrowing to the current chunk might be needed for commands
specific to the major-mode, hence commands which the MMM framework
doesn't know about (and hence doesn't wrap like it does for
indent-line-function).

> You probably mean adding a variable prog-first-column-function, then?

Right.

> Or will we expect multi-mode packages to simply advise the prog-first-column
> function? I'm fine with that.

Give that it's introduced specifically so it can be changed, relying
advice would be a bad idea (especially since the advice would generally
want to be buffer-local, which is easier to get with a variable).

>> Here are some examples I can think of:
>> - auto-fill-mode may like to compute the hypothetical indentation that
>> would result from inserting a newline somewhere (and do that before
>> touching the buffer).
> Won't it need to call indent-line-function to find out how much the next
> line should be indented?

No because indent-line-function would actually perform the
re-indentation, whereas we only want to calculate the hypothetical new
indentation column.  See `smie-auto-fill`.

>> - some package may like to highlight lines which aren't currently
>> indented right (so, it won't call indent-according-to-mode, but
>> it will need to compute the "desired" indentation).
> This example will *definitely* need to call indent-line-function.

Again no, because it doesn't want to modify the buffer.

> And both of them should be solved by exchanging indent-line-function for
> (non-mutating) line-intentation-function. But that's a change for
> another day.

Ah, yes, that would be a good change.
Arguably `smie-indent-functions` already provides that functionality.

> I'm sure there can be.  A complete proposal to let the modes know chunk
> boundaries, etc, has yet to be finalized, however.  And just having font-lock
> and indentation work reliably in multi-mode contexts will
> be a significant win.

Agreed.  As far as I'm concerned your proposal is good to go, which is
why I was talking about subsequent changes.

BTW, could we get some kind of multi-mode package in elpa.git or
emacs.git to go along with that (it doesn't have to be fancy, but it's
important that it doesn't have any submode-specific hacks).
Maybe a generalization of mhtml-mode, or something separate but whose
design is such that it can aim to (slowly) merge with mhtml-mode?


        Stefan

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

>> "OK, implement it and we'll see what we can do with it", no
>> encouragement whatsoever.  There was no discussion of major topics, such
>> as the possibilities it would open up, or any plans to use it.  Mainly,
>> the talk was about minor aspects of the proposal, at one point
>> degenerating into a squabble about the use of the word "island".

Here's my opinion: experience shows that implementing MMM support is messy
for all kinds of reasons, and that the problems that show up in practice
are very varied and many of them are completely unexpected.

So any paper design such as your "islands" to me sounds like a plain and
simple pipe-dream.  Can it work in theory?  Sure, why not.  Can it work
in practice?  Probably.  Will it work without having to "pollute major
modes to adapt to that particular MMM"?  Not a chance in hell, IMO.

So overall, will it be better than the solutions we're already
familiar with?  maybe, but my money is on "probably not".

Also, I don't think major modes need more freedom in how to implement
things.  Instead, they need more help so they can just use existing
solutions rather than reinvent the wheel in their very own way.
Conventions/guidelines actually makes the life of major mode
implementers easier, rather than harder.  E.g. they can just use
`syntax-propertize` and move on.  If it's not efficient enough or if it
doesn't interact correctly with an MMM framework, it's not their problem
it's syntax-propertize's. 


        Stefan


PS: On a very related topic, I'd recommend anyone interested in such
ideas to look at the way other editors handle font-lock-style
highlighting and based on that develop a replacement for font-lock.
AFAICT, font-lock is much less powerful/flexible than some of our
competitor's frameworks, especially in terms of handling differently
different parts of the buffer.  So a "font-lock-NG" might be a good
starting point to add better support for some MMM-style features.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

>> 99% of the multi-major-modes out there use narrowing for that purpose,
>> and in practice it doesn't restrict other uses of narrowing.
> If that were the case, there would be no restrictions on a major mode
> widening and narrowing.  From what I understand about the new proposal,
> your last paragraph is false.

Depends what you expect "use" to mean.  For me "call widen"
doesn't qualify as "uses narrowing".  So I stand by what I said.

> By the way, what about that other 1% of multiple-major-modes?  How do
> they work?

Multiple buffers (either by copying text to/from side-buffers, or by
use of indirect-buffers).

>> I've opposed the use of narrowing for that in the past, but the evidence
>> against that opposition is overwhelming.  Let's do that and move on.
> No.  The evidence is that these modes can only work with specially
> adapted major modes,

Right.  And use of narrowing is the way that makes this adaption
simplest (in many cases it becomes trivial.  See for example how easy
it's been to implement mhtml-mode).

I don't know of any MMM solution which provides a satisfactory user
experience and at the same time works for all major modes,
without any need to tweak them at all, no matter how twisted is the
major mode's code.


        Stefan

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 1 Dec 2017 23:24:28 +0000
> Cc: Tom Tromey <tom@tromey.com>, Stefan Monnier <monnier@iro.umontreal.ca>,
> 	Vitalie Spinu <spinuvit@gmail.com>, emacs-devel@gnu.org
> 
> Like mentioned, for most major mode the changes are trivial, if at all 
> needed.

IMO, C modes and Lisp modes must be supported well, before this lands
on master.

Thanks.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/2/17 8:27 AM, Eli Zaretskii wrote:
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Date: Fri, 1 Dec 2017 23:24:28 +0000
>> Cc: Tom Tromey <tom@tromey.com>, Stefan Monnier <monnier@iro.umontreal.ca>,
>> 	Vitalie Spinu <spinuvit@gmail.com>, emacs-devel@gnu.org
>>
>> Like mentioned, for most major mode the changes are trivial, if at all
>> needed.
> 
> IMO, C modes and Lisp modes must be supported well, before this lands
> on master.

Why? Web languages are a lot more common in mixed mode buffers (think 
HTML files and templates), so it's a lot more important to have that 
support in them (and we already do).

Lisp modes should work already too, because lisp-indent-line doesn't 
call widen.

And about C, first, it's much harder for different reasons, two, we'd 
have to spend more time convincing its author. And then we'd have to fix 
other MMM-related problems in it that the other mentioned modes don't have.

And, I'm sorry about this, but I want to land these changes in emacs-26, 
for one main reason: it simplifies, reworks and replaces the 
prog-indentation-context feature, which is not in Emacs 25.

At one point we/I removed it from Emacs 25 for reasons of being 
undercooked, but didn't merge that change to then-master. And then kind 
of forgot. https://debbugs.gnu.org/cgi/bugreport.cgi?bug=28475#11 was a 
later reminder.

This way, the "widen less" proposal could be considered a "bug fix", if 
we stretch the definition far enough. And also, the new approach is 
pretty small, it removes more code than adds.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> Cc: acm@muc.de, tom@tromey.com, monnier@iro.umontreal.ca, spinuvit@gmail.com,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 2 Dec 2017 10:50:24 +0000
> 
> > IMO, C modes and Lisp modes must be supported well, before this lands
> > on master.
> 
> Why?

Because Emacs is written in these 2 languages.  Sorry, I thought that
was clear.

> Web languages are a lot more common in mixed mode buffers (think 
> HTML files and templates), so it's a lot more important to have that 
> support in them (and we already do).

I expected those languages to be supported already, which is why I
didn't mention them.

> And, I'm sorry about this, but I want to land these changes in emacs-26, 
> for one main reason: it simplifies, reworks and replaces the 
> prog-indentation-context feature, which is not in Emacs 25.

It's too late for landing such features on emacs-26.  Sorry, it will
have to wait until Emacs 27.

Emacs 26 is nearing its release; the stream of serious bugs (or any
bugs, actually) seems to have died out.  So hopefully one more
pretest, and we will be ready to go.  Adding a significant new
feature, which will no doubt need changes in other places, is
incompatible with that.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Tom Tromey]

>>>>> "Stefan" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

Stefan> Right.  And use of narrowing is the way that makes this adaption
Stefan> simplest (in many cases it becomes trivial.  See for example how easy
Stefan> it's been to implement mhtml-mode).

It doesn't much matter to me whether narrowing or some other approach is
used.  Narrowing was simpler at the time -- and, I tend to think, will
always be simpler, because the various buffer primitives already respect
it, so you don't need to pass extra information around.  It has the same
sort of utility as dynamic binding in this way, just like RMS pointed
out in that Emacs paper back in the beginning.

However - if something else is more desirable somehow, that's also fine
with me.

Stefan> I don't know of any MMM solution which provides a satisfactory user
Stefan> experience and at the same time works for all major modes,
Stefan> without any need to tweak them at all, no matter how twisted is the
Stefan> major mode's code.

Yes, I completely agree here.  Some changes will be needed.  If you look
through the mhtml development process, there were a few tweaks needed
here and there, at least one of which (a syntax change in sgml mode
IIRC) was quite unexpected.

I haven't really appreciated until recently that maybe some people were
looking for a solution that required no changes.  Better, I think, to
accept that some changes are needed; maybe work to keep the changes
reasonably small and specified; but then just update the interesting
modes (maybe all modes) and move on.

Also, there's still some unanswered questions in mhtml mode.  For
example, the way it captures sub-mode buffer-locals could really be
improved.  Maybe by applying .dir-locals from all the modes; maybe by
having something like the mode hook but for sub-modes -- I'm not sure.

Tom

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

>> > IMO, C modes and Lisp modes must be supported well, before this lands
>> > on master.
>> Why?
> Because Emacs is written in these 2 languages.  Sorry, I thought that
> was clear.

So you'd rather see "no multi-major-mode support at all" than
"multi-major-mode support for some major modes"?

> It's too late for landing such features on emacs-26.  Sorry, it will
> have to wait until Emacs 27.

Then we should urgently remove the prog-indentation-context business
(i.e. revert to the Emacs-25 state of those features).


        Stefan

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Dmitry Gutov <dgutov@yandex.ru>,  acm@muc.de,  tom@tromey.com,  spinuvit@gmail.com,  emacs-devel@gnu.org
> Date: Sat, 02 Dec 2017 11:41:49 -0500
> 
> >> > IMO, C modes and Lisp modes must be supported well, before this lands
> >> > on master.
> >> Why?
> > Because Emacs is written in these 2 languages.  Sorry, I thought that
> > was clear.
> 
> So you'd rather see "no multi-major-mode support at all" than
> "multi-major-mode support for some major modes"?

I think the feature should be fairly complete before it lands, yes.

> > It's too late for landing such features on emacs-26.  Sorry, it will
> > have to wait until Emacs 27.
> 
> Then we should urgently remove the prog-indentation-context business
> (i.e. revert to the Emacs-25 state of those features).

I don't really understand what this means in practice, sorry.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

> I think the feature should be fairly complete before it lands, yes.

Would "make it work with Lisp and C" be satisfied if the "and C" part
uses sm-c-mode?  If not, then I'm afraid this makes it nigh-on impossible.


        Stefan

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: dgutov@yandex.ru,  acm@muc.de,  tom@tromey.com,  spinuvit@gmail.com,  emacs-devel@gnu.org
> Date: Sat, 02 Dec 2017 12:53:58 -0500
> 
> > I think the feature should be fairly complete before it lands, yes.
> 
> Would "make it work with Lisp and C" be satisfied if the "and C" part
> uses sm-c-mode?

I don't see how it could be, not IMO.  I think we need it working with
the C modes the majority of Emacs users are using, the ones that are
part of Emacs.

> If not, then I'm afraid this makes it nigh-on impossible.

Can you explain why is this so hard that you are willing to give up?

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/2/17 2:24 AM, Stefan Monnier wrote:

> But widening/narrowing to the current chunk might be needed for commands
> specific to the major-mode, hence commands which the MMM framework
> doesn't know about (and hence doesn't wrap like it does for
> indent-line-function).

Depending on whether the major mode is the "primary" mode or a 
"submode", some commands might prefer no resrictions, and some the 
current chunk. font-lock, on the other hand, will (almost?) always want 
to use the current chunk.

And only the MMM framework knows whether the mode is primary or not. So 
the issue is not that obvious.

>> You probably mean adding a variable prog-first-column-function, then?
> 
> Right.

Are you fine with only having prog-first-column variable in Emacs 26?

Adding prog-first-column-function without having something similar for 
chunks would seem inconsistent.

> Give that it's introduced specifically so it can be changed, relying
> advice would be a bad idea (especially since the advice would generally
> want to be buffer-local, which is easier to get with a variable).

Makes sense.

> No because indent-line-function would actually perform the
> re-indentation, whereas we only want to calculate the hypothetical new
> indentation column.  See `smie-auto-fill`.

I see.

>>> - some package may like to highlight lines which aren't currently
>>> indented right (so, it won't call indent-according-to-mode, but
>>> it will need to compute the "desired" indentation).
>> This example will *definitely* need to call indent-line-function.
> 
> Again no, because it doesn't want to modify the buffer.

This one I don't understand: how would you know the indentation is 
correct without trying to reindent?

>> And both of them should be solved by exchanging indent-line-function for
>> (non-mutating) line-intentation-function. But that's a change for
>> another day.
> 
> Ah, yes, that would be a good change.
> Arguably `smie-indent-functions` already provides that functionality.

As an example, yes, but it's not like we can move all modes to SMIE.

>> I'm sure there can be.  A complete proposal to let the modes know chunk
>> boundaries, etc, has yet to be finalized, however.  And just having font-lock
>> and indentation work reliably in multi-mode contexts will
>> be a significant win.
> 
> Agreed.  As far as I'm concerned your proposal is good to go, which is
> why I was talking about subsequent changes.

The current state of the branch, then?

> BTW, could we get some kind of multi-mode package in elpa.git or
> emacs.git to go along with that (it doesn't have to be fancy, but it's
> important that it doesn't have any submode-specific hacks).

Do you have any particular hacks in mind, in e.g. mmm-mode?

> Maybe a generalization of mhtml-mode,
That might actually be possible, a kind of mixed-mode builder.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/2/17 11:10 AM, Eli Zaretskii wrote:

>>> IMO, C modes and Lisp modes must be supported well, before this lands
>>> on master.
>>
>> Why?
> 
> Because Emacs is written in these 2 languages.  Sorry, I thought that
> was clear.

Still. Why would you need it? What benefits haven't you been able to 
achieve without that support, while working on Emacs?

>> Web languages are a lot more common in mixed mode buffers (think
>> HTML files and templates), so it's a lot more important to have that
>> support in them (and we already do).
> 
> I expected those languages to be supported already, which is why I
> didn't mention them.

But they are the most important ones/ones most desired by the users.

> It's too late for landing such features on emacs-26.  Sorry, it will
> have to wait until Emacs 27.

I understand it's a problem, and I'm sorry for bringing it up this late.

> Emacs 26 is nearing its release; the stream of serious bugs (or any
> bugs, actually) seems to have died out.

Allow me to put it like this: AFAIU, prog-indenation-context, which is 
in emacs-26, is *not* properly supported in most of the modes where we'd 
expect it to be. E.g. css-mode, js-mode and all modes using SMIE for 
indentation (such as ruby-mode).

The original author only contributed support for python-mode, and it 
stayed this way since. Now, adding that support wouldn't be complex, but 
my proposed change isn't complex either. And we have more of a consensus 
on it.

> Adding a significant new
> feature, which will no doubt need changes in other places, is
> incompatible with that.

Have you taken a look at the branch? It's your call, but based on the 
number of changes, I wouldn't call it "significant". Important, yes, but 
it's fairly small.

The other option is simply removing prog-indentation-context before the 
release.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/2/17 5:13 PM, Eli Zaretskii wrote:

>> So you'd rather see "no multi-major-mode support at all" than
>> "multi-major-mode support for some major modes"?
> 
> I think the feature should be fairly complete before it lands, yes.

It's not much of a feature, it's more of a codified protocol. The change 
in the manual is the actually important part.

And it's expected that some modes can choose not to follow it. Then 
they'll be no worse off than before its introduction. Multi-mode 
packages just won't be able to support them.

>> Then we should urgently remove the prog-indentation-context business
>> (i.e. revert to the Emacs-25 state of those features).
> 
> I don't really understand what this means in practice, sorry.

Removing all references to prog-indentation-context, prog-first-column 
and prog-widen.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Alan Mackenzie]

Hello, Stefan.

On Fri, Dec 01, 2017 at 21:47:26 -0500, Stefan Monnier wrote:
> >> "OK, implement it and we'll see what we can do with it", no
> >> encouragement whatsoever.  There was no discussion of major topics, such
> >> as the possibilities it would open up, or any plans to use it.  Mainly,
> >> the talk was about minor aspects of the proposal, at one point
> >> degenerating into a squabble about the use of the word "island".

> Here's my opinion: experience shows that implementing MMM support is messy
> for all kinds of reasons, and that the problems that show up in practice
> are very varied and many of them are completely unexpected.

> So any paper design such as your "islands" to me sounds like a plain and
> simple pipe-dream.  Can it work in theory?  Sure, why not.  Can it work
> in practice?  Probably.  Will it work without having to "pollute major
> modes to adapt to that particular MMM"?  Not a chance in hell, IMO.

It would work, and work well, for the simple reason it is a coherent
proposal.

Emacs was designed with the assumption that each buffer has exactly one
major mode.  It turns out this assumption hasn't held all that well.
When that assumption does hold, the following returns a well defined and
meaningful state:

    (parse-partial-sexp 1 n)

.  I am proposing extending this property for buffers with several
modes and several syntax tables.  Nothing more, nothing less.

You're saying that providing support for multiple major modes in the
Emacs core is a pipe dream.  I put it to you you're mistaken.  ONLY by
supporting this in the Emacs core can we arrive at a coherent efficient
design.  The fact we are having this conversation is a strong indication
that the current designs for MMM are not coherent enough.

[ .... ]

> Also, I don't think major modes need more freedom in how to implement
> things.  Instead, they need more help so they can just use existing
> solutions rather than reinvent the wheel in their very own way.
> Conventions/guidelines actually makes the life of major mode
> implementers easier, rather than harder.

That is immensely patronising.

> E.g. they can just use `syntax-propertize` and move on.  If it's not
> efficient enough or if it doesn't interact correctly with an MMM
> framework, it's not their problem it's syntax-propertize's. 

syntax-propertize is your way of doing things, so naturally you want it
to become a "standard".  However it is not the only way of doing things,
and is suboptimal in several respects.

When designing syntax-propertize, with whom did you discuss the
technical aspects?  Certainly not with me, but this may have been before
my time in Emacs.  With Martin Stjernholm or Barry Warsaw, my
predecessors as CC Mode maintainers?  With RMS?  With anybody at all?

>         Stefan

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> Cc: acm@muc.de, tom@tromey.com, monnier@iro.umontreal.ca, spinuvit@gmail.com,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 2 Dec 2017 20:14:54 +0000
> 
> > Because Emacs is written in these 2 languages.  Sorry, I thought that
> > was clear.
> 
> Still. Why would you need it? What benefits haven't you been able to 
> achieve without that support, while working on Emacs?

I don't understand the question.  Surely, a feature that aims to
support multiple major modes should strive to support CC Mode?

> >> Web languages are a lot more common in mixed mode buffers (think
> >> HTML files and templates), so it's a lot more important to have that
> >> support in them (and we already do).
> > 
> > I expected those languages to be supported already, which is why I
> > didn't mention them.
> 
> But they are the most important ones/ones most desired by the users.

They are important, but so are C-based languages: C++, Java, Awk, etc.

> > It's too late for landing such features on emacs-26.  Sorry, it will
> > have to wait until Emacs 27.
> 
> I understand it's a problem, and I'm sorry for bringing it up this late.

It's not a catastrophe.  Assuming Emacs 26.1 is not a complete
disaster, we could decide releasing Emacs 27.1 as the next version.

> > Adding a significant new
> > feature, which will no doubt need changes in other places, is
> > incompatible with that.
> 
> Have you taken a look at the branch? It's your call, but based on the 
> number of changes, I wouldn't call it "significant". Important, yes, but 
> it's fairly small.

It might be "small", but it is by no means "insignificant".  Changing
buffer restrictions at strategic places is not something that should
be taken lightly.  The current pretest has been out there for the last
2 months; introducing this feature now (and actually in at least a few
days, as it isn't ready yet) would set us back that many days, if not
more.

> The other option is simply removing prog-indentation-context before the 
> release.

Why would we need to remove that for Emacs 26.1?

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/2/17 8:58 PM, Eli Zaretskii wrote:

>> Still. Why would you need it? What benefits haven't you been able to
>> achieve without that support, while working on Emacs?
> 
> I don't understand the question.

I'm asking why "Emacs is written in these 2 languages" is important.

> Surely, a feature that aims to
> support multiple major modes should strive to support CC Mode?

Not necessarily. It should first and foremost support the languages that 
are most important for our users.

> They are important, but so are C-based languages: C++, Java, Awk, etc.

I don't know CC Mode, and cannot provide patches for us. I tried 
debugging related problems a few times, and those turned out to be 
fairly long spelunking expeditions.

However long it will take just to convince Alan to agree, is anybody's 
guess.

> It's not a catastrophe.  Assuming Emacs 26.1 is not a complete
> disaster, we could decide releasing Emacs 27.1 as the next version.

Not a catastrophe indeed. But no step forward for mixed-modes support in 
Emacs 26 would be unfortunate.

> Why would we need to remove that for Emacs 26.1?

It's not well supported, and it's incompatible with the code we are 
discussing. There's no point in having it present in Emacs 26, and then 
removing it in Emacs 27.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

> Are you fine with only having prog-first-column variable in Emacs 26?

Since Eli only wants MMM-support if it work 100% everywhere, I don't
think it matters what gets into Emacs itself in this respect, because
the real stuff will have to live elsewhere.

The thing we should try and fight for of course are those such as making
indent-according-to-mode widen before calling indent-line-function.

>> Give that it's introduced specifically so it can be changed, relying
       ^                                                               ^^
       n                                                               on
>> advice would be a bad idea (especially since the advice would generally
>> want to be buffer-local, which is easier to get with a variable).
> Makes sense.

I'm glad English includes enough redundancy that you could make out what
I meant.

>>>> - some package may like to highlight lines which aren't currently
>>>> indented right (so, it won't call indent-according-to-mode, but
>>>> it will need to compute the "desired" indentation).
>>> This example will *definitely* need to call indent-line-function.
>> Again no, because it doesn't want to modify the buffer.
> This one I don't understand: how would you know the indentation is correct
> without trying to reindent?

The idea is just to highlight those lines whose indentation would be
modified by indent-line-function, so you only want to compute the
target column and compare it with the current column.

>> Arguably `smie-indent-functions` already provides that functionality.
> As an example, yes, but it's not like we can move all modes to SMIE.

Hence the "arguably": it can be used without using SMIE, but it's not as
convenient as you might like.  E.g. you can do:

    (require 'smie)

    (define-derived-mode ...
      ...
      (setq-local smie-indent-functions '(my-compute-indent))
      ...)

You can even reuse some parts of smie-indent-functions which don't depend
on the smie-grammar/rules business, e.g.:

      (setq-local smie-indent-functions
                  '(smie-indent-fixindent smie-indent-bob
                    smie-indent-comment smie-indent-comment-continue
                    smie-indent-comment-close smie-indent-comment-inside
                    smie-indent-inside-string
                    my-compute-indent))

but the more serious problem is that other packages (such as auto-fill)
can't rely on `smie-indent-functions' because it's not used by all major modes.

>> Agreed.  As far as I'm concerned your proposal is good to go, which is
>> why I was talking about subsequent changes.
> The current state of the branch, then?

Sure.

>> BTW, could we get some kind of multi-mode package in elpa.git or
>> emacs.git to go along with that (it doesn't have to be fancy, but it's
>> important that it doesn't have any submode-specific hacks).
> Do you have any particular hacks in mind, in e.g. mmm-mode?

No.  Can we put mmm-mode into elpa.git?


        Stefan

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

We could add another C-level feature similar to narrowing,
specifically for the sake of multiple major modes, if that helps
implement them cleanly.

Multiple major modes is important enough to justify such a thing.

The reason I see that an additional C-level feature MIGHT help is that
we could arrange at C level for it to interact in just the right way
with ALL uses of the existing narrowing feature.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

>     (parse-partial-sexp 1 n)
>
> .  I am proposing extending this property for buffers with several
> modes and several syntax tables.  Nothing more, nothing less.

That would potentially solve some percentage of the problems with
MMM support.  But what about the rest?

> syntax-propertize is your way of doing things, so naturally you want it
> to become a "standard".

I don't actually care whether the standard is my standard or not.
What I wanted was *a* standard.

> However it is not the only way of doing things,
> and is suboptimal in several respects.

All standards are suboptimal.  The reason people use them is because the
existence of a standard makes other things possible.

> When designing syntax-propertize, with whom did you discuss the
> technical aspects?

My cat.  Then I (cause at that point my cat couldn't be bothered to
follow me to the desk) tried what we both thought might work, then
I applied it to more modes, and a few years later it ends up working in
*all* major modes I know (including CC-mode, tho I haven't bothered to
make it robust enough to have a patch, knowing that you're never going
to accept such a patch in CC-mode, even though it's mostly just throwing
away code).


        Stefan

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sat, 02 Dec 2017 18:47:57 -0500
> Cc: Alan Mackenzie <acm@muc.de>, Tom Tromey <tom@tromey.com>,
> 	Vitalie Spinu <spinuvit@gmail.com>, emacs-devel@gnu.org
> 
> > Are you fine with only having prog-first-column variable in Emacs 26?
> 
> Since Eli only wants MMM-support if it work 100% everywhere

Please don't misrepresent so completely what I wrote.

> I don't think it matters what gets into Emacs itself in this
> respect, because the real stuff will have to live elsewhere.
> [...]
> Can we put mmm-mode into elpa.git?

Even if it's possible, it makes absolutely no sense to me to have such
a central feature on ELPA and not in the core.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/2/17 8:28 PM, Alan Mackenzie wrote:

> It would work, and work well, for the simple reason it is a coherent
> proposal.

I've never seen a full proposal. Details matter. And it's going to have 
to include a lot more of them.

> Emacs was designed with the assumption that each buffer has exactly one
> major mode.  It turns out this assumption hasn't held all that well.
> When that assumption does hold, the following returns a well defined and
> meaningful state:
> 
>      (parse-partial-sexp 1 n)
> 
> .  I am proposing extending this property for buffers with several
> modes and several syntax tables.  Nothing more, nothing less.

Extending parse-partial-sexp to nested syntax tables can be worthwhile, 
but that can't be it. And while we're on this subject, there are some 
example of Ruby syntax that we'd like to see supported, where the 
current syntax tables (and even syntax-propertize-function) don't cut it.

I'm not convinced "several syntax tables" would be enough, though.

> You're saying that providing support for multiple major modes in the
> Emacs core is a pipe dream.  I put it to you you're mistaken.  ONLY by
> supporting this in the Emacs core can we arrive at a coherent efficient
> design.

The major modes will have to change to allow for this support, that's 
for sure. You can't expect no code changes.

> The fact we are having this conversation is a strong indication
> that the current designs for MMM are not coherent enough.

The current idea is "let's do the minimum coherent step forward, so that 
somebody can continue improving things later". A step that's easy to 
use, and also easy to undo later, if we find something else that's much 
better.

Coming up with it was not easy at all, BTW.

>> Also, I don't think major modes need more freedom in how to implement
>> things.  Instead, they need more help so they can just use existing
>> solutions rather than reinvent the wheel in their very own way.
>> Conventions/guidelines actually makes the life of major mode
>> implementers easier, rather than harder.
> 
> That is immensely patronising.

It's simply description of good engineering. You can try to criticize 
the details of syntax-propertize, but the idea of reusing common 
solutions is more than sound.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Alan Mackenzie]

Hello, Stefan.

On Sat, Dec 02, 2017 at 19:03:44 -0500, Stefan Monnier wrote:
> >     (parse-partial-sexp 1 n)

> > .  I am proposing extending this property for buffers with several
> > modes and several syntax tables.  Nothing more, nothing less.

> That would potentially solve some percentage of the problems with
> MMM support.  But what about the rest?

They would clearly need other work doing on them.  Richard suggested
building something similar to narrowing for the excusive use of MMM.

> > syntax-propertize is your way of doing things, so naturally you want it
> > to become a "standard".

> I don't actually care whether the standard is my standard or not.
> What I wanted was *a* standard.

Standards are created by like-minded experts coming together, thrashing
things out, and finally agreeing on some reasonable set of compromises;
not by a lone hacker, no matter how expert, throwing something together
in his bedroom.

> > However it is not the only way of doing things,
> > and is suboptimal in several respects.

> All standards are suboptimal.

You seem to be implying that all standards are suboptimal, therefore any
standard is as good as any other.  That is palpably false.  

> The reason people use them is because the existence of a standard
> makes other things possible.

Not really.  At least, this particular standard (if such it be),
syntax-propertize, shuts off useful, even essential, possibilities.  It
is also based on syntax-ppss, which we recently agreed needed replacing
with something which does the right job.

The way you want to restrict the use of the syntax-table text property
is like having a screw driver, and only being permitted to turn it
clockwise.  That text property is a very useful general purpose tool,
something you seem to want to do away with.

> > When designing syntax-propertize, with whom did you discuss the
> > technical aspects?

> My cat.  Then I (cause at that point my cat couldn't be bothered to
> follow me to the desk) tried what we both thought might work, then
> I applied it to more modes, and a few years later it ends up working in
> *all* major modes I know (including CC-mode, tho I haven't bothered to
> make it robust enough to have a patch, knowing that you're never going
> to accept such a patch in CC-mode, even though it's mostly just throwing
> away code).

It may be possible to rebuild CC Mode to conform to this "standard", but
it would be a lot of work, the "standard" would need enhancements[*], CC
Mode would run more slowly, and its users wouldn't see any benefit.

[*] What I'm thinking of, in particular, is after buffer changes, the
syntax-propertize mechanism doesn't amend the syntax-table text
properties adquately.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Alan Mackenzie]

Hello, Dmitry.

On Sun, Dec 03, 2017 at 03:52:59 +0000, Dmitry Gutov wrote:
> On 12/2/17 8:28 PM, Alan Mackenzie wrote:

> > It would work, and work well, for the simple reason it is a coherent
> > proposal.

> I've never seen a full proposal. Details matter. And it's going to have 
> to include a lot more of them.

Details do matter.  I posted a full proposal, Subject: "Syntax
ambiguities in narrowed buffers and multiple major modes: a proposed
solution." here in emacs-devel on Sat, 25 Feb 2017 13:53:56 +0000.

I've changed my mind about some of the details, partly in response to
what people said, partly because I see things now I didn't see then.

> > Emacs was designed with the assumption that each buffer has exactly one
> > major mode.  It turns out this assumption hasn't held all that well.
> > When that assumption does hold, the following returns a well defined and
> > meaningful state:

> >      (parse-partial-sexp 1 n)

> > .  I am proposing extending this property for buffers with several
> > modes and several syntax tables.  Nothing more, nothing less.

> Extending parse-partial-sexp to nested syntax tables can be worthwhile, 
> but that can't be it. And while we're on this subject, there are some 
> example of Ruby syntax that we'd like to see supported, where the 
> current syntax tables (and even syntax-propertize-function) don't cut it.

What is preventing a s-p-function from working satisfactorally?

> I'm not convinced "several syntax tables" would be enough, though.

OK, let's make that "many syntax tables".  :-)

> > You're saying that providing support for multiple major modes in the
> > Emacs core is a pipe dream.  I put it to you you're mistaken.  ONLY by
> > supporting this in the Emacs core can we arrive at a coherent efficient
> > design.

> The major modes will have to change to allow for this support, that's 
> for sure. You can't expect no code changes.

Maybe, maybe not.  Super-modes would definitely have to change to use
it.  Any changes in major modes should be minor.  (I don't rate
restrictions on the use of Emacs primitives to be minor.)

> > The fact we are having this conversation is a strong indication
> > that the current designs for MMM are not coherent enough.

> The current idea is "let's do the minimum coherent step forward, so that 
> somebody can continue improving things later". A step that's easy to 
> use, and also easy to undo later, if we find something else that's much 
> better.

OK.  With such a tricky subject matter, it would surprising if we
couldn't find improvements for a long time.

> Coming up with it was not easy at all, BTW.

Yes, I can see that.  Well done!

> >> Also, I don't think major modes need more freedom in how to implement
> >> things.  Instead, they need more help so they can just use existing
> >> solutions rather than reinvent the wheel in their very own way.
> >> Conventions/guidelines actually makes the life of major mode
> >> implementers easier, rather than harder.

> > That is immensely patronising.

> It's simply description of good engineering. You can try to criticize 
> the details of syntax-propertize, but the idea of reusing common 
> solutions is more than sound.

If those common solution are adequate, yes.  But it is also critical to
have the freedom to depart from things like syntax-propertize when they
are inadequate, or too awkward.  syntax-propertize is inadequate for CC
Mode:

E.g., in C++ Mode, with

    foo (a < b, c > d);

we have a constructor foo with one parameter d, of a templated type.  The
characters "<" and ">" have syntax-table text properties giving them
paren syntaxes and making them match each other.

Now, mark the characters ", c > d" and kill them with C-w.  What we are
left with is a function call with an integer argument a < b:

    foo (a < b);

.  Something, somewhere has to remove the syntax-table property from
"<", since it is no longer a template delimiter.

The syntax-propertize mechanism won't do this, since the place to
de-propertise is before BEG.  In CC Mode, the job is done by
c-before-change-check-<>-operators.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> Cc: acm@muc.de, tom@tromey.com, monnier@iro.umontreal.ca, spinuvit@gmail.com,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 2 Dec 2017 21:35:43 +0000
> 
> I'm asking why "Emacs is written in these 2 languages" is important.

Because every development environment should first support its own
development well.  If it doesn't do that, it's incomplete.

> > Surely, a feature that aims to
> > support multiple major modes should strive to support CC Mode?
> 
> Not necessarily. It should first and foremost support the languages that 
> are most important for our users.

Indeed; and C-like languages are quite popular among them.

How will we look if we support JS and Ruby embedded in HTML, but not C
code embedded in Yacc grammars nor Awk snippets embedded in shell
scripts?  Both are quite frequent out there.  We will be laughed at.

> > They are important, but so are C-based languages: C++, Java, Awk, etc.
> 
> I don't know CC Mode, and cannot provide patches for us. I tried 
> debugging related problems a few times, and those turned out to be 
> fairly long spelunking expeditions.
> 
> However long it will take just to convince Alan to agree, is anybody's 
> guess.

Convince Alan to do what?  Do we even understand well enough what are
the problems in CC Mode that get in the way?  About the only thing
spelled out in this discussion was the need not to widen.  Personally,
I think this is not a grave issue, just some technical problem that is
certainly solvable (you proposed one such solution).  But is that all?
You mentioned some other problems, but never described them.  What are
they?  Are they grave enough to prevent your proposal from ever
supporting CC Mode, even if your proposal is amended?  I don't know.
Does anyone know?  If so, would they please speak up?

I also don't understand well the essence of your proposal.  Yes, I've
(tried to) read the discussions you pointed to, but what I'm looking
for is buried there under many layers of secondary stuff.  Why do you
want to abandon the prog-widen stuff, which you supported just 2 years
ago?  How do you address the issues which prog-indentation-context did
(e.g., if the embedded chunk of code is incomplete, and perhaps even
syntactically invalid)?  All these questions and considerations need
to be understood, because you are explicitly proposing an incomplete
solution, and asking us to agree with its deficiencies, but without
providing a clear picture of what are we going to give up on.  And
last, but not least, what is different in your proposal that will not
cause it to end up like all the previous ones: either committed to
Emacs and basically unused, or, worse, collecting dust in long
forgotten patches or branches?

We desperately need these details spelled out to make the discussion
of this to become technical again, not emotional.

> > It's not a catastrophe.  Assuming Emacs 26.1 is not a complete
> > disaster, we could decide releasing Emacs 27.1 as the next version.
> 
> Not a catastrophe indeed. But no step forward for mixed-modes support in 
> Emacs 26 would be unfortunate.

Emacs 26 has enough new features to justify a major release, so I see
no reason to press so hard to get this feature into it as well.  You
are still debating its design and implementation, even if we disregard
the CC Mode issue.  It makes no sense to put this into a code base
which is frozen since September.

> > Why would we need to remove that for Emacs 26.1?
> 
> It's not well supported, and it's incompatible with the code we are 
> discussing. There's no point in having it present in Emacs 26, and then 
> removing it in Emacs 27.

That code is in Emacs for more than 2 years.  It was admitted with
Stefan's full support, and at least ANTLR needs it in conjunction with
Python.  Removing it without having some alternative again makes no
sense to me.  We should discuss this when the incompatible code lands;
at that time, we will see how to remove/replace prog-widen etc. with
minimal pain for its users.  For now, there's no incompatibility that
requires us to remove the code.  And anyway, making such changes when
we are in pretest is against our practices.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/3/17 12:18 PM, Alan Mackenzie wrote:

>> That would potentially solve some percentage of the problems with
>> MMM support.  But what about the rest?
> 
> They would clearly need other work doing on them.  Richard suggested
> building something similar to narrowing for the excusive use of MMM.

This sounds fairly different from your islands proposal. And the notion 
that you've already described a full solution, and was only waiting for 
approval.

> Standards are created by like-minded experts coming together, thrashing
> things out, and finally agreeing on some reasonable set of compromises;
> not by a lone hacker, no matter how expert, throwing something together
> in his bedroom.

That's very debatable. Most good standards come from working experience, 
and usually some working product (or a codebase).

A lot of standards that start in a meeting room don't leave one. Just 
look at the multitude of discussions here that don't move anywhere. Also 
see "design by committee".

Discussion is important, of course, but it has to be constructive from 
all sides.

> It may be possible to rebuild CC Mode to conform to this "standard", but
> it would be a lot of work, the "standard" would need enhancements[*], CC
> Mode would run more slowly, and its users wouldn't see any benefit.

Would you like to provide a benchmark that Stefan could try and see if 
the syntax-ppss based code indeed runs "more slowly"?

Anyway, this seems irrelevant to this discussion.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/3/17 3:28 PM, Eli Zaretskii wrote:

> Because every development environment should first support its own
> development well.  If it doesn't do that, it's incomplete.

OK, I see. But!

This feature is *not* targeted at Emacs development.

Please don't make me fly out to Scotland and chase Alan down with a 
broomstick to make this development come through.

> Indeed; and C-like languages are quite popular among them.

Not in comparison to JS, CSS and HTML.

> How will we look if we support JS and Ruby embedded in HTML, but not C
> code embedded in Yacc grammars nor Awk snippets embedded in shell
> scripts?  Both are quite frequent out there.  We will be laughed at.

We'll be fine. And the actual support is still going to reside in 
third-party packages for now. This change just makes them possible (or 
their codebases saner, at least).

Except for antlr-mode. It will be able to use it right away (the version 
inside Emacs, at least).

> Convince Alan to do what?

To adhere to the current proposal (avoid widening in 
indent-line-function and font-lock-keywords, to start with).

> Do we even understand well enough what are
> the problems in CC Mode that get in the way?

Who do you think is most qualified to study that issue? We'd probably 
have to convince Alan to do that as well, unfortunately.

I have a rough understanding of the issue, but since I haven't reached a 
working state, I don't know how many pitfalls there are left.

> About the only thing
> spelled out in this discussion was the need not to widen.  Personally,
> I think this is not a grave issue, just some technical problem that is
> certainly solvable (you proposed one such solution).

Indeed.

> But is that all?

I imagine the process itself might be trickier than expected. Various 
primitives use caches that save context information. What is such 
primitive to do if the cache contains "beginning of nesting" outside of 
the current restriction, and the logic of said primitive says "go to the 
beginning of the current function and do such and such"? The answer 
isn't obvious to me.

But it's going to need to be answered anyway, no matter which MMM 
approach we choose in the end, I think.

> You mentioned some other problems, but never described them.  What are
> they?  Are they grave enough to prevent your proposal from ever
> supporting CC Mode, even if your proposal is amended?  I don't know.
> Does anyone know?  If so, would they please speak up?

Yes, please.

> I also don't understand well the essence of your proposal.  Yes, I've
> (tried to) read the discussions you pointed to, but what I'm looking
> for is buried there under many layers of secondary stuff. 

The proposal itself is very small, and there's not much to explain. Just 
look at the changes in the manual, in this branch.

It simply facilitates what mmm-mode (and other modes, including 
mhtml-mode) has been doing for years, with varying success. And does 
that in a way that requires minimal changes from major modes (except for 
CC Mode, it seems), and would let us easily change our mind later.

> Why do you
> want to abandon the prog-widen stuff, which you supported just 2 years
> ago?

I never actually supported it, just stopped arguing because I didn't 
have a good alternative idea. Now I do, I think. And there is some 
agreement from Stefan.

> How do you address the issues which prog-indentation-context did
> (e.g., if the embedded chunk of code is incomplete, and perhaps even
> syntactically invalid)?

prog-indentation-context never addressed that issue, and we don't 
either. The major mode code in such situation will either have to 
swallow errors, or, well, they won't work. Or a MMM package will have to 
do error-swallowing, that's also possible.

In practice, a lot of such situations don't actually manifest, though.

> All these questions and considerations need
> to be understood, because you are explicitly proposing an incomplete
> solution, and asking us to agree with its deficiencies, but without
> providing a clear picture of what are we going to give up on.

That's the thing: we're not giving up much (I'd argue we're giving up 
nothing, but Alan obviously disagrees).

Consolidating the 'widen' calls is simply good engineering, even aside 
from making life easier for MMM packages: it's much better to do that in 
several well-defined places, instead of having every helper function do 
it as well, like python-mode CC Mode do.

> And
> last, but not least, what is different in your proposal that will not
> cause it to end up like all the previous ones: either committed to
> Emacs and basically unused, or, worse, collecting dust in long
> forgotten patches or branches?

Like mentioned, the narrowing-based approach is already used in mmm-mode 
and mhtml-mode (and polymode too, I think).

And it works fine in many situations, because many major modes already 
never call widen in their indentation functions (SMIE-based modes, Ruby, 
HTML and JS don't). So this proposal is, to a large extent, codifying an 
existing practice. And at the same time makes the aforementioned modes 
work *better* with interactive narrowing.

> We desperately need these details spelled out to make the discussion
> of this to become technical again, not emotional.

Please feel free to ask any further questions.

> You
> are still debating its design and implementation, even if we disregard
> the CC Mode issue.

Not really. There's a minor issue of whether to make prog-first-column a 
variable, or a hook right away, but the importance of that choice isn't big.

> That code is in Emacs for more than 2 years.  It was admitted with
> Stefan's full support, and at least ANTLR needs it in conjunction with
> Python.

Transitioning from prog-indentation-context to the new approach is very 
easy. And I would show you the patch, but unfortunately, the antlr-mode 
in Emacs doesn't use prog-indentation-context.

So really, it's been here for 2 years, and virtually nobody's using it, 
or improving it.

> Removing it without having some alternative again makes no
> sense to me. 

You have the alternative, though.

> We should discuss this when the incompatible code lands;

How about we discuss it now?

> at that time, we will see how to remove/replace prog-widen etc. with
> minimal pain for its users.  For now, there's no incompatibility that
> requires us to remove the code.  And anyway, making such changes when
> we are in pretest is against our practices.

That seems very unwise to me. Not to mention discouraging.

Take a look at any third-party packages. I'm willing to bet none use 
prog-indentation-context yet. Maybe the standalone version of 
antlr-mode, if only.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/2/17 11:47 PM, Stefan Monnier wrote:

> Since Eli only wants MMM-support if it work 100% everywhere, I don't
> think it matters what gets into Emacs itself in this respect, because
> the real stuff will have to live elsewhere.

But we need prog-first-column in Emacs. Except it already is. >_<

> The thing we should try and fight for of course are those such as making
> indent-according-to-mode widen before calling indent-line-function.

Right. That's true for many major modes already, though.

> I'm glad English includes enough redundancy that you could make out what
> I meant.

:-)

> The idea is just to highlight those lines whose indentation would be
> modified by indent-line-function, so you only want to compute the
> target column and compare it with the current column.

I still don't see how you're going to compute the target column without 
calling the indentation function. That's why switching to a non-mutating 
function will help.

>>> Arguably `smie-indent-functions` already provides that functionality.
>> As an example, yes, but it's not like we can move all modes to SMIE.
> 
> Hence the "arguably": it can be used without using SMIE, but it's not as
> convenient as you might like.  E.g. you can do:

Yeah... Let's not do that. Generic code won't really be able to rely on 
this practice anyway.

>>> BTW, could we get some kind of multi-mode package in elpa.git or
>>> emacs.git to go along with that (it doesn't have to be fancy, but it's
>>> important that it doesn't have any submode-specific hacks).
>> Do you have any particular hacks in mind, in e.g. mmm-mode?
> 
> No.  Can we put mmm-mode into elpa.git?

I'll try emailing the original authors again. The last time I did that 
there were no replies, unfortunately.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> Cc: acm@muc.de, tom@tromey.com, monnier@iro.umontreal.ca, spinuvit@gmail.com,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 3 Dec 2017 16:35:23 +0000
> 
> This feature is *not* targeted at Emacs development.

It is for me.

> > Indeed; and C-like languages are quite popular among them.
> 
> Not in comparison to JS, CSS and HTML.

Maybe in your world, but not in mine.  So we will have to agree (or
disagree, if you want) that both worlds need to be catered to.

> > How will we look if we support JS and Ruby embedded in HTML, but not C
> > code embedded in Yacc grammars nor Awk snippets embedded in shell
> > scripts?  Both are quite frequent out there.  We will be laughed at.
> 
> We'll be fine.

From your POV, maybe.  Not from mine.

> > Convince Alan to do what?
> 
> To adhere to the current proposal (avoid widening in 
> indent-line-function and font-lock-keywords, to start with).

We need to understand the full extent of problems, and then we need to
see how to go about them, at least design-wise.  "To start with" is a
start, but it cannot be the middle and the end.

> > Do we even understand well enough what are
> > the problems in CC Mode that get in the way?
> 
> Who do you think is most qualified to study that issue? We'd probably 
> have to convince Alan to do that as well, unfortunately.

If he agrees, that'd be swell.  If not, I see no reason why someone
else couldn't do that and report back, it's not like CC Mode is hard
to activate and see what problems pop up.

> I have a rough understanding of the issue, but since I haven't reached a 
> working state, I don't know how many pitfalls there are left.

How about describing what you do know?

> I imagine the process itself might be trickier than expected. Various 
> primitives use caches that save context information. What is such 
> primitive to do if the cache contains "beginning of nesting" outside of 
> the current restriction, and the logic of said primitive says "go to the 
> beginning of the current function and do such and such"? The answer 
> isn't obvious to me.

I don't understand: AFAIU in the use cases supported by MMM there can
be nothing outside of the current restriction that is of interest for
the mode which supports the chunk under the restriction.  So what kind
of "nesting outside of the current restriction" are we talking about,
and how does it come into play?

> The proposal itself is very small, and there's not much to explain. Just 
> look at the changes in the manual, in this branch.

That just removes text, more or less, and what it adds doesn't explain
itself well enough, sorry.  Otherwise I wouldn't have asked these
questions, believe me.

> It simply facilitates what mmm-mode (and other modes, including 
> mhtml-mode) has been doing for years, with varying success.

Sorry, that doesn't help me understand the proposal, as I don't know
mmm-mode well enough.  How hard is it to just describe the idea in a
few sentences?

> > Why do you
> > want to abandon the prog-widen stuff, which you supported just 2 years
> > ago?
> 
> I never actually supported it, just stopped arguing because I didn't 
> have a good alternative idea. Now I do, I think. And there is some 
> agreement from Stefan.

(whose change of hearts is also a mystery for me)

> > How do you address the issues which prog-indentation-context did
> > (e.g., if the embedded chunk of code is incomplete, and perhaps even
> > syntactically invalid)?
> 
> prog-indentation-context never addressed that issue, and we don't 
> either.

I think it does, at least to some extent.  And even if I'm wrong, then
what is the equivalent of what it does address in your proposal?

In any case, we should at least have provisions for supporting that
within the proposed framework, and we should be fairly certain that
supporting that later will not blow things up.

> > All these questions and considerations need
> > to be understood, because you are explicitly proposing an incomplete
> > solution, and asking us to agree with its deficiencies, but without
> > providing a clear picture of what are we going to give up on.
> 
> That's the thing: we're not giving up much

For the modes that are dear to your heart, maybe.  But that's not all
Emacs should support well, IMO.

> Consolidating the 'widen' calls is simply good engineering, even aside 
> from making life easier for MMM packages: it's much better to do that in 
> several well-defined places, instead of having every helper function do 
> it as well, like python-mode CC Mode do.

The 'widen' thing is a non-issue; it's the other stuff I'm worried
about, mostly because it is left unsaid, undiscussed, and I fear
uncatered-to by the design.

> > You
> > are still debating its design and implementation, even if we disregard
> > the CC Mode issue.
> 
> Not really. There's a minor issue of whether to make prog-first-column a 
> variable, or a hook right away, but the importance of that choice isn't big.

The fact is, it isn't done yet, let alone well-tested.

> > That code is in Emacs for more than 2 years.  It was admitted with
> > Stefan's full support, and at least ANTLR needs it in conjunction with
> > Python.
> 
> Transitioning from prog-indentation-context to the new approach is very 
> easy.

That's what I thought.  And that's why we should do that
simultaneously with landing the new approach.  There's no reason to do
that earlier.

> So really, it's been here for 2 years, and virtually nobody's using it, 
> or improving it.

It's definitely used by its author, so "nobody uses it" is untrue, and
even unfair: that feature was accepted by Stefan, under the assumption
that it will be used.

> > Removing it without having some alternative again makes no
> > sense to me. 
> 
> You have the alternative, though.

No, we don't.  You are asking to remove the code _before_ the
alternative lands.

> > We should discuss this when the incompatible code lands;
> 
> How about we discuss it now?

The incompatible code didn't land yet.  But feel free to include the
replacement in your branch.

> > at that time, we will see how to remove/replace prog-widen etc. with
> > minimal pain for its users.  For now, there's no incompatibility that
> > requires us to remove the code.  And anyway, making such changes when
> > we are in pretest is against our practices.
> 
> That seems very unwise to me. Not to mention discouraging.

I don't see anything unwise in it, let alone discouraging.  We've
turned down much simpler and safer changes that came too late, and no
one claimed that was unwise or discouraging.

> Take a look at any third-party packages. I'm willing to bet none use 
> prog-indentation-context yet.

We cannot know that.  Once the code is that long with us, we cannot
throw it away without a equivalent replacement.  And I see no reason
to do that now, since the fact that this code will exist in Emacs 26
doesn't hurt anyone, especially since the replacement will be easy, as
you say.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

> Details do matter.  I posted a full proposal, Subject: "Syntax
> ambiguities in narrowed buffers and multiple major modes: a proposed
> solution."  here in emacs-devel on Sat, 25 Feb 2017 13:53:56 +0000.

For the reasons exposed before, contrary to Dmitry, I don't really care
about seeing a detailed proposal.  I know full well that I'm absolutely
unable to run such the paper-design in my head and predict if it'll
really work.

> I don't rate restrictions on the use of Emacs primitives to be minor.

I think there's a fundamental misunderstanding abut Dmitry's suggestion
here: his proposal does not restrict the use of any primitive anywhere.

The only thing it does is to make the calling convention of
indent-line-function a bit more precise: instead of not saying anything
about the kind of narrowing that can be in place when that is called, it
specifies that the caller should arrange the narrowing to expose "all"
the code, and so the callee doesn't need to widen (which is useful since
in an MMM context, the sub-mode usually doesn't know how far it should
widen).

The indent-line-functions can still narrow/widen all it wants.
And the callers of indent-line-function also can still widen/narrow all
they want.  Just as is the case now, depending on how they narrow/widen
the result may be incorrect.  And the novelty here is to provide
a convention that, if followed, should make it more likely that the
result is correct, without the caller and the callee being aware of each
other, because they only care about the shared convention.

> If those common solution are adequate, yes.  But it is also critical to
> have the freedom to depart from things like syntax-propertize when they
> are inadequate, or too awkward.  syntax-propertize is inadequate for CC
> Mode:
>
> E.g., in C++ Mode, with
>
>     foo (a < b, c > d);
>
> we have a constructor foo with one parameter d, of a templated type.  The
> characters "<" and ">" have syntax-table text properties giving them
> paren syntaxes and making them match each other.
>
> Now, mark the characters ", c > d" and kill them with C-w.  What we are
> left with is a function call with an integer argument a < b:
>
>     foo (a < b);
>
> .  Something, somewhere has to remove the syntax-table property from
> "<", since it is no longer a template delimiter.
>
> The syntax-propertize mechanism won't do this, since the place to
> de-propertise is before BEG.

syntax-propertize is no silver bullet, indeed, so it won't do everything
magically for you.  But, it can handle the above situation just fine.
In your very specific above example, it will actually be handled
magically simply by virtue of syntax-propertize-wholelines being
included in the default value of syntax-propertize-extend-region-functions.

Now, of course, if you tweak your example so that the < and the > are
not on the same line any more, that magic won't work.  But that doesn't
mean you can't make syntax-propertize handle it right.  You could for
example solve it with:

    (add-hook 'syntax-propertize-extend-region-functions
              #'syntax-propertize-multiline nil t)

in your major mode function and

    (put-text-property position-of-left-< position-of-right->
                       'syntax-multiline t)

when you find (and put the syntax-table property on) the <...> pair.
Of course, it's not the only possible way to do it.  It's just the way
I solved similar problems in perl-mode and sh-mode.

If you're interested in making CC-mode work reliably with
syntax-propertize I'd be *thrilled* to help you do that.

By design, there should be basically nothing syntax-propertize can't handle
(modulo bugs and within the limits of what we can do with syntax-tables
and syntax-table properties, obviously).  I have no doubt that the way
CC-mode works right now will occasionally be more efficient than the way
syntax-propertize would do it.  But I also have no doubt that the way
syntax-propertize would do it will occasionally by more efficient than
the way CC-mode does it right now.  I strongly doubt that the difference
in performance behavior will ever matter, tho.


        Stefan

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Alan Mackenzie]

Hello, Dmitry.

On Sun, Dec 03, 2017 at 16:35:23 +0000, Dmitry Gutov wrote:
> On 12/3/17 3:28 PM, Eli Zaretskii wrote:

> > Because every development environment should first support its own
> > development well.  If it doesn't do that, it's incomplete.

> OK, I see. But!

> This feature is *not* targeted at Emacs development.

> Please don't make me fly out to Scotland and chase Alan down with a 
> broomstick to make this development come through.

Please read my signature, sometime.

> > Indeed; and C-like languages are quite popular among them.

> Not in comparison to JS, CSS and HTML.

> > How will we look if we support JS and Ruby embedded in HTML, but not C
> > code embedded in Yacc grammars nor Awk snippets embedded in shell
> > scripts?  Both are quite frequent out there.  We will be laughed at.

> We'll be fine. And the actual support is still going to reside in 
> third-party packages for now. This change just makes them possible (or 
> their codebases saner, at least).

> Except for antlr-mode. It will be able to use it right away (the version 
> inside Emacs, at least).

> > Convince Alan to do what?

> To adhere to the current proposal (avoid widening in 
> indent-line-function and font-lock-keywords, to start with).

I know Eli has asked to move on from the emotional to the technical
stuff, but that last paragraph belongs to the former, and I will deal
with it as such.

Maybe I missed it, but I don't recall anybody posting the suggestion
"Hey, we're considering writing an MMM mode which will mean nobody else
can use narrowing and widening, what do people think?".  If somebody had
posted that, there's a good chance that discussions could have sorted
out the technical stuff without provoking bad feelings.

Right now, I am being painted as the bad person for objecting to being
restricted to a subset of Emacs Lisp for CC Mode.  Forgive me for
feeling a bit peeved, but despite being in Emacs development for around
15 years, nobody asked me beforehand what I thought of this.  It is yet
one more thing that is presented as a done thing and foisted on Emacs
developers without them having any say in the matter.

You mentioned today, I think, that writing an MMM is hard.  Well, CC
Mode is hard, too.  There are 30 calls to `widen' in CC Mode and 47 to
`narrow-to-region'.  They are all there for a reason.  It will be
grinding tedious work to sort out the whys and to remove them.  You
categorise the injunction against widening as "only applying to
indent-line-function and "font-lock-keywords" (which isn't a function)".
This is from my point of view non-sensical since so many of the `widen'
calls are in low level functions called from "everywhere".

Yesterday, Richard Stallman suggested as an alternative to the
purloining of `widen' and `narrow-to-region' that a new "region" be
implemented somehow which would be independent of the existing region
and used solely by MMM super-modes.  How about exploring this
possibility?

Last February, I suggested extensions to the syntax code ("syntactic
islands") which would allow operations such as parse-partial-sexp to
work essentially without restriction in buffers with several syntax
tables.  How about exploring this possibility?

> > Do we even understand well enough what are
> > the problems in CC Mode that get in the way?

> Who do you think is most qualified to study that issue? We'd probably 
> have to convince Alan to do that as well, unfortunately.

Believe it or not, I am in favour of CC Mode working in an MMM mode.

> I have a rough understanding of the issue, but since I haven't reached a 
> working state, I don't know how many pitfalls there are left.

Neither do I.  But RMS's "new region" and my "syntax islands" may be a
more satisfactory way of resolving them.

> > About the only thing spelled out in this discussion was the need not
> > to widen.  Personally, I think this is not a grave issue, just some
> > technical problem that is certainly solvable (you proposed one such
> > solution).

> Indeed.

> > But is that all?

> I imagine the process itself might be trickier than expected. Various 
> primitives use caches that save context information. What is such 
> primitive to do if the cache contains "beginning of nesting" outside of 
> the current restriction, and the logic of said primitive says "go to the 
> beginning of the current function and do such and such"? The answer 
> isn't obvious to me.

Again, looking at RMS's suggested "new region" might help, here.

> But it's going to need to be answered anyway, no matter which MMM 
> approach we choose in the end, I think.

[ .... ]

> Consolidating the 'widen' calls is simply good engineering, even aside 
> from making life easier for MMM packages: it's much better to do that in 
> several well-defined places, instead of having every helper function do 
> it as well, like python-mode CC Mode do.

No, it isn't "simply good engineering" at all.  It's a different
approach with its pros and cons, just as widening at the immediate place
of need is.  And "consolidating" is probably the wrong word.  The number
of `widen's will probably grow, since every conceivable entry point
which could possibly call the pertinent primitve needs a `widen'.

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> Date: Sun, 3 Dec 2017 18:59:46 +0000
> Cc: Eli Zaretskii <eliz@gnu.org>, tom@tromey.com, monnier@iro.umontreal.ca,
>   spinuvit@gmail.com, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > > Convince Alan to do what?
> 
> > To adhere to the current proposal (avoid widening in 
> > indent-line-function and font-lock-keywords, to start with).
> 
> I know Eli has asked to move on from the emotional to the technical
> stuff, but that last paragraph belongs to the former, and I will deal
> with it as such.

And I would _really_ want to move to the technical stuff.  So...  I
actually don't understand all the fuss about widening.  With a chunk
of C code embedded in something that is not C, CC Mode cannot possibly
need to look outside of the chunk, so why would you need to widen
beyond that?

> You mentioned today, I think, that writing an MMM is hard.  Well, CC
> Mode is hard, too.  There are 30 calls to `widen' in CC Mode and 47 to
> `narrow-to-region'.  They are all there for a reason.  It will be
> grinding tedious work to sort out the whys and to remove them.

Is that the only reason for your objections?  The need to tediously go
through a few dozen of calls to 'widen' and 'narrow-to-region', and
take care of each one of them as appropriate?  Or are there more
substantial problems in adapting CC Mode to non-widening MMM?

> so many of the `widen' calls are in low level functions called from
> "everywhere".

This can be taken care of with a more-or-less trivial wrapper and a
variable.  Right?

> Yesterday, Richard Stallman suggested as an alternative to the
> purloining of `widen' and `narrow-to-region' that a new "region" be
> implemented somehow which would be independent of the existing region
> and used solely by MMM super-modes.  How about exploring this
> possibility?

We can do that if needed, but I don't see the need yet.  (And a
proposal to do something along these lines was put forward by Vitalie
some time ago.)

> Last February, I suggested extensions to the syntax code ("syntactic
> islands") which would allow operations such as parse-partial-sexp to
> work essentially without restriction in buffers with several syntax
> tables.

What problems does this solve?  The problem with 'widen' or some other
problem(s)?

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/3/17 5:20 PM, Eli Zaretskii wrote:

>> This feature is *not* targeted at Emacs development.
> 
> It is for me.

You are not working on it. You've expressed no interest in this problem 
until now. Maybe delegate the basic requirements, at least, to the 
people who did?

>>> Indeed; and C-like languages are quite popular among them.
>>
>> Not in comparison to JS, CSS and HTML.
> 
> Maybe in your world, but not in mine.  So we will have to agree (or
> disagree, if you want) that both worlds need to be catered to.

Yes, and they can be catered to, *as soon as* the authors of each 
individual major mode make the effort to support the feature.

Having it work in HTML, CSS, JS and Ruby is in no way dependent in 
investigating and fixing the behemoth that CC Mode is.

>>> How will we look if we support JS and Ruby embedded in HTML, but not C
>>> code embedded in Yacc grammars nor Awk snippets embedded in shell
>>> scripts?  Both are quite frequent out there.  We will be laughed at.
>>
>> We'll be fine.
> 
>  From your POV, maybe.  Not from mine.

Eli, we already have a feature in Emacs (prog-indentation-context) that 
does not adhere to your (unreasonably high) requirements.

Let's at least take it out, then.

>>> Convince Alan to do what?
>>
>> To adhere to the current proposal (avoid widening in
>> indent-line-function and font-lock-keywords, to start with).
> 
> We need to understand the full extent of problems, and then we need to
> see how to go about them, at least design-wise.  "To start with" is a
> start, but it cannot be the middle and the end.

The full extent of the problem can be realized as one tried to do what I 
said, step by step. And see what else can be is going wrong.

> If he agrees, that'd be swell.  If not, I see no reason why someone
> else couldn't do that and report back, it's not like CC Mode is hard
> to activate and see what problems pop up.

Please go ahead and do it, then.

If you're really expecting me to do it, it's unrealistic. Getting 
oriented in CC Mode will take days, if not weeks. And I have no interest 
nor energy to do that.

>> I have a rough understanding of the issue, but since I haven't reached a
>> working state, I don't know how many pitfalls there are left.
> 
> How about describing what you do know?

Already did. Last time I tried investigating it was 1-2 years ago, and 
at no point I felt I reached an understanding of why CC Mode exhibits 
the problems it does, or what I could change to fix them.

So all I have are vague recollections.

>> I imagine the process itself might be trickier than expected. Various
>> primitives use caches that save context information. What is such
>> primitive to do if the cache contains "beginning of nesting" outside of
>> the current restriction, and the logic of said primitive says "go to the
>> beginning of the current function and do such and such"? The answer
>> isn't obvious to me.
> 
> I don't understand: AFAIU in the use cases supported by MMM there can
> be nothing outside of the current restriction that is of interest for
> the mode which supports the chunk under the restriction.  So what kind
> of "nesting outside of the current restriction" are we talking about,
> and how does it come into play?

Imagine if the chunk contains only this:

    <

And, to fontify it, CC Mode, tried to find out what syntactic element it 
is, and to do that, calls some function that uses a specialized cache 
that's contained in a variable that mmm-mode knows nothing about.

And that cache point to a position outside of the current restriction. 
Confusion ensues. I've seen errors originating from that (I think), but 
can't recall the exact sequence of calls.

>> The proposal itself is very small, and there's not much to explain. Just
>> look at the changes in the manual, in this branch.
> 
> That just removes text, more or less, and what it adds doesn't explain
> itself well enough, sorry.  Otherwise I wouldn't have asked these
> questions, believe me.

The text it adds is more important. Including the mentions of 
font-lock-keywords and syntax-propertize-function, by the way.

>> It simply facilitates what mmm-mode (and other modes, including
>> mhtml-mode) has been doing for years, with varying success.
> 
> Sorry, that doesn't help me understand the proposal, as I don't know
> mmm-mode well enough.  How hard is it to just describe the idea in a
> few sentences?

Not really sure what to describe.

To fontify (or indent) a chunk, mmm-mode narrows to its bounds, and 
calls the corresponding indent-line-function (or applies 
font-lock-keywords). If either of those calls (widen), that causes problems.

So we want to document a convention that they don't call widen. That's 
really it.

>> I never actually supported it, just stopped arguing because I didn't
>> have a good alternative idea. Now I do, I think. And there is some
>> agreement from Stefan.
> 
> (whose change of hearts is also a mystery for me)

Sometimes a better technical proposal does that.

For more context: we're been waiting for some kind of improvement for 
this problem for years. Then prog-indentation-context came along, and it 
wasn't terrible (though it didn't solve all of the problems we wanted). 
That seemed a good enough reason to adopt it, to some of us.

Since then, for the mentioned two years, the original author contributed 
no improvements aside from the mentioned support in python-mode, and no 
other mode added support for it, or used it, inside or outside Emacs. So 
it seems to me like it pretty much failed.

>>> How do you address the issues which prog-indentation-context did
>>> (e.g., if the embedded chunk of code is incomplete, and perhaps even
>>> syntactically invalid)?
>>
>> prog-indentation-context never addressed that issue, and we don't
>> either.
> 
> I think it does, at least to some extent.

Not any more than scratch/widen-less.

> And even if I'm wrong, then
> what is the equivalent of what it does address in your proposal?

There are full equivalents.

We keep 'prog-first-column' from that proposal, and instead of allowing 
MMM to indicate the chunk bounds through prog-indentation-context, we 
allow MMM to apply narrowing directly, and that modes honor it.

> In any case, we should at least have provisions for supporting that
> within the proposed framework, and we should be fairly certain that
> supporting that later will not blow things up.

I have honestly no idea where to begin in addressing that issue.

On the other hand, there are third-party MMM framework that function 
within these limitations, and are helpful to users.

>> That's the thing: we're not giving up much
> 
> For the modes that are dear to your heart, maybe.  But that's not all
> Emacs should support well, IMO.

For all modes. The user won't lose any functionality, at all. Even if 
the programmer might have to spend some effort for that to happen, for 
some of the modes.

> The 'widen' thing is a non-issue; it's the other stuff I'm worried
> about, mostly because it is left unsaid, undiscussed, and I fear
> uncatered-to by the design.

What other stuff? We basically say "widen happens in one place, don't do 
that again", and that's it.

>> Not really. There's a minor issue of whether to make prog-first-column a
>> variable, or a hook right away, but the importance of that choice isn't big.
> 
> The fact is, it isn't done yet, let alone well-tested.

It can be "done" in a day or two.

>>> That code is in Emacs for more than 2 years.  It was admitted with
>>> Stefan's full support, and at least ANTLR needs it in conjunction with
>>> Python.
>>
>> Transitioning from prog-indentation-context to the new approach is very
>> easy.
> 
> That's what I thought.  And that's why we should do that
> simultaneously with landing the new approach.  There's no reason to do
> that earlier.

Why confuse the language modes authors?

>> So really, it's been here for 2 years, and virtually nobody's using it,
>> or improving it.
> 
> It's definitely used by its author, so "nobody uses it" is untrue, and
> even unfair: that feature was accepted by Stefan, under the assumption
> that it will be used.

...and improved, and support for CC Mode would be contributed. Which it 
wasn't.

The author, who's the only user of prog-indentation-context, can 
transition to the new convention trivially.

>>> Removing it without having some alternative again makes no
>>> sense to me.
>>
>> You have the alternative, though.
> 
> No, we don't.  You are asking to remove the code _before_ the
> alternative lands.

You have the choice of accepting the alternative sooner, though.

>>> We should discuss this when the incompatible code lands;
>>
>> How about we discuss it now?
> 
> The incompatible code didn't land yet.  But feel free to include the
> replacement in your branch.

It's in there already.

>> Take a look at any third-party packages. I'm willing to bet none use
>> prog-indentation-context yet.
> 
> We cannot know that.  Once the code is that long with us, we cannot
> throw it away without a equivalent replacement.  And I see no reason
> to do that now, since the fact that this code will exist in Emacs 26
> doesn't hurt anyone, especially since the replacement will be easy, as
> you say.

Well... you have a point there. The result will be pretty ugly, though. 
Adding an API in one version, and removing it in another.

They are incompatible with each other, so it's not like there can be a 
smooth transition.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Alan Mackenzie]

Hello, Eli.

On Sun, Dec 03, 2017 at 21:25:09 +0200, Eli Zaretskii wrote:
> > Date: Sun, 3 Dec 2017 18:59:46 +0000
> > Cc: Eli Zaretskii <eliz@gnu.org>, tom@tromey.com, monnier@iro.umontreal.ca,
> >   spinuvit@gmail.com, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > > Convince Alan to do what?

> > > To adhere to the current proposal (avoid widening in 
> > > indent-line-function and font-lock-keywords, to start with).

> > I know Eli has asked to move on from the emotional to the technical
> > stuff, but that last paragraph belongs to the former, and I will deal
> > with it as such.

> And I would _really_ want to move to the technical stuff.  So...  I
> actually don't understand all the fuss about widening.  With a chunk
> of C code embedded in something that is not C, CC Mode cannot possibly
> need to look outside of the chunk, so why would you need to widen
> beyond that?

There is a style of coding called "literate programming" (I know nothing
more about it) where pieces of C code (in C Mode, obviously) alternate
with explanatory text (in text mode).  If point is at the start of a C
defun, C-M-a will have to take it to either the start of the text mode
block or the previous C defun.  Either way CC Mode needs to access
buffer pieces outside the current chunk.

> > You mentioned today, I think, that writing an MMM is hard.  Well, CC
> > Mode is hard, too.  There are 30 calls to `widen' in CC Mode and 47 to
> > `narrow-to-region'.  They are all there for a reason.  It will be
> > grinding tedious work to sort out the whys and to remove them.

> Is that the only reason for your objections?  The need to tediously go
> through a few dozen of calls to 'widen' and 'narrow-to-region', and
> take care of each one of them as appropriate?  Or are there more
> substantial problems in adapting CC Mode to non-widening MMM?

No, my main reason is philosophical.  In Lisp, we're supposed to have
the entire language available at read, evaluation, and print time.  With
this proposal, that notion becomes complicated and ugly, once you
qualify how everything but `widen' and `narrow-to-region' are available.

Using the region to do this is an abuse.  This isn't what point-min and
point-max were designed for.  Programmers are going to carry on using
`widen', and this will lead to possibly difficult to debug bugs.  I am
sure that this use of the region is going to lead to other nasty
problems, even though I can't forsee in any detail what these will be.

> > so many of the `widen' calls are in low level functions called from
> > "everywhere".

> This can be taken care of with a more-or-less trivial wrapper and a
> variable.  Right?

I suppose so, but it will degrade such a function effectively into two,
one special purpose function which can only be called from
beginning-of-defun-function (etc.) and the other one called from
everywhere else.  There will be confusion between them.

> > Yesterday, Richard Stallman suggested as an alternative to the
> > purloining of `widen' and `narrow-to-region' that a new "region" be
> > implemented somehow which would be independent of the existing region
> > and used solely by MMM super-modes.  How about exploring this
> > possibility?

> We can do that if needed, but I don't see the need yet.

Then we won't do it at all.  Once all the effort has been put into
making MMM work with the region, nobody will bother with putting the
(substantial) effort into "new region".  We'll lose the opportunity for
a possibly better solution.

> (And a proposal to do something along these lines was put forward by
> Vitalie some time ago.)

I didn't know that.  Respect, Vitalie!

> > Last February, I suggested extensions to the syntax code ("syntactic
> > islands") which would allow operations such as parse-partial-sexp to
> > work essentially without restriction in buffers with several syntax
> > tables.

> What problems does this solve?  The problem with 'widen' or some other
> problem(s)?

The idea is to render the use of the region by MMM unnecessary, though
it will need other code, too.  The need to use parse-partial-sexp in any
progmode buffer seems obvious, and having to do this in bits (as at the
moment) seems an obvious enough inconvenience to make syntactic islands
worthwhile.  To be honest, I've no exact use case for them at the
moment.  They would allow, for example, syntax-ppss to work in an entire
MMM buffer.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

>> No.  Can we put mmm-mode into elpa.git?
> I'll try emailing the original authors again. The last time I did that there
> were no replies, unfortunately.

I really think we should have such a functionality in elpa.git or
emacs.git, so the other option is a new mmm-like thingy, maybe based on
mhtml-mode or otherwise started from pure unadulterated scratch.


        Stefan

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

> Convince Alan to do what?  Do we even understand well enough what are
> the problems in CC Mode that get in the way?

Yes.

> About the only thing
> spelled out in this discussion was the need not to widen.  Personally,
> I think this is not a grave issue, just some technical problem that is
> certainly solvable (you proposed one such solution).

Indeed.

> But is that all?  You mentioned some other problems, but never
> described them.  What are they?  Are they grave enough to prevent your
> proposal from ever supporting CC Mode,

No, it's just a small matter of adjusting CC-mode here and there.
But given that the only person who understands this part of CC-mode well
enough is Alan, it's difficult for anyone but Alan to do it (I think
I could do it as well, but I know that my solution wouldn't please Alan
because it would involve switching over to syntax-propertize, so my
motivation to do it is rather low).

> even if your proposal is amended?

No need to amend anything.

> How do you address the issues which prog-indentation-context did
> (e.g., if the embedded chunk of code is incomplete, and perhaps even
> syntactically invalid)?

We can easily provide a similar functionality with Dmitry's design.
Dmitry hasn't bothered doing that, because so far it's never been used
(so it's not even clear whether it's ever needed nor whether it would
provide the needed info).

> cause it to end up like all the previous ones: either committed to
> Emacs and basically unused, or, worse, collecting dust in long
> forgotten patches or branches?

His proposal is really really small, and matches what is already done
in mmm-mode and mhtml-mode, so it's actually the solution that's least
likely to collect dust, because it's already in use.

> Emacs 26 has enough new features to justify a major release, so I see
> no reason to press so hard to get this feature into it as well.

I think the failure here is to describe clearly the proposal.
The core of the proposal, AFAIK, is the following:

    state that indent-line-function can rely on the current narrowing to
    find all the relevant text, because the caller has already widened
    when necessary for that.

To make it true, we need a few changes in the code, mostly we need to
tweak indent-according-to-mode so that it widens before calling
indent-line-function.  This change should be harmless because
even if the user has narrowed the buffer, the indentation should always
be done depending on the whole buffer's content anyway.

>> It's not well supported, and it's incompatible with the code we are 
>> discussing. There's no point in having it present in Emacs 26, and then 
>> removing it in Emacs 27.
> That code is in Emacs for more than 2 years.

It's been in emacs.git's master, yes, but not in any released version of Emacs.

> It was admitted with Stefan's full support, and at least ANTLR needs
> it in conjunction with Python.

These can very easily be adjusted to rely on the new convention.


        Stefan

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

> You mentioned today, I think, that writing an MMM is hard.  Well, CC
> Mode is hard, too.  There are 30 calls to `widen' in CC Mode and 47 to
> `narrow-to-region'.

The calls to narrow-to-region don't need to be investigated.  Only those
to `widen'.


        Stefan

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Alan Mackenzie]

Hello, Stefan.

On Sun, Dec 03, 2017 at 13:40:22 -0500, Stefan Monnier wrote:

[ .... ]

> > I don't rate restrictions on the use of Emacs primitives to be minor.

> I think there's a fundamental misunderstanding abut Dmitry's suggestion
> here: his proposal does not restrict the use of any primitive anywhere.

You say there's no restriction happening, then go on to detail the
restriction that is to be put in place.  ;-)

> The only thing it does is to make the calling convention of
> indent-line-function a bit more precise: instead of not saying anything
> about the kind of narrowing that can be in place when that is called, it
> specifies that the caller should arrange the narrowing to expose "all"
> the code, and so the callee doesn't need to widen (which is useful since
> in an MMM context, the sub-mode usually doesn't know how far it should
> widen).

Which is why I proposed the "syntactic islands", which would allow code
to do parse-partial-sexp (or even syntax-ppss).  Typically the code does
need to know about "distant" chunks.  For example if point is after an
embedded AWK Mode script in shell-script-mode, C-M-a will need access to
the buffer part before the AWK Mode interlude.

> The indent-line-functions can still narrow/widen all it wants.
> And the callers of indent-line-function also can still widen/narrow all
> they want.  Just as is the case now, depending on how they narrow/widen
> the result may be incorrect.  And the novelty here is to provide
> a convention that, if followed, should make it more likely that the
> result is correct, without the caller and the callee being aware of each
> other, because they only care about the shared convention.

> > If those common solution are adequate, yes.  But it is also critical to
> > have the freedom to depart from things like syntax-propertize when they
> > are inadequate, or too awkward.  syntax-propertize is inadequate for CC
> > Mode:

> > E.g., in C++ Mode, with

> >     foo (a < b, c > d);

> > we have a constructor foo with one parameter d, of a templated type.  The
> > characters "<" and ">" have syntax-table text properties giving them
> > paren syntaxes and making them match each other.

> > Now, mark the characters ", c > d" and kill them with C-w.  What we are
> > left with is a function call with an integer argument a < b:

> >     foo (a < b);

> > .  Something, somewhere has to remove the syntax-table property from
> > "<", since it is no longer a template delimiter.

> > The syntax-propertize mechanism won't do this, since the place to
> > de-propertise is before BEG.

> syntax-propertize is no silver bullet, indeed, so it won't do everything
> magically for you.  But, it can handle the above situation just fine.

I don't see that.

> In your very specific above example, it will actually be handled
> magically simply by virtue of syntax-propertize-wholelines being
> included in the default value of syntax-propertize-extend-region-functions.

> Now, of course, if you tweak your example so that the < and the > are
> not on the same line any more, that magic won't work.  But that doesn't
> mean you can't make syntax-propertize handle it right.  You could for
> example solve it with:

>     (add-hook 'syntax-propertize-extend-region-functions
>               #'syntax-propertize-multiline nil t)

All constructs in CC Mode languages are "multiline".

> in your major mode function and

>     (put-text-property position-of-left-< position-of-right->
>                        'syntax-multiline t)

> when you find (and put the syntax-table property on) the <...> pair.

Why not just put that property on the entire buffer?  Or, if
syntax-multiline is intended to flag up just one construct, it won't cope
well with nested (or overlapping) constructs.

> Of course, it's not the only possible way to do it.  It's just the way
> I solved similar problems in perl-mode and sh-mode.

How is that going to help when a closing template delimiter is deleted?

> If you're interested in making CC-mode work reliably with
> syntax-propertize I'd be *thrilled* to help you do that.

> By design, there should be basically nothing syntax-propertize can't handle
> (modulo bugs and within the limits of what we can do with syntax-tables
> and syntax-table properties, obviously).

One problem is the vagueness in the documentation of
syntax-propertize-extend-region-functions.  Are these functions called in
before-change-functions or a-c-functions?  Obviously, in my case above,
they would be needed in b-c-f, otherwise they wouldn't notice the ">"
being killed.  But they need to be called in a-c-f too, in case the
change is adding "> >", when the code needs to search back to the earlier
two "<"s, which now have become "dirty" when they weren't previously.

> I have no doubt that the way CC-mode works right now will occasionally
> be more efficient than the way syntax-propertize would do it.  But I
> also have no doubt that the way syntax-propertize would do it will
> occasionally by more efficient than the way CC-mode does it right now.

syntax-propertize splats all syntax-table properties after a change
point, if I understand correctly.  That means they must at some point
soon be regenerated[*].  This will be expensive in a large buffer.

[*] imagine moving point forward to just before a "<" which (along with
its matching ">") needs to have paren s-t. properties to allow
paren-matching to happen.  I can't see what is putting those properties
on it, given the ">" could be an arbitrary distance forward.

> I strongly doubt that the difference in performance behavior will ever
> matter, tho.

Several differences in performance that "hardly matter" can add up to
something substantial.  I recently celebrated a 3.9% increase in
fontification performance after deleting some old duplicate code.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

>> > I don't rate restrictions on the use of Emacs primitives to be minor.
>> I think there's a fundamental misunderstanding abut Dmitry's suggestion
>> here: his proposal does not restrict the use of any primitive anywhere.
> You say there's no restriction happening, then go on to detail the
> restriction that is to be put in place.  ;-)

I have no idea what you're talking about.  Where did you see a restriction?
There are only guidelines, no restrictions.

> Typically the code does need to know about "distant" chunks.

The years of experience with mmm-mode, as well as the more recent
experience with mhtml-mode disagree with this.

> For example if point is after an embedded AWK Mode script in
> shell-script-mode, C-M-a will need access to the buffer part before
> the AWK Mode interlude.

This already works and won't be affected by the proposal.

>> (add-hook 'syntax-propertize-extend-region-functions
>> #'syntax-propertize-multiline nil t)
> All constructs in CC Mode languages are "multiline".

So what?

>> when you find (and put the syntax-table property on) the <...> pair.
> Why not just put that property on the entire buffer?  Or, if
> syntax-multiline is intended to flag up just one construct, it won't cope
> well with nested (or overlapping) constructs.

As I said, you don't have to use that.  You can write your own
syntax-propertize-extend-region-function any which way you like.

>> Of course, it's not the only possible way to do it.  It's just the way
>> I solved similar problems in perl-mode and sh-mode.
> How is that going to help when a closing template delimiter is deleted?

It will cause the re-propertization to be done from the matching
opening delimiter (at least, if you've set things up correctly, of
course: syntax-propertize itself has no idea about those delimiters).

> One problem is the vagueness in the documentation of
> syntax-propertize-extend-region-functions.  Are these functions called in
> before-change-functions or a-c-functions?  Obviously, in my case above,
> they would be needed in b-c-f, otherwise they wouldn't notice the ">"
> being killed.

The text between < and > is marked, what when BEG falls in the middle, 
syntax-propertize-extend-region-functions will know without having to
have access to the ">" (so it doesn't matter if it's already been
deleted or not).

> But they need to be called in a-c-f too, in case the
> change is adding "> >", when the code needs to search back to the earlier
> two "<"s, which now have become "dirty" when they weren't previously.

This kind of complexity disappears in syntax-propertize's approach.

> syntax-propertize splats all syntax-table properties after a change
> point, if I understand correctly.  That means they must at some point
> soon be regenerated[*].  This will be expensive in a large buffer.

This *can* be expensive, but they're only regenerated lazily, so
typically it's not a problem.  We have many years of experience in many
different major modes to say that it's likely not to be a problem.

> Several differences in performance that "hardly matter" can add up to
> something substantial.

"can", indeed.  My experience makes me confident it won't happen.


        Stefan

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/3/17 3:38 AM, Eli Zaretskii wrote:

>> Can we put mmm-mode into elpa.git?
> 
> Even if it's possible, it makes absolutely no sense to me to have such
> a central feature on ELPA and not in the core.

Maybe if we grow the "bundled packages from ELPA" feature.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/3/17 2:54 PM, Alan Mackenzie wrote:

> Details do matter.  I posted a full proposal, Subject: "Syntax
> ambiguities in narrowed buffers and multiple major modes: a proposed
> solution." here in emacs-devel on Sat, 25 Feb 2017 13:53:56 +0000.
> 
> I've changed my mind about some of the details, partly in response to
> what people said, partly because I see things now I didn't see then.

Thanks. I've read it, and you've seen my comments.

>> I'm not convinced "several syntax tables" would be enough, though.
> 
> OK, let's make that "many syntax tables".  :-)

TBH, it's hard for me to tell with confidence either way. But here's the 
kind of syntax that Ruby allows:

a = %Q( #{ %q{abc} + %q{ "#{"abc #{ %q[ abc ] }" }" } } )

These are all nested literals (with the exception of #{}, which is 
interpolation). Thus exiting from a literal requires the knowledge of 
all openers that have been used before it, and a pre-defined set of 
syntax table (if I understood you correctly) won't do.

>> The major modes will have to change to allow for this support, that's
>> for sure. You can't expect no code changes.
> 
> Maybe, maybe not.  Super-modes would definitely have to change to use
> it.  Any changes in major modes should be minor.  (I don't rate
> restrictions on the use of Emacs primitives to be minor.)

Major modes will have to work inside arbitrary restrictions one way or 
another. That's the crucial part, and it will require changes.

>> The current idea is "let's do the minimum coherent step forward, so that
>> somebody can continue improving things later". A step that's easy to
>> use, and also easy to undo later, if we find something else that's much
>> better.
> 
> OK.  With such a tricky subject matter, it would surprising if we
> couldn't find improvements for a long time.

Since it's tricky, I would bet in the opposite direction. But there's no 
way to predict.

>> Coming up with it was not easy at all, BTW.
> 
> Yes, I can see that.  Well done!

If you are not being sarcastic, thanks! Could you please stop opposing it?

>> It's simply description of good engineering. You can try to criticize
>> the details of syntax-propertize, but the idea of reusing common
>> solutions is more than sound.
> 
> If those common solution are adequate, yes.  But it is also critical to
> have the freedom to depart from things like syntax-propertize when they
> are inadequate, or too awkward.  syntax-propertize is inadequate for CC
> Mode:

Sorry, I'd rather you put it in a separate discussion. But, like Stefan 
described, this example is very much solvable with syntax-propertize 
(which I could have told you independently, too).

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/3/17 9:23 PM, Stefan Monnier wrote:

> I really think we should have such a functionality in elpa.git or
> emacs.git, so the other option is a new mmm-like thingy, maybe based on
> mhtml-mode or otherwise started from pure unadulterated scratch.

Agreed. But if extraction from mhtml-mode is expected from me, sorry, 
it's going to the bottom of the pile for now.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

Hi Stefan,

On 12/3/17 9:52 PM, Stefan Monnier wrote:

> No, it's just a small matter of adjusting CC-mode here and there.
> But given that the only person who understands this part of CC-mode well
> enough is Alan, it's difficult for anyone but Alan to do it (I think
> I could do it as well, but I know that my solution wouldn't please Alan
> because it would involve switching over to syntax-propertize, so my
> motivation to do it is rather low).

Maybe you understand the issue better than me. Or maybe it's the 
conversion to syntax-propertize that would do away with the extra caches 
that IIRC were giving me troubles.

>> How do you address the issues which prog-indentation-context did
>> (e.g., if the embedded chunk of code is incomplete, and perhaps even
>> syntactically invalid)?
> 
> We can easily provide a similar functionality with Dmitry's design.
> Dmitry hasn't bothered doing that, because so far it's never been used
> (so it's not even clear whether it's ever needed nor whether it would
> provide the needed info).

To be clear, I'm not sure what you mean. But then, the problem/question 
isn't well-formulated here.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/3/17 6:59 PM, Alan Mackenzie wrote:

> Please read my signature, sometime.

I sincerely apologize. Fly out to Germany, of course.

> Maybe I missed it, but I don't recall anybody posting the suggestion
> "Hey, we're considering writing an MMM mode which will mean nobody else
> can use narrowing and widening, what do people think?".  If somebody had
> posted that, there's a good chance that discussions could have sorted
> out the technical stuff without provoking bad feelings.

Erm, mmm-mode is 17 years old, originally written by a graduate student 
to help with literate programming (AFAIK). Did you mean they should have 
asked back then? And in the intervening years, several other packages 
took a similar approach.

And if you mean that I should have asked specifically, I did bring up it 
in a discussion... a year ago or so. And no realistic and complete 
counter-proposal came up, as far as I'm concerned.

> Right now, I am being painted as the bad person for objecting to being
> restricted to a subset of Emacs Lisp for CC Mode.  Forgive me for
> feeling a bit peeved, but despite being in Emacs development for around
> 15 years, nobody asked me beforehand what I thought of this.  It is yet
> one more thing that is presented as a done thing and foisted on Emacs
> developers without them having any say in the matter.

You've never worked on a mixed-mode package, though. So yes, I 
personally have a reason to assume that I have a better knowledge of the 
problem area.

But I never said that CC Mode must support it.

> You mentioned today, I think, that writing an MMM is hard.  Well, CC
> Mode is hard, too.  There are 30 calls to `widen' in CC Mode and 47 to
> `narrow-to-region'.  They are all there for a reason.  It will be
> grinding tedious work to sort out the whys and to remove them.

Still, that sounds manageable. And there's no hurry at all, as far as 
I'm concerned.

> You
> categorise the injunction against widening as "only applying to
> indent-line-function and "font-lock-keywords" (which isn't a function)".

I think you know what I meant anyway.

> Yesterday, Richard Stallman suggested as an alternative to the
> purloining of `widen' and `narrow-to-region' that a new "region" be
> implemented somehow which would be independent of the existing region
> and used solely by MMM super-modes.  How about exploring this
> possibility?

Vitalie proposed something like that years ago, and we asked for help 
with the C code. Nobody turned up, including yourself (and I asked 
directly).

Anyway, I don't see how it would be qualitatively better. The problem of 
being able to function in a restricted area of a buffer will still be there.

> Last February, I suggested extensions to the syntax code ("syntactic
> islands") which would allow operations such as parse-partial-sexp to
> work essentially without restriction in buffers with several syntax
> tables.  How about exploring this possibility?

Please go ahead and explore. I've done my part with my proposal (which 
was intentionally small and thus required small changes everywhere but 
CC Mode).

> Believe it or not, I am in favour of CC Mode working in an MMM mode.

You just don't want the protocol we are proposing. Did you participate 
in the prog-indentation-context discussion when it was proposed, by the way?

>> I have a rough understanding of the issue, but since I haven't reached a
>> working state, I don't know how many pitfalls there are left.
> 
> Neither do I.  But RMS's "new region" and my "syntax islands" may be a
> more satisfactory way of resolving them.

May be, or may not. We'd need to see the code.

By the way, again, you can add a "new region" function exclusively to CC 
Mode, and use it in the indentation function. It's trivial to implement, 
and will look awfully like the shim I proposed.

>> I imagine the process itself might be trickier than expected. Various
>> primitives use caches that save context information. What is such
>> primitive to do if the cache contains "beginning of nesting" outside of
>> the current restriction, and the logic of said primitive says "go to the
>> beginning of the current function and do such and such"? The answer
>> isn't obvious to me.
> 
> Again, looking at RMS's suggested "new region" might help, here.

See above.

But no, the problem I imagined (not 100% sure if it's real) won't be 
solved by "hard narrowing".

>> Consolidating the 'widen' calls is simply good engineering, even aside
>> from making life easier for MMM packages: it's much better to do that in
>> several well-defined places, instead of having every helper function do
>> it as well, like python-mode CC Mode do.
> 
> No, it isn't "simply good engineering" at all.  It's a different
> approach with its pros and cons, just as widening at the immediate place
> of need is.  And "consolidating" is probably the wrong word.  The number
> of `widen's will probably grow, since every conceivable entry point
> which could possibly call the pertinent primitve needs a `widen'.

No more than there are interactive functions. A lot of them have to do 
that anyway already. And then, you could also have helpers that aren't 
used in indentation code.

There are several ways to skin that cat.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> Cc: acm@muc.de, tom@tromey.com, spinuvit@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 3 Dec 2017 23:43:48 +0000
> 
> On 12/3/17 3:38 AM, Eli Zaretskii wrote:
> 
> >> Can we put mmm-mode into elpa.git?
> > 
> > Even if it's possible, it makes absolutely no sense to me to have such
> > a central feature on ELPA and not in the core.
> 
> Maybe if we grow the "bundled packages from ELPA" feature.

That would be a different Emacs and a different discussion.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/4/17 3:38 AM, Eli Zaretskii wrote:

>> Maybe if we grow the "bundled packages from ELPA" feature.
> 
> That would be a different Emacs and a different discussion.

Point is, putting mmm-mode in ELPA or not is the developer's choice.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> Cc: acm@muc.de, tom@tromey.com, monnier@iro.umontreal.ca, spinuvit@gmail.com,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 3 Dec 2017 19:43:31 +0000
> 
>         This feature is *not* targeted at Emacs development.
> 
>     It is for me.
> 
> You are not working on it. You've expressed no interest in this problem until now. Maybe delegate the basic requirements, at least, to the people who did?

Your work on this is highly appreciated.  But I have certain
responsibilities, and I'm trying to do my job here.

I'm also your fellow developer of the same project and its active
user.  I think it means my opinions should be of some importance, not
something to be ignored solely because I'm not coding this.  Just like
I would never dream of ignoring your opinions about features I'm
implementing.

As for expressing interest: you are mistaken when you judge that by my
silence -- it only means that I had nothing significant to say about
the issue, until I did.

> [C-like major modes] can be catered to, *as soon as* the authors of each individual major mode make the effort to support the feature.

Agreed.  What I'm trying to establish is whether the design supports
those major modes, or does it ignore them.  The latter I would like to
avoid as much as possible.

> Eli, we already have a feature in Emacs (prog-indentation-context) that does not adhere to your (unreasonably high) requirements.

Emacs development is not only about not losing existing features.  It
is mainly about gaining new important features.  Here you are adding a
feature that I think is important enough for us to try make it support
C-like languages.  I don't think this requirement is unreasonably
high, as those languages are still pretty much the mainstay of the
industry, and definitely used by many Emacs users.

> Imagine if the chunk contains only this:
> 
>    <
> 
> And, to fontify it, CC Mode, tried to find out what syntactic element it is, and to do that, calls some function that uses a specialized cache that's contained in a variable that mmm-mode knows nothing about.
> 
> And that cache point to a position outside of the current restriction. Confusion ensues. I've seen errors originating from that (I think), but can't recall the exact sequence of calls.

Thanks, so the caches will have to be sensitive to restrictions as
they are sensitive to deletion of text.

> To fontify (or indent) a chunk, mmm-mode narrows to its bounds, and calls the corresponding indent-line-function (or applies font-lock-keywords). If either of those calls (widen), that causes problems.
> 
> So we want to document a convention that they don't call widen. That's really it.

But given your example with CC Mode's caches, that's not all of it, is
it?  You also need the mode to adapt any such caches to changes in
buffer restrictions, right?

>             How do you address the issues which prog-indentation-context did
>             (e.g., if the embedded chunk of code is incomplete, and perhaps even
>             syntactically invalid)?
> 
>         prog-indentation-context never addressed that issue, and we don't
>         either.
> 
>     I think it does, at least to some extent.
> 
> Not any more than scratch/widen-less.

Maybe I'm missing something, but what's the equivalent of PREV-CHUNK?

> We keep 'prog-first-column' from that proposal

But it was a function, and you made it a variable.  This will get in
the way of compatibility, and also potentially will make future
extensions harder.  Why not keep it a function?

> and instead of allowing MMM to indicate the chunk bounds through prog-indentation-context, we allow MMM to apply narrowing directly, and that modes honor it.

This simplification also took away some of the features, like the
ability to nest restrictions.  I wonder why did you discard that.
It's existing functionality, pretty lightweight, which was in Emacs
for some time, and is reasonably well documented.  And it already
satisfied your requirement of not allowing sub-modes to widen.  Why
not leave it alone?  What am I missing?

>         Transitioning from prog-indentation-context to the new approach is very
>         easy.
> 
>     That's what I thought.  And that's why we should do that
>     simultaneously with landing the new approach.  There's no reason to do
>     that earlier.
> 
> Why confuse the language modes authors?

If they should be confused (I don't think so), they are already
confused, as we've been having this for the last 2 years.

>     We cannot know that.  Once the code is that long with us, we cannot
>     throw it away without a equivalent replacement.  And I see no reason
>     to do that now, since the fact that this code will exist in Emacs 26
>     doesn't hurt anyone, especially since the replacement will be easy, as
>     you say.
> 
> Well... you have a point there. The result will be pretty ugly, though. Adding an API in one version, and removing it in another.

We will have to discuss the "removal" part.  An alternative is to
provide compatibility shims, or even leave (some or all of) the
existing API intact.

> They are incompatible with each other, so it's not like there can be a smooth transition. 

I don't see the incompatibilities.  Basically, you replaced prog-widen
with widen, made prog-first-column a variable instead of a function,
and tossed prog-indentation-context.  We could instead keep the
existing API with minimal changes: prog-widen calls widen internally,
and we could allow direct calls to widen as well.

IOW, the transition could be much smoother than it will be if we
actually remove that stuff, because I don't think there's any
incompatibility here which would disallow direct calls to 'widen'
and/or leaving prog-indentation-context at its default nil value.  Or
maybe I'm missing something again.  But if I'm right, and if we can
make the necessary changes limited only to the documentation, then it
could well make it into Emacs 26.  So I think it is worth our while
to make some effort in that direction, if at all possible.

Thanks.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Alan Mackenzie]

Hello, Dmitry.

On Mon, Dec 04, 2017 at 00:37:51 +0000, Dmitry Gutov wrote:
> On 12/3/17 6:59 PM, Alan Mackenzie wrote:

[ .... ]

> You've never worked on a mixed-mode package, though. So yes, I 
> personally have a reason to assume that I have a better knowledge of the 
> problem area.

OK.

> But I never said that CC Mode must support it.

Any chance you could give me some basic details of how to get into this
MMM mode?  Like which git branch (? scratch/widen-less ?) can I find the
code in, and where is the best documentation to get started.

> > You mentioned today, I think, that writing an MMM is hard.  Well, CC
> > Mode is hard, too.  There are 30 calls to `widen' in CC Mode and 47 to
> > `narrow-to-region'.  They are all there for a reason.  It will be
> > grinding tedious work to sort out the whys and to remove them.

> Still, that sounds manageable. And there's no hurry at all, as far as 
> I'm concerned.

That's a good job!  I've no idea how long it will take to make MMM and
CC Mode work with eachother.

> Anyway, I don't see how it [ a new region, distinct from THE region ]
> would be qualitatively better. The problem of being able to function
> in a restricted area of a buffer will still be there.

I don't see that at the moment either, but perhaps that will become
evident.

> > Last February, I suggested extensions to the syntax code ("syntactic
> > islands") which would allow operations such as parse-partial-sexp to
> > work essentially without restriction in buffers with several syntax
> > tables.  How about exploring this possibility?

> Please go ahead and explore. I've done my part with my proposal (which 
> was intentionally small and thus required small changes everywhere but 
> CC Mode).

> > Believe it or not, I am in favour of CC Mode working in an MMM mode.

> You just don't want the protocol we are proposing. Did you participate 
> in the prog-indentation-context discussion when it was proposed, by the way?

Not much, if at all.  I found those discussions to be lacking the
context I would have needed to understand them.

> >> I have a rough understanding of the issue, but since I haven't reached a
> >> working state, I don't know how many pitfalls there are left.

> > Neither do I.  But RMS's "new region" and my "syntax islands" may be a
> > more satisfactory way of resolving them.

> May be, or may not. We'd need to see the code.

That's fair.  But the "syntax islands" proposal would be a lot of work,
which I don't want to commit to before I see some sort of undertaking to
take it seriously.

[ .... ]

> There are several ways to skin that cat.

That's the second cat in as many days, here!

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> Date: Sun, 3 Dec 2017 21:20:40 +0000
> Cc: tom@tromey.com, emacs-devel@gnu.org, monnier@iro.umontreal.ca,
>   spinuvit@gmail.com, dgutov@yandex.ru
> From: Alan Mackenzie <acm@muc.de>
> 
> > actually don't understand all the fuss about widening.  With a chunk
> > of C code embedded in something that is not C, CC Mode cannot possibly
> > need to look outside of the chunk, so why would you need to widen
> > beyond that?
> 
> There is a style of coding called "literate programming" (I know nothing
> more about it) where pieces of C code (in C Mode, obviously) alternate
> with explanatory text (in text mode).  If point is at the start of a C
> defun, C-M-a will have to take it to either the start of the text mode
> block or the previous C defun.  Either way CC Mode needs to access
> buffer pieces outside the current chunk.

Does MMM support Literate Programming?  If not, this is a separate
project, and we don't have to solve its problems now.  I would like
first to reach a more limited goal: allow editing C/C++ snippets
inside Yacc grammars, Awk snippets inside shell scripts, and other
similar use cases.

Can we do that first, and worry about higher-hanging fruit later?

> No, my main reason is philosophical.

You don't need to accept the philosophy just yet.  It is possible that
we will find in the future a solution that is more easily generalized,
perhaps even more elegant, when we consider the more complex features
and tasks.  Right now, the issue at hand is whether CC Mode will
support MMM.  It sounds wrong to me to have MMM support that excludes
CC Mode.  And the solution doesn't sound hard, just a bit tedious, and
at least some of it (replacing the calls to 'widen') quite mechanical.

> In Lisp, we're supposed to have
> the entire language available at read, evaluation, and print time.  With
> this proposal, that notion becomes complicated and ugly, once you
> qualify how everything but `widen' and `narrow-to-region' are available.

I think you exaggerate.  We have already some features whose correct
use requires some discipline, and breaks if one doesn't exercise such
discipline.  It's less than ideal, but sometimes the ideal is not very
practical.

In any case, I'm not asking you to like this restriction, just help
bring CC Mode into the MMM framework, because I think without that the
feature will be woefully incomplete.

> > This can be taken care of with a more-or-less trivial wrapper and a
> > variable.  Right?
> 
> I suppose so, but it will degrade such a function effectively into two,
> one special purpose function which can only be called from
> beginning-of-defun-function (etc.) and the other one called from
> everywhere else.  There will be confusion between them.

I don't think I understand the need for two functions.  Can you
elaborate, please?

> > > Yesterday, Richard Stallman suggested as an alternative to the
> > > purloining of `widen' and `narrow-to-region' that a new "region" be
> > > implemented somehow which would be independent of the existing region
> > > and used solely by MMM super-modes.  How about exploring this
> > > possibility?
> 
> > We can do that if needed, but I don't see the need yet.
> 
> Then we won't do it at all.

We will if we get to the point where the current proposal is too
restrictive, or cannot support some features we want.  Compare this
with the line numbers: linum.el exists, and we still added a native
implementation in the display engine.

I suggest to take a good look at what it will take to adapt CC Mode to
MMM.  If it turns out that there are fundamental obstacles to that
which we currently don't envision, at least we will have a list of
concrete problems to solve, and that will facilitate designing a
solution.

TIA

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sun, 03 Dec 2017 16:52:25 -0500
> 
> > How do you address the issues which prog-indentation-context did
> > (e.g., if the embedded chunk of code is incomplete, and perhaps even
> > syntactically invalid)?
> 
> We can easily provide a similar functionality with Dmitry's design.
> Dmitry hasn't bothered doing that, because so far it's never been used
> (so it's not even clear whether it's ever needed nor whether it would
> provide the needed info).

What I don't understand is why we are asked to remove
prog-indentation-context.  I don't see how its existence contradicts
the possibility of calling 'widen' directly or leaving the variable at
its default nil value.  What am I missing?

> I think the failure here is to describe clearly the proposal.
> The core of the proposal, AFAIK, is the following:
> 
>     state that indent-line-function can rely on the current narrowing to
>     find all the relevant text, because the caller has already widened
>     when necessary for that.
> 
> To make it true, we need a few changes in the code, mostly we need to
> tweak indent-according-to-mode so that it widens before calling
> indent-line-function.

Why does it need to widen, though?  If the mode's indent-line-function
is not supposed to call 'widen', why do we do that in
indent-according-to-mode?  I don't think this has been explained
anywhere.  When you widen, the text that comes into the view will be
entirely alien syntactically to the sub-mode that is trying to indent,
right?

The existing code calls prog-widen in those cases, but that only
widens to include the current chunk (or whatever else the writer of
prog-indentation-context had in mind), which had this issue under
control, but the proposed changes toss that.

> This change should be harmless because even if the user has narrowed
> the buffer, the indentation should always be done depending on the
> whole buffer's content anyway.

But that's an incompatible change, isn't it?  And I don't see what it
has to do with MMM, where any restriction prior to the call to the
indentation function should be honored.

> > That code is in Emacs for more than 2 years.

> It's been in emacs.git's master, yes, but not in any released version of Emacs.

That matters very little these days, as many people track the
development branches and adapt their packages to the new features
before they are released.

> > It was admitted with Stefan's full support, and at least ANTLR needs
> > it in conjunction with Python.
> 
> These can very easily be adjusted to rely on the new convention.

Why not just leave them alone?  IOW, where do they get in the way of
MMM calling 'widen' directly?

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Alan Mackenzie]

Hello, Eli.

On Mon, Dec 04, 2017 at 18:10:07 +0200, Eli Zaretskii wrote:
> > Date: Sun, 3 Dec 2017 21:20:40 +0000
> > Cc: tom@tromey.com, emacs-devel@gnu.org, monnier@iro.umontreal.ca,
> >   spinuvit@gmail.com, dgutov@yandex.ru
> > From: Alan Mackenzie <acm@muc.de>

> > > actually don't understand all the fuss about widening.  With a chunk
> > > of C code embedded in something that is not C, CC Mode cannot possibly
> > > need to look outside of the chunk, so why would you need to widen
> > > beyond that?

> > There is a style of coding called "literate programming" (I know nothing
> > more about it) where pieces of C code (in C Mode, obviously) alternate
> > with explanatory text (in text mode).  If point is at the start of a C
> > defun, C-M-a will have to take it to either the start of the text mode
> > block or the previous C defun.  Either way CC Mode needs to access
> > buffer pieces outside the current chunk.

> Does MMM support Literate Programming?  If not, this is a separate
> project, and we don't have to solve its problems now.  I would like
> first to reach a more limited goal: allow editing C/C++ snippets
> inside Yacc grammars, Awk snippets inside shell scripts, and other
> similar use cases.

OK.

> Can we do that first, and worry about higher-hanging fruit later?

OK2.

> > No, my main reason is philosophical.

> You don't need to accept the philosophy just yet.  It is possible that
> we will find in the future a solution that is more easily generalized,
> perhaps even more elegant, when we consider the more complex features
> and tasks.  Right now, the issue at hand is whether CC Mode will
> support MMM.  It sounds wrong to me to have MMM support that excludes
> CC Mode.  And the solution doesn't sound hard, just a bit tedious, and
> at least some of it (replacing the calls to 'widen') quite mechanical.

> > In Lisp, we're supposed to have
> > the entire language available at read, evaluation, and print time.  With
> > this proposal, that notion becomes complicated and ugly, once you
> > qualify how everything but `widen' and `narrow-to-region' are available.

> I think you exaggerate.  We have already some features whose correct
> use requires some discipline, and breaks if one doesn't exercise such
> discipline.  It's less than ideal, but sometimes the ideal is not very
> practical.

I've been worried about Emacs losing its generality - that major modes
can increasingly only be written in one restricted way, that `widen' can
only be used by supermodes, etc.  Maybe I've been seeing something which
isn't really there.

> In any case, I'm not asking you to like this restriction, just help
> bring CC Mode into the MMM framework, because I think without that the
> feature will be woefully incomplete.

I will do what I can.  Could you tell me where to find MMM?  ;-)

[ .... ]

> > > > Yesterday, Richard Stallman suggested as an alternative to the
> > > > purloining of `widen' and `narrow-to-region' that a new "region" be
> > > > implemented somehow which would be independent of the existing region
> > > > and used solely by MMM super-modes.  How about exploring this
> > > > possibility?

> > > We can do that if needed, but I don't see the need yet.

> > Then we won't do it at all.

> We will if we get to the point where the current proposal is too
> restrictive, or cannot support some features we want.  Compare this
> with the line numbers: linum.el exists, and we still added a native
> implementation in the display engine.

> I suggest to take a good look at what it will take to adapt CC Mode to
> MMM.  If it turns out that there are fundamental obstacles to that
> which we currently don't envision, at least we will have a list of
> concrete problems to solve, and that will facilitate designing a
> solution.

Sorry for being so uncooperative over the last couple of days.  I agree
that CC Mode must work together with MMM, and I will do everything I can
to facilitate that.

> TIA

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

>> Eli, we already have a feature in Emacs (prog-indentation-context) that
>> does not adhere to your (unreasonably high) requirements.
> Emacs development is not only about not losing existing features.  It
> is mainly about gaining new important features.  Here you are adding a
> feature that I think is important enough for us to try make it support
> C-like languages.

His proposal is reworking prog-indentation-context.
CC-mode doesn't support prog-indentation-context, so it's no surprise
that it doesn't support this replacement= and I don't see why suddenly
we should first get CC-mode to support it before the replacement can
be installed.

> I don't think this requirement is unreasonably high, as those
> languages are still pretty much the mainstay of the industry, and
> definitely used by many Emacs users.

Apparently, this has not been enough motivation for the last decade, so
I'm not holding my breath.  Including the new proposal into Emacs may
provide an added incentive if MMM-support becomes sufficiently common
that CC-mode becomes really the odd-one out (tho it's just an incentive
and CC-mode is the odd-one out in many respects already).

> Maybe I'm missing something, but what's the equivalent of PREV-CHUNK?

There isn't any.  PREV-CHUNK is probably a good idea somewhere in some
cases, but we haven't found those cases yet.

>> We keep 'prog-first-column' from that proposal
> But it was a function, and you made it a variable.  This will get in
> the way of compatibility, and also potentially will make future
> extensions harder.  Why not keep it a function?

Agreed, it should be a function (which can internally just lookup
a variable, in its current implementation, but API exposed to major
modes should be the function).

>> and instead of allowing MMM to indicate the chunk bounds through
>> prog-indentation-context, we allow MMM to apply narrowing directly, and
>> that modes honor it.
> This simplification also took away some of the features, like the
> ability to nest restrictions.

I do not know what you're referring to.

> I wonder why did you discard that.
> It's existing functionality, pretty lightweight, which was in Emacs
> for some time, and is reasonably well documented.  And it already
> satisfied your requirement of not allowing sub-modes to widen.  Why
> not leave it alone?  What am I missing?

>>>> Transitioning from prog-indentation-context to the new approach is very
>>>> easy.
>>> That's what I thought.  And that's why we should do that
>>> simultaneously with landing the new approach.  There's no reason to do
>>> that earlier.

You mean we should provide patches for those codes using
prog-indentation-context?  Doesn't Dmitry's branch already do that?

> I don't see the incompatibilities.  Basically, you replaced prog-widen
> with widen,

Actually, no.  The replacement for prog-widen is more like #'ignore
(remember: prog-widen is for major modes who want to "narrow to the
current chunk's boundaries", but with the new arrangement, the major
mode's code is already called with such a narrowing in place).

> IOW, the transition could be much smoother than it will be if we
> actually remove that stuff, because I don't think there's any
> incompatibility here which would disallow direct calls to 'widen'
> and/or leaving prog-indentation-context at its default nil value.  Or
> maybe I'm missing something again.  But if I'm right, and if we can
> make the necessary changes limited only to the documentation, then it
> could well make it into Emacs 26.  So I think it is worth our while
> to make some effort in that direction, if at all possible.

Then let's focus on the doc plus adding a few calls to widen in places
like indent-according-to-mode.


        Stefan

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> Cc: acm@muc.de, tom@tromey.com, monnier@iro.umontreal.ca, spinuvit@gmail.com,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 4 Dec 2017 11:53:09 +0000
> 
> Point is, putting mmm-mode in ELPA or not is the developer's choice.

That's true, but why would a developer object to including his/her
package in Emacs?  It will get better integration, better maintenance,
better publicity, etc.  Sounds like win-win to me.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> Date: Mon, 4 Dec 2017 15:52:38 +0000
> Cc: Eli Zaretskii <eliz@gnu.org>, tom@tromey.com, monnier@iro.umontreal.ca,
>   spinuvit@gmail.com, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> the "syntax islands" proposal would be a lot of work, which I don't
> want to commit to before I see some sort of undertaking to take it
> seriously.

I hear you.  I also think that having concrete problems in hand that
cannot be easily solved by the existing framework will help us revisit
the "islands" proposal (and perhaps other related proposals) with more
"meat" to discuss them, and thus with higher probability of reaching
an agreement.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> Date: Mon, 4 Dec 2017 16:23:49 +0000
> Cc: tom@tromey.com, emacs-devel@gnu.org, monnier@iro.umontreal.ca,
>   spinuvit@gmail.com, dgutov@yandex.ru
> From: Alan Mackenzie <acm@muc.de>
> 
> > In any case, I'm not asking you to like this restriction, just help
> > bring CC Mode into the MMM framework, because I think without that the
> > feature will be woefully incomplete.
> 
> I will do what I can.

Thank you.

> Could you tell me where to find MMM?  ;-)

I think it's here:  https://github.com/purcell/mmm-mode

> Sorry for being so uncooperative over the last couple of days.  I agree
> that CC Mode must work together with MMM, and I will do everything I can
> to facilitate that.

Thank you very much.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

> Why does it need to widen, though?

Because in general, indentation may need to look at the context, and
that context may be outside of the current user-chosen narrowing.

> If the mode's indent-line-function is not supposed to call 'widen',
> why do we do that in indent-according-to-mode?

We do it in indent-according-to-mode specifically so that
indent-line-function doesn't need to do it.

> When you widen, the text that comes into the view will be entirely
> alien syntactically to the sub-mode that is trying to indent, right?

The widening is done in indent-according-to-mode and brings into view
the whole multi-mode buffer, indeed.  Then MMM's indent-line-function is
called, which narrows the buffer to the relevant chunk and then calls
the submode's indent-line-function which hence only sees the part of the
buffer it needs.

>> This change should be harmless because even if the user has narrowed
>> the buffer, the indentation should always be done depending on the
>> whole buffer's content anyway.
> But that's an incompatible change, isn't it?

It's a change that can have visible effects, of course, but these are
fairly rare and I'm hard pressed to come up with scenarios where the
effect is anything but beneficial.


        Stefan

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Dmitry Gutov <dgutov@yandex.ru>,  acm@muc.de,  tom@tromey.com,  spinuvit@gmail.com,  emacs-devel@gnu.org
> Date: Mon, 04 Dec 2017 11:35:53 -0500
> 
> > Emacs development is not only about not losing existing features.  It
> > is mainly about gaining new important features.  Here you are adding a
> > feature that I think is important enough for us to try make it support
> > C-like languages.
> 
> His proposal is reworking prog-indentation-context.
> CC-mode doesn't support prog-indentation-context, so it's no surprise
> that it doesn't support this replacement= and I don't see why suddenly
> we should first get CC-mode to support it before the replacement can
> be installed.

Not "first", "together with", or at least "soon enough after".

> Apparently, this has not been enough motivation for the last decade, so
> I'm not holding my breath.

Well, Alan just said he will try to make that happen, so I _am_
holding my breath.

> > Maybe I'm missing something, but what's the equivalent of PREV-CHUNK?
> 
> There isn't any.  PREV-CHUNK is probably a good idea somewhere in some
> cases, but we haven't found those cases yet.

But it's already in Emacs.  So the question IMO is "is it such a bad
idea that we need to get rid of it right away?"

> >> We keep 'prog-first-column' from that proposal
> > But it was a function, and you made it a variable.  This will get in
> > the way of compatibility, and also potentially will make future
> > extensions harder.  Why not keep it a function?
> 
> Agreed, it should be a function (which can internally just lookup
> a variable, in its current implementation, but API exposed to major
> modes should be the function).

That's what it is: a function that looks up a variable internally.

> >> and instead of allowing MMM to indicate the chunk bounds through
> >> prog-indentation-context, we allow MMM to apply narrowing directly, and
> >> that modes honor it.
> > This simplification also took away some of the features, like the
> > ability to nest restrictions.
> 
> I do not know what you're referring to.

AFAIU, prog-indentation-context could support a way to nest
restrictions, because prev-chunk could be a function.

> You mean we should provide patches for those codes using
> prog-indentation-context?  Doesn't Dmitry's branch already do that?

Not for ANTLR whose latest versions are maintained outside Emacs,
AFAIU.

> > I don't see the incompatibilities.  Basically, you replaced prog-widen
> > with widen,
> 
> Actually, no.

Again, I might be missing something, but "git diff" says I'm right.

> Then let's focus on the doc plus adding a few calls to widen in places
> like indent-according-to-mode.

I'd still like to understand why those additional calls to 'widen' are
needed.  Doing that seems to contradict what we want to ask major
modes not to do in their indentation code.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Mon, 04 Dec 2017 11:49:59 -0500
> 
> > Why does it need to widen, though?
> 
> Because in general, indentation may need to look at the context, and
> that context may be outside of the current user-chosen narrowing.
> 
> > If the mode's indent-line-function is not supposed to call 'widen',
> > why do we do that in indent-according-to-mode?
> 
> We do it in indent-according-to-mode specifically so that
> indent-line-function doesn't need to do it.
> 
> > When you widen, the text that comes into the view will be entirely
> > alien syntactically to the sub-mode that is trying to indent, right?
> 
> The widening is done in indent-according-to-mode and brings into view
> the whole multi-mode buffer, indeed.  Then MMM's indent-line-function is
> called, which narrows the buffer to the relevant chunk and then calls
> the submode's indent-line-function which hence only sees the part of the
> buffer it needs.

Ah, so this assumes that MMM is active in the buffer?  If so, can we
widen only if MMM is indeed active?  Doing so will I believe minimize
(or even eliminate) the effects on existing behaviors when MMM is not
used.

> It's a change that can have visible effects, of course, but these are
> fairly rare and I'm hard pressed to come up with scenarios where the
> effect is anything but beneficial.

Use being so deep into the pretest requires to be either 110% careful
not to change existing behaviors, or agree to postpone this to Emacs 27.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

> That's true, but why would a developer object to including his/her
> package in Emacs?  It will get better integration, better maintenance,
> better publicity, etc.  Sounds like win-win to me.

Beats me as well, but I know that many people see it very differently.


        Stefan

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/4/17 7:28 PM, Eli Zaretskii wrote:

>> The widening is done in indent-according-to-mode and brings into view
>> the whole multi-mode buffer, indeed.  Then MMM's indent-line-function is
>> called, which narrows the buffer to the relevant chunk and then calls
>> the submode's indent-line-function which hence only sees the part of the
>> buffer it needs.
> 
> Ah, so this assumes that MMM is active in the buffer?  If so, can we
> widen only if MMM is indeed active?  Doing so will I believe minimize
> (or even eliminate) the effects on existing behaviors when MMM is not
> used.

That widening is for correctness when working during an interactive 
narrowing. MMM mode being enabled or not is irrelevant (it's not the 
beneficiary).

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/4/17 6:56 PM, Eli Zaretskii wrote:

>> His proposal is reworking prog-indentation-context.
>> CC-mode doesn't support prog-indentation-context, so it's no surprise
>> that it doesn't support this replacement= and I don't see why suddenly
>> we should first get CC-mode to support it before the replacement can
>> be installed.
> 
> Not "first", "together with", or at least "soon enough after".

That still doesn't follow.

>> Apparently, this has not been enough motivation for the last decade, so
>> I'm not holding my breath.
> 
> Well, Alan just said he will try to make that happen, so I _am_
> holding my breath.

That's wonderful. But I'm still not going to hold my breath regarding 
the time estimates. It might turn out to be fairly time-consuming.

>> There isn't any.  PREV-CHUNK is probably a good idea somewhere in some
>> cases, but we haven't found those cases yet.
> 
> But it's already in Emacs.  So the question IMO is "is it such a bad
> idea that we need to get rid of it right away?"

It's pointless: PREV-CHUNK must be followed/used by a major mode. The 
majority of the important major modes are inside Emacs (the ones used by 
ANTLR are, at least). So there won't be any real benefit to keeping it in.

On the other hand, we might get a better idea how to solve that problem 
in a different way, during the next release cycle.

> AFAIU, prog-indentation-context could support a way to nest
> restrictions, because prev-chunk could be a function.

Nobody really knows how to use this.

>> You mean we should provide patches for those codes using
>> prog-indentation-context?  Doesn't Dmitry's branch already do that?
> 
> Not for ANTLR whose latest versions are maintained outside Emacs,
> AFAIU.

The diff will be like 4 lines. I can submit it whenever, wherever you want.

>>> I don't see the incompatibilities.  Basically, you replaced prog-widen
>>> with widen,
>>
>> Actually, no.
> 
> Again, I might be missing something, but "git diff" says I'm right.

Not really. There is one place where it looks like prog-widen is 
replaced by widen, but it's inside a function that's not part of the 
indentation engine (python-indent-guess-indent-offset, despite what its 
name suggests, is called from the major mode function, not through 
indent-line-function).

The other places, inside indentation engines, feature the removals of 
prog-widen and (in one case) widen calls.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/4/17 5:52 PM, Eli Zaretskii wrote:

> Your work on this is highly appreciated.

Thank you.

> But I have certain
> responsibilities, and I'm trying to do my job here.

True, and I fully expect to discuss API breakage, development cycles, 
and that sort of issues. Adding a big requirement, and it being 
something we've effectively given up on solving in the shorter term, has 
been very surprising.

> I'm also your fellow developer of the same project and its active
> user.  I think it means my opinions should be of some importance, not
> something to be ignored solely because I'm not coding this.

I kind of expected opinions in different directions.

> Just like
> I would never dream of ignoring your opinions about features I'm
> implementing.

Not opinions like "don't push this change, unless you make it work for 
my use case X too!", I expect. At least, not for all such opinions.

>> [C-like major modes] can be catered to, *as soon as* the authors of each individual major mode make the effort to support the feature.
> 
> Agreed.  What I'm trying to establish is whether the design supports
> those major modes, or does it ignore them.  The latter I would like to
> avoid as much as possible.

Whatever special requirements particular major modes turn out to have, 
can be added later. I've tried my best to make sure that improvements 
would be smooth. Hopefully just additive.

>> Eli, we already have a feature in Emacs (prog-indentation-context) that does not adhere to your (unreasonably high) requirements.
> 
> Emacs development is not only about not losing existing features.  It
> is mainly about gaining new important features.  Here you are adding a
> feature that I think is important enough for us to try make it support
> C-like languages.

So far, the discussion of supporting it in CC Mode, has not uncovered 
any extra requirements to the proposed protocol, from where I'm standing.

> I don't think this requirement is unreasonably
> high, as those languages are still pretty much the mainstay of the
> industry, and definitely used by many Emacs users.

The requirement seems a bit late, is all. Either you apply it to 
prog-indentation-context and vote to remove it, or you don't get to 
apply it to this proposal. That's my opinion, at least.

>> And that cache point to a position outside of the current restriction. Confusion ensues. I've seen errors originating from that (I think), but can't recall the exact sequence of calls.
> 
> Thanks, so the caches will have to be sensitive to restrictions as
> they are sensitive to deletion of text.

Right, and that's the responsibility of the major mode. So there's not 
much we can add in terms of code (I think), but we can add some more 
sentences to the manual.

> But given your example with CC Mode's caches, that's not all of it, is
> it?  You also need the mode to adapt any such caches to changes in
> buffer restrictions, right?

I think so. But only CC Mode, in my experience, includes such caches 
that lead to problems in MMM context.

>> We keep 'prog-first-column' from that proposal
> 
> But it was a function, and you made it a variable.  This will get in
> the way of compatibility, and also potentially will make future
> extensions harder.  Why not keep it a function?

To have a better API. If we keep it a function, do we also have a 
variable prog-first-column?

Then, some consumers might have doubt which ones to use, and might opt 
to reference the variable. Then, we lose all benefits of having it a 
function (like making its implementation somehow smarter later), and the 
only benefit remaining is backward compatibility of having a function 
with that name.

But! Like with PREV-CHUNKS, prog-first-column (and the later prog-widen) 
are supposed to be used by the major modes only. There are no references 
to either of these functions in the latest antlr-mode.el, and there 
shouldn't be.

And as long as all the calls to that function are inside Emacs, we're 
free to change it however, including turning it into a variable.

>> and instead of allowing MMM to indicate the chunk bounds through prog-indentation-context, we allow MMM to apply narrowing directly, and that modes honor it.
> 
> This simplification also took away some of the features, like the
> ability to nest restrictions.  I wonder why did you discard that.
> It's existing functionality, pretty lightweight, which was in Emacs
> for some time, and is reasonably well documented.  And it already
> satisfied your requirement of not allowing sub-modes to widen.  Why
> not leave it alone?  What am I missing?

It's just pointless. I've asked, in the original discussion 2 years ago, 
for at least one patch that would use it. None have arrived since.

And as long as major modes do not use it, MMM packages don't have any 
benefits from it.

>>          Transitioning from prog-indentation-context to the new approach is very
>>          easy.
>>
>>      That's what I thought.  And that's why we should do that
>>      simultaneously with landing the new approach.  There's no reason to do
>>      that earlier.
>>
>> Why confuse the language modes authors?
> 
> If they should be confused (I don't think so), they are already
> confused, as we've been having this for the last 2 years.

Not in a release version. And the confusion I'm talking about will come 
later, when we either try to keep both ways of doing this (I don't think 
we'll manage), or abruptly switch from one way to another.

The latter is easy code-wise, but by that time, whatever compatibility 
code they had, they might have removed it intending to support only 
Emacs 26+, for instance. And here we come saying Emacs 27 will be 
strictly incompatible.

>>      We cannot know that.  Once the code is that long with us, we cannot
>>      throw it away without a equivalent replacement.  And I see no reason
>>      to do that now, since the fact that this code will exist in Emacs 26
>>      doesn't hurt anyone, especially since the replacement will be easy, as
>>      you say.
>>
>> Well... you have a point there. The result will be pretty ugly, though. Adding an API in one version, and removing it in another.
> 
> We will have to discuss the "removal" part.  An alternative is to
> provide compatibility shims, or even leave (some or all of) the
> existing API intact.

Yeah, I cannot imagine how a compatibility shim would work. Hence my 
request to replace it as soon as possible.

>> They are incompatible with each other, so it's not like there can be a smooth transition.
> 
> I don't see the incompatibilities.  Basically, you replaced prog-widen
> with widen,

Nope, just removed prog-widen. And widen in one place.

Also replaced one prog-widen call with widen, but it was not in an 
indentation function (os it probably shouldn't have been there in the 
first place).

> made prog-first-column a variable instead of a function,

See (*).

> and tossed prog-indentation-context.  We could instead keep the
> existing API with minimal changes: prog-widen calls widen internally,
> and we could allow direct calls to widen as well.

That won't work. Either indentation functions call prog-widen (which 
fully widens in the absence of prog-indentation-context binding), or 
they don't widen at all. These are two different, incompatible ways of 
doing things.

> IOW, the transition could be much smoother than it will be if we
> actually remove that stuff, because I don't think there's any
> incompatibility here which would disallow direct calls to 'widen'
> and/or leaving prog-indentation-context at its default nil value.

Yes, there is. Please go back to one of the several descriptions, in 
this thread only, of how MMM uses narrowing to restrict indentation 
engine to the current chunk.

> Or
> maybe I'm missing something again.  But if I'm right, and if we can
> make the necessary changes limited only to the documentation, then it
> could well make it into Emacs 26.  So I think it is worth our while
> to make some effort in that direction, if at all possible.

Unfortunately, I don't imagine it's really possible.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/4/17 6:41 PM, Eli Zaretskii wrote:

>> Point is, putting mmm-mode in ELPA or not is the developer's choice.
> 
> That's true, but why would a developer object to including his/her
> package in Emacs?  It will get better integration, better maintenance,
> better publicity, etc.  Sounds like win-win to me.

Better control over releases, faster to get fixes to the users. That's 
if we're comparing "in Emacs only" with "in ELPA only".

If the package is released to both, I have no experience with that, but 
I imagine there needs to be some extra work synchronizing, as well as 
making sure the APIs are compatible, at least (to which set of versions 
of Emacs? I'm not sure...).

Overall, I'm in favor of "modular Emacs", and so I'd rather ask why 
include this particular package, and how it would benefit. Users being 
able to avoid using ELPA is a very minor difference, in my mind, 
especially compared to benefits above.

mmm-mode, in particular, is designed to work "outside in", with little 
support from the code inside Emacs, so it shouldn't need much support 
over the years, not much more than it would get being part of ELPA, at 
least.

Further, I'm not sure that its way of doing things (the user-facing 
part) is best. Maybe polymode has it thought out better, I have not 
studied it well enough. Or an extraction from mhtml-mode would turn out 
better still. Putting an MMM package into Emacs would seem like a choice 
of the better package, and I'd rather we have several equally supported 
ones, at least until one comes out decisively at the top.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

On December 4, 2017 11:52:26 PM GMT+02:00, Dmitry Gutov <dgutov@yandex.ru> wrote:
> On 12/4/17 7:28 PM, Eli Zaretskii wrote:
> 
> >> The widening is done in indent-according-to-mode and brings into
> view
> >> the whole multi-mode buffer, indeed.  Then MMM's
> indent-line-function is
> >> called, which narrows the buffer to the relevant chunk and then
> calls
> >> the submode's indent-line-function which hence only sees the part
> of the
> >> buffer it needs.
> > 
> > Ah, so this assumes that MMM is active in the buffer?  If so, can we
> > widen only if MMM is indeed active?  Doing so will I believe
> minimize
> > (or even eliminate) the effects on existing behaviors when MMM is
> not
> > used.
> 
> That widening is for correctness when working during an interactive 
> narrowing. MMM mode being enabled or not is irrelevant (it's not the 
> beneficiary).

I understand that you and Stefan think so, but doing that unconditionally
means these changes cannot be committed to emacs-26.  I was
trying to find a way of squeezing them into the release branch, but if
you think there's no way, it's fine with me to have this on master.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

On December 5, 2017 7:08:14 AM GMT+02:00, Eli Zaretskii <eliz@gnu.org> wrote:
> On December 4, 2017 11:52:26 PM GMT+02:00, Dmitry Gutov
> <dgutov@yandex.ru> wrote:
> > On 12/4/17 7:28 PM, Eli Zaretskii wrote:
> > 
> > >> The widening is done in indent-according-to-mode and brings into
> > view
> > >> the whole multi-mode buffer, indeed.  Then MMM's
> > indent-line-function is
> > >> called, which narrows the buffer to the relevant chunk and then
> > calls
> > >> the submode's indent-line-function which hence only sees the part
> > of the
> > >> buffer it needs.
> > > 
> > > Ah, so this assumes that MMM is active in the buffer?  If so, can
> we
> > > widen only if MMM is indeed active?  Doing so will I believe
> > minimize
> > > (or even eliminate) the effects on existing behaviors when MMM is
> > not
> > > used.
> > 
> > That widening is for correctness when working during an interactive 
> > narrowing. MMM mode being enabled or not is irrelevant (it's not the
> 
> > beneficiary).
> 
> I understand that you and Stefan think so, but doing that
> unconditionally
> means these changes cannot be committed to emacs-26.  I was
> trying to find a way of squeezing them into the release branch, but if
> you think there's no way, it's fine with me to have this on master.

And btw, I'm not really convinced the unconditional widening is a good idea in general, even on master.  I could probably agree that in most cases it is TRT, but why would we _force_ all modes to indent with restrictions lifted?  No exceptions?  Not even a fire escape for some specialized mode with weird needs and requirements?  I'm not sure.  It certainly should be discussed, regardless of the MMM issues.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/5/17 7:33 AM, Eli Zaretskii wrote:

> And btw, I'm not really convinced the unconditional widening is a good idea in general, even on master.  I could probably agree that in most cases it is TRT, but why would we _force_ all modes to indent with restrictions lifted?  No exceptions?  Not even a fire escape for some specialized mode with weird needs and requirements?  I'm not sure.  It certainly should be discussed, regardless of the MMM issues.

After indent-for-tab-command calls widen, indent-line-function is 
called. And it can apply all the narrowing it wants (inside 
save-restriction), it just shouldn't call 'widen'.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/5/17 7:08 AM, Eli Zaretskii wrote:

> I understand that you and Stefan think so, but doing that unconditionally
> means these changes cannot be committed to emacs-26.  I was
> trying to find a way of squeezing them into the release branch, but if
> you think there's no way, it's fine with me to have this on master.

Suppose the patch goes on master.

Do I actually remove all references to prog-indentation-context like 
shown on the branch? Despite that it will be present in Emacs 26?

What about NEWS? Will it say "prog-indentation-context was removed, and 
a different protocol for multi-major-mode buffers is established now"?

These questions are still unresolved, and there's no point in delaying 
those decisions.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

Hey Alan,

On 12/4/17 5:52 PM, Alan Mackenzie wrote:

> Any chance you could give me some basic details of how to get into this
> MMM mode?  Like which git branch (? scratch/widen-less ?) can I find the
> code in, and where is the best documentation to get started.

I was talking in abstract (MMM meaning any similar framework), but it's 
a good question. To debug the problems, a concrete example is needed.

The latest version of mmm-mode lives here: 
https://github.com/purcell/mmm-mode.git, you can check out the master 
branch, and it even has a manual that's reasonably up-to-date WRT to usage.

Here are a couple of related bug reports:

https://github.com/purcell/mmm-mode/issues/55
https://github.com/purcell/mmm-mode/issues/57

Both pertaining to the use of c-mode in the context of mmm-noweb.el.

Please let me know if you need more localized examples.

> I've no idea how long it will take to make MMM and
> CC Mode work with eachother.

If you manage to do it at all, that would be an achievement. But just 
figuring out the biggest problems should help us to design the feature.

> Not much, if at all.  I found those discussions to be lacking the
> context I would have needed to understand them.

I hope the context is apparent now.

> That's fair.  But the "syntax islands" proposal would be a lot of work,
> which I don't want to commit to before I see some sort of undertaking to
> take it seriously.

Then it should be a good idea to try the proposed, smaller approach, and 
collect the drawbacks you meet along the way.

> [ .... ]
> 
>> There are several ways to skin that cat.
> 
> That's the second cat in as many days, here!

Lots of cats around here in Cyprus. ;-)

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

On December 5, 2017 10:55:33 AM GMT+00:00, Dmitry Gutov <dgutov@yandex.ru> wrote:
> On 12/5/17 7:33 AM, Eli Zaretskii wrote:
> 
> > And btw, I'm not really convinced the unconditional widening is a
> good idea in general, even on master.  I could probably agree that in
> most cases it is TRT, but why would we _force_ all modes to indent
> with restrictions lifted?  No exceptions?  Not even a fire escape for
> some specialized mode with weird needs and requirements?  I'm not
> sure.  It certainly should be discussed, regardless of the MMM issues.
> 
> After indent-for-tab-command calls widen, indent-line-function is 
> called. And it can apply all the narrowing it wants (inside 
> save-restriction), it just shouldn't call 'widen'.

You assume that the indentation function has the same information about the higher-level context as its callers.  But that assumption is not necessarily true; narrowing exists precisely because higher-level code wants to set up lower-level code without explicitly passing it this information.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

On December 5, 2017 12:55:46 PM GMT+00:00, Dmitry Gutov <dgutov@yandex.ru> wrote:
> On 12/5/17 7:08 AM, Eli Zaretskii wrote:
> 
> > I understand that you and Stefan think so, but doing that
> unconditionally
> > means these changes cannot be committed to emacs-26.  I was
> > trying to find a way of squeezing them into the release branch, but
> if
> > you think there's no way, it's fine with me to have this on master.
> 
> Suppose the patch goes on master.
> 
> Do I actually remove all references to prog-indentation-context like 
> shown on the branch? Despite that it will be present in Emacs 26?
> 
> What about NEWS? Will it say "prog-indentation-context was removed,
> and 
> a different protocol for multi-major-mode buffers is established now"?
> 
> These questions are still unresolved, and there's no point in delaying
> 
> those decisions.

I posted a more or less complete proposal for how to make the changes you want without breaking previous facilities.  If you agree with that proposal (which IMO makes sense even for master), then most of the issues you raise disappear.  If you disagree, please respond to that message of mine explaining why, and let's discuss that first.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/5/17 7:53 PM, Eli Zaretskii wrote:

> You assume that the indentation function has the same information about the higher-level context as its callers.  But that assumption is not necessarily true; narrowing exists precisely because higher-level code wants to set up lower-level code without explicitly passing it this information.

Without MMM in the equation, the caller here is indent-for-tab-command.

It's an interactive function, and the only narrowing present before it 
is the one possibly applied by the user, interactively.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/5/17 7:57 PM, Eli Zaretskii wrote:

> I posted a more or less complete proposal for how to make the changes you want without breaking previous facilities.  If you agree with that proposal (which IMO makes sense even for master), then most of the issues you raise disappear.  If you disagree, please respond to that message of mine explaining why, and let's discuss that first.

Sorry, is it this message?

http://lists.gnu.org/archive/html/emacs-devel/2017-12/msg00158.html

I thought I've addresses every point in there (that I could make sense 
of) in other messages in this thread, but if you prefer, I can respond 
to it directly, to put the critique in one place.

Have you read the following messages, BTW? There were no responses.

http://lists.gnu.org/archive/html/emacs-devel/2017-12/msg00169.html
http://lists.gnu.org/archive/html/emacs-devel/2017-12/msg00170.html

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

On December 5, 2017 6:54:59 PM GMT+00:00, Dmitry Gutov <dgutov@yandex.ru> wrote:
> On 12/5/17 7:57 PM, Eli Zaretskii wrote:
> 
> > I posted a more or less complete proposal for how to make the
> changes you want without breaking previous facilities.  If you agree
> with that proposal (which IMO makes sense even for master), then most
> of the issues you raise disappear.  If you disagree, please respond to
> that message of mine explaining why, and let's discuss that first.
> 
> Sorry, is it this message?
> 
> http://lists.gnu.org/archive/html/emacs-devel/2017-12/msg00158.html
> 
> I thought I've addresses every point in there (that I could make sense
> 
> of) in other messages in this thread, but if you prefer, I can respond
> 
> to it directly, to put the critique in one place.
> 
> Have you read the following messages, BTW? There were no responses.
> 
> http://lists.gnu.org/archive/html/emacs-devel/2017-12/msg00169.html
> http://lists.gnu.org/archive/html/emacs-devel/2017-12/msg00170.html

Yes, please respond to that message directly so that we could have all the points together.
To make myself clear: I'd like to see your changes land in a way that as much as possible doesn't remove prog-indentation-context except where there's a clear and present contradiction.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

On December 5, 2017 6:40:00 PM GMT+00:00, Dmitry Gutov <dgutov@yandex.ru> wrote:
> On 12/5/17 7:53 PM, Eli Zaretskii wrote:
> 
> > You assume that the indentation function has the same information
> about the higher-level context as its callers.  But that assumption is
> not necessarily true; narrowing exists precisely because higher-level
> code wants to set up lower-level code without explicitly passing it
> this information.
> 
> Without MMM in the equation, the caller here is
> indent-for-tab-command.
> 
> It's an interactive function, and the only narrowing present before it
> 
> is the one possibly applied by the user, interactively.

Interactive functions can also be called from Lisp.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Ingo Lohmar]

On Tue, Dec 05 2017 20:48 (+0000), Eli Zaretskii wrote:

> To make myself clear: I'd like to see your changes land in a way that as much as possible doesn't remove prog-indentation-context except where there's a clear and present contradiction.

Previously I dabbled with mmm-mode and even maintained an unpublished
fork to fix some shortcomings.  Therefore I followed this and the
related threads with quite some interest.

Apologies if I misunderstood or if this is just noise (in which case
simply ignore my message), but I have only seen this point briefly
raised in passing, and it appears to be very important:

1. I gathered that from what is known now, prog-indentation-context
   cannot provide all that which multiple major mode frameworks
   need/want from Emacs core.

2. Also, it is not orthogonal, but rather incompatible with all other
   proposals (be they concrete or vague) discussed thus far.

So if we really want to facilitate mmm-like frameworks (which is
wonderful and surely a great thing to do), prog-indentation-context will
have to be removed whenever that happens.

Since it's not in any released version yet, it seems obvious to me that
it should be removed right now, or else it will be "in the wild".  And
then it will be much more awkward to remove it in the next release (and
I would assume that you will strongly object to removing it then).

What am I missing here?

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/5/17 10:49 PM, Eli Zaretskii wrote:

>> It's an interactive function, and the only narrowing present before it
>>
>> is the one possibly applied by the user, interactively.
> 
> Interactive functions can also be called from Lisp.

True, and that can be a reason to worry about stability (emacs-26 vs 
master).

I don't think it should affect the eventual design here, though.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

On December 5, 2017 9:08:56 PM GMT+00:00, Ingo Lohmar <i.lohmar@gmail.com> wrote:
> On Tue, Dec 05 2017 20:48 (+0000), Eli Zaretskii wrote:
> 
> > To make myself clear: I'd like to see your changes land in a way
> that as much as possible doesn't remove prog-indentation-context
> except where there's a clear and present contradiction.
> 
> Previously I dabbled with mmm-mode and even maintained an unpublished
> fork to fix some shortcomings.  Therefore I followed this and the
> related threads with quite some interest.
> 
> Apologies if I misunderstood or if this is just noise (in which case
> simply ignore my message), but I have only seen this point briefly
> raised in passing, and it appears to be very important:
> 
> 1. I gathered that from what is known now, prog-indentation-context
>    cannot provide all that which multiple major mode frameworks
>    need/want from Emacs core.
> 
> 2. Also, it is not orthogonal, but rather incompatible with all other
>    proposals (be they concrete or vague) discussed thus far.
> 
> So if we really want to facilitate mmm-like frameworks (which is
> wonderful and surely a great thing to do), prog-indentation-context
> will
> have to be removed whenever that happens.
> 
> Since it's not in any released version yet, it seems obvious to me
> that
> it should be removed right now, or else it will be "in the wild".  And
> then it will be much more awkward to remove it in the next release
> (and
> I would assume that you will strongly object to removing it then).
> 
> What am I missing here?

My conclusions from reviewing the discussions and the proposed code changes, as they appear on Dmitry's branch, were that prog-indentation-context is a super-set of what is needed for MMM and similar frameworks.  As I understand, Dmitry submits that the extra features are not needed and have never been used, but I haven't yet heard any arguments to the effect that prog-indentation-context _contradicts_ MMM.  Which is why I still think it'd be prudent not to remove it, at least not yet, because the extra features it provides might be useful, and because the code is part of the development sources for more than 2 years.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

On December 5, 2017 11:16:08 PM GMT+00:00, Dmitry Gutov <dgutov@yandex.ru> wrote:
> On 12/5/17 10:49 PM, Eli Zaretskii wrote:
> 
> >> It's an interactive function, and the only narrowing present before
> it
> >>
> >> is the one possibly applied by the user, interactively.
> > 
> > Interactive functions can also be called from Lisp.
> 
> True, and that can be a reason to worry about stability (emacs-26 vs 
> master).
> 
> I don't think it should affect the eventual design here, though.

My point is that this aspect of the design should be discussed before we make the code changes.  (I think this aspect is separate from the rest of MMM related discussions.)

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/6/17 11:28 AM, Eli Zaretskii wrote:

> My point is that this aspect of the design should be discussed before we make the code changes.

Let's go ahead and discuss it then?

Short overview:

- We have indentation functions that widen (some major modes), and 
indentation function that don't (where the authors didn't bother, or, in 
one place, where I personally removed it, from js-mode, after a 
MMM-related discussion).

- IME, whenever the question of narrowing comes up, it's usually from a 
user who wants command X to (save-restriction (widen) ...) before doing 
what it always did. Never to avoid calling widen, which makes sense from 
a perspective of a user who uses interactive narrowing.

Hence, it seems to make sense that indent-for-tab-command should always 
operate of a widened buffer.

Your turn.

 > (I think this aspect is separate from the rest of MMM related 
discussions.)

Not exactly separate. My proposal hinges on it somewhat. In particular, 
on the idea that all major modes should be treated the same in this respect.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/5/17 10:48 PM, Eli Zaretskii wrote:

> Yes, please respond to that message directly so that we could have all the points together.
> To make myself clear: I'd like to see your changes land in a way that as much as possible doesn't remove prog-indentation-context except where there's a clear and present contradiction.

Right. IMO, that's impossible to do in any useful way.

I'll respond to that message in full today.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Wed, 6 Dec 2017 15:36:16 +0200
> 
> On 12/6/17 11:28 AM, Eli Zaretskii wrote:
> 
> > My point is that this aspect of the design should be discussed before we make the code changes.
> 
> Let's go ahead and discuss it then?

Not with me, with everyone else.  I already described my doubts.

I will start a new thread about this, I think this issue all but
drowned in this long discussion, and I'd expect some people to avoid
reading it.

>  > (I think this aspect is separate from the rest of MMM related 
> discussions.)
> 
> Not exactly separate. My proposal hinges on it somewhat. In particular, 
> on the idea that all major modes should be treated the same in this respect.

The separate issue is to widen unconditionally.  I could probably
agree to have these functions widen when MMM-like mode is active.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/8/17 6:41 PM, Eli Zaretskii wrote:

> The separate issue is to widen unconditionally.  I could probably
> agree to have these functions widen when MMM-like mode is active.

While all indent-line-functions stop calling 'widen'? Or will then also 
have to do that conditionally?

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 9 Dec 2017 17:17:34 +0200
> 
> On 12/8/17 6:41 PM, Eli Zaretskii wrote:
> 
> > The separate issue is to widen unconditionally.  I could probably
> > agree to have these functions widen when MMM-like mode is active.
> 
> While all indent-line-functions stop calling 'widen'? Or will then also 
> have to do that conditionally?

If a particular indent-line-function needs to widen, then yes, it will
need to do that by itself, and if that gets in the way of MMM, then
the widening will have to be conditional.

(I thought that prog-widen and prog-indentation-context were a means
to solve this, isn't that so?)

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/9/17 5:43 PM, Eli Zaretskii wrote:

> If a particular indent-line-function needs to widen, then yes, it will
> need to do that by itself, and if that gets in the way of MMM, then
> the widening will have to be conditional.

That sounds surprisingly more complex than the whole solution that I'm 
proposing.

> (I thought that prog-widen and prog-indentation-context were a means
> to solve this, isn't that so?)

It was one of the goals, but that approach comes with certain inherent, 
as well as certain incidental, drawbacks.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 10 Dec 2017 21:59:45 +0200
> 
> On 12/9/17 5:43 PM, Eli Zaretskii wrote:
> 
> > If a particular indent-line-function needs to widen, then yes, it will
> > need to do that by itself, and if that gets in the way of MMM, then
> > the widening will have to be conditional.
> 
> That sounds surprisingly more complex than the whole solution that I'm 
> proposing.

I'm surprised that you think this to be so much more complex.

But in any case, this is the subject of a separate thread now, so I
suggest to discuss there, to avoid people who are interested missing
this.

> > (I thought that prog-widen and prog-indentation-context were a means
> > to solve this, isn't that so?)
> 
> It was one of the goals, but that approach comes with certain inherent, 
> as well as certain incidental, drawbacks.

It looks sufficiently lightweight to me, so I hope the drawbacks are
not unbearable.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Wedler, Christoph]

Sorry for not having followed this discussion earlier - as the one who
has somehow introduced `prog-indentation-context', I probably should have...

If it works and you tell me how to do it, I have no issues with an
alternative way to make the indentation function of a "sub mode" work
for code chunks inside an "outer language" (like ANTLR grammar files).

I you want to try some ANTLR grammar examples yourself, I would
recommend:

 - antlr-mode 3.1.4
   from https://sourceforge.net/projects/antlr-mode/files/

 - example grammar files, especially lines 46 to 51 in
   https://github.com/antlr/examples-v3/blob/master/Python/C/C.g

 - and others in that repo (mentioned in line 100 of antlr-mode.el)

Now to the elements in `prog-indentation-context':

 - FIRST-COLUMN: I understood that you want to keep (s/th like) that...

 - (START . END) = the region of the code chunk: well, some indentation
   engines got confused to see unknown code before code written in the
   "sub-modes" - for some I could do `narrow-to-region' myself, other
   engines did widen, like python.  IMHO, modes with widening provide a
   better indentation experience than those without, see below.

 - PREVIOUS-CHUNKS: well, I was told to introduce this to be more
   general... - some modes (like cc-mode) could be told that the code is
   not some usual "top-level" code, but just some "normal statement".
   (antlr-mode includes some ugly hack for cc-mode).

If all this is covered by a better way - fine with me!  Even if does not cover
PREVIOUS-CHUNKS…

Now to a mode with non-widening indentation function: emacs-lisp mode:

Consider the following code
    (defun foo ()
      (+ 1 3))

I would assume that the indentation of line 2 does not change
when I narrow to line 2 - but Emacs-25.1.1 now deletes the first 2
spaces of line 2...  (yes; I also need to update to a newer Emacs)

- Christoph

P.S. antlr-mode 3.1.5 will have docstrings for all function and
variables - just the last 100 lines are missing.  After that, I need to
adapt it to "the new way" and it could replace the very old antlr-mode
in Emacs (or move it to ELPA or whatever you like).

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

>  - (START . END) = the region of the code chunk: well, some indentation
>    engines got confused to see unknown code before code written in the
>    "sub-modes" - for some I could do `narrow-to-region' myself, other
>    engines did widen, like python.  IMHO, modes with widening provide a
>    better indentation experience than those without, see below.

The change to use narrowing for submodes also starts by widening the
buffer, so all modes now benefit from widening without having to call
widen themselves.


        Stefan

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 3421 bytes --]

Hi Christoph,

On 12/15/17 5:48 PM, Wedler, Christoph wrote:
> Sorry for not having followed this discussion earlier - as the one who
> has somehow introduced `prog-indentation-context', I probably should have...

You checked out early from the previous discussion, so I didn't want to 
drag you in again. Hope that's okay.

> If it works and you tell me how to do it, I have no issues with an
> alternative way to make the indentation function of a "sub mode" work
> for code chunks inside an "outer language" (like ANTLR grammar files).

Is there a reason why you called narrow-to-region in 
antlr-python-indent-action-line anyway, even when using 
prog-indentation-context?

Because you do that, though, the "new way" should work with your code 
automatically. I'm attaching a patch with some minor cleanup, though.

Please try it with a build of Emacs from the 'widen-less' branch. Do you 
know how to build it?

> I you want to try some ANTLR grammar examples yourself, I would
> recommend:
> 
>   - antlr-mode 3.1.4
>     from https://sourceforge.net/projects/antlr-mode/files/
> 
>   - example grammar files, especially lines 46 to 51 in
>     https://github.com/antlr/examples-v3/blob/master/Python/C/C.g
> 
>   - and others in that repo (mentioned in line 100 of antlr-mode.el)

antlr-mode.el says these examples are to see the syntax highlighting. 
Any particular file and place to see how indentation works out? I 
couldn't find one from a cursory search.

> Now to the elements in `prog-indentation-context':
> 
>   - FIRST-COLUMN: I understood that you want to keep (s/th like) that...
> 
>   - (START . END) = the region of the code chunk: well, some indentation
>     engines got confused to see unknown code before code written in the
>     "sub-modes" - for some I could do `narrow-to-region' myself, other
>     engines did widen, like python.  IMHO, modes with widening provide a
>     better indentation experience than those without, see below.
> 
>   - PREVIOUS-CHUNKS: well, I was told to introduce this to be more
>     general... - some modes (like cc-mode) could be told that the code is
>     not some usual "top-level" code, but just some "normal statement".
>     (antlr-mode includes some ugly hack for cc-mode).
> 
> If all this is covered by a better way - fine with me!  Even if does not cover
> PREVIOUS-CHUNKS…

Yup.

> Now to a mode with non-widening indentation function: emacs-lisp mode:
> 
> Consider the following code
>      (defun foo ()
>        (+ 1 3))
> 
> I would assume that the indentation of line 2 does not change
> when I narrow to line 2 - but Emacs-25.1.1 now deletes the first 2
> spaces of line 2...  (yes; I also need to update to a newer Emacs)

Try out the widen-less branch!

And also, can we get your help with the manual? It still says:

Examples include
  @dfn{literate programming} source files that combine documentation and
  snippets of source code, Yacc/Bison programs that include snippets of
  plain C code, etc.  To correctly indent the embedded chunks, the primary
  mode needs to delegate the indentation to another mode's indentation
  engine (e.g., call @code{c-indent-defun} for C code or
  @code{python-indent-line} for Python)

I.e., it still mentions C and c-indent-defun (?). But c-indent-defun 
does not work for embedded chunks (yet)!

Can we replace that mentions with JS and js-indent-line, for instance? 
Would that still make sense?

[-- Attachment #2: antlr-mode-cleanup.diff --]
[-- Type: text/x-patch, Size: 1135 bytes --]

diff -u --label /home/dgutov/examples/antlr/antlr-mode.el --label \#\<buffer\ antlr-mode.el\> /home/dgutov/examples/antlr/antlr-mode.el /tmp/buffer-content-4j5MsU
--- /home/dgutov/examples/antlr/antlr-mode.el
+++ #<buffer antlr-mode.el>
@@ -3408,12 +3408,10 @@
     (when (and boa
                (eq antlr-indent-comment t) ; indent-region
                (boundp 'prog-indentation-context)) ; Emacs 24.5 or later
-      (let ((syntax-ppss-cache nil) ;#dynamic
-            (syntax-ppss-last nil) ;#dynamic
-            ;; TODO: do we also need to call `syntax-propertize'?
-            ;; and/or bind `syntax-propertize-function'?
-            (prog-indentation-context ;#dynamic
-             (list (+ leftouter c-basic-offset) (list (1+ boa)))))
+      (let ((prog-indentation-context ;#dynamic
+             (list (+ leftouter c-basic-offset))))
+        ;; TODO: do we also need to call `syntax-propertize'?
+        ;; and/or bind `syntax-propertize-function'?
         (narrow-to-region (1+ boa) (point-max))
         (python-indent-line (eq this-command 'antlr-indent-command))))))
 

Diff finished.  Sat Dec 16 19:27:17 2017

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Wedler, Christoph]

Dmitry Gutov wrote:

> Is there a reason why you called narrow-to-region in
> antlr-python-indent-action-line anyway, even when using
> prog-indentation-context?  Because you do that, though, the "new way"
> should work with your code automatically.

Prophesy, I guess ;-)

> I'm attaching a patch with some minor cleanup, though.

To be honest, I am not so sure about the cleanup.  I think it is better
to dynamically bind `syntax-ppss-cache' and `syntax-ppss-last' while
letting python-mode do its thing: neither should python-mode use the
cache state set from the ANTLR syntax, nor should antlr-mode later use
the syntax state set by pathon-mode.  Or do I miss something?

> Please try it with a build of Emacs from the 'widen-less' branch. Do you 
> know how to build it?

It seems to have worked, even on a MacOS - good!  To do it without much
Additional installation for the moment, I used
    ./configure --with-ns --without-makeinfo --with-gnutls=no

>> I you want to try some ANTLR grammar examples yourself, I would
>> recommend:
>>   - antlr-mode 3.1.4
>>     from https://sourceforge.net/projects/antlr-mode/files/
>>   - example grammar files, especially lines 46 to 51 in
>>     https://github.com/antlr/examples-v3/blob/master/Python/C/C.g
>>   - and others in that repo (mentioned in line 100 of antlr-mode.el)

> antlr-mode.el says these examples are to see the syntax highlighting. 
> Any particular file and place to see how indentation works out? I 
> couldn't find one from a cursory search.

Are you sure that you use antlr-mode 3.1.5 (not the one from Emacs)?
Everything works for me with Emacs-26.0.90, too – I also tried an empty
init.el…

>> Now to a mode with non-widening indentation function: emacs-lisp mode:
>> Consider the following code
>>      (defun foo ()
>>        (+ 1 3))
>>
>> I would assume that the indentation of line 2 does not change
>> when I narrow to line 2 - but Emacs-25.1.1 now deletes the first 2
>> spaces of line 2...  (yes; I also need to update to a newer Emacs)

> Try out the widen-less branch!

Ehem, still the same...

> And also, can we get your help with the manual? It still says:

> Examples include
>  @dfn{literate programming} source files that combine documentation and
>  snippets of source code, Yacc/Bison programs that include snippets of
>  plain C code, etc.  To correctly indent the embedded chunks, the primary
>  mode needs to delegate the indentation to another mode's indentation
>  engine (e.g., call @code{c-indent-defun} for C code or
>  @code{python-indent-line} for Python)

> I.e., it still mentions C and c-indent-defun (?). But c-indent-defun 
> does not work for embedded chunks (yet)!

> Can we replace that mentions with JS and js-indent-line, for instance? 
> Would that still make sense?

Yes, using `js-indent-line' is a better example.
`c-indent-line' (c-indent-defun looks like a typo) is tricky - see my
ugly function `antlr-indent-line' (but I have a TODO...)

P.S. I will probably "borrow" (aka "steal") from mhtml-mode.el later...

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/18/17 2:39 PM, Wedler, Christoph wrote:

>> I'm attaching a patch with some minor cleanup, though.
> 
> To be honest, I am not so sure about the cleanup.  I think it is better
> to dynamically bind `syntax-ppss-cache' and `syntax-ppss-last' while
> letting python-mode do its thing: neither should python-mode use the
> cache state set from the ANTLR syntax, nor should antlr-mode later use
> the syntax state set by pathon-mode.  Or do I miss something?

It shouldn't matter much either way, but in Emacs 26 syntax-ppss already 
checks that point-min has not changed since the last invocation.

And it uses two caches under the hood: one for the "full" buffer, and 
another for the most recent narrowing.

>> Please try it with a build of Emacs from the 'widen-less' branch. Do you
>> know how to build it?
> 
> It seems to have worked, even on a MacOS - good!  To do it without much
> Additional installation for the moment, I used
>      ./configure --with-ns --without-makeinfo --with-gnutls=no

Good.

Did you use the same recipe when testing the widen-less branch, though? 
I'm pretty sure you can't simply run ./configure when building from 
development sources (not a pre-built tarball).

> Are you sure that you use antlr-mode 3.1.5 (not the one from Emacs)?

Yes.

> Everything works for me with Emacs-26.0.90, too – I also tried an empty
> init.el…

I'm saying I don't know what to check. Indentation certainly doesn't 
work the same, e.g. because the examples use a non-default indentation 
offset. But I'm not sure that's all.

>>> Now to a mode with non-widening indentation function: emacs-lisp mode:
>>> Consider the following code
>>>       (defun foo ()
>>>         (+ 1 3))
>>>
>>> I would assume that the indentation of line 2 does not change
>>> when I narrow to line 2 - but Emacs-25.1.1 now deletes the first 2
>>> spaces of line 2...  (yes; I also need to update to a newer Emacs)
> 
>> Try out the widen-less branch!
> 
> Ehem, still the same...

Are you sure you tried the right branch? It works differently over here 
(namely: doesn't change the indentation).

> Yes, using `js-indent-line' is a better example.
> `c-indent-line' (c-indent-defun looks like a typo) is tricky - see my
> ugly function `antlr-indent-line' (but I have a TODO...)

I've pushed that change. If you have any more comments on the manual 
entry and it's written now, please let me know.

> P.S. I will probably "borrow" (aka "steal") from mhtml-mode.el later...

Sounds good!

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Wedler, Christoph]

Dmitry Gutov wrote:
> On 12/18/17 2:39 PM, Wedler, Christoph wrote:

> It shouldn't matter much either way, but in Emacs 26 syntax-ppss already 
> checks that point-min has not changed since the last invocation.

OK, then I'll keep the dynamic binding to make it work in Emacs-25, too.

> Did you use the same recipe when testing the widen-less branch, though? 
> I'm pretty sure you can't simply run ./configure when building from 
> development sources (not a pre-built tarball).

Actually, I build that branch only

    Head:     widen-less Consolidate 'widen' calls
    Merge:    origin/widen-less Consolidate 'widen' calls
    Tag:      emacs-26.0.90 (351)

The build sequence was

    ./autogen.sh
    ./configure --with-ns --without-makeinfo --with-gnutls=no
    make

> I'm saying I don't know what to check. Indentation certainly doesn't 
> work the same, e.g. because the examples use a non-default indentation 
> offset. But I'm not sure that's all.

OK, ANTLR source have TAB length 4, the file is not even completly
consistently indented with that, but the lines of interest are 46 to 51
of Python/C/C.g of the antlr3 examples repo, as mentioned earlier,
namely:

@members {
    def isTypeName(self, name):
        for scope in reversed(self.Symbols_stack):
            if name in scope.types:
                return True

        return False

}

It looks strange to me that you neither get syntax highlighting
(keywords "def", "self", "for", …, function name "isTypeName", ...)
nor the indentation cycling for the lines starting with "for", "if",
"return"

>>> Now to a mode with non-widening indentation function: emacs-lisp mode:
>>> Consider the following code
>>>       (defun foo ()
>>>         (+ 1 3))
>>>
>>> I would assume that the indentation of line 2 does not change
>>> when I narrow to line 2 - but Emacs-25.1.1 now deletes the first 2
>>> spaces of line 2...  (yes; I also need to update to a newer Emacs)
> 
>> Try out the widen-less branch!
> 
> Ehem, still the same...

> Are you sure you tried the right branch? It works differently over here 
> (namely: doesn't change the indentation).

Yes, see above - and no "init.el" except using antlr-mode 3.1.5.  Very
strange...

>> P.S. I will probably "borrow" (aka "steal") from mhtml-mode.el later...

Sorry Stefan, I forgot the ";-" after "steal"...

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/18/17 7:41 PM, Wedler, Christoph wrote:

> OK, then I'll keep the dynamic binding to make it work in Emacs-25, too.

But that branch checks for (boundp 'prog-indentation-context). Which got 
taken out of Emacs 25, didn't it?

> Actually, I build that branch only
> 
>      Head:     widen-less Consolidate 'widen' calls
>      Merge:    origin/widen-less Consolidate 'widen' calls
>      Tag:      emacs-26.0.90 (351)

Not sure which tool's output this is.

> The build sequence was
> 
>      ./autogen.sh
>      ./configure --with-ns --without-makeinfo --with-gnutls=no
>      make

Could you try with 'make bootstrap' instead of just 'make'?

> OK, ANTLR source have TAB length 4, the file is not even completly
> consistently indented with that, but the lines of interest are 46 to 51
> of Python/C/C.g of the antlr3 examples repo, as mentioned earlier,
> namely:
> 
> @members {
>      def isTypeName(self, name):
>          for scope in reversed(self.Symbols_stack):
>              if name in scope.types:
>                  return True
> 
>          return False
> 
> }

It seems to work okay. Thanks.

> It looks strange to me that you neither get syntax highlighting
> (keywords "def", "self", "for", …, function name "isTypeName", ...)
> nor the indentation cycling for the lines starting with "for", "if",
> "return"

I don't get syntax highlighting there, but indentation works fine. I 
just needed to know which part of the example file to test.

>>>> Now to a mode with non-widening indentation function: emacs-lisp mode:
>>>> Consider the following code
>>>>        (defun foo ()
>>>>          (+ 1 3))
>>>>
>>>> I would assume that the indentation of line 2 does not change
>>>> when I narrow to line 2 - but Emacs-25.1.1 now deletes the first 2
>>>> spaces of line 2...  (yes; I also need to update to a newer Emacs)
>>
>>> Try out the widen-less branch!
>>
>> Ehem, still the same...

If 'make bootstrap' doesn't help, could you try a step-by-step 
instruction, or a video, or something?

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Wedler, Christoph]

[-- Attachment #1: Type: text/plain, Size: 1839 bytes --]

Dmitry Gutov wrote:
> On 12/18/17 7:41 PM, Wedler, Christoph wrote:

>>      Head:     widen-less Consolidate 'widen' calls
>>      Merge:    origin/widen-less Consolidate 'widen' calls
>>      Tag:      emacs-26.0.90 (351)

> Not sure which tool's output this is.

It is copied from the magit status buffer for Emacs…

> Could you try with 'make bootstrap' instead of just 'make'?

I did that – no effect.
And if I look at

(defun indent-for-tab-command (&optional arg)
  … 
      (or (not (eq (funcall indent-line-function) 'noindent))
          (indent--default-inside-comment)
          (when (or (<= (current-column) (current-indentation))
                    (not (eq tab-always-indent 'complete)))
            (save-restriction
              (widen)
              (funcall (default-value 'indent-line-function)))))

I do not see why it should now behave correctly
(the “called” funcall is outside “save-restriction – widen”)

> I don't get syntax highlighting there, but indentation works fine. I 
> just needed to know which part of the example file to test.

Hm, with src/emacs –q &  and M-x load-file <path-to-antlr-mode-3.1.5>/antlr-mode.elc
I get the result of the attached screenshot area. 

If 'make bootstrap' doesn't help, could you try a step-by-step 
instruction, or a video, or something?

In bash:
      ./autogen.sh
      ./configure --with-ns --without-makeinfo --with-gnutls=no
      make
      src/emacs –q &

in Emacs_
      C-x C-f new.el RET
      Insert: “(defun foo ()” RET
      Insert: <2 spaces> “3)” RET
      C-x SPC  C-p  C-x n n (enable narrowing with  SPC)
      (region still active) TAB -> works (i.e., no deletion of the spaces)
      (region no longer active) TAB -> deletes the 2 spaces (wrong as before)
 
      


[-- Attachment #2: antlr3-example.png --]
[-- Type: image/png, Size: 43882 bytes --]

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/19/17 1:27 PM, Wedler, Christoph wrote:

> I do not see why it should now behave correctly
> (the “called” funcall is outside “save-restriction – widen”)

Ouch. Thanks for that, should be fixed now. I only tested this with 
indent-region, in fact. And emacs-lisp-mode has a dedicated 
indent-region-function.

>> I don't get syntax highlighting there, but indentation works fine. I
>> just needed to know which part of the example file to test.
> 
> Hm, with src/emacs –q &  and M-x load-file <path-to-antlr-mode-3.1.5>/antlr-mode.elc
> I get the result of the attached screenshot area.

Hmm, yes. This way it works. But not with my full configuration (just 
omitting -q from this scenario).

Not sure what can be going wrong there; maybe incompatibility with some 
minor modes. All I can say is that font-lock-keywords don't end up 
having the entry for the Python keywords in the latter case. Nor the 
entries for the constants or built-in functions, for that matter.

Anyway, that's not a big problem for me. No immediate plans to edit 
Antlr grammars.

>        (region still active) TAB -> works (i.e., no deletion of the spaces)
>        (region no longer active) TAB -> deletes the 2 spaces (wrong as before)

Right, I only tested the first case originally.