Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

>     Replace prog-widen with consolidating widen calls

So, IIUC the idea is that the multi-major-mode framework will *have* to
use narrowing to indicate to the sub-modes the boundaries of the current
chunk, right?


        Stefan


> ---
>  doc/lispref/text.texi       | 45 +++++----------------------------------------
>  etc/NEWS                    |  8 ++++----
>  lisp/indent.el              | 12 +++++++++---
>  lisp/progmodes/prog-mode.el | 28 ++--------------------------
>  lisp/progmodes/python.el    | 34 ++++++++--------------------------
>  lisp/progmodes/ruby-mode.el |  1 -
>  lisp/vc/add-log.el          |  4 +++-
>  7 files changed, 31 insertions(+), 101 deletions(-)

> diff --git a/doc/lispref/text.texi b/doc/lispref/text.texi
> index 1e19f75..5a63e5b 100644
> --- a/doc/lispref/text.texi
> +++ b/doc/lispref/text.texi
> @@ -2407,42 +2407,17 @@ such multi-mode indentation.
>  This variable, when non-@code{nil}, holds the indentation context for
>  the sub-mode's indentation engine provided by the superior major mode.
>  The value should be a list of the form @code{(@var{first-column}
> -@w{(@var{start} . @var{end})} @code{prev-chunk})}.  The members of the
> -list have the following meaning:
> +@code{...rest})}.  The members of the list have the following meaning:
 
>  @table @var
>  @item first-column
>  The column to be used for top-level constructs.  This replaces the
>  default value of the top-level column used by the sub-mode, usually
>  zero.
> -@item start
> -@itemx end
> -The region of the code chunk to be indented by the sub-mode.  The
> -value of @var{end} can be @code{nil}, which stands for the value of
> -@code{point-max}.
> -@item prev-chunk
> -If this is non-@code{nil}, it should provide the sub-mode's
> -indentation engine with a virtual context of the code chunk.  Valid
> -values include:
> -
> -@itemize @minus
> -@item
> -A string whose contents is the text the sub-mode's indentation engine
> -should consider to precede the code chunk.  The sub-mode's indentation
> -engine can add text properties to that string, to be reused in
> -repeated calls with the same string, thus using it as a cache.  An
> -example where this is useful is code chunks that need to be indented
> -as function bodies, but lack the function's preamble---the string
> -could then include that missing preamble.
> -@item
> -A function.  It is expected to be called with the start position of
> -the current chunk, and should return a cons cell
> -@w{@code{(@var{prev-start} . @var{prev-end})}} that specifies the
> -region of the previous code chunk, or @code{nil} if there is no previous
> -chunk.  This is useful in literate-programming sources, where code is
> -split into chunks, and correct indentation needs to access previous
> -chunks.
> -@end itemize
> +@item rest
> +Reserved for future use.  The intent is to provide the sub-mode's
> +indentation engine with a virtual context of the code chunk.  Working
> +patches that use this in a significant fashion are welcome.
>  @end table
>  @end defvar
 
> @@ -2457,16 +2432,6 @@ function's value is the column number to use for top-level constructs.
>  When no superior mode is in effect, this function returns zero.
>  @end defun
 
> -@defun prog-widen
> -Call this function instead of @code{widen} to remove any restrictions
> -imposed by the mode's indentation engine and restore the restrictions
> -recorded in @code{prog-indentation-context}.  This prevents the
> -indentation engine of a sub-mode from inadvertently operating on text
> -outside of the chunk it was supposed to indent, and preserves the
> -restriction imposed by the superior mode.  When no superior mode is in
> -effect, this function just calls @code{widen}.
> -@end defun
> -
 
>  @node Region Indent
>  @subsection Indenting an Entire Region
> diff --git a/etc/NEWS b/etc/NEWS
> index 4ccf468..6e1561d 100644
> --- a/etc/NEWS
> +++ b/etc/NEWS
> @@ -1108,10 +1108,10 @@ programming languages in the same buffer, like literate programming
>  environments or ANTLR programs with embedded Python code.
 
>  A major mode can provide indentation context for a sub-mode through
> -the 'prog-indentation-context' variable.  To support this, modes that
> -provide indentation should use 'prog-widen' instead of 'widen' and
> -'prog-first-column' instead of a literal zero.  See the node
> -"(elisp) Mode-Specific Indent" in the ELisp manual for more details.
> +the 'prog-indentation-context' variable.  To support this, modes
> +should use 'prog-first-column' instead of a literal zero and never
> +call 'widen' in their indentation functions.  See the node "(elisp)
> +Mode-Specific Indent" in the ELisp manual for more details.
 
>  ** ERC
 
> diff --git a/lisp/indent.el b/lisp/indent.el
> index d5ba0bd..ccf0e99 100644
> --- a/lisp/indent.el
> +++ b/lisp/indent.el
> @@ -69,6 +69,8 @@ variable is `indent-relative' or `indent-relative-maybe', handle
>  it specially (since those functions are used for tabbing); in
>  that case, indent by aligning to the previous non-blank line."
>    (interactive)
> +  (save-restriction
> +    (widen)
>    (syntax-propertize (line-end-position))
>    (if (memq indent-line-function
>  	    '(indent-relative indent-relative-maybe))
> @@ -84,7 +86,7 @@ that case, indent by aligning to the previous non-blank line."
>  	    (indent-line-to column)
>  	  (save-excursion (indent-line-to column))))
>      ;; The normal case.
> -    (funcall indent-line-function)))
> +    (funcall indent-line-function))))
 
>  (defun indent--default-inside-comment ()
>    (unless (or (> (current-column) (current-indentation))
> @@ -144,7 +146,9 @@ prefix argument is ignored."
>            (indent--default-inside-comment)
>            (when (or (<= (current-column) (current-indentation))
>                      (not (eq tab-always-indent 'complete)))
> -            (funcall (default-value 'indent-line-function))))
> +            (save-restriction
> +              (widen)
> +              (funcall (default-value 'indent-line-function)))))
 
>        (cond
>         ;; If the text was already indented right, try completion.
> @@ -538,7 +542,9 @@ column to indent to; if it is nil, use one of the three methods above."
>  	  (forward-line 1)))))
>     ;; Use indent-region-function is available.
>     (indent-region-function
> -    (funcall indent-region-function start end))
> +    (save-restriction
> +      (widen)
> +      (funcall indent-region-function start end)))
>     ;; Else, use a default implementation that calls indent-line-function on
>     ;; each line.
>     (t (indent-region-line-by-line start end)))
> diff --git a/lisp/progmodes/prog-mode.el b/lisp/progmodes/prog-mode.el
> index f727e45..b36d242 100644
> --- a/lisp/progmodes/prog-mode.el
> +++ b/lisp/progmodes/prog-mode.el
> @@ -64,37 +64,13 @@ mode, it should bind this variable to non-nil around the call.
 
>  The non-nil value should be a list of the form:
 
> -   (FIRST-COLUMN (START . END) PREVIOUS-CHUNKS)
> +   (FIRST-COLUMN ...REST)
 
>  FIRST-COLUMN is the column the indentation engine of the sub-mode
>  should use for top-level language constructs inside the code
>  chunk (instead of 0).
 
> -START and END specify the region of the code chunk.  END can be
> -nil, which stands for the value of `point-max'.  The function
> -`prog-widen' uses this to restore restrictions imposed by the
> -sub-mode's indentation engine.
> -
> -PREVIOUS-CHUNKS, if non-nil, provides the indentation engine of
> -the sub-mode with the virtual context of the code chunk.  Valid
> -values are:
> -
> - - A string containing text which the indentation engine can
> -   consider as standing in front of the code chunk.  To cache the
> -   string's calculated syntactic information for repeated calls
> -   with the same string, the sub-mode can add text-properties to
> -   the string.
> -
> -   A typical use case is for grammars with code chunks which are
> -   to be indented like function bodies -- the string would contain
> -   the corresponding function preamble.
> -
> - - A function, to be called with the start position of the current
> -   chunk.  It should return either the region of the previous chunk
> -   as (PREV-START . PREV-END), or nil if there is no previous chunk.
> -
> -   A typical use case are literate programming sources -- the
> -   function would successively return the previous code chunks.")
> +REST is currently unused, but can be defined in future versions.")
 
>  (defun prog-indent-sexp (&optional defun)
>    "Indent the expression after point.
> diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
> index d4226e5..104889a 100644
> --- a/lisp/progmodes/python.el
> +++ b/lisp/progmodes/python.el
> @@ -287,10 +287,6 @@
>  ;;; 24.x Compat
>  \f
 
> -(unless (fboundp 'prog-widen)
> -  (defun prog-widen ()
> -    (widen)))
> -
>  (unless (fboundp 'prog-first-column)
>    (defun prog-first-column ()
>      0))
> @@ -785,7 +781,7 @@ work on `python-indent-calculate-indentation' instead."
>    (interactive)
>    (save-excursion
>      (save-restriction
> -      (prog-widen)
> +      (widen)
>        (goto-char (point-min))
>        (let ((block-end))
>          (while (and (not block-end)
> @@ -883,8 +879,6 @@ keyword
>  :at-dedenter-block-start
>   - Point is on a line starting a dedenter block.
>   - START is the position where the dedenter block starts."
> -  (save-restriction
> -    (prog-widen)
>      (let ((ppss (save-excursion
>                    (beginning-of-line)
>                    (syntax-ppss))))
> @@ -1022,7 +1016,7 @@ keyword
>                        (looking-at (python-rx block-ender)))
>                      :after-block-end)
>                     (t :after-line))
> -             (point)))))))))
> +             (point))))))))
 
>  (defun python-indent--calculate-indentation ()
>    "Internal implementation of `python-indent-calculate-indentation'.
> @@ -1030,8 +1024,6 @@ May return an integer for the maximum possible indentation at
>  current context or a list of integers.  The latter case is only
>  happening for :at-dedenter-block-start context since the
>  possibilities can be narrowed to specific indentation points."
> -  (save-restriction
> -    (prog-widen)
>      (save-excursion
>        (pcase (python-indent-context)
>          (`(:no-indent . ,_) (prog-first-column)) ; usually 0
> @@ -1081,7 +1073,7 @@ possibilities can be narrowed to specific indentation points."
>          (`(,(or :inside-paren-newline-start-from-block) . ,start)
>           ;; Add two indentation levels to make the suite stand out.
>           (goto-char start)
> -         (+ (current-indentation) (* python-indent-offset 2)))))))
> +         (+ (current-indentation) (* python-indent-offset 2))))))
 
>  (defun python-indent--calculate-levels (indentation)
>    "Calculate levels list given INDENTATION.
> @@ -4589,8 +4581,6 @@ To this:
>  Optional argument INCLUDE-TYPE indicates to include the type of the defun.
>  This function can be used as the value of `add-log-current-defun-function'
>  since it returns nil if point is not inside a defun."
> -  (save-restriction
> -    (prog-widen)
>      (save-excursion
>        (end-of-line 1)
>        (let ((names)
> @@ -4644,7 +4634,7 @@ since it returns nil if point is not inside a defun."
>              (and (= (current-indentation) 0) (throw 'exit t))))
>          (and names
>               (concat (and type (format "%s " type))
> -                     (mapconcat 'identity names ".")))))))
> +                     (mapconcat 'identity names "."))))))
 
>  (defun python-info-current-symbol (&optional replace-self)
>    "Return current symbol using dotty syntax.
> @@ -4788,12 +4778,10 @@ likely an invalid python file."
>    "Message the first line of the block the current statement closes."
>    (let ((point (python-info-dedenter-opening-block-position)))
>      (when point
> -      (save-restriction
> -        (prog-widen)
>          (message "Closes %s" (save-excursion
>                                 (goto-char point)
>                                 (buffer-substring
> -                                (point) (line-end-position))))))))
> +                                (point) (line-end-position)))))))
 
>  (defun python-info-dedenter-statement-p ()
>    "Return point if current statement is a dedenter.
> @@ -4809,8 +4797,6 @@ statement."
>    "Return non-nil if current line ends with backslash.
>  With optional argument LINE-NUMBER, check that line instead."
>    (save-excursion
> -    (save-restriction
> -      (prog-widen)
>        (when line-number
>          (python-util-goto-line line-number))
>        (while (and (not (eobp))
> @@ -4819,14 +4805,12 @@ With optional argument LINE-NUMBER, check that line instead."
>                    (not (equal (char-before (point)) ?\\)))
>          (forward-line 1))
>        (when (equal (char-before) ?\\)
> -        (point-marker)))))
> +        (point-marker))))
 
>  (defun python-info-beginning-of-backslash (&optional line-number)
>    "Return the point where the backslashed line start.
>  Optional argument LINE-NUMBER forces the line number to check against."
>    (save-excursion
> -    (save-restriction
> -      (prog-widen)
>        (when line-number
>          (python-util-goto-line line-number))
>        (when (python-info-line-ends-backslash-p)
> @@ -4835,15 +4819,13 @@ Optional argument LINE-NUMBER forces the line number to check against."
>                   (python-syntax-context 'paren))
>            (forward-line -1))
>          (back-to-indentation)
> -        (point-marker)))))
> +        (point-marker))))
 
>  (defun python-info-continuation-line-p ()
>    "Check if current line is continuation of another.
>  When current line is continuation of another return the point
>  where the continued line ends."
>    (save-excursion
> -    (save-restriction
> -      (prog-widen)
>        (let* ((context-type (progn
>                               (back-to-indentation)
>                               (python-syntax-context-type)))
> @@ -4869,7 +4851,7 @@ where the continued line ends."
>                 (python-util-forward-comment -1)
>                 (when (and (equal (1- line-start) (line-number-at-pos))
>                            (python-info-line-ends-backslash-p))
> -                 (point-marker))))))))
> +                 (point-marker)))))))
 
>  (defun python-info-block-continuation-line-p ()
>    "Return non-nil if current line is a continuation of a block."
> diff --git a/lisp/progmodes/ruby-mode.el b/lisp/progmodes/ruby-mode.el
> index dc1b0f8..7c10114 100644
> --- a/lisp/progmodes/ruby-mode.el
> +++ b/lisp/progmodes/ruby-mode.el
> @@ -1364,7 +1364,6 @@ delimiter."
>                                               "\\)\\>")))
>                      (eq (ruby-deep-indent-paren-p t) 'space)
>                      (not (bobp)))
> -                   (widen)
>                     (goto-char (or begin parse-start))
>                     (skip-syntax-forward " ")
>                     (current-column))
> diff --git a/lisp/vc/add-log.el b/lisp/vc/add-log.el
> index 392147b..9da8abc 100644
> --- a/lisp/vc/add-log.el
> +++ b/lisp/vc/add-log.el
> @@ -1133,6 +1133,8 @@ identifiers followed by `:' or `='.  See variables
>  Has a preference of looking backwards."
>    (condition-case nil
>        (save-excursion
> +        (save-restriction
> +          (widen)
>  	(if add-log-current-defun-function
>  	    (funcall add-log-current-defun-function)
>  	  ;; If all else fails, try heuristics
> @@ -1147,7 +1149,7 @@ Has a preference of looking backwards."
>  	      (when (string-match "\\([^ \t\n\r\f].*[^ \t\n\r\f]\\)"
>  				  result)
>  		(setq result (match-string-no-properties 1 result)))
> -	      result))))
> +	      result)))))
>      (error nil)))
 
>  (defvar change-log-get-method-definition-md)

> _______________________________________________
> Emacs-diffs mailing list
> Emacs-diffs@gnu.org
> https://lists.gnu.org/mailman/listinfo/emacs-diffs

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 11/30/17 1:53 AM, Stefan Monnier wrote:
>>      Replace prog-widen with consolidating widen calls
> 
> So, IIUC the idea is that the multi-major-mode framework will *have* to
> use narrowing to indicate to the sub-modes the boundaries of the current
> chunk, right?

Yes. Just like it would *have* to bind prog-indentation-context to a 
complex-ish value in the previous proposal.

I can't imagine a case where it would be inconvenient to do so.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 11/30/17 8:59 AM, Dmitry Gutov wrote:
> On 11/30/17 1:53 AM, Stefan Monnier wrote:
>>>      Replace prog-widen with consolidating widen calls
>>
>> So, IIUC the idea is that the multi-major-mode framework will *have* to
>> use narrowing to indicate to the sub-modes the boundaries of the current
>> chunk, right?
> 
> Yes. Just like it would *have* to bind prog-indentation-context to a 
> complex-ish value in the previous proposal.

And while the difference isn't night and day, allow me to elaborate why 
I think this is a move in a good direction:

- Major mode authors have been forgetting to widen in all the cases they 
actually have to. Or they would do that in indent-line-function, but 
forget in beginning-of-defun-function, etc. Now, they are *mandated* to 
obey the current narrowing (deleting code is easy, and some won't have 
delete anything at all). The indentation code might fail to work inside 
narrowing (like all of CC Mode, probably), but that's an orthogonal issue.

- Multi-mode packages have been using narrowing for this purpose for 
years, so they won't have to change much.

And, it's kind of funny, mhtml-indent-line which we've reviewed and 
added recently, uses narrow-to-region too. :-) While disregarding the 
second element of prog-indentation-context (which it set to the value 
"widen all you want"). Which works well enough because SMIE doesn't 
widen, and I've removed the `widen' call from js-mode indentation code 
3.5 years ago.

I'm adding Tom and Vitalie to Cc. Maybe they have something to add.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Tom Tromey]

>>>>> "Dmitry" == Dmitry Gutov <dgutov@yandex.ru> writes:

Dmitry> And, it's kind of funny, mhtml-indent-line which we've reviewed and
Dmitry> added recently, uses narrow-to-region too. :-) While disregarding the
Dmitry> second element of prog-indentation-context (which it set to the value
Dmitry> "widen all you want"). Which works well enough because SMIE doesn't
Dmitry> widen, and I've removed the `widen' call from js-mode indentation code
Dmitry> 3.5 years ago.

Yes, IIRC mhtml-mode narrows so that a sub-mode won't try to examine
parts of the buffer that have a different syntax.  Whether you consider
this ugly or not is perhaps related to your stance on the whole widening
issue.  My view, though, is that there is a problem here intrinsic to
multi-modes: something has to tell the sub-modes about the relevant
boundaries in some way; so if the narrowing/widening stuff is changed,
something new will have to be invented here.

Tom

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 11/30/17 3:31 PM, Tom Tromey wrote:
> Yes, IIRC mhtml-mode narrows so that a sub-mode won't try to examine
> parts of the buffer that have a different syntax.  Whether you consider
> this ugly or not is perhaps related to your stance on the whole widening
> issue.  My view, though, is that there is a problem here intrinsic to
> multi-modes: something has to tell the sub-modes about the relevant
> boundaries in some way;

Err, no. Ugly or not is beside the point (I personally prefer this 
approach, please also take a look at the branch referenced in this 
thread subject).

Thing is, if you're using prog-indentation-context as it's now 
implemented, you're "supposed to" indicate the narrowing bounds via 
setting the second element of that list, instead of calling 
narrow-to-region directly. Of course, that also requires support from 
the major modes in question.

 > so if the narrowing/widening stuff is changed,
 > something new will have to be invented here.

See above. The "something new" has a few rough edges, so this discussion 
is about revisiting those choices.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Alan Mackenzie]

Hello, Dmitry.

On Thu, Nov 30, 2017 at 10:58:47 +0000, Dmitry Gutov wrote:
> On 11/30/17 8:59 AM, Dmitry Gutov wrote:
> > On 11/30/17 1:53 AM, Stefan Monnier wrote:
> >>>      Replace prog-widen with consolidating widen calls

> >> So, IIUC the idea is that the multi-major-mode framework will *have* to
> >> use narrowing to indicate to the sub-modes the boundaries of the current
> >> chunk, right?

Using narrowing for marking the bounds of a sub-mode is a bad thing,
since it is likely to cause contention with other uses of narrowing.
There are better ways.

> > Yes. Just like it would *have* to bind prog-indentation-context to a 
> > complex-ish value in the previous proposal.

> And while the difference isn't night and day, allow me to elaborate why 
> I think this is a move in a good direction:

> - Major mode authors have been forgetting to widen in all the cases they 
> actually have to. Or they would do that in indent-line-function, but 
> forget in beginning-of-defun-function, etc. Now, they are *mandated* to 
> obey the current narrowing

It's not clear what is meant here, but mandating maintainers of major
modes to use narrowing in a particular way is at best controversial, and
probably will render many major modes non-functional.

> (deleting code is easy, and some won't have delete anything at all).
> The indentation code might fail to work inside narrowing (like all of
> CC Mode, probably), but that's an orthogonal issue.

Narrowing belongs to users and major modes.  I am lacking the context of
your post, which appears to start deep down in some mail thread which I
don't have, so I can't follow the details of your proposal.  But I would
appreciate you confirming that these proposals won't place any
constraints whatsoever on how users and major modes can use narrowing.

You can also understand me being a bit concerned at the reference to CC
Mode.  ;-)

> - Multi-mode packages have been using narrowing for this purpose for 
> years, so they won't have to change much.

They have used narrowing because that is the only tool they have had.

> And, it's kind of funny, mhtml-indent-line which we've reviewed and 
> added recently, uses narrow-to-region too. :-) While disregarding the 
> second element of prog-indentation-context (which it set to the value 
> "widen all you want"). Which works well enough because SMIE doesn't 
> widen, and I've removed the `widen' call from js-mode indentation code 
> 3.5 years ago.

> I'm adding Tom and Vitalie to Cc. Maybe they have something to add.

-- 
Alan Mackenzie (Nuremberg, Germany).

[SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

> Using narrowing for marking the bounds of a sub-mode is a bad thing,
> since it is likely to cause contention with other uses of narrowing.

Dmitry's solution to this threat is to say that indent-line-function
should always be called fully-widened, so there can be no "other uses of
narrowing" that can get in the way.

> It's not clear what is meant here, but mandating maintainers of major
> modes to use narrowing in a particular way is at best controversial, and
> probably will render many major modes non-functional.

It doesn't mandate that major modes use narrowing.  Quite the opposite:
it says "don't touch narrowing inside your indentation function (unless
you know what you're doing), because it's already setup for you".


        Stefan

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

Hi Alan,

On 11/30/17 9:46 PM, Alan Mackenzie wrote:

> Using narrowing for marking the bounds of a sub-mode is a bad thing,
> since it is likely to cause contention with other uses of narrowing.

Please give an example.

> It's not clear what is meant here, 

To rephrase, don't widen in indent-line-function or 
beginning-of-defun-function.

> but mandating maintainers of major
> modes to use narrowing in a particular way is at best controversial, and
> probably will render many major modes non-functional.

I don't see a reason why. Even more, it should be compatible with all 
uses of narrowing that I know about.

> Narrowing belongs to users and major modes. 

It can serve all.

> I am lacking the context of
> your post, which appears to start deep down in some mail thread which I
> don't have, so I can't follow the details of your proposal.  But I would
> appreciate you confirming that these proposals won't place any
> constraints whatsoever on how users and major modes can use narrowing.

Please see 
http://lists.gnu.org/archive/html/emacs-devel/2017-11/msg00691.html and 
http://lists.gnu.org/archive/html/emacs-devel/2016-12/msg00796.html.

> You can also understand me being a bit concerned at the reference to CC
> Mode.  ;-)

You shouldn't be: CC Mode can just ignore this new rule, as long as it's 
too hard to support embedding it in multi-mode buffers.

>> - Multi-mode packages have been using narrowing for this purpose for
>> years, so they won't have to change much.
> 
> They have used narrowing because that is the only tool they have had.

The only other tool we have is also narrowing, semantically.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Alan Mackenzie]

Hello, Dmitry.

Just to be clear, in the following, by "narrowing" I mean all operations
of narrowing and widening regarded generically.

On Thu, Nov 30, 2017 at 23:09:00 +0000, Dmitry Gutov wrote:
> Hi Alan,

> On 11/30/17 9:46 PM, Alan Mackenzie wrote:

> > Using narrowing for marking the bounds of a sub-mode is a bad thing,
> > since it is likely to cause contention with other uses of narrowing.

> Please give an example.

A low level function which as an essential part of its functionality
(for example, to make sure point-min isn't within a comment or string)
widens.

> > It's not clear what is meant here, 

> To rephrase, don't widen in indent-line-function or 
> beginning-of-defun-function.

This is an intolerable restriction.  The low level function mentioned
above cannot, should not, must not know whether it's being called
(indirectly) from indent-line-function or b-o-d-function.  It will have
to widen in all cases or none.  Therefore there will be failures whilst
being called either from one of the two noted functions or from
elsewhere.

> > but mandating maintainers of major
> > modes to use narrowing in a particular way is at best controversial, and
> > probably will render many major modes non-functional.

> I don't see a reason why. Even more, it should be compatible with all 
> uses of narrowing that I know about.

See above.

> > Narrowing belongs to users and major modes. 

> It can serve all.

Indeed it can, and it must.  A super-mode thus may not "reserve"
narrowing for its own purposes to the exclusion of other uses.

[ .... ]

> > You can also understand me being a bit concerned at the reference to CC
> > Mode.  ;-)

> You shouldn't be: CC Mode can just ignore this new rule, as long as it's 
> too hard to support embedding it in multi-mode buffers.

The multi-mode mechanism should be designed to be usable with any major
mode.  There's nothing particularly hard about supporting CC Mode in a
well designed multi-mode scheme.

> >> - Multi-mode packages have been using narrowing for this purpose for
> >> years, so they won't have to change much.

> > They have used narrowing because that is the only tool they have had.

> The only other tool we have is also narrowing, semantically.

We need better tools.  I have already proposed and offered to implement
such tools.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Alan Mackenzie]

Hello, Stefan.

On Thu, Nov 30, 2017 at 18:03:05 -0500, Stefan Monnier wrote:
> > Using narrowing for marking the bounds of a sub-mode is a bad thing,
> > since it is likely to cause contention with other uses of narrowing.

> Dmitry's solution to this threat is to say that indent-line-function
> should always be called fully-widened, .....

That is a horrible, confusing, restriction, and an incompatibility with
older and other Emacsen.

> .... so there can be no "other uses of narrowing" that can get in the
> way.

The indent-line-function will be calling low-level functions which
themselves widen, possibly to determine whether point-min is in a
literal, or more generally to determine context.  These low-level
functions will also be called from other contexts, hence the necessity
of widening.

> > It's not clear what is meant here, but mandating maintainers of major
> > modes to use narrowing in a particular way is at best controversial, and
> > probably will render many major modes non-functional.

> It doesn't mandate that major modes use narrowing.  Quite the opposite:
> it says "don't touch narrowing inside your indentation function (unless
> you know what you're doing), because it's already setup for you".

That is mandating the way major modes use (or don't use) narrowing.
It's entirely the wrong thing to do, and will lead to trouble.  Whatever
this abuse of narrowing is trying to achieve can be done in other less
damaging ways.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

>> To rephrase, don't widen in indent-line-function or
>> beginning-of-defun-function.
> This is an intolerable restriction.  The low level function mentioned
> above cannot, should not, must not know whether it's being called
> (indirectly) from indent-line-function or b-o-d-function.

It's a change.  So, to make it work right, some code will need to adapt.
As mentioned by Dmitry, this shouldn't introduce breakage in general:
the code will only need to adapt *if* it wants to work with
multi-major-mode solutions.  For the normal use case, you can still
widen/narrow to your heart's content and nobody will be the wiser.

There's nothing special here: any multi-major-mode solution will need to
define some convention/protocol that major need to follow in order to
work well.

Now if you worry that (after the code is adapted) the result is code
that is too complex/brittle, time will tell, but I get the impression
that it won't be too bad, likely no worse than any other solution.

The way your above situation would be handled is to say "the low level
function mentioned above assumes that the buffer has already been
properly widened".  This way, it doesn't need to know whether it's being
called (indirectly) from indent-line-function or b-o-d-function.

But some of its callers may need to be changed to do the widening, and
indeed, in order to know how far to widen, they'll need the help from
the multi-major-mode package.

So maybe we need something like:

    (defvar prog-widen-function #'widen)
    (defun prog-widen () (funcall prog-widen-function))

[ And maybe for the same reason, we need a function (rather than
  a variable) that returns the `first-column`, so we can get the info
  anywhere/anytime rather than only during indentation.  ]


        Stefan

RE: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Drew Adams]

> > > Narrowing belongs to users and major modes.
> > It can serve all.
> 
> Indeed it can, and it must.  A super-mode thus may not "reserve"
> narrowing for its own purposes to the exclusion of other uses.

+1

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

> Indeed it can, and it must.  A super-mode thus may not "reserve"
> narrowing for its own purposes to the exclusion of other uses.

99% of the multi-major-modes out there use narrowing for that purpose,
and in practice it doesn't restrict other uses of narrowing.

I've opposed the use of narrowing for that in the past, but the evidence
against that opposition is overwhelming.  Let's do that and move on.

It's not perfect, but we've already looked much too long for a less
imperfect solution.


        Stefan

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Alan Mackenzie]

Hello, Stefan.

On Fri, Dec 01, 2017 at 11:26:08 -0500, Stefan Monnier wrote:
> >> To rephrase, don't widen in indent-line-function or
> >> beginning-of-defun-function.
> > This is an intolerable restriction.  The low level function mentioned
> > above cannot, should not, must not know whether it's being called
> > (indirectly) from indent-line-function or b-o-d-function.

> It's a change.  So, to make it work right, some code will need to adapt.
> As mentioned by Dmitry, this shouldn't introduce breakage in general:

That has yet to be shown.

> the code will only need to adapt *if* it wants to work with
> multi-major-mode solutions.  For the normal use case, you can still
> widen/narrow to your heart's content and nobody will be the wiser.

If we're going to have a multi-major-mode solution (and I think we
should), let's at least get it right.  I.e., it should work with _any_
major mode without grotesque restrictions on that mode.

I would very much like to have AWK Mode working inside a
shell-script-mode buffer.  Why not?  The MMM solution which appears to
be being proposed won't achieve this.

I would very much like the option of handling C buffers as MMM buffers,
where macros would be handled by a (slightly) different mode from normal
code.  I'm not sure whether this is a good or practicable idea, but it
would be nice to have the option.

> There's nothing special here: any multi-major-mode solution will need to
> define some convention/protocol that major need to follow in order to
> work well.

No.  Our MMM solution should work without imposing restrictions on the
major modes.  All the awkwardness should be dealt with by the MMM.  This
is possible.

> Now if you worry that (after the code is adapted) the result is code
> that is too complex/brittle, time will tell, but I get the impression
> that it won't be too bad, likely no worse than any other solution.

As I keep saying, we should adapt a solution that doesn't place manacles
on hackers.

> The way your above situation would be handled is to say "the low level
> function mentioned above assumes that the buffer has already been
> properly widened".  This way, it doesn't need to know whether it's being
> called (indirectly) from indent-line-function or b-o-d-function.

Then it ceases to be a general low level function and becomes one that
can only be called in certain circumstances.  This effectively transmits
high level state to the low levels, and isn't good software engineering.

> But some of its callers may need to be changed to do the widening, and
> indeed, in order to know how far to widen, they'll need the help from
> the multi-major-mode package.

This is intolerable.  It will lead to twisted code.

Why should a major mode need to know "how far to widen"?  This is only
due to the proposal to misuse narrowing to control regions with
different syntax-tables, I think.  If these regions are controlled
without using narrowing, this and many other problems cease to exist.

> So maybe we need something like:

>     (defvar prog-widen-function #'widen)
>     (defun prog-widen () (funcall prog-widen-function))

No, we need

    (widen)

like we've had for many decades.

> [ And maybe for the same reason, we need a function (rather than
>   a variable) that returns the `first-column`, so we can get the info
>   anywhere/anytime rather than only during indentation.  ]


>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/1/17 4:26 PM, Stefan Monnier wrote:
> The way your above situation would be handled is to say "the low level
> function mentioned above assumes that the buffer has already been
> properly widened".  This way, it doesn't need to know whether it's being
> called (indirectly) from indent-line-function or b-o-d-function.

That one way, I'd call it "clean". But the change might take quite a bit 
of effort, depending on the size of the code base.

> But some of its callers may need to be changed to do the widening, and
> indeed, in order to know how far to widen, they'll need the help from
> the multi-major-mode package.

That would be the "dirty" way.

> So maybe we need something like:
> 
>      (defvar prog-widen-function #'widen)
>      (defun prog-widen () (funcall prog-widen-function))

I don't know if we need that in the common API, but individual packages 
can define such shims if they find that convenient.

E.g.

(defvar cc-mode-inside-indent-line nil)

(defun cc-mode-maybe-widen ()
   ;; You can also throw in an Emacs version check here,
   ;; for good measure.
   (unless cc-mode-inside-indent-line (widen)))

And then the "low-level primitives" can call cc-mode-maybe-widen.

> [ And maybe for the same reason, we need a function (rather than
>    a variable) that returns the `first-column`, so we can get the info
>    anywhere/anytime rather than only during indentation.  ]

Please elaborate.

How does using a function instead of a variable change the duration when 
prog-first-column is set?

And when else will we need this value?

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/1/17 3:49 PM, Alan Mackenzie wrote:

>> Please give an example.
> 
> A low level function which as an essential part of its functionality
> (for example, to make sure point-min isn't within a comment or string)
> widens.

See Stefan and I's responses. Including an example of a shim that 
wouldn't require you to change your code too much.

But first and foremost, CC Mode probably doesn't need to worry about 
complying with this guideline (yet). In my experience, it doesn't work 
well in multi-mode context for reasons other than it calling widen. So 
worrying about this change in protocol might be premature.

>> To rephrase, don't widen in indent-line-function or
>> beginning-of-defun-function.
> 
> This is an intolerable restriction.  The low level function mentioned
> above cannot, should not, must not know whether it's being called
> (indirectly) from indent-line-function or b-o-d-function.  It will have
> to widen in all cases or none.  Therefore there will be failures whilst
> being called either from one of the two noted functions or from
> elsewhere.

See that shim in the other email.

> Indeed it can, and it must.  A super-mode thus may not "reserve"
> narrowing for its own purposes to the exclusion of other uses.

Right.

> The multi-mode mechanism should be designed to be usable with any major
> mode.  There's nothing particularly hard about supporting CC Mode in a
> well designed multi-mode scheme.

It's harder than supporting Ruby, CSS and JS modes, that I can tell you 
for sure. For one thing, the extra caches CC Mode uses have given me 
multiple "argument out of bounds" errors when used with narrowing.

So these are my takeaways:

1) The "low-level primitives" in CC Mode do not widen consistently. Some 
of them miss that responsibility (or else I wouldn't get those kind of 
errors). That tells me that moving (widen) call to some higher level is 
generally a good idea, but you can disagree.

2) The other errors should be solvable, but they require a separate 
investigation, which nobody has found the time and energy to do so far, 
over many years. So maybe it's not so necessary.

3) The solution to the other problems is most likely orthogonal to the 
current discussion. Allowing multi-mode packages to use narrowing 
certainly shouldn't hurt.

> We need better tools.  I have already proposed and offered to implement
> such tools.

It was much more complex, nowhere near even a proof of concept, and 
doesn't seem to move anywhere, so far.

The change we're discussing is much smaller and pretty much implemented.

And again, there's no reason why they couldn't coexist.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

>> (defvar prog-widen-function #'widen)
>> (defun prog-widen () (funcall prog-widen-function))
> I don't know if we need that in the common API, but individual packages can
> define such shims if they find that convenient.

If we don't add such a thing in the common API, how could individual
packages know the boundary of the current chunk without adding
MMM-specific hacks?

> (defvar cc-mode-inside-indent-line nil)
> (defun cc-mode-maybe-widen ()
>   ;; You can also throw in an Emacs version check here,
>   ;; for good measure.
>   (unless cc-mode-inside-indent-line (widen)))
> And then the "low-level primitives" can call cc-mode-maybe-widen.

But just `widen` will do thew wrong thing (it'll widen too much) when
we're inside an MMM chunk.

>> [ And maybe for the same reason, we need a function (rather than
>> a variable) that returns the `first-column`, so we can get the info
>> anywhere/anytime rather than only during indentation.  ]
> Please elaborate.
> How does using a function instead of a variable change the duration when
> prog-first-column is set?

Having it as a function means you don't have to compute the value of
prog-first-column every time to "enter" a chunk, but instead we only
compute it if/when the submode decides it needs to know what is the
value of prog-first-column.

> And when else will we need this value?

Here are some examples I can think of:
- auto-fill-mode may like to compute the hypothetical indentation that
  would result from inserting a newline somewhere (and do that before
  touching the buffer).
- some package may like to highlight lines which aren't currently
  indented right (so, it won't call indent-according-to-mode, but
  it will need to compute the "desired" indentation).
I'm sure there can be other cases.


        Stefan

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/1/17 7:51 PM, Stefan Monnier wrote:
>>> (defvar prog-widen-function #'widen)
>>> (defun prog-widen () (funcall prog-widen-function))
>> I don't know if we need that in the common API, but individual packages can
>> define such shims if they find that convenient.
> 
> If we don't add such a thing in the common API, how could individual
> packages know the boundary of the current chunk without adding
> MMM-specific hacks?

The idea for multi-mode packages has always been to make do without 
making the major modes actually aware of hunk boundaries.

And the snippet of code won't make them aware either, since the chunk 
boundaries are conveyed via narrowing only in particular dynamic 
contexts, such as when indent-line-functions is called.

Adding other means too might make sense, but that's expanding the scope 
of the current proposal. I'd rather make it small and try to get it into 
Emacs 26.

>> (defvar cc-mode-inside-indent-line nil)
>> (defun cc-mode-maybe-widen ()
>>    ;; You can also throw in an Emacs version check here,
>>    ;; for good measure.
>>    (unless cc-mode-inside-indent-line (widen)))
>> And then the "low-level primitives" can call cc-mode-maybe-widen.
> 
> But just `widen` will do thew wrong thing (it'll widen too much) when
> we're inside an MMM chunk.

Yes, so CC Mode primitives all should call cc-mode-maybe-widen. Having 
them call prog-widen vs widen is very much the same difference.

> Having it as a function means you don't have to compute the value of
> prog-first-column every time to "enter" a chunk, but instead we only
> compute it if/when the submode decides it needs to know what is the
> value of prog-first-column.

You probably mean adding a variable prog-first-column-function, then?

Or will we expect multi-mode packages to simply advise the 
prog-first-column function? I'm fine with that.

>> And when else will we need this value?
> 
> Here are some examples I can think of:
> - auto-fill-mode may like to compute the hypothetical indentation that
>    would result from inserting a newline somewhere (and do that before
>    touching the buffer).

Won't it need to call indent-line-function to find out how much the next 
line should be indented?

> - some package may like to highlight lines which aren't currently
>    indented right (so, it won't call indent-according-to-mode, but
>    it will need to compute the "desired" indentation).

This example will *definitely* need to call indent-line-function.

And both of them should be solved by exchanging indent-line-function for 
(non-mutating) line-intentation-function. But that's a change for 
another day.

> I'm sure there can be other cases.

I'm sure there can be. A complete proposal to let the modes know chunk 
boundaries, etc, has yet to be finalized, however. And just having 
font-lock and indentation work reliably in multi-mode contexts will be a 
significant win.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Vitalie Spinu]

Please correct me if I am wrong but the current patch requires all modes to use
`prog-widen` and reserve `widen` to the interactive use, right? If so, this is
the opposite of the older idea of adding `hard-limits` for widening which only
multi-modes should use.

I have proposed a patch for `set-widen-limits` 1.5 years ago which still lives
in scratch/hard-narrow branch. It turned to be a pretty straightforward feat and
worked without requiring any changes of the excising elisp code.

The lengthy discussion started here:

  https://lists.gnu.org/archive/html/emacs-devel/2016-04/msg00629.html

Somewhere along the way we decided in favor of adding an extra `hard` argument
to `narrow` instead of having a new `set-widen-limits` function. That turned to
be impossible as it required backward-incompatible changes to the byte-code. I
run out of steam by then and the whole idea died :(

If adding a new function is back on the table then I think `set-widen-limits` is
a better approach than `prog-widen`. It doesn't impose new restrictions on the
existing code and it avoids introducing new types of narrowing or widening.

Unless something changed in the meanwhile, the changes to the C code to make
that work were minimal. If there is interest I can get it another shot.

  Vitalie

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/1/17 9:27 PM, Vitalie Spinu wrote:
> 
> Please correct me if I am wrong but the current patch requires all modes to use
> `prog-widen` and reserve `widen` to the interactive use, right? If so, this is
> the opposite of the older idea of adding `hard-limits` for widening which only
> multi-modes should use.

Quite the opposite: the patch under discussion, in scratch/widen-less, 
removes prog-widen altogether.

> I have proposed a patch for `set-widen-limits` 1.5 years ago which still lives
> in scratch/hard-narrow branch. It turned to be a pretty straightforward feat and
> worked without requiring any changes of the excising elisp code.
> 
> The lengthy discussion started here:
> 
>    https://lists.gnu.org/archive/html/emacs-devel/2016-04/msg00629.html
> 
> Somewhere along the way we decided in favor of adding an extra `hard` argument
> to `narrow` instead of having a new `set-widen-limits` function. That turned to
> be impossible as it required backward-incompatible changes to the byte-code. I
> run out of steam by then and the whole idea died :(

If that was the only difficulty, having set-widen-limits a separate 
function would have been a better choice back then (I wasn't aware, FWIW).

The new proposal is a bit simpler, OTOH.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Alan Mackenzie]

Hello, Dmitry.

On Fri, Dec 01, 2017 at 19:13:27 +0000, Dmitry Gutov wrote:
> On 12/1/17 3:49 PM, Alan Mackenzie wrote:

> >> Please give an example.

> > A low level function which as an essential part of its functionality
> > (for example, to make sure point-min isn't within a comment or string)
> > widens.

> See Stefan and I's responses. Including an example of a shim that 
> wouldn't require you to change your code too much.

A shim?  Yuck!  My idea is an MMM that doesn't require any such shims.
Major modes should not be "polluted" to adapt to one particular MMM.  It
solidifies that MMM, making it difficult to change to a better one.

> But first and foremost, CC Mode probably doesn't need to worry about 
> complying with this guideline (yet). In my experience, it doesn't work 
> well in multi-mode context for reasons other than it calling widen. So 
> worrying about this change in protocol might be premature.

You mean, it would be better to wait until this protocol is firmly
established before arguing against it?  I'm guessing that the main
reason CC Mode doesn't work well in MMM is the contention between CC
Mode's narrowing and MMM's narrowing.  (This is the example you asked
for earlier on.)

> >> To rephrase, don't widen in indent-line-function or
> >> beginning-of-defun-function.

> > This is an intolerable restriction.  The low level function mentioned
> > above cannot, should not, must not know whether it's being called
> > (indirectly) from indent-line-function or b-o-d-function.  It will have
> > to widen in all cases or none.  Therefore there will be failures whilst
> > being called either from one of the two noted functions or from
> > elsewhere.

> See that shim in the other email.

The shim doesn't help.  It's still horribly ugly.

> > Indeed it can, and it must.  A super-mode thus may not "reserve"
> > narrowing for its own purposes to the exclusion of other uses.

> Right.

Glad we agree upon this.  However that contradicts your proposal that
anything a beginning-of-defun-function calls can't use narrowing and
widening.

> > The multi-mode mechanism should be designed to be usable with any major
> > mode.  There's nothing particularly hard about supporting CC Mode in a
> > well designed multi-mode scheme.

> It's harder than supporting Ruby, CSS and JS modes, that I can tell you 
> for sure. For one thing, the extra caches CC Mode uses have given me 
> multiple "argument out of bounds" errors when used with narrowing.

There's your example for contention between major modes' narrowing and
MMM's narrowing.  If you didn't narrow, CC Mode would behave properly.

> So these are my takeaways:

> 1) The "low-level primitives" in CC Mode do not widen consistently.

They widen when they need to, which is not always.

> Some of them miss that responsibility (or else I wouldn't get those
> kind of errors).

It is the multiple-mode code that is responsible for not getting these
errors.  At least it should be.

> That tells me that moving (widen) call to some higher level is
> generally a good idea, but you can disagree.

> 2) The other errors should be solvable, but they require a separate 
> investigation, which nobody has found the time and energy to do so far, 
> over many years. So maybe it's not so necessary.

I would like AWK Mode to work inside shell-script-mode, since short AWK
scripts are so often embedded in scripts.

> 3) The solution to the other problems is most likely orthogonal to the 
> current discussion. Allowing multi-mode packages to use narrowing 
> certainly shouldn't hurt.

Of course not.  It's prohibiting major modes from using
narrowing/widening that is objectionable.

> > We need better tools.  I have already proposed and offered to implement
> > such tools.

> It was much more complex, nowhere near even a proof of concept, and 
> doesn't seem to move anywhere, so far.

Investing the time and energy on an implementation was too risky.  In
practice, Stefan has veto power over what goes into the syntax area, and
I've been at the sharp end of that veto once too often (the rejection of
my "comment-cache" code which solved the problem of open parens at
column zero in comments).  In the last exchange about the "syntactic
islands" concept, Stefan showed little enthusiasm for it: there was no
"OK, implement it and we'll see what we can do with it", no
encouragement whatsoever.  There was no discussion of major topics, such
as the possibilities it would open up, or any plans to use it.  Mainly,
the talk was about minor aspects of the proposal, at one point
degenerating into a squabble about the use of the word "island".

I'm still prepared to implement the "islands" proposal, which would
allow a better MMM mode than one based on narrowing, but ONLY if I get
some assurance from John, Eli, or even Stefan himself beforehand that
the implementation is, in principle, going to be incorporated into
Emacs.  That the only thing which would stop that would be solid
technical reasons.

> The change we're discussing is much smaller and pretty much implemented.

But it has negative implications.  It will only work with major modes
specially adapted for it, and it imposes severe restrictions on those
major modes.  I think it needs those modes to be "polluted" with MMM
code.

> And again, there's no reason why they couldn't coexist.

The "pollution" mentioned above would make it more difficult.

If the new code you're proposing is incorporated into Emacs, it will
acquire some momentum, making it difficult to introduce a better
solution.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Alan Mackenzie]

Hello, Stefan.

On Fri, Dec 01, 2017 at 13:03:05 -0500, Stefan Monnier wrote:
> > Indeed it can, and it must.  A super-mode thus may not "reserve"
> > narrowing for its own purposes to the exclusion of other uses.

> 99% of the multi-major-modes out there use narrowing for that purpose,
> and in practice it doesn't restrict other uses of narrowing.

If that were the case, there would be no restrictions on a major mode
widening and narrowing.  From what I understand about the new proposal,
your last paragraph is false.

By the way, what about that other 1% of multiple-major-modes?  How do
they work?

> I've opposed the use of narrowing for that in the past, but the evidence
> against that opposition is overwhelming.  Let's do that and move on.

No.  The evidence is that these modes can only work with specially
adapted major modes, and possibly that MMM code pollutes those major
modes.

> It's not perfect, but we've already looked much too long for a less
> imperfect solution.

I don't think that's the case.  My "syntactic islands" proposal was
never given proper consideration.  That was in last February.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/1/17 10:35 PM, Alan Mackenzie wrote:

> A shim?  Yuck!  My idea is an MMM that doesn't require any such shims.M

Ideas have the tendency to be more ideal than their eventual 
implementations.

> Major modes should not be "polluted" to adapt to one particular MMM.  It
> solidifies that MMM, making it difficult to change to a better one.

It's not "one particular MMM". All existing MMM packages use narrowing.

And it's not just to adapt. Calling `widen' in fewer places just makes 
sense from engineering point of view.

> You mean, it would be better to wait until this protocol is firmly
> established before arguing against it? 

The proposal isn't going to hurt you in any significant way. That's what 
I mean.

> The shim doesn't help.  It's still horribly ugly.

Calling widen from dozens of functions in a package is ugly. A shim just 
helps you avoid rearchitecting it.

>>> Indeed it can, and it must.  A super-mode thus may not "reserve"
>>> narrowing for its own purposes to the exclusion of other uses.
> 
>> Right.
> 
> Glad we agree upon this.  However that contradicts your proposal that
> anything a beginning-of-defun-function calls can't use narrowing and
> widening.

Nope.

> There's your example for contention between major modes' narrowing and
> MMM's narrowing. 

Not sure what you mean by that. It's already apparent that some major 
modes will have to change to adapt to this proposal. Isn't it?

> If you didn't narrow, CC Mode would behave properly.

Without narrowing it gives us wrong syntax highlighting and indentation.

>> So these are my takeaways:
> 
>> 1) The "low-level primitives" in CC Mode do not widen consistently.
> 
> They widen when they need to, which is not always.

I don't think I'd get this error if they did.

But hmm, I think this error was in fontification code, actually. 
mmm-mode, of course, narrows to each individual hunk before using the 
corresponding mode's font-lock-keywords.

>> 2) The other errors should be solvable, but they require a separate
>> investigation, which nobody has found the time and energy to do so far,
>> over many years. So maybe it's not so necessary.
> 
> I would like AWK Mode to work inside shell-script-mode, since short AWK
> scripts are so often embedded in scripts.

When we're talking about short snippets, sometimes it's easier to use 
e.g. js-mode instead. That's what I've been advising for C code, too.

But if complex syntax in AWK happens often, and CC Mode has good support 
for it, maybe supporting it is a worthy endeavor.

>> 3) The solution to the other problems is most likely orthogonal to the
>> current discussion. Allowing multi-mode packages to use narrowing
>> certainly shouldn't hurt.
> 
> Of course not.  It's prohibiting major modes from using
> narrowing/widening that is objectionable.

Sometimes code has to follow certain protocols. That's nothing new.

> Investing the time and energy on an implementation was too risky.  In
> practice, Stefan has veto power over what goes into the syntax area, and
> I've been at the sharp end of that veto once too often (the rejection of
> my "comment-cache" code which solved the problem of open parens at
> column zero in comments).  In the last exchange about the "syntactic
> islands" concept, Stefan showed little enthusiasm for it: there was no
> "OK, implement it and we'll see what we can do with it", no
> encouragement whatsoever.  There was no discussion of major topics, such
> as the possibilities it would open up, or any plans to use it.  Mainly,
> the talk was about minor aspects of the proposal, at one point
> degenerating into a squabble about the use of the word "island".

No discussion, really? I've pointed out the major difficulties I could 
think of, which is normally the area that should get the most attention 
during the discussion of a technical proposal.

The fact that few people seemed interested... well, that happens. 
There's no use complaining that a proposal is unpopular. I've had a few 
over the years myself. Please feel free to push for it, of course.

> But it has negative implications.  It will only work with major modes
> specially adapted for it, and it imposes severe restrictions on those
> major modes.  I think it needs those modes to be "polluted" with MMM
> code.

We already have several major modes adhering to it, and for them the 
change was trivial. As far as major modes are concerned, web programming 
is the most important area, so CC Mode support is less urgent anyway.

> If the new code you're proposing is incorporated into Emacs, it will
> acquire some momentum, making it difficult to introduce a better
> solution.

Like mentioned, for most major mode the changes are trivial, if at all 
needed.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

>>>> (defvar prog-widen-function #'widen)
>>>> (defun prog-widen () (funcall prog-widen-function))
> The idea for multi-mode packages has always been to make do without making
> the major modes actually aware of hunk boundaries.
[...]
> And the snippet of code won't make them aware either, since the chunk
> boundaries are conveyed via narrowing only in particular dynamic contexts,
> such as when indent-line-functions is called.

But widening/narrowing to the current chunk might be needed for commands
specific to the major-mode, hence commands which the MMM framework
doesn't know about (and hence doesn't wrap like it does for
indent-line-function).

> You probably mean adding a variable prog-first-column-function, then?

Right.

> Or will we expect multi-mode packages to simply advise the prog-first-column
> function? I'm fine with that.

Give that it's introduced specifically so it can be changed, relying
advice would be a bad idea (especially since the advice would generally
want to be buffer-local, which is easier to get with a variable).

>> Here are some examples I can think of:
>> - auto-fill-mode may like to compute the hypothetical indentation that
>> would result from inserting a newline somewhere (and do that before
>> touching the buffer).
> Won't it need to call indent-line-function to find out how much the next
> line should be indented?

No because indent-line-function would actually perform the
re-indentation, whereas we only want to calculate the hypothetical new
indentation column.  See `smie-auto-fill`.

>> - some package may like to highlight lines which aren't currently
>> indented right (so, it won't call indent-according-to-mode, but
>> it will need to compute the "desired" indentation).
> This example will *definitely* need to call indent-line-function.

Again no, because it doesn't want to modify the buffer.

> And both of them should be solved by exchanging indent-line-function for
> (non-mutating) line-intentation-function. But that's a change for
> another day.

Ah, yes, that would be a good change.
Arguably `smie-indent-functions` already provides that functionality.

> I'm sure there can be.  A complete proposal to let the modes know chunk
> boundaries, etc, has yet to be finalized, however.  And just having font-lock
> and indentation work reliably in multi-mode contexts will
> be a significant win.

Agreed.  As far as I'm concerned your proposal is good to go, which is
why I was talking about subsequent changes.

BTW, could we get some kind of multi-mode package in elpa.git or
emacs.git to go along with that (it doesn't have to be fancy, but it's
important that it doesn't have any submode-specific hacks).
Maybe a generalization of mhtml-mode, or something separate but whose
design is such that it can aim to (slowly) merge with mhtml-mode?


        Stefan

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

>> "OK, implement it and we'll see what we can do with it", no
>> encouragement whatsoever.  There was no discussion of major topics, such
>> as the possibilities it would open up, or any plans to use it.  Mainly,
>> the talk was about minor aspects of the proposal, at one point
>> degenerating into a squabble about the use of the word "island".

Here's my opinion: experience shows that implementing MMM support is messy
for all kinds of reasons, and that the problems that show up in practice
are very varied and many of them are completely unexpected.

So any paper design such as your "islands" to me sounds like a plain and
simple pipe-dream.  Can it work in theory?  Sure, why not.  Can it work
in practice?  Probably.  Will it work without having to "pollute major
modes to adapt to that particular MMM"?  Not a chance in hell, IMO.

So overall, will it be better than the solutions we're already
familiar with?  maybe, but my money is on "probably not".

Also, I don't think major modes need more freedom in how to implement
things.  Instead, they need more help so they can just use existing
solutions rather than reinvent the wheel in their very own way.
Conventions/guidelines actually makes the life of major mode
implementers easier, rather than harder.  E.g. they can just use
`syntax-propertize` and move on.  If it's not efficient enough or if it
doesn't interact correctly with an MMM framework, it's not their problem
it's syntax-propertize's. 


        Stefan


PS: On a very related topic, I'd recommend anyone interested in such
ideas to look at the way other editors handle font-lock-style
highlighting and based on that develop a replacement for font-lock.
AFAICT, font-lock is much less powerful/flexible than some of our
competitor's frameworks, especially in terms of handling differently
different parts of the buffer.  So a "font-lock-NG" might be a good
starting point to add better support for some MMM-style features.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

>> 99% of the multi-major-modes out there use narrowing for that purpose,
>> and in practice it doesn't restrict other uses of narrowing.
> If that were the case, there would be no restrictions on a major mode
> widening and narrowing.  From what I understand about the new proposal,
> your last paragraph is false.

Depends what you expect "use" to mean.  For me "call widen"
doesn't qualify as "uses narrowing".  So I stand by what I said.

> By the way, what about that other 1% of multiple-major-modes?  How do
> they work?

Multiple buffers (either by copying text to/from side-buffers, or by
use of indirect-buffers).

>> I've opposed the use of narrowing for that in the past, but the evidence
>> against that opposition is overwhelming.  Let's do that and move on.
> No.  The evidence is that these modes can only work with specially
> adapted major modes,

Right.  And use of narrowing is the way that makes this adaption
simplest (in many cases it becomes trivial.  See for example how easy
it's been to implement mhtml-mode).

I don't know of any MMM solution which provides a satisfactory user
experience and at the same time works for all major modes,
without any need to tweak them at all, no matter how twisted is the
major mode's code.


        Stefan

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 1 Dec 2017 23:24:28 +0000
> Cc: Tom Tromey <tom@tromey.com>, Stefan Monnier <monnier@iro.umontreal.ca>,
> 	Vitalie Spinu <spinuvit@gmail.com>, emacs-devel@gnu.org
> 
> Like mentioned, for most major mode the changes are trivial, if at all 
> needed.

IMO, C modes and Lisp modes must be supported well, before this lands
on master.

Thanks.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/2/17 8:27 AM, Eli Zaretskii wrote:
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Date: Fri, 1 Dec 2017 23:24:28 +0000
>> Cc: Tom Tromey <tom@tromey.com>, Stefan Monnier <monnier@iro.umontreal.ca>,
>> 	Vitalie Spinu <spinuvit@gmail.com>, emacs-devel@gnu.org
>>
>> Like mentioned, for most major mode the changes are trivial, if at all
>> needed.
> 
> IMO, C modes and Lisp modes must be supported well, before this lands
> on master.

Why? Web languages are a lot more common in mixed mode buffers (think 
HTML files and templates), so it's a lot more important to have that 
support in them (and we already do).

Lisp modes should work already too, because lisp-indent-line doesn't 
call widen.

And about C, first, it's much harder for different reasons, two, we'd 
have to spend more time convincing its author. And then we'd have to fix 
other MMM-related problems in it that the other mentioned modes don't have.

And, I'm sorry about this, but I want to land these changes in emacs-26, 
for one main reason: it simplifies, reworks and replaces the 
prog-indentation-context feature, which is not in Emacs 25.

At one point we/I removed it from Emacs 25 for reasons of being 
undercooked, but didn't merge that change to then-master. And then kind 
of forgot. https://debbugs.gnu.org/cgi/bugreport.cgi?bug=28475#11 was a 
later reminder.

This way, the "widen less" proposal could be considered a "bug fix", if 
we stretch the definition far enough. And also, the new approach is 
pretty small, it removes more code than adds.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> Cc: acm@muc.de, tom@tromey.com, monnier@iro.umontreal.ca, spinuvit@gmail.com,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 2 Dec 2017 10:50:24 +0000
> 
> > IMO, C modes and Lisp modes must be supported well, before this lands
> > on master.
> 
> Why?

Because Emacs is written in these 2 languages.  Sorry, I thought that
was clear.

> Web languages are a lot more common in mixed mode buffers (think 
> HTML files and templates), so it's a lot more important to have that 
> support in them (and we already do).

I expected those languages to be supported already, which is why I
didn't mention them.

> And, I'm sorry about this, but I want to land these changes in emacs-26, 
> for one main reason: it simplifies, reworks and replaces the 
> prog-indentation-context feature, which is not in Emacs 25.

It's too late for landing such features on emacs-26.  Sorry, it will
have to wait until Emacs 27.

Emacs 26 is nearing its release; the stream of serious bugs (or any
bugs, actually) seems to have died out.  So hopefully one more
pretest, and we will be ready to go.  Adding a significant new
feature, which will no doubt need changes in other places, is
incompatible with that.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Tom Tromey]

>>>>> "Stefan" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

Stefan> Right.  And use of narrowing is the way that makes this adaption
Stefan> simplest (in many cases it becomes trivial.  See for example how easy
Stefan> it's been to implement mhtml-mode).

It doesn't much matter to me whether narrowing or some other approach is
used.  Narrowing was simpler at the time -- and, I tend to think, will
always be simpler, because the various buffer primitives already respect
it, so you don't need to pass extra information around.  It has the same
sort of utility as dynamic binding in this way, just like RMS pointed
out in that Emacs paper back in the beginning.

However - if something else is more desirable somehow, that's also fine
with me.

Stefan> I don't know of any MMM solution which provides a satisfactory user
Stefan> experience and at the same time works for all major modes,
Stefan> without any need to tweak them at all, no matter how twisted is the
Stefan> major mode's code.

Yes, I completely agree here.  Some changes will be needed.  If you look
through the mhtml development process, there were a few tweaks needed
here and there, at least one of which (a syntax change in sgml mode
IIRC) was quite unexpected.

I haven't really appreciated until recently that maybe some people were
looking for a solution that required no changes.  Better, I think, to
accept that some changes are needed; maybe work to keep the changes
reasonably small and specified; but then just update the interesting
modes (maybe all modes) and move on.

Also, there's still some unanswered questions in mhtml mode.  For
example, the way it captures sub-mode buffer-locals could really be
improved.  Maybe by applying .dir-locals from all the modes; maybe by
having something like the mode hook but for sub-modes -- I'm not sure.

Tom

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

>> > IMO, C modes and Lisp modes must be supported well, before this lands
>> > on master.
>> Why?
> Because Emacs is written in these 2 languages.  Sorry, I thought that
> was clear.

So you'd rather see "no multi-major-mode support at all" than
"multi-major-mode support for some major modes"?

> It's too late for landing such features on emacs-26.  Sorry, it will
> have to wait until Emacs 27.

Then we should urgently remove the prog-indentation-context business
(i.e. revert to the Emacs-25 state of those features).


        Stefan

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Dmitry Gutov <dgutov@yandex.ru>,  acm@muc.de,  tom@tromey.com,  spinuvit@gmail.com,  emacs-devel@gnu.org
> Date: Sat, 02 Dec 2017 11:41:49 -0500
> 
> >> > IMO, C modes and Lisp modes must be supported well, before this lands
> >> > on master.
> >> Why?
> > Because Emacs is written in these 2 languages.  Sorry, I thought that
> > was clear.
> 
> So you'd rather see "no multi-major-mode support at all" than
> "multi-major-mode support for some major modes"?

I think the feature should be fairly complete before it lands, yes.

> > It's too late for landing such features on emacs-26.  Sorry, it will
> > have to wait until Emacs 27.
> 
> Then we should urgently remove the prog-indentation-context business
> (i.e. revert to the Emacs-25 state of those features).

I don't really understand what this means in practice, sorry.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

> I think the feature should be fairly complete before it lands, yes.

Would "make it work with Lisp and C" be satisfied if the "and C" part
uses sm-c-mode?  If not, then I'm afraid this makes it nigh-on impossible.


        Stefan

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: dgutov@yandex.ru,  acm@muc.de,  tom@tromey.com,  spinuvit@gmail.com,  emacs-devel@gnu.org
> Date: Sat, 02 Dec 2017 12:53:58 -0500
> 
> > I think the feature should be fairly complete before it lands, yes.
> 
> Would "make it work with Lisp and C" be satisfied if the "and C" part
> uses sm-c-mode?

I don't see how it could be, not IMO.  I think we need it working with
the C modes the majority of Emacs users are using, the ones that are
part of Emacs.

> If not, then I'm afraid this makes it nigh-on impossible.

Can you explain why is this so hard that you are willing to give up?

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/2/17 2:24 AM, Stefan Monnier wrote:

> But widening/narrowing to the current chunk might be needed for commands
> specific to the major-mode, hence commands which the MMM framework
> doesn't know about (and hence doesn't wrap like it does for
> indent-line-function).

Depending on whether the major mode is the "primary" mode or a 
"submode", some commands might prefer no resrictions, and some the 
current chunk. font-lock, on the other hand, will (almost?) always want 
to use the current chunk.

And only the MMM framework knows whether the mode is primary or not. So 
the issue is not that obvious.

>> You probably mean adding a variable prog-first-column-function, then?
> 
> Right.

Are you fine with only having prog-first-column variable in Emacs 26?

Adding prog-first-column-function without having something similar for 
chunks would seem inconsistent.

> Give that it's introduced specifically so it can be changed, relying
> advice would be a bad idea (especially since the advice would generally
> want to be buffer-local, which is easier to get with a variable).

Makes sense.

> No because indent-line-function would actually perform the
> re-indentation, whereas we only want to calculate the hypothetical new
> indentation column.  See `smie-auto-fill`.

I see.

>>> - some package may like to highlight lines which aren't currently
>>> indented right (so, it won't call indent-according-to-mode, but
>>> it will need to compute the "desired" indentation).
>> This example will *definitely* need to call indent-line-function.
> 
> Again no, because it doesn't want to modify the buffer.

This one I don't understand: how would you know the indentation is 
correct without trying to reindent?

>> And both of them should be solved by exchanging indent-line-function for
>> (non-mutating) line-intentation-function. But that's a change for
>> another day.
> 
> Ah, yes, that would be a good change.
> Arguably `smie-indent-functions` already provides that functionality.

As an example, yes, but it's not like we can move all modes to SMIE.

>> I'm sure there can be.  A complete proposal to let the modes know chunk
>> boundaries, etc, has yet to be finalized, however.  And just having font-lock
>> and indentation work reliably in multi-mode contexts will
>> be a significant win.
> 
> Agreed.  As far as I'm concerned your proposal is good to go, which is
> why I was talking about subsequent changes.

The current state of the branch, then?

> BTW, could we get some kind of multi-mode package in elpa.git or
> emacs.git to go along with that (it doesn't have to be fancy, but it's
> important that it doesn't have any submode-specific hacks).

Do you have any particular hacks in mind, in e.g. mmm-mode?

> Maybe a generalization of mhtml-mode,
That might actually be possible, a kind of mixed-mode builder.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/2/17 11:10 AM, Eli Zaretskii wrote:

>>> IMO, C modes and Lisp modes must be supported well, before this lands
>>> on master.
>>
>> Why?
> 
> Because Emacs is written in these 2 languages.  Sorry, I thought that
> was clear.

Still. Why would you need it? What benefits haven't you been able to 
achieve without that support, while working on Emacs?

>> Web languages are a lot more common in mixed mode buffers (think
>> HTML files and templates), so it's a lot more important to have that
>> support in them (and we already do).
> 
> I expected those languages to be supported already, which is why I
> didn't mention them.

But they are the most important ones/ones most desired by the users.

> It's too late for landing such features on emacs-26.  Sorry, it will
> have to wait until Emacs 27.

I understand it's a problem, and I'm sorry for bringing it up this late.

> Emacs 26 is nearing its release; the stream of serious bugs (or any
> bugs, actually) seems to have died out.

Allow me to put it like this: AFAIU, prog-indenation-context, which is 
in emacs-26, is *not* properly supported in most of the modes where we'd 
expect it to be. E.g. css-mode, js-mode and all modes using SMIE for 
indentation (such as ruby-mode).

The original author only contributed support for python-mode, and it 
stayed this way since. Now, adding that support wouldn't be complex, but 
my proposed change isn't complex either. And we have more of a consensus 
on it.

> Adding a significant new
> feature, which will no doubt need changes in other places, is
> incompatible with that.

Have you taken a look at the branch? It's your call, but based on the 
number of changes, I wouldn't call it "significant". Important, yes, but 
it's fairly small.

The other option is simply removing prog-indentation-context before the 
release.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/2/17 5:13 PM, Eli Zaretskii wrote:

>> So you'd rather see "no multi-major-mode support at all" than
>> "multi-major-mode support for some major modes"?
> 
> I think the feature should be fairly complete before it lands, yes.

It's not much of a feature, it's more of a codified protocol. The change 
in the manual is the actually important part.

And it's expected that some modes can choose not to follow it. Then 
they'll be no worse off than before its introduction. Multi-mode 
packages just won't be able to support them.

>> Then we should urgently remove the prog-indentation-context business
>> (i.e. revert to the Emacs-25 state of those features).
> 
> I don't really understand what this means in practice, sorry.

Removing all references to prog-indentation-context, prog-first-column 
and prog-widen.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Alan Mackenzie]

Hello, Stefan.

On Fri, Dec 01, 2017 at 21:47:26 -0500, Stefan Monnier wrote:
> >> "OK, implement it and we'll see what we can do with it", no
> >> encouragement whatsoever.  There was no discussion of major topics, such
> >> as the possibilities it would open up, or any plans to use it.  Mainly,
> >> the talk was about minor aspects of the proposal, at one point
> >> degenerating into a squabble about the use of the word "island".

> Here's my opinion: experience shows that implementing MMM support is messy
> for all kinds of reasons, and that the problems that show up in practice
> are very varied and many of them are completely unexpected.

> So any paper design such as your "islands" to me sounds like a plain and
> simple pipe-dream.  Can it work in theory?  Sure, why not.  Can it work
> in practice?  Probably.  Will it work without having to "pollute major
> modes to adapt to that particular MMM"?  Not a chance in hell, IMO.

It would work, and work well, for the simple reason it is a coherent
proposal.

Emacs was designed with the assumption that each buffer has exactly one
major mode.  It turns out this assumption hasn't held all that well.
When that assumption does hold, the following returns a well defined and
meaningful state:

    (parse-partial-sexp 1 n)

.  I am proposing extending this property for buffers with several
modes and several syntax tables.  Nothing more, nothing less.

You're saying that providing support for multiple major modes in the
Emacs core is a pipe dream.  I put it to you you're mistaken.  ONLY by
supporting this in the Emacs core can we arrive at a coherent efficient
design.  The fact we are having this conversation is a strong indication
that the current designs for MMM are not coherent enough.

[ .... ]

> Also, I don't think major modes need more freedom in how to implement
> things.  Instead, they need more help so they can just use existing
> solutions rather than reinvent the wheel in their very own way.
> Conventions/guidelines actually makes the life of major mode
> implementers easier, rather than harder.

That is immensely patronising.

> E.g. they can just use `syntax-propertize` and move on.  If it's not
> efficient enough or if it doesn't interact correctly with an MMM
> framework, it's not their problem it's syntax-propertize's. 

syntax-propertize is your way of doing things, so naturally you want it
to become a "standard".  However it is not the only way of doing things,
and is suboptimal in several respects.

When designing syntax-propertize, with whom did you discuss the
technical aspects?  Certainly not with me, but this may have been before
my time in Emacs.  With Martin Stjernholm or Barry Warsaw, my
predecessors as CC Mode maintainers?  With RMS?  With anybody at all?

>         Stefan

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> Cc: acm@muc.de, tom@tromey.com, monnier@iro.umontreal.ca, spinuvit@gmail.com,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 2 Dec 2017 20:14:54 +0000
> 
> > Because Emacs is written in these 2 languages.  Sorry, I thought that
> > was clear.
> 
> Still. Why would you need it? What benefits haven't you been able to 
> achieve without that support, while working on Emacs?

I don't understand the question.  Surely, a feature that aims to
support multiple major modes should strive to support CC Mode?

> >> Web languages are a lot more common in mixed mode buffers (think
> >> HTML files and templates), so it's a lot more important to have that
> >> support in them (and we already do).
> > 
> > I expected those languages to be supported already, which is why I
> > didn't mention them.
> 
> But they are the most important ones/ones most desired by the users.

They are important, but so are C-based languages: C++, Java, Awk, etc.

> > It's too late for landing such features on emacs-26.  Sorry, it will
> > have to wait until Emacs 27.
> 
> I understand it's a problem, and I'm sorry for bringing it up this late.

It's not a catastrophe.  Assuming Emacs 26.1 is not a complete
disaster, we could decide releasing Emacs 27.1 as the next version.

> > Adding a significant new
> > feature, which will no doubt need changes in other places, is
> > incompatible with that.
> 
> Have you taken a look at the branch? It's your call, but based on the 
> number of changes, I wouldn't call it "significant". Important, yes, but 
> it's fairly small.

It might be "small", but it is by no means "insignificant".  Changing
buffer restrictions at strategic places is not something that should
be taken lightly.  The current pretest has been out there for the last
2 months; introducing this feature now (and actually in at least a few
days, as it isn't ready yet) would set us back that many days, if not
more.

> The other option is simply removing prog-indentation-context before the 
> release.

Why would we need to remove that for Emacs 26.1?

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/2/17 8:58 PM, Eli Zaretskii wrote:

>> Still. Why would you need it? What benefits haven't you been able to
>> achieve without that support, while working on Emacs?
> 
> I don't understand the question.

I'm asking why "Emacs is written in these 2 languages" is important.

> Surely, a feature that aims to
> support multiple major modes should strive to support CC Mode?

Not necessarily. It should first and foremost support the languages that 
are most important for our users.

> They are important, but so are C-based languages: C++, Java, Awk, etc.

I don't know CC Mode, and cannot provide patches for us. I tried 
debugging related problems a few times, and those turned out to be 
fairly long spelunking expeditions.

However long it will take just to convince Alan to agree, is anybody's 
guess.

> It's not a catastrophe.  Assuming Emacs 26.1 is not a complete
> disaster, we could decide releasing Emacs 27.1 as the next version.

Not a catastrophe indeed. But no step forward for mixed-modes support in 
Emacs 26 would be unfortunate.

> Why would we need to remove that for Emacs 26.1?

It's not well supported, and it's incompatible with the code we are 
discussing. There's no point in having it present in Emacs 26, and then 
removing it in Emacs 27.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

> Are you fine with only having prog-first-column variable in Emacs 26?

Since Eli only wants MMM-support if it work 100% everywhere, I don't
think it matters what gets into Emacs itself in this respect, because
the real stuff will have to live elsewhere.

The thing we should try and fight for of course are those such as making
indent-according-to-mode widen before calling indent-line-function.

>> Give that it's introduced specifically so it can be changed, relying
       ^                                                               ^^
       n                                                               on
>> advice would be a bad idea (especially since the advice would generally
>> want to be buffer-local, which is easier to get with a variable).
> Makes sense.

I'm glad English includes enough redundancy that you could make out what
I meant.

>>>> - some package may like to highlight lines which aren't currently
>>>> indented right (so, it won't call indent-according-to-mode, but
>>>> it will need to compute the "desired" indentation).
>>> This example will *definitely* need to call indent-line-function.
>> Again no, because it doesn't want to modify the buffer.
> This one I don't understand: how would you know the indentation is correct
> without trying to reindent?

The idea is just to highlight those lines whose indentation would be
modified by indent-line-function, so you only want to compute the
target column and compare it with the current column.

>> Arguably `smie-indent-functions` already provides that functionality.
> As an example, yes, but it's not like we can move all modes to SMIE.

Hence the "arguably": it can be used without using SMIE, but it's not as
convenient as you might like.  E.g. you can do:

    (require 'smie)

    (define-derived-mode ...
      ...
      (setq-local smie-indent-functions '(my-compute-indent))
      ...)

You can even reuse some parts of smie-indent-functions which don't depend
on the smie-grammar/rules business, e.g.:

      (setq-local smie-indent-functions
                  '(smie-indent-fixindent smie-indent-bob
                    smie-indent-comment smie-indent-comment-continue
                    smie-indent-comment-close smie-indent-comment-inside
                    smie-indent-inside-string
                    my-compute-indent))

but the more serious problem is that other packages (such as auto-fill)
can't rely on `smie-indent-functions' because it's not used by all major modes.

>> Agreed.  As far as I'm concerned your proposal is good to go, which is
>> why I was talking about subsequent changes.
> The current state of the branch, then?

Sure.

>> BTW, could we get some kind of multi-mode package in elpa.git or
>> emacs.git to go along with that (it doesn't have to be fancy, but it's
>> important that it doesn't have any submode-specific hacks).
> Do you have any particular hacks in mind, in e.g. mmm-mode?

No.  Can we put mmm-mode into elpa.git?


        Stefan

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

We could add another C-level feature similar to narrowing,
specifically for the sake of multiple major modes, if that helps
implement them cleanly.

Multiple major modes is important enough to justify such a thing.

The reason I see that an additional C-level feature MIGHT help is that
we could arrange at C level for it to interact in just the right way
with ALL uses of the existing narrowing feature.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

>     (parse-partial-sexp 1 n)
>
> .  I am proposing extending this property for buffers with several
> modes and several syntax tables.  Nothing more, nothing less.

That would potentially solve some percentage of the problems with
MMM support.  But what about the rest?

> syntax-propertize is your way of doing things, so naturally you want it
> to become a "standard".

I don't actually care whether the standard is my standard or not.
What I wanted was *a* standard.

> However it is not the only way of doing things,
> and is suboptimal in several respects.

All standards are suboptimal.  The reason people use them is because the
existence of a standard makes other things possible.

> When designing syntax-propertize, with whom did you discuss the
> technical aspects?

My cat.  Then I (cause at that point my cat couldn't be bothered to
follow me to the desk) tried what we both thought might work, then
I applied it to more modes, and a few years later it ends up working in
*all* major modes I know (including CC-mode, tho I haven't bothered to
make it robust enough to have a patch, knowing that you're never going
to accept such a patch in CC-mode, even though it's mostly just throwing
away code).


        Stefan

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sat, 02 Dec 2017 18:47:57 -0500
> Cc: Alan Mackenzie <acm@muc.de>, Tom Tromey <tom@tromey.com>,
> 	Vitalie Spinu <spinuvit@gmail.com>, emacs-devel@gnu.org
> 
> > Are you fine with only having prog-first-column variable in Emacs 26?
> 
> Since Eli only wants MMM-support if it work 100% everywhere

Please don't misrepresent so completely what I wrote.

> I don't think it matters what gets into Emacs itself in this
> respect, because the real stuff will have to live elsewhere.
> [...]
> Can we put mmm-mode into elpa.git?

Even if it's possible, it makes absolutely no sense to me to have such
a central feature on ELPA and not in the core.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/2/17 8:28 PM, Alan Mackenzie wrote:

> It would work, and work well, for the simple reason it is a coherent
> proposal.

I've never seen a full proposal. Details matter. And it's going to have 
to include a lot more of them.

> Emacs was designed with the assumption that each buffer has exactly one
> major mode.  It turns out this assumption hasn't held all that well.
> When that assumption does hold, the following returns a well defined and
> meaningful state:
> 
>      (parse-partial-sexp 1 n)
> 
> .  I am proposing extending this property for buffers with several
> modes and several syntax tables.  Nothing more, nothing less.

Extending parse-partial-sexp to nested syntax tables can be worthwhile, 
but that can't be it. And while we're on this subject, there are some 
example of Ruby syntax that we'd like to see supported, where the 
current syntax tables (and even syntax-propertize-function) don't cut it.

I'm not convinced "several syntax tables" would be enough, though.

> You're saying that providing support for multiple major modes in the
> Emacs core is a pipe dream.  I put it to you you're mistaken.  ONLY by
> supporting this in the Emacs core can we arrive at a coherent efficient
> design.

The major modes will have to change to allow for this support, that's 
for sure. You can't expect no code changes.

> The fact we are having this conversation is a strong indication
> that the current designs for MMM are not coherent enough.

The current idea is "let's do the minimum coherent step forward, so that 
somebody can continue improving things later". A step that's easy to 
use, and also easy to undo later, if we find something else that's much 
better.

Coming up with it was not easy at all, BTW.

>> Also, I don't think major modes need more freedom in how to implement
>> things.  Instead, they need more help so they can just use existing
>> solutions rather than reinvent the wheel in their very own way.
>> Conventions/guidelines actually makes the life of major mode
>> implementers easier, rather than harder.
> 
> That is immensely patronising.

It's simply description of good engineering. You can try to criticize 
the details of syntax-propertize, but the idea of reusing common 
solutions is more than sound.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Alan Mackenzie]

Hello, Stefan.

On Sat, Dec 02, 2017 at 19:03:44 -0500, Stefan Monnier wrote:
> >     (parse-partial-sexp 1 n)

> > .  I am proposing extending this property for buffers with several
> > modes and several syntax tables.  Nothing more, nothing less.

> That would potentially solve some percentage of the problems with
> MMM support.  But what about the rest?

They would clearly need other work doing on them.  Richard suggested
building something similar to narrowing for the excusive use of MMM.

> > syntax-propertize is your way of doing things, so naturally you want it
> > to become a "standard".

> I don't actually care whether the standard is my standard or not.
> What I wanted was *a* standard.

Standards are created by like-minded experts coming together, thrashing
things out, and finally agreeing on some reasonable set of compromises;
not by a lone hacker, no matter how expert, throwing something together
in his bedroom.

> > However it is not the only way of doing things,
> > and is suboptimal in several respects.

> All standards are suboptimal.

You seem to be implying that all standards are suboptimal, therefore any
standard is as good as any other.  That is palpably false.  

> The reason people use them is because the existence of a standard
> makes other things possible.

Not really.  At least, this particular standard (if such it be),
syntax-propertize, shuts off useful, even essential, possibilities.  It
is also based on syntax-ppss, which we recently agreed needed replacing
with something which does the right job.

The way you want to restrict the use of the syntax-table text property
is like having a screw driver, and only being permitted to turn it
clockwise.  That text property is a very useful general purpose tool,
something you seem to want to do away with.

> > When designing syntax-propertize, with whom did you discuss the
> > technical aspects?

> My cat.  Then I (cause at that point my cat couldn't be bothered to
> follow me to the desk) tried what we both thought might work, then
> I applied it to more modes, and a few years later it ends up working in
> *all* major modes I know (including CC-mode, tho I haven't bothered to
> make it robust enough to have a patch, knowing that you're never going
> to accept such a patch in CC-mode, even though it's mostly just throwing
> away code).

It may be possible to rebuild CC Mode to conform to this "standard", but
it would be a lot of work, the "standard" would need enhancements[*], CC
Mode would run more slowly, and its users wouldn't see any benefit.

[*] What I'm thinking of, in particular, is after buffer changes, the
syntax-propertize mechanism doesn't amend the syntax-table text
properties adquately.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Alan Mackenzie]

Hello, Dmitry.

On Sun, Dec 03, 2017 at 03:52:59 +0000, Dmitry Gutov wrote:
> On 12/2/17 8:28 PM, Alan Mackenzie wrote:

> > It would work, and work well, for the simple reason it is a coherent
> > proposal.

> I've never seen a full proposal. Details matter. And it's going to have 
> to include a lot more of them.

Details do matter.  I posted a full proposal, Subject: "Syntax
ambiguities in narrowed buffers and multiple major modes: a proposed
solution." here in emacs-devel on Sat, 25 Feb 2017 13:53:56 +0000.

I've changed my mind about some of the details, partly in response to
what people said, partly because I see things now I didn't see then.

> > Emacs was designed with the assumption that each buffer has exactly one
> > major mode.  It turns out this assumption hasn't held all that well.
> > When that assumption does hold, the following returns a well defined and
> > meaningful state:

> >      (parse-partial-sexp 1 n)

> > .  I am proposing extending this property for buffers with several
> > modes and several syntax tables.  Nothing more, nothing less.

> Extending parse-partial-sexp to nested syntax tables can be worthwhile, 
> but that can't be it. And while we're on this subject, there are some 
> example of Ruby syntax that we'd like to see supported, where the 
> current syntax tables (and even syntax-propertize-function) don't cut it.

What is preventing a s-p-function from working satisfactorally?

> I'm not convinced "several syntax tables" would be enough, though.

OK, let's make that "many syntax tables".  :-)

> > You're saying that providing support for multiple major modes in the
> > Emacs core is a pipe dream.  I put it to you you're mistaken.  ONLY by
> > supporting this in the Emacs core can we arrive at a coherent efficient
> > design.

> The major modes will have to change to allow for this support, that's 
> for sure. You can't expect no code changes.

Maybe, maybe not.  Super-modes would definitely have to change to use
it.  Any changes in major modes should be minor.  (I don't rate
restrictions on the use of Emacs primitives to be minor.)

> > The fact we are having this conversation is a strong indication
> > that the current designs for MMM are not coherent enough.

> The current idea is "let's do the minimum coherent step forward, so that 
> somebody can continue improving things later". A step that's easy to 
> use, and also easy to undo later, if we find something else that's much 
> better.

OK.  With such a tricky subject matter, it would surprising if we
couldn't find improvements for a long time.

> Coming up with it was not easy at all, BTW.

Yes, I can see that.  Well done!

> >> Also, I don't think major modes need more freedom in how to implement
> >> things.  Instead, they need more help so they can just use existing
> >> solutions rather than reinvent the wheel in their very own way.
> >> Conventions/guidelines actually makes the life of major mode
> >> implementers easier, rather than harder.

> > That is immensely patronising.

> It's simply description of good engineering. You can try to criticize 
> the details of syntax-propertize, but the idea of reusing common 
> solutions is more than sound.

If those common solution are adequate, yes.  But it is also critical to
have the freedom to depart from things like syntax-propertize when they
are inadequate, or too awkward.  syntax-propertize is inadequate for CC
Mode:

E.g., in C++ Mode, with

    foo (a < b, c > d);

we have a constructor foo with one parameter d, of a templated type.  The
characters "<" and ">" have syntax-table text properties giving them
paren syntaxes and making them match each other.

Now, mark the characters ", c > d" and kill them with C-w.  What we are
left with is a function call with an integer argument a < b:

    foo (a < b);

.  Something, somewhere has to remove the syntax-table property from
"<", since it is no longer a template delimiter.

The syntax-propertize mechanism won't do this, since the place to
de-propertise is before BEG.  In CC Mode, the job is done by
c-before-change-check-<>-operators.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> Cc: acm@muc.de, tom@tromey.com, monnier@iro.umontreal.ca, spinuvit@gmail.com,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 2 Dec 2017 21:35:43 +0000
> 
> I'm asking why "Emacs is written in these 2 languages" is important.

Because every development environment should first support its own
development well.  If it doesn't do that, it's incomplete.

> > Surely, a feature that aims to
> > support multiple major modes should strive to support CC Mode?
> 
> Not necessarily. It should first and foremost support the languages that 
> are most important for our users.

Indeed; and C-like languages are quite popular among them.

How will we look if we support JS and Ruby embedded in HTML, but not C
code embedded in Yacc grammars nor Awk snippets embedded in shell
scripts?  Both are quite frequent out there.  We will be laughed at.

> > They are important, but so are C-based languages: C++, Java, Awk, etc.
> 
> I don't know CC Mode, and cannot provide patches for us. I tried 
> debugging related problems a few times, and those turned out to be 
> fairly long spelunking expeditions.
> 
> However long it will take just to convince Alan to agree, is anybody's 
> guess.

Convince Alan to do what?  Do we even understand well enough what are
the problems in CC Mode that get in the way?  About the only thing
spelled out in this discussion was the need not to widen.  Personally,
I think this is not a grave issue, just some technical problem that is
certainly solvable (you proposed one such solution).  But is that all?
You mentioned some other problems, but never described them.  What are
they?  Are they grave enough to prevent your proposal from ever
supporting CC Mode, even if your proposal is amended?  I don't know.
Does anyone know?  If so, would they please speak up?

I also don't understand well the essence of your proposal.  Yes, I've
(tried to) read the discussions you pointed to, but what I'm looking
for is buried there under many layers of secondary stuff.  Why do you
want to abandon the prog-widen stuff, which you supported just 2 years
ago?  How do you address the issues which prog-indentation-context did
(e.g., if the embedded chunk of code is incomplete, and perhaps even
syntactically invalid)?  All these questions and considerations need
to be understood, because you are explicitly proposing an incomplete
solution, and asking us to agree with its deficiencies, but without
providing a clear picture of what are we going to give up on.  And
last, but not least, what is different in your proposal that will not
cause it to end up like all the previous ones: either committed to
Emacs and basically unused, or, worse, collecting dust in long
forgotten patches or branches?

We desperately need these details spelled out to make the discussion
of this to become technical again, not emotional.

> > It's not a catastrophe.  Assuming Emacs 26.1 is not a complete
> > disaster, we could decide releasing Emacs 27.1 as the next version.
> 
> Not a catastrophe indeed. But no step forward for mixed-modes support in 
> Emacs 26 would be unfortunate.

Emacs 26 has enough new features to justify a major release, so I see
no reason to press so hard to get this feature into it as well.  You
are still debating its design and implementation, even if we disregard
the CC Mode issue.  It makes no sense to put this into a code base
which is frozen since September.

> > Why would we need to remove that for Emacs 26.1?
> 
> It's not well supported, and it's incompatible with the code we are 
> discussing. There's no point in having it present in Emacs 26, and then 
> removing it in Emacs 27.

That code is in Emacs for more than 2 years.  It was admitted with
Stefan's full support, and at least ANTLR needs it in conjunction with
Python.  Removing it without having some alternative again makes no
sense to me.  We should discuss this when the incompatible code lands;
at that time, we will see how to remove/replace prog-widen etc. with
minimal pain for its users.  For now, there's no incompatibility that
requires us to remove the code.  And anyway, making such changes when
we are in pretest is against our practices.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/3/17 12:18 PM, Alan Mackenzie wrote:

>> That would potentially solve some percentage of the problems with
>> MMM support.  But what about the rest?
> 
> They would clearly need other work doing on them.  Richard suggested
> building something similar to narrowing for the excusive use of MMM.

This sounds fairly different from your islands proposal. And the notion 
that you've already described a full solution, and was only waiting for 
approval.

> Standards are created by like-minded experts coming together, thrashing
> things out, and finally agreeing on some reasonable set of compromises;
> not by a lone hacker, no matter how expert, throwing something together
> in his bedroom.

That's very debatable. Most good standards come from working experience, 
and usually some working product (or a codebase).

A lot of standards that start in a meeting room don't leave one. Just 
look at the multitude of discussions here that don't move anywhere. Also 
see "design by committee".

Discussion is important, of course, but it has to be constructive from 
all sides.

> It may be possible to rebuild CC Mode to conform to this "standard", but
> it would be a lot of work, the "standard" would need enhancements[*], CC
> Mode would run more slowly, and its users wouldn't see any benefit.

Would you like to provide a benchmark that Stefan could try and see if 
the syntax-ppss based code indeed runs "more slowly"?

Anyway, this seems irrelevant to this discussion.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/3/17 3:28 PM, Eli Zaretskii wrote:

> Because every development environment should first support its own
> development well.  If it doesn't do that, it's incomplete.

OK, I see. But!

This feature is *not* targeted at Emacs development.

Please don't make me fly out to Scotland and chase Alan down with a 
broomstick to make this development come through.

> Indeed; and C-like languages are quite popular among them.

Not in comparison to JS, CSS and HTML.

> How will we look if we support JS and Ruby embedded in HTML, but not C
> code embedded in Yacc grammars nor Awk snippets embedded in shell
> scripts?  Both are quite frequent out there.  We will be laughed at.

We'll be fine. And the actual support is still going to reside in 
third-party packages for now. This change just makes them possible (or 
their codebases saner, at least).

Except for antlr-mode. It will be able to use it right away (the version 
inside Emacs, at least).

> Convince Alan to do what?

To adhere to the current proposal (avoid widening in 
indent-line-function and font-lock-keywords, to start with).

> Do we even understand well enough what are
> the problems in CC Mode that get in the way?

Who do you think is most qualified to study that issue? We'd probably 
have to convince Alan to do that as well, unfortunately.

I have a rough understanding of the issue, but since I haven't reached a 
working state, I don't know how many pitfalls there are left.

> About the only thing
> spelled out in this discussion was the need not to widen.  Personally,
> I think this is not a grave issue, just some technical problem that is
> certainly solvable (you proposed one such solution).

Indeed.

> But is that all?

I imagine the process itself might be trickier than expected. Various 
primitives use caches that save context information. What is such 
primitive to do if the cache contains "beginning of nesting" outside of 
the current restriction, and the logic of said primitive says "go to the 
beginning of the current function and do such and such"? The answer 
isn't obvious to me.

But it's going to need to be answered anyway, no matter which MMM 
approach we choose in the end, I think.

> You mentioned some other problems, but never described them.  What are
> they?  Are they grave enough to prevent your proposal from ever
> supporting CC Mode, even if your proposal is amended?  I don't know.
> Does anyone know?  If so, would they please speak up?

Yes, please.

> I also don't understand well the essence of your proposal.  Yes, I've
> (tried to) read the discussions you pointed to, but what I'm looking
> for is buried there under many layers of secondary stuff. 

The proposal itself is very small, and there's not much to explain. Just 
look at the changes in the manual, in this branch.

It simply facilitates what mmm-mode (and other modes, including 
mhtml-mode) has been doing for years, with varying success. And does 
that in a way that requires minimal changes from major modes (except for 
CC Mode, it seems), and would let us easily change our mind later.

> Why do you
> want to abandon the prog-widen stuff, which you supported just 2 years
> ago?

I never actually supported it, just stopped arguing because I didn't 
have a good alternative idea. Now I do, I think. And there is some 
agreement from Stefan.

> How do you address the issues which prog-indentation-context did
> (e.g., if the embedded chunk of code is incomplete, and perhaps even
> syntactically invalid)?

prog-indentation-context never addressed that issue, and we don't 
either. The major mode code in such situation will either have to 
swallow errors, or, well, they won't work. Or a MMM package will have to 
do error-swallowing, that's also possible.

In practice, a lot of such situations don't actually manifest, though.

> All these questions and considerations need
> to be understood, because you are explicitly proposing an incomplete
> solution, and asking us to agree with its deficiencies, but without
> providing a clear picture of what are we going to give up on.

That's the thing: we're not giving up much (I'd argue we're giving up 
nothing, but Alan obviously disagrees).

Consolidating the 'widen' calls is simply good engineering, even aside 
from making life easier for MMM packages: it's much better to do that in 
several well-defined places, instead of having every helper function do 
it as well, like python-mode CC Mode do.

> And
> last, but not least, what is different in your proposal that will not
> cause it to end up like all the previous ones: either committed to
> Emacs and basically unused, or, worse, collecting dust in long
> forgotten patches or branches?

Like mentioned, the narrowing-based approach is already used in mmm-mode 
and mhtml-mode (and polymode too, I think).

And it works fine in many situations, because many major modes already 
never call widen in their indentation functions (SMIE-based modes, Ruby, 
HTML and JS don't). So this proposal is, to a large extent, codifying an 
existing practice. And at the same time makes the aforementioned modes 
work *better* with interactive narrowing.

> We desperately need these details spelled out to make the discussion
> of this to become technical again, not emotional.

Please feel free to ask any further questions.

> You
> are still debating its design and implementation, even if we disregard
> the CC Mode issue.

Not really. There's a minor issue of whether to make prog-first-column a 
variable, or a hook right away, but the importance of that choice isn't big.

> That code is in Emacs for more than 2 years.  It was admitted with
> Stefan's full support, and at least ANTLR needs it in conjunction with
> Python.

Transitioning from prog-indentation-context to the new approach is very 
easy. And I would show you the patch, but unfortunately, the antlr-mode 
in Emacs doesn't use prog-indentation-context.

So really, it's been here for 2 years, and virtually nobody's using it, 
or improving it.

> Removing it without having some alternative again makes no
> sense to me. 

You have the alternative, though.

> We should discuss this when the incompatible code lands;

How about we discuss it now?

> at that time, we will see how to remove/replace prog-widen etc. with
> minimal pain for its users.  For now, there's no incompatibility that
> requires us to remove the code.  And anyway, making such changes when
> we are in pretest is against our practices.

That seems very unwise to me. Not to mention discouraging.

Take a look at any third-party packages. I'm willing to bet none use 
prog-indentation-context yet. Maybe the standalone version of 
antlr-mode, if only.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/2/17 11:47 PM, Stefan Monnier wrote:

> Since Eli only wants MMM-support if it work 100% everywhere, I don't
> think it matters what gets into Emacs itself in this respect, because
> the real stuff will have to live elsewhere.

But we need prog-first-column in Emacs. Except it already is. >_<

> The thing we should try and fight for of course are those such as making
> indent-according-to-mode widen before calling indent-line-function.

Right. That's true for many major modes already, though.

> I'm glad English includes enough redundancy that you could make out what
> I meant.

:-)

> The idea is just to highlight those lines whose indentation would be
> modified by indent-line-function, so you only want to compute the
> target column and compare it with the current column.

I still don't see how you're going to compute the target column without 
calling the indentation function. That's why switching to a non-mutating 
function will help.

>>> Arguably `smie-indent-functions` already provides that functionality.
>> As an example, yes, but it's not like we can move all modes to SMIE.
> 
> Hence the "arguably": it can be used without using SMIE, but it's not as
> convenient as you might like.  E.g. you can do:

Yeah... Let's not do that. Generic code won't really be able to rely on 
this practice anyway.

>>> BTW, could we get some kind of multi-mode package in elpa.git or
>>> emacs.git to go along with that (it doesn't have to be fancy, but it's
>>> important that it doesn't have any submode-specific hacks).
>> Do you have any particular hacks in mind, in e.g. mmm-mode?
> 
> No.  Can we put mmm-mode into elpa.git?

I'll try emailing the original authors again. The last time I did that 
there were no replies, unfortunately.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> Cc: acm@muc.de, tom@tromey.com, monnier@iro.umontreal.ca, spinuvit@gmail.com,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 3 Dec 2017 16:35:23 +0000
> 
> This feature is *not* targeted at Emacs development.

It is for me.

> > Indeed; and C-like languages are quite popular among them.
> 
> Not in comparison to JS, CSS and HTML.

Maybe in your world, but not in mine.  So we will have to agree (or
disagree, if you want) that both worlds need to be catered to.

> > How will we look if we support JS and Ruby embedded in HTML, but not C
> > code embedded in Yacc grammars nor Awk snippets embedded in shell
> > scripts?  Both are quite frequent out there.  We will be laughed at.
> 
> We'll be fine.

From your POV, maybe.  Not from mine.

> > Convince Alan to do what?
> 
> To adhere to the current proposal (avoid widening in 
> indent-line-function and font-lock-keywords, to start with).

We need to understand the full extent of problems, and then we need to
see how to go about them, at least design-wise.  "To start with" is a
start, but it cannot be the middle and the end.

> > Do we even understand well enough what are
> > the problems in CC Mode that get in the way?
> 
> Who do you think is most qualified to study that issue? We'd probably 
> have to convince Alan to do that as well, unfortunately.

If he agrees, that'd be swell.  If not, I see no reason why someone
else couldn't do that and report back, it's not like CC Mode is hard
to activate and see what problems pop up.

> I have a rough understanding of the issue, but since I haven't reached a 
> working state, I don't know how many pitfalls there are left.

How about describing what you do know?

> I imagine the process itself might be trickier than expected. Various 
> primitives use caches that save context information. What is such 
> primitive to do if the cache contains "beginning of nesting" outside of 
> the current restriction, and the logic of said primitive says "go to the 
> beginning of the current function and do such and such"? The answer 
> isn't obvious to me.

I don't understand: AFAIU in the use cases supported by MMM there can
be nothing outside of the current restriction that is of interest for
the mode which supports the chunk under the restriction.  So what kind
of "nesting outside of the current restriction" are we talking about,
and how does it come into play?

> The proposal itself is very small, and there's not much to explain. Just 
> look at the changes in the manual, in this branch.

That just removes text, more or less, and what it adds doesn't explain
itself well enough, sorry.  Otherwise I wouldn't have asked these
questions, believe me.

> It simply facilitates what mmm-mode (and other modes, including 
> mhtml-mode) has been doing for years, with varying success.

Sorry, that doesn't help me understand the proposal, as I don't know
mmm-mode well enough.  How hard is it to just describe the idea in a
few sentences?

> > Why do you
> > want to abandon the prog-widen stuff, which you supported just 2 years
> > ago?
> 
> I never actually supported it, just stopped arguing because I didn't 
> have a good alternative idea. Now I do, I think. And there is some 
> agreement from Stefan.

(whose change of hearts is also a mystery for me)

> > How do you address the issues which prog-indentation-context did
> > (e.g., if the embedded chunk of code is incomplete, and perhaps even
> > syntactically invalid)?
> 
> prog-indentation-context never addressed that issue, and we don't 
> either.

I think it does, at least to some extent.  And even if I'm wrong, then
what is the equivalent of what it does address in your proposal?

In any case, we should at least have provisions for supporting that
within the proposed framework, and we should be fairly certain that
supporting that later will not blow things up.

> > All these questions and considerations need
> > to be understood, because you are explicitly proposing an incomplete
> > solution, and asking us to agree with its deficiencies, but without
> > providing a clear picture of what are we going to give up on.
> 
> That's the thing: we're not giving up much

For the modes that are dear to your heart, maybe.  But that's not all
Emacs should support well, IMO.

> Consolidating the 'widen' calls is simply good engineering, even aside 
> from making life easier for MMM packages: it's much better to do that in 
> several well-defined places, instead of having every helper function do 
> it as well, like python-mode CC Mode do.

The 'widen' thing is a non-issue; it's the other stuff I'm worried
about, mostly because it is left unsaid, undiscussed, and I fear
uncatered-to by the design.

> > You
> > are still debating its design and implementation, even if we disregard
> > the CC Mode issue.
> 
> Not really. There's a minor issue of whether to make prog-first-column a 
> variable, or a hook right away, but the importance of that choice isn't big.

The fact is, it isn't done yet, let alone well-tested.

> > That code is in Emacs for more than 2 years.  It was admitted with
> > Stefan's full support, and at least ANTLR needs it in conjunction with
> > Python.
> 
> Transitioning from prog-indentation-context to the new approach is very 
> easy.

That's what I thought.  And that's why we should do that
simultaneously with landing the new approach.  There's no reason to do
that earlier.

> So really, it's been here for 2 years, and virtually nobody's using it, 
> or improving it.

It's definitely used by its author, so "nobody uses it" is untrue, and
even unfair: that feature was accepted by Stefan, under the assumption
that it will be used.

> > Removing it without having some alternative again makes no
> > sense to me. 
> 
> You have the alternative, though.

No, we don't.  You are asking to remove the code _before_ the
alternative lands.

> > We should discuss this when the incompatible code lands;
> 
> How about we discuss it now?

The incompatible code didn't land yet.  But feel free to include the
replacement in your branch.

> > at that time, we will see how to remove/replace prog-widen etc. with
> > minimal pain for its users.  For now, there's no incompatibility that
> > requires us to remove the code.  And anyway, making such changes when
> > we are in pretest is against our practices.
> 
> That seems very unwise to me. Not to mention discouraging.

I don't see anything unwise in it, let alone discouraging.  We've
turned down much simpler and safer changes that came too late, and no
one claimed that was unwise or discouraging.

> Take a look at any third-party packages. I'm willing to bet none use 
> prog-indentation-context yet.

We cannot know that.  Once the code is that long with us, we cannot
throw it away without a equivalent replacement.  And I see no reason
to do that now, since the fact that this code will exist in Emacs 26
doesn't hurt anyone, especially since the replacement will be easy, as
you say.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

> Details do matter.  I posted a full proposal, Subject: "Syntax
> ambiguities in narrowed buffers and multiple major modes: a proposed
> solution."  here in emacs-devel on Sat, 25 Feb 2017 13:53:56 +0000.

For the reasons exposed before, contrary to Dmitry, I don't really care
about seeing a detailed proposal.  I know full well that I'm absolutely
unable to run such the paper-design in my head and predict if it'll
really work.

> I don't rate restrictions on the use of Emacs primitives to be minor.

I think there's a fundamental misunderstanding abut Dmitry's suggestion
here: his proposal does not restrict the use of any primitive anywhere.

The only thing it does is to make the calling convention of
indent-line-function a bit more precise: instead of not saying anything
about the kind of narrowing that can be in place when that is called, it
specifies that the caller should arrange the narrowing to expose "all"
the code, and so the callee doesn't need to widen (which is useful since
in an MMM context, the sub-mode usually doesn't know how far it should
widen).

The indent-line-functions can still narrow/widen all it wants.
And the callers of indent-line-function also can still widen/narrow all
they want.  Just as is the case now, depending on how they narrow/widen
the result may be incorrect.  And the novelty here is to provide
a convention that, if followed, should make it more likely that the
result is correct, without the caller and the callee being aware of each
other, because they only care about the shared convention.

> If those common solution are adequate, yes.  But it is also critical to
> have the freedom to depart from things like syntax-propertize when they
> are inadequate, or too awkward.  syntax-propertize is inadequate for CC
> Mode:
>
> E.g., in C++ Mode, with
>
>     foo (a < b, c > d);
>
> we have a constructor foo with one parameter d, of a templated type.  The
> characters "<" and ">" have syntax-table text properties giving them
> paren syntaxes and making them match each other.
>
> Now, mark the characters ", c > d" and kill them with C-w.  What we are
> left with is a function call with an integer argument a < b:
>
>     foo (a < b);
>
> .  Something, somewhere has to remove the syntax-table property from
> "<", since it is no longer a template delimiter.
>
> The syntax-propertize mechanism won't do this, since the place to
> de-propertise is before BEG.

syntax-propertize is no silver bullet, indeed, so it won't do everything
magically for you.  But, it can handle the above situation just fine.
In your very specific above example, it will actually be handled
magically simply by virtue of syntax-propertize-wholelines being
included in the default value of syntax-propertize-extend-region-functions.

Now, of course, if you tweak your example so that the < and the > are
not on the same line any more, that magic won't work.  But that doesn't
mean you can't make syntax-propertize handle it right.  You could for
example solve it with:

    (add-hook 'syntax-propertize-extend-region-functions
              #'syntax-propertize-multiline nil t)

in your major mode function and

    (put-text-property position-of-left-< position-of-right->
                       'syntax-multiline t)

when you find (and put the syntax-table property on) the <...> pair.
Of course, it's not the only possible way to do it.  It's just the way
I solved similar problems in perl-mode and sh-mode.

If you're interested in making CC-mode work reliably with
syntax-propertize I'd be *thrilled* to help you do that.

By design, there should be basically nothing syntax-propertize can't handle
(modulo bugs and within the limits of what we can do with syntax-tables
and syntax-table properties, obviously).  I have no doubt that the way
CC-mode works right now will occasionally be more efficient than the way
syntax-propertize would do it.  But I also have no doubt that the way
syntax-propertize would do it will occasionally by more efficient than
the way CC-mode does it right now.  I strongly doubt that the difference
in performance behavior will ever matter, tho.


        Stefan

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Alan Mackenzie]

Hello, Dmitry.

On Sun, Dec 03, 2017 at 16:35:23 +0000, Dmitry Gutov wrote:
> On 12/3/17 3:28 PM, Eli Zaretskii wrote:

> > Because every development environment should first support its own
> > development well.  If it doesn't do that, it's incomplete.

> OK, I see. But!

> This feature is *not* targeted at Emacs development.

> Please don't make me fly out to Scotland and chase Alan down with a 
> broomstick to make this development come through.

Please read my signature, sometime.

> > Indeed; and C-like languages are quite popular among them.

> Not in comparison to JS, CSS and HTML.

> > How will we look if we support JS and Ruby embedded in HTML, but not C
> > code embedded in Yacc grammars nor Awk snippets embedded in shell
> > scripts?  Both are quite frequent out there.  We will be laughed at.

> We'll be fine. And the actual support is still going to reside in 
> third-party packages for now. This change just makes them possible (or 
> their codebases saner, at least).

> Except for antlr-mode. It will be able to use it right away (the version 
> inside Emacs, at least).

> > Convince Alan to do what?

> To adhere to the current proposal (avoid widening in 
> indent-line-function and font-lock-keywords, to start with).

I know Eli has asked to move on from the emotional to the technical
stuff, but that last paragraph belongs to the former, and I will deal
with it as such.

Maybe I missed it, but I don't recall anybody posting the suggestion
"Hey, we're considering writing an MMM mode which will mean nobody else
can use narrowing and widening, what do people think?".  If somebody had
posted that, there's a good chance that discussions could have sorted
out the technical stuff without provoking bad feelings.

Right now, I am being painted as the bad person for objecting to being
restricted to a subset of Emacs Lisp for CC Mode.  Forgive me for
feeling a bit peeved, but despite being in Emacs development for around
15 years, nobody asked me beforehand what I thought of this.  It is yet
one more thing that is presented as a done thing and foisted on Emacs
developers without them having any say in the matter.

You mentioned today, I think, that writing an MMM is hard.  Well, CC
Mode is hard, too.  There are 30 calls to `widen' in CC Mode and 47 to
`narrow-to-region'.  They are all there for a reason.  It will be
grinding tedious work to sort out the whys and to remove them.  You
categorise the injunction against widening as "only applying to
indent-line-function and "font-lock-keywords" (which isn't a function)".
This is from my point of view non-sensical since so many of the `widen'
calls are in low level functions called from "everywhere".

Yesterday, Richard Stallman suggested as an alternative to the
purloining of `widen' and `narrow-to-region' that a new "region" be
implemented somehow which would be independent of the existing region
and used solely by MMM super-modes.  How about exploring this
possibility?

Last February, I suggested extensions to the syntax code ("syntactic
islands") which would allow operations such as parse-partial-sexp to
work essentially without restriction in buffers with several syntax
tables.  How about exploring this possibility?

> > Do we even understand well enough what are
> > the problems in CC Mode that get in the way?

> Who do you think is most qualified to study that issue? We'd probably 
> have to convince Alan to do that as well, unfortunately.

Believe it or not, I am in favour of CC Mode working in an MMM mode.

> I have a rough understanding of the issue, but since I haven't reached a 
> working state, I don't know how many pitfalls there are left.

Neither do I.  But RMS's "new region" and my "syntax islands" may be a
more satisfactory way of resolving them.

> > About the only thing spelled out in this discussion was the need not
> > to widen.  Personally, I think this is not a grave issue, just some
> > technical problem that is certainly solvable (you proposed one such
> > solution).

> Indeed.

> > But is that all?

> I imagine the process itself might be trickier than expected. Various 
> primitives use caches that save context information. What is such 
> primitive to do if the cache contains "beginning of nesting" outside of 
> the current restriction, and the logic of said primitive says "go to the 
> beginning of the current function and do such and such"? The answer 
> isn't obvious to me.

Again, looking at RMS's suggested "new region" might help, here.

> But it's going to need to be answered anyway, no matter which MMM 
> approach we choose in the end, I think.

[ .... ]

> Consolidating the 'widen' calls is simply good engineering, even aside 
> from making life easier for MMM packages: it's much better to do that in 
> several well-defined places, instead of having every helper function do 
> it as well, like python-mode CC Mode do.

No, it isn't "simply good engineering" at all.  It's a different
approach with its pros and cons, just as widening at the immediate place
of need is.  And "consolidating" is probably the wrong word.  The number
of `widen's will probably grow, since every conceivable entry point
which could possibly call the pertinent primitve needs a `widen'.

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> Date: Sun, 3 Dec 2017 18:59:46 +0000
> Cc: Eli Zaretskii <eliz@gnu.org>, tom@tromey.com, monnier@iro.umontreal.ca,
>   spinuvit@gmail.com, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > > Convince Alan to do what?
> 
> > To adhere to the current proposal (avoid widening in 
> > indent-line-function and font-lock-keywords, to start with).
> 
> I know Eli has asked to move on from the emotional to the technical
> stuff, but that last paragraph belongs to the former, and I will deal
> with it as such.

And I would _really_ want to move to the technical stuff.  So...  I
actually don't understand all the fuss about widening.  With a chunk
of C code embedded in something that is not C, CC Mode cannot possibly
need to look outside of the chunk, so why would you need to widen
beyond that?

> You mentioned today, I think, that writing an MMM is hard.  Well, CC
> Mode is hard, too.  There are 30 calls to `widen' in CC Mode and 47 to
> `narrow-to-region'.  They are all there for a reason.  It will be
> grinding tedious work to sort out the whys and to remove them.

Is that the only reason for your objections?  The need to tediously go
through a few dozen of calls to 'widen' and 'narrow-to-region', and
take care of each one of them as appropriate?  Or are there more
substantial problems in adapting CC Mode to non-widening MMM?

> so many of the `widen' calls are in low level functions called from
> "everywhere".

This can be taken care of with a more-or-less trivial wrapper and a
variable.  Right?

> Yesterday, Richard Stallman suggested as an alternative to the
> purloining of `widen' and `narrow-to-region' that a new "region" be
> implemented somehow which would be independent of the existing region
> and used solely by MMM super-modes.  How about exploring this
> possibility?

We can do that if needed, but I don't see the need yet.  (And a
proposal to do something along these lines was put forward by Vitalie
some time ago.)

> Last February, I suggested extensions to the syntax code ("syntactic
> islands") which would allow operations such as parse-partial-sexp to
> work essentially without restriction in buffers with several syntax
> tables.

What problems does this solve?  The problem with 'widen' or some other
problem(s)?

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/3/17 5:20 PM, Eli Zaretskii wrote:

>> This feature is *not* targeted at Emacs development.
> 
> It is for me.

You are not working on it. You've expressed no interest in this problem 
until now. Maybe delegate the basic requirements, at least, to the 
people who did?

>>> Indeed; and C-like languages are quite popular among them.
>>
>> Not in comparison to JS, CSS and HTML.
> 
> Maybe in your world, but not in mine.  So we will have to agree (or
> disagree, if you want) that both worlds need to be catered to.

Yes, and they can be catered to, *as soon as* the authors of each 
individual major mode make the effort to support the feature.

Having it work in HTML, CSS, JS and Ruby is in no way dependent in 
investigating and fixing the behemoth that CC Mode is.

>>> How will we look if we support JS and Ruby embedded in HTML, but not C
>>> code embedded in Yacc grammars nor Awk snippets embedded in shell
>>> scripts?  Both are quite frequent out there.  We will be laughed at.
>>
>> We'll be fine.
> 
>  From your POV, maybe.  Not from mine.

Eli, we already have a feature in Emacs (prog-indentation-context) that 
does not adhere to your (unreasonably high) requirements.

Let's at least take it out, then.

>>> Convince Alan to do what?
>>
>> To adhere to the current proposal (avoid widening in
>> indent-line-function and font-lock-keywords, to start with).
> 
> We need to understand the full extent of problems, and then we need to
> see how to go about them, at least design-wise.  "To start with" is a
> start, but it cannot be the middle and the end.

The full extent of the problem can be realized as one tried to do what I 
said, step by step. And see what else can be is going wrong.

> If he agrees, that'd be swell.  If not, I see no reason why someone
> else couldn't do that and report back, it's not like CC Mode is hard
> to activate and see what problems pop up.

Please go ahead and do it, then.

If you're really expecting me to do it, it's unrealistic. Getting 
oriented in CC Mode will take days, if not weeks. And I have no interest 
nor energy to do that.

>> I have a rough understanding of the issue, but since I haven't reached a
>> working state, I don't know how many pitfalls there are left.
> 
> How about describing what you do know?

Already did. Last time I tried investigating it was 1-2 years ago, and 
at no point I felt I reached an understanding of why CC Mode exhibits 
the problems it does, or what I could change to fix them.

So all I have are vague recollections.

>> I imagine the process itself might be trickier than expected. Various
>> primitives use caches that save context information. What is such
>> primitive to do if the cache contains "beginning of nesting" outside of
>> the current restriction, and the logic of said primitive says "go to the
>> beginning of the current function and do such and such"? The answer
>> isn't obvious to me.
> 
> I don't understand: AFAIU in the use cases supported by MMM there can
> be nothing outside of the current restriction that is of interest for
> the mode which supports the chunk under the restriction.  So what kind
> of "nesting outside of the current restriction" are we talking about,
> and how does it come into play?

Imagine if the chunk contains only this:

    <

And, to fontify it, CC Mode, tried to find out what syntactic element it 
is, and to do that, calls some function that uses a specialized cache 
that's contained in a variable that mmm-mode knows nothing about.

And that cache point to a position outside of the current restriction. 
Confusion ensues. I've seen errors originating from that (I think), but 
can't recall the exact sequence of calls.

>> The proposal itself is very small, and there's not much to explain. Just
>> look at the changes in the manual, in this branch.
> 
> That just removes text, more or less, and what it adds doesn't explain
> itself well enough, sorry.  Otherwise I wouldn't have asked these
> questions, believe me.

The text it adds is more important. Including the mentions of 
font-lock-keywords and syntax-propertize-function, by the way.

>> It simply facilitates what mmm-mode (and other modes, including
>> mhtml-mode) has been doing for years, with varying success.
> 
> Sorry, that doesn't help me understand the proposal, as I don't know
> mmm-mode well enough.  How hard is it to just describe the idea in a
> few sentences?

Not really sure what to describe.

To fontify (or indent) a chunk, mmm-mode narrows to its bounds, and 
calls the corresponding indent-line-function (or applies 
font-lock-keywords). If either of those calls (widen), that causes problems.

So we want to document a convention that they don't call widen. That's 
really it.

>> I never actually supported it, just stopped arguing because I didn't
>> have a good alternative idea. Now I do, I think. And there is some
>> agreement from Stefan.
> 
> (whose change of hearts is also a mystery for me)

Sometimes a better technical proposal does that.

For more context: we're been waiting for some kind of improvement for 
this problem for years. Then prog-indentation-context came along, and it 
wasn't terrible (though it didn't solve all of the problems we wanted). 
That seemed a good enough reason to adopt it, to some of us.

Since then, for the mentioned two years, the original author contributed 
no improvements aside from the mentioned support in python-mode, and no 
other mode added support for it, or used it, inside or outside Emacs. So 
it seems to me like it pretty much failed.

>>> How do you address the issues which prog-indentation-context did
>>> (e.g., if the embedded chunk of code is incomplete, and perhaps even
>>> syntactically invalid)?
>>
>> prog-indentation-context never addressed that issue, and we don't
>> either.
> 
> I think it does, at least to some extent.

Not any more than scratch/widen-less.

> And even if I'm wrong, then
> what is the equivalent of what it does address in your proposal?

There are full equivalents.

We keep 'prog-first-column' from that proposal, and instead of allowing 
MMM to indicate the chunk bounds through prog-indentation-context, we 
allow MMM to apply narrowing directly, and that modes honor it.

> In any case, we should at least have provisions for supporting that
> within the proposed framework, and we should be fairly certain that
> supporting that later will not blow things up.

I have honestly no idea where to begin in addressing that issue.

On the other hand, there are third-party MMM framework that function 
within these limitations, and are helpful to users.

>> That's the thing: we're not giving up much
> 
> For the modes that are dear to your heart, maybe.  But that's not all
> Emacs should support well, IMO.

For all modes. The user won't lose any functionality, at all. Even if 
the programmer might have to spend some effort for that to happen, for 
some of the modes.

> The 'widen' thing is a non-issue; it's the other stuff I'm worried
> about, mostly because it is left unsaid, undiscussed, and I fear
> uncatered-to by the design.

What other stuff? We basically say "widen happens in one place, don't do 
that again", and that's it.

>> Not really. There's a minor issue of whether to make prog-first-column a
>> variable, or a hook right away, but the importance of that choice isn't big.
> 
> The fact is, it isn't done yet, let alone well-tested.

It can be "done" in a day or two.

>>> That code is in Emacs for more than 2 years.  It was admitted with
>>> Stefan's full support, and at least ANTLR needs it in conjunction with
>>> Python.
>>
>> Transitioning from prog-indentation-context to the new approach is very
>> easy.
> 
> That's what I thought.  And that's why we should do that
> simultaneously with landing the new approach.  There's no reason to do
> that earlier.

Why confuse the language modes authors?

>> So really, it's been here for 2 years, and virtually nobody's using it,
>> or improving it.
> 
> It's definitely used by its author, so "nobody uses it" is untrue, and
> even unfair: that feature was accepted by Stefan, under the assumption
> that it will be used.

...and improved, and support for CC Mode would be contributed. Which it 
wasn't.

The author, who's the only user of prog-indentation-context, can 
transition to the new convention trivially.

>>> Removing it without having some alternative again makes no
>>> sense to me.
>>
>> You have the alternative, though.
> 
> No, we don't.  You are asking to remove the code _before_ the
> alternative lands.

You have the choice of accepting the alternative sooner, though.

>>> We should discuss this when the incompatible code lands;
>>
>> How about we discuss it now?
> 
> The incompatible code didn't land yet.  But feel free to include the
> replacement in your branch.

It's in there already.

>> Take a look at any third-party packages. I'm willing to bet none use
>> prog-indentation-context yet.
> 
> We cannot know that.  Once the code is that long with us, we cannot
> throw it away without a equivalent replacement.  And I see no reason
> to do that now, since the fact that this code will exist in Emacs 26
> doesn't hurt anyone, especially since the replacement will be easy, as
> you say.

Well... you have a point there. The result will be pretty ugly, though. 
Adding an API in one version, and removing it in another.

They are incompatible with each other, so it's not like there can be a 
smooth transition.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Alan Mackenzie]

Hello, Eli.

On Sun, Dec 03, 2017 at 21:25:09 +0200, Eli Zaretskii wrote:
> > Date: Sun, 3 Dec 2017 18:59:46 +0000
> > Cc: Eli Zaretskii <eliz@gnu.org>, tom@tromey.com, monnier@iro.umontreal.ca,
> >   spinuvit@gmail.com, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > > Convince Alan to do what?

> > > To adhere to the current proposal (avoid widening in 
> > > indent-line-function and font-lock-keywords, to start with).

> > I know Eli has asked to move on from the emotional to the technical
> > stuff, but that last paragraph belongs to the former, and I will deal
> > with it as such.

> And I would _really_ want to move to the technical stuff.  So...  I
> actually don't understand all the fuss about widening.  With a chunk
> of C code embedded in something that is not C, CC Mode cannot possibly
> need to look outside of the chunk, so why would you need to widen
> beyond that?

There is a style of coding called "literate programming" (I know nothing
more about it) where pieces of C code (in C Mode, obviously) alternate
with explanatory text (in text mode).  If point is at the start of a C
defun, C-M-a will have to take it to either the start of the text mode
block or the previous C defun.  Either way CC Mode needs to access
buffer pieces outside the current chunk.

> > You mentioned today, I think, that writing an MMM is hard.  Well, CC
> > Mode is hard, too.  There are 30 calls to `widen' in CC Mode and 47 to
> > `narrow-to-region'.  They are all there for a reason.  It will be
> > grinding tedious work to sort out the whys and to remove them.

> Is that the only reason for your objections?  The need to tediously go
> through a few dozen of calls to 'widen' and 'narrow-to-region', and
> take care of each one of them as appropriate?  Or are there more
> substantial problems in adapting CC Mode to non-widening MMM?

No, my main reason is philosophical.  In Lisp, we're supposed to have
the entire language available at read, evaluation, and print time.  With
this proposal, that notion becomes complicated and ugly, once you
qualify how everything but `widen' and `narrow-to-region' are available.

Using the region to do this is an abuse.  This isn't what point-min and
point-max were designed for.  Programmers are going to carry on using
`widen', and this will lead to possibly difficult to debug bugs.  I am
sure that this use of the region is going to lead to other nasty
problems, even though I can't forsee in any detail what these will be.

> > so many of the `widen' calls are in low level functions called from
> > "everywhere".

> This can be taken care of with a more-or-less trivial wrapper and a
> variable.  Right?

I suppose so, but it will degrade such a function effectively into two,
one special purpose function which can only be called from
beginning-of-defun-function (etc.) and the other one called from
everywhere else.  There will be confusion between them.

> > Yesterday, Richard Stallman suggested as an alternative to the
> > purloining of `widen' and `narrow-to-region' that a new "region" be
> > implemented somehow which would be independent of the existing region
> > and used solely by MMM super-modes.  How about exploring this
> > possibility?

> We can do that if needed, but I don't see the need yet.

Then we won't do it at all.  Once all the effort has been put into
making MMM work with the region, nobody will bother with putting the
(substantial) effort into "new region".  We'll lose the opportunity for
a possibly better solution.

> (And a proposal to do something along these lines was put forward by
> Vitalie some time ago.)

I didn't know that.  Respect, Vitalie!

> > Last February, I suggested extensions to the syntax code ("syntactic
> > islands") which would allow operations such as parse-partial-sexp to
> > work essentially without restriction in buffers with several syntax
> > tables.

> What problems does this solve?  The problem with 'widen' or some other
> problem(s)?

The idea is to render the use of the region by MMM unnecessary, though
it will need other code, too.  The need to use parse-partial-sexp in any
progmode buffer seems obvious, and having to do this in bits (as at the
moment) seems an obvious enough inconvenience to make syntactic islands
worthwhile.  To be honest, I've no exact use case for them at the
moment.  They would allow, for example, syntax-ppss to work in an entire
MMM buffer.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

>> No.  Can we put mmm-mode into elpa.git?
> I'll try emailing the original authors again. The last time I did that there
> were no replies, unfortunately.

I really think we should have such a functionality in elpa.git or
emacs.git, so the other option is a new mmm-like thingy, maybe based on
mhtml-mode or otherwise started from pure unadulterated scratch.


        Stefan

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

> Convince Alan to do what?  Do we even understand well enough what are
> the problems in CC Mode that get in the way?

Yes.

> About the only thing
> spelled out in this discussion was the need not to widen.  Personally,
> I think this is not a grave issue, just some technical problem that is
> certainly solvable (you proposed one such solution).

Indeed.

> But is that all?  You mentioned some other problems, but never
> described them.  What are they?  Are they grave enough to prevent your
> proposal from ever supporting CC Mode,

No, it's just a small matter of adjusting CC-mode here and there.
But given that the only person who understands this part of CC-mode well
enough is Alan, it's difficult for anyone but Alan to do it (I think
I could do it as well, but I know that my solution wouldn't please Alan
because it would involve switching over to syntax-propertize, so my
motivation to do it is rather low).

> even if your proposal is amended?

No need to amend anything.

> How do you address the issues which prog-indentation-context did
> (e.g., if the embedded chunk of code is incomplete, and perhaps even
> syntactically invalid)?

We can easily provide a similar functionality with Dmitry's design.
Dmitry hasn't bothered doing that, because so far it's never been used
(so it's not even clear whether it's ever needed nor whether it would
provide the needed info).

> cause it to end up like all the previous ones: either committed to
> Emacs and basically unused, or, worse, collecting dust in long
> forgotten patches or branches?

His proposal is really really small, and matches what is already done
in mmm-mode and mhtml-mode, so it's actually the solution that's least
likely to collect dust, because it's already in use.

> Emacs 26 has enough new features to justify a major release, so I see
> no reason to press so hard to get this feature into it as well.

I think the failure here is to describe clearly the proposal.
The core of the proposal, AFAIK, is the following:

    state that indent-line-function can rely on the current narrowing to
    find all the relevant text, because the caller has already widened
    when necessary for that.

To make it true, we need a few changes in the code, mostly we need to
tweak indent-according-to-mode so that it widens before calling
indent-line-function.  This change should be harmless because
even if the user has narrowed the buffer, the indentation should always
be done depending on the whole buffer's content anyway.

>> It's not well supported, and it's incompatible with the code we are 
>> discussing. There's no point in having it present in Emacs 26, and then 
>> removing it in Emacs 27.
> That code is in Emacs for more than 2 years.

It's been in emacs.git's master, yes, but not in any released version of Emacs.

> It was admitted with Stefan's full support, and at least ANTLR needs
> it in conjunction with Python.

These can very easily be adjusted to rely on the new convention.


        Stefan

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

> You mentioned today, I think, that writing an MMM is hard.  Well, CC
> Mode is hard, too.  There are 30 calls to `widen' in CC Mode and 47 to
> `narrow-to-region'.

The calls to narrow-to-region don't need to be investigated.  Only those
to `widen'.


        Stefan

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Alan Mackenzie]

Hello, Stefan.

On Sun, Dec 03, 2017 at 13:40:22 -0500, Stefan Monnier wrote:

[ .... ]

> > I don't rate restrictions on the use of Emacs primitives to be minor.

> I think there's a fundamental misunderstanding abut Dmitry's suggestion
> here: his proposal does not restrict the use of any primitive anywhere.

You say there's no restriction happening, then go on to detail the
restriction that is to be put in place.  ;-)

> The only thing it does is to make the calling convention of
> indent-line-function a bit more precise: instead of not saying anything
> about the kind of narrowing that can be in place when that is called, it
> specifies that the caller should arrange the narrowing to expose "all"
> the code, and so the callee doesn't need to widen (which is useful since
> in an MMM context, the sub-mode usually doesn't know how far it should
> widen).

Which is why I proposed the "syntactic islands", which would allow code
to do parse-partial-sexp (or even syntax-ppss).  Typically the code does
need to know about "distant" chunks.  For example if point is after an
embedded AWK Mode script in shell-script-mode, C-M-a will need access to
the buffer part before the AWK Mode interlude.

> The indent-line-functions can still narrow/widen all it wants.
> And the callers of indent-line-function also can still widen/narrow all
> they want.  Just as is the case now, depending on how they narrow/widen
> the result may be incorrect.  And the novelty here is to provide
> a convention that, if followed, should make it more likely that the
> result is correct, without the caller and the callee being aware of each
> other, because they only care about the shared convention.

> > If those common solution are adequate, yes.  But it is also critical to
> > have the freedom to depart from things like syntax-propertize when they
> > are inadequate, or too awkward.  syntax-propertize is inadequate for CC
> > Mode:

> > E.g., in C++ Mode, with

> >     foo (a < b, c > d);

> > we have a constructor foo with one parameter d, of a templated type.  The
> > characters "<" and ">" have syntax-table text properties giving them
> > paren syntaxes and making them match each other.

> > Now, mark the characters ", c > d" and kill them with C-w.  What we are
> > left with is a function call with an integer argument a < b:

> >     foo (a < b);

> > .  Something, somewhere has to remove the syntax-table property from
> > "<", since it is no longer a template delimiter.

> > The syntax-propertize mechanism won't do this, since the place to
> > de-propertise is before BEG.

> syntax-propertize is no silver bullet, indeed, so it won't do everything
> magically for you.  But, it can handle the above situation just fine.

I don't see that.

> In your very specific above example, it will actually be handled
> magically simply by virtue of syntax-propertize-wholelines being
> included in the default value of syntax-propertize-extend-region-functions.

> Now, of course, if you tweak your example so that the < and the > are
> not on the same line any more, that magic won't work.  But that doesn't
> mean you can't make syntax-propertize handle it right.  You could for
> example solve it with:

>     (add-hook 'syntax-propertize-extend-region-functions
>               #'syntax-propertize-multiline nil t)

All constructs in CC Mode languages are "multiline".

> in your major mode function and

>     (put-text-property position-of-left-< position-of-right->
>                        'syntax-multiline t)

> when you find (and put the syntax-table property on) the <...> pair.

Why not just put that property on the entire buffer?  Or, if
syntax-multiline is intended to flag up just one construct, it won't cope
well with nested (or overlapping) constructs.

> Of course, it's not the only possible way to do it.  It's just the way
> I solved similar problems in perl-mode and sh-mode.

How is that going to help when a closing template delimiter is deleted?

> If you're interested in making CC-mode work reliably with
> syntax-propertize I'd be *thrilled* to help you do that.

> By design, there should be basically nothing syntax-propertize can't handle
> (modulo bugs and within the limits of what we can do with syntax-tables
> and syntax-table properties, obviously).

One problem is the vagueness in the documentation of
syntax-propertize-extend-region-functions.  Are these functions called in
before-change-functions or a-c-functions?  Obviously, in my case above,
they would be needed in b-c-f, otherwise they wouldn't notice the ">"
being killed.  But they need to be called in a-c-f too, in case the
change is adding "> >", when the code needs to search back to the earlier
two "<"s, which now have become "dirty" when they weren't previously.

> I have no doubt that the way CC-mode works right now will occasionally
> be more efficient than the way syntax-propertize would do it.  But I
> also have no doubt that the way syntax-propertize would do it will
> occasionally by more efficient than the way CC-mode does it right now.

syntax-propertize splats all syntax-table properties after a change
point, if I understand correctly.  That means they must at some point
soon be regenerated[*].  This will be expensive in a large buffer.

[*] imagine moving point forward to just before a "<" which (along with
its matching ">") needs to have paren s-t. properties to allow
paren-matching to happen.  I can't see what is putting those properties
on it, given the ">" could be an arbitrary distance forward.

> I strongly doubt that the difference in performance behavior will ever
> matter, tho.

Several differences in performance that "hardly matter" can add up to
something substantial.  I recently celebrated a 3.9% increase in
fontification performance after deleting some old duplicate code.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

>> > I don't rate restrictions on the use of Emacs primitives to be minor.
>> I think there's a fundamental misunderstanding abut Dmitry's suggestion
>> here: his proposal does not restrict the use of any primitive anywhere.
> You say there's no restriction happening, then go on to detail the
> restriction that is to be put in place.  ;-)

I have no idea what you're talking about.  Where did you see a restriction?
There are only guidelines, no restrictions.

> Typically the code does need to know about "distant" chunks.

The years of experience with mmm-mode, as well as the more recent
experience with mhtml-mode disagree with this.

> For example if point is after an embedded AWK Mode script in
> shell-script-mode, C-M-a will need access to the buffer part before
> the AWK Mode interlude.

This already works and won't be affected by the proposal.

>> (add-hook 'syntax-propertize-extend-region-functions
>> #'syntax-propertize-multiline nil t)
> All constructs in CC Mode languages are "multiline".

So what?

>> when you find (and put the syntax-table property on) the <...> pair.
> Why not just put that property on the entire buffer?  Or, if
> syntax-multiline is intended to flag up just one construct, it won't cope
> well with nested (or overlapping) constructs.

As I said, you don't have to use that.  You can write your own
syntax-propertize-extend-region-function any which way you like.

>> Of course, it's not the only possible way to do it.  It's just the way
>> I solved similar problems in perl-mode and sh-mode.
> How is that going to help when a closing template delimiter is deleted?

It will cause the re-propertization to be done from the matching
opening delimiter (at least, if you've set things up correctly, of
course: syntax-propertize itself has no idea about those delimiters).

> One problem is the vagueness in the documentation of
> syntax-propertize-extend-region-functions.  Are these functions called in
> before-change-functions or a-c-functions?  Obviously, in my case above,
> they would be needed in b-c-f, otherwise they wouldn't notice the ">"
> being killed.

The text between < and > is marked, what when BEG falls in the middle, 
syntax-propertize-extend-region-functions will know without having to
have access to the ">" (so it doesn't matter if it's already been
deleted or not).

> But they need to be called in a-c-f too, in case the
> change is adding "> >", when the code needs to search back to the earlier
> two "<"s, which now have become "dirty" when they weren't previously.

This kind of complexity disappears in syntax-propertize's approach.

> syntax-propertize splats all syntax-table properties after a change
> point, if I understand correctly.  That means they must at some point
> soon be regenerated[*].  This will be expensive in a large buffer.

This *can* be expensive, but they're only regenerated lazily, so
typically it's not a problem.  We have many years of experience in many
different major modes to say that it's likely not to be a problem.

> Several differences in performance that "hardly matter" can add up to
> something substantial.

"can", indeed.  My experience makes me confident it won't happen.


        Stefan

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/3/17 3:38 AM, Eli Zaretskii wrote:

>> Can we put mmm-mode into elpa.git?
> 
> Even if it's possible, it makes absolutely no sense to me to have such
> a central feature on ELPA and not in the core.

Maybe if we grow the "bundled packages from ELPA" feature.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/3/17 2:54 PM, Alan Mackenzie wrote:

> Details do matter.  I posted a full proposal, Subject: "Syntax
> ambiguities in narrowed buffers and multiple major modes: a proposed
> solution." here in emacs-devel on Sat, 25 Feb 2017 13:53:56 +0000.
> 
> I've changed my mind about some of the details, partly in response to
> what people said, partly because I see things now I didn't see then.

Thanks. I've read it, and you've seen my comments.

>> I'm not convinced "several syntax tables" would be enough, though.
> 
> OK, let's make that "many syntax tables".  :-)

TBH, it's hard for me to tell with confidence either way. But here's the 
kind of syntax that Ruby allows:

a = %Q( #{ %q{abc} + %q{ "#{"abc #{ %q[ abc ] }" }" } } )

These are all nested literals (with the exception of #{}, which is 
interpolation). Thus exiting from a literal requires the knowledge of 
all openers that have been used before it, and a pre-defined set of 
syntax table (if I understood you correctly) won't do.

>> The major modes will have to change to allow for this support, that's
>> for sure. You can't expect no code changes.
> 
> Maybe, maybe not.  Super-modes would definitely have to change to use
> it.  Any changes in major modes should be minor.  (I don't rate
> restrictions on the use of Emacs primitives to be minor.)

Major modes will have to work inside arbitrary restrictions one way or 
another. That's the crucial part, and it will require changes.

>> The current idea is "let's do the minimum coherent step forward, so that
>> somebody can continue improving things later". A step that's easy to
>> use, and also easy to undo later, if we find something else that's much
>> better.
> 
> OK.  With such a tricky subject matter, it would surprising if we
> couldn't find improvements for a long time.

Since it's tricky, I would bet in the opposite direction. But there's no 
way to predict.

>> Coming up with it was not easy at all, BTW.
> 
> Yes, I can see that.  Well done!

If you are not being sarcastic, thanks! Could you please stop opposing it?

>> It's simply description of good engineering. You can try to criticize
>> the details of syntax-propertize, but the idea of reusing common
>> solutions is more than sound.
> 
> If those common solution are adequate, yes.  But it is also critical to
> have the freedom to depart from things like syntax-propertize when they
> are inadequate, or too awkward.  syntax-propertize is inadequate for CC
> Mode:

Sorry, I'd rather you put it in a separate discussion. But, like Stefan 
described, this example is very much solvable with syntax-propertize 
(which I could have told you independently, too).

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/3/17 9:23 PM, Stefan Monnier wrote:

> I really think we should have such a functionality in elpa.git or
> emacs.git, so the other option is a new mmm-like thingy, maybe based on
> mhtml-mode or otherwise started from pure unadulterated scratch.

Agreed. But if extraction from mhtml-mode is expected from me, sorry, 
it's going to the bottom of the pile for now.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

Hi Stefan,

On 12/3/17 9:52 PM, Stefan Monnier wrote:

> No, it's just a small matter of adjusting CC-mode here and there.
> But given that the only person who understands this part of CC-mode well
> enough is Alan, it's difficult for anyone but Alan to do it (I think
> I could do it as well, but I know that my solution wouldn't please Alan
> because it would involve switching over to syntax-propertize, so my
> motivation to do it is rather low).

Maybe you understand the issue better than me. Or maybe it's the 
conversion to syntax-propertize that would do away with the extra caches 
that IIRC were giving me troubles.

>> How do you address the issues which prog-indentation-context did
>> (e.g., if the embedded chunk of code is incomplete, and perhaps even
>> syntactically invalid)?
> 
> We can easily provide a similar functionality with Dmitry's design.
> Dmitry hasn't bothered doing that, because so far it's never been used
> (so it's not even clear whether it's ever needed nor whether it would
> provide the needed info).

To be clear, I'm not sure what you mean. But then, the problem/question 
isn't well-formulated here.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/3/17 6:59 PM, Alan Mackenzie wrote:

> Please read my signature, sometime.

I sincerely apologize. Fly out to Germany, of course.

> Maybe I missed it, but I don't recall anybody posting the suggestion
> "Hey, we're considering writing an MMM mode which will mean nobody else
> can use narrowing and widening, what do people think?".  If somebody had
> posted that, there's a good chance that discussions could have sorted
> out the technical stuff without provoking bad feelings.

Erm, mmm-mode is 17 years old, originally written by a graduate student 
to help with literate programming (AFAIK). Did you mean they should have 
asked back then? And in the intervening years, several other packages 
took a similar approach.

And if you mean that I should have asked specifically, I did bring up it 
in a discussion... a year ago or so. And no realistic and complete 
counter-proposal came up, as far as I'm concerned.

> Right now, I am being painted as the bad person for objecting to being
> restricted to a subset of Emacs Lisp for CC Mode.  Forgive me for
> feeling a bit peeved, but despite being in Emacs development for around
> 15 years, nobody asked me beforehand what I thought of this.  It is yet
> one more thing that is presented as a done thing and foisted on Emacs
> developers without them having any say in the matter.

You've never worked on a mixed-mode package, though. So yes, I 
personally have a reason to assume that I have a better knowledge of the 
problem area.

But I never said that CC Mode must support it.

> You mentioned today, I think, that writing an MMM is hard.  Well, CC
> Mode is hard, too.  There are 30 calls to `widen' in CC Mode and 47 to
> `narrow-to-region'.  They are all there for a reason.  It will be
> grinding tedious work to sort out the whys and to remove them.

Still, that sounds manageable. And there's no hurry at all, as far as 
I'm concerned.

> You
> categorise the injunction against widening as "only applying to
> indent-line-function and "font-lock-keywords" (which isn't a function)".

I think you know what I meant anyway.

> Yesterday, Richard Stallman suggested as an alternative to the
> purloining of `widen' and `narrow-to-region' that a new "region" be
> implemented somehow which would be independent of the existing region
> and used solely by MMM super-modes.  How about exploring this
> possibility?

Vitalie proposed something like that years ago, and we asked for help 
with the C code. Nobody turned up, including yourself (and I asked 
directly).

Anyway, I don't see how it would be qualitatively better. The problem of 
being able to function in a restricted area of a buffer will still be there.

> Last February, I suggested extensions to the syntax code ("syntactic
> islands") which would allow operations such as parse-partial-sexp to
> work essentially without restriction in buffers with several syntax
> tables.  How about exploring this possibility?

Please go ahead and explore. I've done my part with my proposal (which 
was intentionally small and thus required small changes everywhere but 
CC Mode).

> Believe it or not, I am in favour of CC Mode working in an MMM mode.

You just don't want the protocol we are proposing. Did you participate 
in the prog-indentation-context discussion when it was proposed, by the way?

>> I have a rough understanding of the issue, but since I haven't reached a
>> working state, I don't know how many pitfalls there are left.
> 
> Neither do I.  But RMS's "new region" and my "syntax islands" may be a
> more satisfactory way of resolving them.

May be, or may not. We'd need to see the code.

By the way, again, you can add a "new region" function exclusively to CC 
Mode, and use it in the indentation function. It's trivial to implement, 
and will look awfully like the shim I proposed.

>> I imagine the process itself might be trickier than expected. Various
>> primitives use caches that save context information. What is such
>> primitive to do if the cache contains "beginning of nesting" outside of
>> the current restriction, and the logic of said primitive says "go to the
>> beginning of the current function and do such and such"? The answer
>> isn't obvious to me.
> 
> Again, looking at RMS's suggested "new region" might help, here.

See above.

But no, the problem I imagined (not 100% sure if it's real) won't be 
solved by "hard narrowing".

>> Consolidating the 'widen' calls is simply good engineering, even aside
>> from making life easier for MMM packages: it's much better to do that in
>> several well-defined places, instead of having every helper function do
>> it as well, like python-mode CC Mode do.
> 
> No, it isn't "simply good engineering" at all.  It's a different
> approach with its pros and cons, just as widening at the immediate place
> of need is.  And "consolidating" is probably the wrong word.  The number
> of `widen's will probably grow, since every conceivable entry point
> which could possibly call the pertinent primitve needs a `widen'.

No more than there are interactive functions. A lot of them have to do 
that anyway already. And then, you could also have helpers that aren't 
used in indentation code.

There are several ways to skin that cat.

[SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

>> No, it's just a small matter of adjusting CC-mode here and there.
>> But given that the only person who understands this part of CC-mode well
>> enough is Alan, it's difficult for anyone but Alan to do it (I think
>> I could do it as well, but I know that my solution wouldn't please Alan
>> because it would involve switching over to syntax-propertize, so my
>> motivation to do it is rather low).
> Maybe you understand the issue better than me. Or maybe it's the conversion
> to syntax-propertize that would do away with the extra caches that IIRC were
> giving me troubles.

Not really, no: adjusting the current code so that those caches work
right in mmm-mode can also be done and can't be terribly hard either
(e.g. flush the caches whenever the narrowing has changed, or index the
caches with the current narrowing.  Basically the same solutions we've
discussed to get syntax-ppss to coexist reliably with narrowing).
Of course, it would take you or me a lot of time trying to understand
how those things currently work, then choose a solution, then
implement it, but it's still just a small matter of adjusting CC-mode
here and there.  Nothing fundamentally hard.

>>> How do you address the issues which prog-indentation-context did
>>> (e.g., if the embedded chunk of code is incomplete, and perhaps even
>>> syntactically invalid)?
>> We can easily provide a similar functionality with Dmitry's design.
>> Dmitry hasn't bothered doing that, because so far it's never been used
>> (so it's not even clear whether it's ever needed nor whether it would
>> provide the needed info).
> To be clear, I'm not sure what you mean. But then, the problem/question
> isn't well-formulated here.

I think he was referring to the PREVIOUS-CHUNKS part of
prog-indentation-context.  The MMM framework could easily provide some
similar facility (which would take care of changing the narrowing when
needed).  But I think we had better see some concrete uses first before
deciding upon a design for that.


        Stefan

syntax-propertize and CC-mode (was: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls)

[Stefan Monnier]

> The text between < and > is marked, what when BEG falls in the middle, 
                                      ^^^^
                                    so that

> syntax-propertize-extend-region-functions will know without having to
> have access to the ">" (so it doesn't matter if it's already been
> deleted or not).

[...]

> "can", indeed.  My experience makes me confident it won't happen.

This said, this side-discussion about syntax-propertize is irrelevant to
the MMM issue: CC-mode doesn't need to switch to syntax-propertize to
cooperate with mmm-mode (it's just the way I would make it work because
I dono't understand the workings of the current code, but if you make
the change yourself, then you can adapt the current code, with which
you're familiar).


        Stefan

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> Cc: acm@muc.de, tom@tromey.com, spinuvit@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 3 Dec 2017 23:43:48 +0000
> 
> On 12/3/17 3:38 AM, Eli Zaretskii wrote:
> 
> >> Can we put mmm-mode into elpa.git?
> > 
> > Even if it's possible, it makes absolutely no sense to me to have such
> > a central feature on ELPA and not in the core.
> 
> Maybe if we grow the "bundled packages from ELPA" feature.

That would be a different Emacs and a different discussion.

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/4/17 2:24 AM, Stefan Monnier wrote:

> Not really, no: adjusting the current code so that those caches work
> right in mmm-mode can also be done and can't be terribly hard either
> (e.g. flush the caches whenever the narrowing has changed, or index the
> caches with the current narrowing.  Basically the same solutions we've
> discussed to get syntax-ppss to coexist reliably with narrowing).
> Of course, it would take you or me a lot of time trying to understand
> how those things currently work, then choose a solution, then
> implement it, but it's still just a small matter of adjusting CC-mode
> here and there.  Nothing fundamentally hard.

Makes sense. Yes, one of the problems is that an MMM doesn't know about 
all the caches.

> I think he was referring to the PREVIOUS-CHUNKS part of
> prog-indentation-context.  The MMM framework could easily provide some
> similar facility (which would take care of changing the narrowing when
> needed).  But I think we had better see some concrete uses first before
> deciding upon a design for that.

Ah, right then. I agree.

There are still no examples of using PREVIOUS-CHUNKS.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/4/17 3:38 AM, Eli Zaretskii wrote:

>> Maybe if we grow the "bundled packages from ELPA" feature.
> 
> That would be a different Emacs and a different discussion.

Point is, putting mmm-mode in ELPA or not is the developer's choice.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> Cc: acm@muc.de, tom@tromey.com, monnier@iro.umontreal.ca, spinuvit@gmail.com,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 3 Dec 2017 19:43:31 +0000
> 
>         This feature is *not* targeted at Emacs development.
> 
>     It is for me.
> 
> You are not working on it. You've expressed no interest in this problem until now. Maybe delegate the basic requirements, at least, to the people who did?

Your work on this is highly appreciated.  But I have certain
responsibilities, and I'm trying to do my job here.

I'm also your fellow developer of the same project and its active
user.  I think it means my opinions should be of some importance, not
something to be ignored solely because I'm not coding this.  Just like
I would never dream of ignoring your opinions about features I'm
implementing.

As for expressing interest: you are mistaken when you judge that by my
silence -- it only means that I had nothing significant to say about
the issue, until I did.

> [C-like major modes] can be catered to, *as soon as* the authors of each individual major mode make the effort to support the feature.

Agreed.  What I'm trying to establish is whether the design supports
those major modes, or does it ignore them.  The latter I would like to
avoid as much as possible.

> Eli, we already have a feature in Emacs (prog-indentation-context) that does not adhere to your (unreasonably high) requirements.

Emacs development is not only about not losing existing features.  It
is mainly about gaining new important features.  Here you are adding a
feature that I think is important enough for us to try make it support
C-like languages.  I don't think this requirement is unreasonably
high, as those languages are still pretty much the mainstay of the
industry, and definitely used by many Emacs users.

> Imagine if the chunk contains only this:
> 
>    <
> 
> And, to fontify it, CC Mode, tried to find out what syntactic element it is, and to do that, calls some function that uses a specialized cache that's contained in a variable that mmm-mode knows nothing about.
> 
> And that cache point to a position outside of the current restriction. Confusion ensues. I've seen errors originating from that (I think), but can't recall the exact sequence of calls.

Thanks, so the caches will have to be sensitive to restrictions as
they are sensitive to deletion of text.

> To fontify (or indent) a chunk, mmm-mode narrows to its bounds, and calls the corresponding indent-line-function (or applies font-lock-keywords). If either of those calls (widen), that causes problems.
> 
> So we want to document a convention that they don't call widen. That's really it.

But given your example with CC Mode's caches, that's not all of it, is
it?  You also need the mode to adapt any such caches to changes in
buffer restrictions, right?

>             How do you address the issues which prog-indentation-context did
>             (e.g., if the embedded chunk of code is incomplete, and perhaps even
>             syntactically invalid)?
> 
>         prog-indentation-context never addressed that issue, and we don't
>         either.
> 
>     I think it does, at least to some extent.
> 
> Not any more than scratch/widen-less.

Maybe I'm missing something, but what's the equivalent of PREV-CHUNK?

> We keep 'prog-first-column' from that proposal

But it was a function, and you made it a variable.  This will get in
the way of compatibility, and also potentially will make future
extensions harder.  Why not keep it a function?

> and instead of allowing MMM to indicate the chunk bounds through prog-indentation-context, we allow MMM to apply narrowing directly, and that modes honor it.

This simplification also took away some of the features, like the
ability to nest restrictions.  I wonder why did you discard that.
It's existing functionality, pretty lightweight, which was in Emacs
for some time, and is reasonably well documented.  And it already
satisfied your requirement of not allowing sub-modes to widen.  Why
not leave it alone?  What am I missing?

>         Transitioning from prog-indentation-context to the new approach is very
>         easy.
> 
>     That's what I thought.  And that's why we should do that
>     simultaneously with landing the new approach.  There's no reason to do
>     that earlier.
> 
> Why confuse the language modes authors?

If they should be confused (I don't think so), they are already
confused, as we've been having this for the last 2 years.

>     We cannot know that.  Once the code is that long with us, we cannot
>     throw it away without a equivalent replacement.  And I see no reason
>     to do that now, since the fact that this code will exist in Emacs 26
>     doesn't hurt anyone, especially since the replacement will be easy, as
>     you say.
> 
> Well... you have a point there. The result will be pretty ugly, though. Adding an API in one version, and removing it in another.

We will have to discuss the "removal" part.  An alternative is to
provide compatibility shims, or even leave (some or all of) the
existing API intact.

> They are incompatible with each other, so it's not like there can be a smooth transition. 

I don't see the incompatibilities.  Basically, you replaced prog-widen
with widen, made prog-first-column a variable instead of a function,
and tossed prog-indentation-context.  We could instead keep the
existing API with minimal changes: prog-widen calls widen internally,
and we could allow direct calls to widen as well.

IOW, the transition could be much smoother than it will be if we
actually remove that stuff, because I don't think there's any
incompatibility here which would disallow direct calls to 'widen'
and/or leaving prog-indentation-context at its default nil value.  Or
maybe I'm missing something again.  But if I'm right, and if we can
make the necessary changes limited only to the documentation, then it
could well make it into Emacs 26.  So I think it is worth our while
to make some effort in that direction, if at all possible.

Thanks.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Alan Mackenzie]

Hello, Dmitry.

On Mon, Dec 04, 2017 at 00:37:51 +0000, Dmitry Gutov wrote:
> On 12/3/17 6:59 PM, Alan Mackenzie wrote:

[ .... ]

> You've never worked on a mixed-mode package, though. So yes, I 
> personally have a reason to assume that I have a better knowledge of the 
> problem area.

OK.

> But I never said that CC Mode must support it.

Any chance you could give me some basic details of how to get into this
MMM mode?  Like which git branch (? scratch/widen-less ?) can I find the
code in, and where is the best documentation to get started.

> > You mentioned today, I think, that writing an MMM is hard.  Well, CC
> > Mode is hard, too.  There are 30 calls to `widen' in CC Mode and 47 to
> > `narrow-to-region'.  They are all there for a reason.  It will be
> > grinding tedious work to sort out the whys and to remove them.

> Still, that sounds manageable. And there's no hurry at all, as far as 
> I'm concerned.

That's a good job!  I've no idea how long it will take to make MMM and
CC Mode work with eachother.

> Anyway, I don't see how it [ a new region, distinct from THE region ]
> would be qualitatively better. The problem of being able to function
> in a restricted area of a buffer will still be there.

I don't see that at the moment either, but perhaps that will become
evident.

> > Last February, I suggested extensions to the syntax code ("syntactic
> > islands") which would allow operations such as parse-partial-sexp to
> > work essentially without restriction in buffers with several syntax
> > tables.  How about exploring this possibility?

> Please go ahead and explore. I've done my part with my proposal (which 
> was intentionally small and thus required small changes everywhere but 
> CC Mode).

> > Believe it or not, I am in favour of CC Mode working in an MMM mode.

> You just don't want the protocol we are proposing. Did you participate 
> in the prog-indentation-context discussion when it was proposed, by the way?

Not much, if at all.  I found those discussions to be lacking the
context I would have needed to understand them.

> >> I have a rough understanding of the issue, but since I haven't reached a
> >> working state, I don't know how many pitfalls there are left.

> > Neither do I.  But RMS's "new region" and my "syntax islands" may be a
> > more satisfactory way of resolving them.

> May be, or may not. We'd need to see the code.

That's fair.  But the "syntax islands" proposal would be a lot of work,
which I don't want to commit to before I see some sort of undertaking to
take it seriously.

[ .... ]

> There are several ways to skin that cat.

That's the second cat in as many days, here!

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> Date: Sun, 3 Dec 2017 21:20:40 +0000
> Cc: tom@tromey.com, emacs-devel@gnu.org, monnier@iro.umontreal.ca,
>   spinuvit@gmail.com, dgutov@yandex.ru
> From: Alan Mackenzie <acm@muc.de>
> 
> > actually don't understand all the fuss about widening.  With a chunk
> > of C code embedded in something that is not C, CC Mode cannot possibly
> > need to look outside of the chunk, so why would you need to widen
> > beyond that?
> 
> There is a style of coding called "literate programming" (I know nothing
> more about it) where pieces of C code (in C Mode, obviously) alternate
> with explanatory text (in text mode).  If point is at the start of a C
> defun, C-M-a will have to take it to either the start of the text mode
> block or the previous C defun.  Either way CC Mode needs to access
> buffer pieces outside the current chunk.

Does MMM support Literate Programming?  If not, this is a separate
project, and we don't have to solve its problems now.  I would like
first to reach a more limited goal: allow editing C/C++ snippets
inside Yacc grammars, Awk snippets inside shell scripts, and other
similar use cases.

Can we do that first, and worry about higher-hanging fruit later?

> No, my main reason is philosophical.

You don't need to accept the philosophy just yet.  It is possible that
we will find in the future a solution that is more easily generalized,
perhaps even more elegant, when we consider the more complex features
and tasks.  Right now, the issue at hand is whether CC Mode will
support MMM.  It sounds wrong to me to have MMM support that excludes
CC Mode.  And the solution doesn't sound hard, just a bit tedious, and
at least some of it (replacing the calls to 'widen') quite mechanical.

> In Lisp, we're supposed to have
> the entire language available at read, evaluation, and print time.  With
> this proposal, that notion becomes complicated and ugly, once you
> qualify how everything but `widen' and `narrow-to-region' are available.

I think you exaggerate.  We have already some features whose correct
use requires some discipline, and breaks if one doesn't exercise such
discipline.  It's less than ideal, but sometimes the ideal is not very
practical.

In any case, I'm not asking you to like this restriction, just help
bring CC Mode into the MMM framework, because I think without that the
feature will be woefully incomplete.

> > This can be taken care of with a more-or-less trivial wrapper and a
> > variable.  Right?
> 
> I suppose so, but it will degrade such a function effectively into two,
> one special purpose function which can only be called from
> beginning-of-defun-function (etc.) and the other one called from
> everywhere else.  There will be confusion between them.

I don't think I understand the need for two functions.  Can you
elaborate, please?

> > > Yesterday, Richard Stallman suggested as an alternative to the
> > > purloining of `widen' and `narrow-to-region' that a new "region" be
> > > implemented somehow which would be independent of the existing region
> > > and used solely by MMM super-modes.  How about exploring this
> > > possibility?
> 
> > We can do that if needed, but I don't see the need yet.
> 
> Then we won't do it at all.

We will if we get to the point where the current proposal is too
restrictive, or cannot support some features we want.  Compare this
with the line numbers: linum.el exists, and we still added a native
implementation in the display engine.

I suggest to take a good look at what it will take to adapt CC Mode to
MMM.  If it turns out that there are fundamental obstacles to that
which we currently don't envision, at least we will have a list of
concrete problems to solve, and that will facilitate designing a
solution.

TIA

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sun, 03 Dec 2017 16:52:25 -0500
> 
> > How do you address the issues which prog-indentation-context did
> > (e.g., if the embedded chunk of code is incomplete, and perhaps even
> > syntactically invalid)?
> 
> We can easily provide a similar functionality with Dmitry's design.
> Dmitry hasn't bothered doing that, because so far it's never been used
> (so it's not even clear whether it's ever needed nor whether it would
> provide the needed info).

What I don't understand is why we are asked to remove
prog-indentation-context.  I don't see how its existence contradicts
the possibility of calling 'widen' directly or leaving the variable at
its default nil value.  What am I missing?

> I think the failure here is to describe clearly the proposal.
> The core of the proposal, AFAIK, is the following:
> 
>     state that indent-line-function can rely on the current narrowing to
>     find all the relevant text, because the caller has already widened
>     when necessary for that.
> 
> To make it true, we need a few changes in the code, mostly we need to
> tweak indent-according-to-mode so that it widens before calling
> indent-line-function.

Why does it need to widen, though?  If the mode's indent-line-function
is not supposed to call 'widen', why do we do that in
indent-according-to-mode?  I don't think this has been explained
anywhere.  When you widen, the text that comes into the view will be
entirely alien syntactically to the sub-mode that is trying to indent,
right?

The existing code calls prog-widen in those cases, but that only
widens to include the current chunk (or whatever else the writer of
prog-indentation-context had in mind), which had this issue under
control, but the proposed changes toss that.

> This change should be harmless because even if the user has narrowed
> the buffer, the indentation should always be done depending on the
> whole buffer's content anyway.

But that's an incompatible change, isn't it?  And I don't see what it
has to do with MMM, where any restriction prior to the call to the
indentation function should be honored.

> > That code is in Emacs for more than 2 years.

> It's been in emacs.git's master, yes, but not in any released version of Emacs.

That matters very little these days, as many people track the
development branches and adapt their packages to the new features
before they are released.

> > It was admitted with Stefan's full support, and at least ANTLR needs
> > it in conjunction with Python.
> 
> These can very easily be adjusted to rely on the new convention.

Why not just leave them alone?  IOW, where do they get in the way of
MMM calling 'widen' directly?

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sun, 03 Dec 2017 21:24:45 -0500
> Cc: emacs-devel@gnu.org
> 
> I think he was referring to the PREVIOUS-CHUNKS part of
> prog-indentation-context.  The MMM framework could easily provide some
> similar facility (which would take care of changing the narrowing when
> needed).  But I think we had better see some concrete uses first before
> deciding upon a design for that.

The code which supports PREV-CHUNK is already in Emacs, so the
question IMO is "why remove it"?  If we want to try to get this into
Emacs 26, we should strive to make the code changes minimal, ideally
zero.  Once all we are left with are documentation changes, the
feature can land on emacs-26 right now.

So I think it's worth our while to take a very good look at the
proposed changes and think hard whether we absolutely must make those
changes, and why.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Alan Mackenzie]

Hello, Eli.

On Mon, Dec 04, 2017 at 18:10:07 +0200, Eli Zaretskii wrote:
> > Date: Sun, 3 Dec 2017 21:20:40 +0000
> > Cc: tom@tromey.com, emacs-devel@gnu.org, monnier@iro.umontreal.ca,
> >   spinuvit@gmail.com, dgutov@yandex.ru
> > From: Alan Mackenzie <acm@muc.de>

> > > actually don't understand all the fuss about widening.  With a chunk
> > > of C code embedded in something that is not C, CC Mode cannot possibly
> > > need to look outside of the chunk, so why would you need to widen
> > > beyond that?

> > There is a style of coding called "literate programming" (I know nothing
> > more about it) where pieces of C code (in C Mode, obviously) alternate
> > with explanatory text (in text mode).  If point is at the start of a C
> > defun, C-M-a will have to take it to either the start of the text mode
> > block or the previous C defun.  Either way CC Mode needs to access
> > buffer pieces outside the current chunk.

> Does MMM support Literate Programming?  If not, this is a separate
> project, and we don't have to solve its problems now.  I would like
> first to reach a more limited goal: allow editing C/C++ snippets
> inside Yacc grammars, Awk snippets inside shell scripts, and other
> similar use cases.

OK.

> Can we do that first, and worry about higher-hanging fruit later?

OK2.

> > No, my main reason is philosophical.

> You don't need to accept the philosophy just yet.  It is possible that
> we will find in the future a solution that is more easily generalized,
> perhaps even more elegant, when we consider the more complex features
> and tasks.  Right now, the issue at hand is whether CC Mode will
> support MMM.  It sounds wrong to me to have MMM support that excludes
> CC Mode.  And the solution doesn't sound hard, just a bit tedious, and
> at least some of it (replacing the calls to 'widen') quite mechanical.

> > In Lisp, we're supposed to have
> > the entire language available at read, evaluation, and print time.  With
> > this proposal, that notion becomes complicated and ugly, once you
> > qualify how everything but `widen' and `narrow-to-region' are available.

> I think you exaggerate.  We have already some features whose correct
> use requires some discipline, and breaks if one doesn't exercise such
> discipline.  It's less than ideal, but sometimes the ideal is not very
> practical.

I've been worried about Emacs losing its generality - that major modes
can increasingly only be written in one restricted way, that `widen' can
only be used by supermodes, etc.  Maybe I've been seeing something which
isn't really there.

> In any case, I'm not asking you to like this restriction, just help
> bring CC Mode into the MMM framework, because I think without that the
> feature will be woefully incomplete.

I will do what I can.  Could you tell me where to find MMM?  ;-)

[ .... ]

> > > > Yesterday, Richard Stallman suggested as an alternative to the
> > > > purloining of `widen' and `narrow-to-region' that a new "region" be
> > > > implemented somehow which would be independent of the existing region
> > > > and used solely by MMM super-modes.  How about exploring this
> > > > possibility?

> > > We can do that if needed, but I don't see the need yet.

> > Then we won't do it at all.

> We will if we get to the point where the current proposal is too
> restrictive, or cannot support some features we want.  Compare this
> with the line numbers: linum.el exists, and we still added a native
> implementation in the display engine.

> I suggest to take a good look at what it will take to adapt CC Mode to
> MMM.  If it turns out that there are fundamental obstacles to that
> which we currently don't envision, at least we will have a list of
> concrete problems to solve, and that will facilitate designing a
> solution.

Sorry for being so uncooperative over the last couple of days.  I agree
that CC Mode must work together with MMM, and I will do everything I can
to facilitate that.

> TIA

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

>> Eli, we already have a feature in Emacs (prog-indentation-context) that
>> does not adhere to your (unreasonably high) requirements.
> Emacs development is not only about not losing existing features.  It
> is mainly about gaining new important features.  Here you are adding a
> feature that I think is important enough for us to try make it support
> C-like languages.

His proposal is reworking prog-indentation-context.
CC-mode doesn't support prog-indentation-context, so it's no surprise
that it doesn't support this replacement= and I don't see why suddenly
we should first get CC-mode to support it before the replacement can
be installed.

> I don't think this requirement is unreasonably high, as those
> languages are still pretty much the mainstay of the industry, and
> definitely used by many Emacs users.

Apparently, this has not been enough motivation for the last decade, so
I'm not holding my breath.  Including the new proposal into Emacs may
provide an added incentive if MMM-support becomes sufficiently common
that CC-mode becomes really the odd-one out (tho it's just an incentive
and CC-mode is the odd-one out in many respects already).

> Maybe I'm missing something, but what's the equivalent of PREV-CHUNK?

There isn't any.  PREV-CHUNK is probably a good idea somewhere in some
cases, but we haven't found those cases yet.

>> We keep 'prog-first-column' from that proposal
> But it was a function, and you made it a variable.  This will get in
> the way of compatibility, and also potentially will make future
> extensions harder.  Why not keep it a function?

Agreed, it should be a function (which can internally just lookup
a variable, in its current implementation, but API exposed to major
modes should be the function).

>> and instead of allowing MMM to indicate the chunk bounds through
>> prog-indentation-context, we allow MMM to apply narrowing directly, and
>> that modes honor it.
> This simplification also took away some of the features, like the
> ability to nest restrictions.

I do not know what you're referring to.

> I wonder why did you discard that.
> It's existing functionality, pretty lightweight, which was in Emacs
> for some time, and is reasonably well documented.  And it already
> satisfied your requirement of not allowing sub-modes to widen.  Why
> not leave it alone?  What am I missing?

>>>> Transitioning from prog-indentation-context to the new approach is very
>>>> easy.
>>> That's what I thought.  And that's why we should do that
>>> simultaneously with landing the new approach.  There's no reason to do
>>> that earlier.

You mean we should provide patches for those codes using
prog-indentation-context?  Doesn't Dmitry's branch already do that?

> I don't see the incompatibilities.  Basically, you replaced prog-widen
> with widen,

Actually, no.  The replacement for prog-widen is more like #'ignore
(remember: prog-widen is for major modes who want to "narrow to the
current chunk's boundaries", but with the new arrangement, the major
mode's code is already called with such a narrowing in place).

> IOW, the transition could be much smoother than it will be if we
> actually remove that stuff, because I don't think there's any
> incompatibility here which would disallow direct calls to 'widen'
> and/or leaving prog-indentation-context at its default nil value.  Or
> maybe I'm missing something again.  But if I'm right, and if we can
> make the necessary changes limited only to the documentation, then it
> could well make it into Emacs 26.  So I think it is worth our while
> to make some effort in that direction, if at all possible.

Then let's focus on the doc plus adding a few calls to widen in places
like indent-according-to-mode.


        Stefan

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> Cc: acm@muc.de, tom@tromey.com, monnier@iro.umontreal.ca, spinuvit@gmail.com,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 4 Dec 2017 11:53:09 +0000
> 
> Point is, putting mmm-mode in ELPA or not is the developer's choice.

That's true, but why would a developer object to including his/her
package in Emacs?  It will get better integration, better maintenance,
better publicity, etc.  Sounds like win-win to me.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> Date: Mon, 4 Dec 2017 15:52:38 +0000
> Cc: Eli Zaretskii <eliz@gnu.org>, tom@tromey.com, monnier@iro.umontreal.ca,
>   spinuvit@gmail.com, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> the "syntax islands" proposal would be a lot of work, which I don't
> want to commit to before I see some sort of undertaking to take it
> seriously.

I hear you.  I also think that having concrete problems in hand that
cannot be easily solved by the existing framework will help us revisit
the "islands" proposal (and perhaps other related proposals) with more
"meat" to discuss them, and thus with higher probability of reaching
an agreement.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> Date: Mon, 4 Dec 2017 16:23:49 +0000
> Cc: tom@tromey.com, emacs-devel@gnu.org, monnier@iro.umontreal.ca,
>   spinuvit@gmail.com, dgutov@yandex.ru
> From: Alan Mackenzie <acm@muc.de>
> 
> > In any case, I'm not asking you to like this restriction, just help
> > bring CC Mode into the MMM framework, because I think without that the
> > feature will be woefully incomplete.
> 
> I will do what I can.

Thank you.

> Could you tell me where to find MMM?  ;-)

I think it's here:  https://github.com/purcell/mmm-mode

> Sorry for being so uncooperative over the last couple of days.  I agree
> that CC Mode must work together with MMM, and I will do everything I can
> to facilitate that.

Thank you very much.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

> Why does it need to widen, though?

Because in general, indentation may need to look at the context, and
that context may be outside of the current user-chosen narrowing.

> If the mode's indent-line-function is not supposed to call 'widen',
> why do we do that in indent-according-to-mode?

We do it in indent-according-to-mode specifically so that
indent-line-function doesn't need to do it.

> When you widen, the text that comes into the view will be entirely
> alien syntactically to the sub-mode that is trying to indent, right?

The widening is done in indent-according-to-mode and brings into view
the whole multi-mode buffer, indeed.  Then MMM's indent-line-function is
called, which narrows the buffer to the relevant chunk and then calls
the submode's indent-line-function which hence only sees the part of the
buffer it needs.

>> This change should be harmless because even if the user has narrowed
>> the buffer, the indentation should always be done depending on the
>> whole buffer's content anyway.
> But that's an incompatible change, isn't it?

It's a change that can have visible effects, of course, but these are
fairly rare and I'm hard pressed to come up with scenarios where the
effect is anything but beneficial.


        Stefan

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Dmitry Gutov <dgutov@yandex.ru>,  acm@muc.de,  tom@tromey.com,  spinuvit@gmail.com,  emacs-devel@gnu.org
> Date: Mon, 04 Dec 2017 11:35:53 -0500
> 
> > Emacs development is not only about not losing existing features.  It
> > is mainly about gaining new important features.  Here you are adding a
> > feature that I think is important enough for us to try make it support
> > C-like languages.
> 
> His proposal is reworking prog-indentation-context.
> CC-mode doesn't support prog-indentation-context, so it's no surprise
> that it doesn't support this replacement= and I don't see why suddenly
> we should first get CC-mode to support it before the replacement can
> be installed.

Not "first", "together with", or at least "soon enough after".

> Apparently, this has not been enough motivation for the last decade, so
> I'm not holding my breath.

Well, Alan just said he will try to make that happen, so I _am_
holding my breath.

> > Maybe I'm missing something, but what's the equivalent of PREV-CHUNK?
> 
> There isn't any.  PREV-CHUNK is probably a good idea somewhere in some
> cases, but we haven't found those cases yet.

But it's already in Emacs.  So the question IMO is "is it such a bad
idea that we need to get rid of it right away?"

> >> We keep 'prog-first-column' from that proposal
> > But it was a function, and you made it a variable.  This will get in
> > the way of compatibility, and also potentially will make future
> > extensions harder.  Why not keep it a function?
> 
> Agreed, it should be a function (which can internally just lookup
> a variable, in its current implementation, but API exposed to major
> modes should be the function).

That's what it is: a function that looks up a variable internally.

> >> and instead of allowing MMM to indicate the chunk bounds through
> >> prog-indentation-context, we allow MMM to apply narrowing directly, and
> >> that modes honor it.
> > This simplification also took away some of the features, like the
> > ability to nest restrictions.
> 
> I do not know what you're referring to.

AFAIU, prog-indentation-context could support a way to nest
restrictions, because prev-chunk could be a function.

> You mean we should provide patches for those codes using
> prog-indentation-context?  Doesn't Dmitry's branch already do that?

Not for ANTLR whose latest versions are maintained outside Emacs,
AFAIU.

> > I don't see the incompatibilities.  Basically, you replaced prog-widen
> > with widen,
> 
> Actually, no.

Again, I might be missing something, but "git diff" says I'm right.

> Then let's focus on the doc plus adding a few calls to widen in places
> like indent-according-to-mode.

I'd still like to understand why those additional calls to 'widen' are
needed.  Doing that seems to contradict what we want to ask major
modes not to do in their indentation code.

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

> The code which supports PREV-CHUNK is already in Emacs, so the
> question IMO is "why remove it"?

?Which code is that?

> If we want to try to get this into
> Emacs 26, we should strive to make the code changes minimal, ideally
> zero.  Once all we are left with are documentation changes, the
> feature can land on emacs-26 right now.

Zero is not the intention: for the doc changes to be valid, we need to
add a few `widen` calls in places like indent-according-to-mode.

But I'd be fine keeping the old prog-indentation-context if you want.


        Stefan

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Mon, 04 Dec 2017 11:49:59 -0500
> 
> > Why does it need to widen, though?
> 
> Because in general, indentation may need to look at the context, and
> that context may be outside of the current user-chosen narrowing.
> 
> > If the mode's indent-line-function is not supposed to call 'widen',
> > why do we do that in indent-according-to-mode?
> 
> We do it in indent-according-to-mode specifically so that
> indent-line-function doesn't need to do it.
> 
> > When you widen, the text that comes into the view will be entirely
> > alien syntactically to the sub-mode that is trying to indent, right?
> 
> The widening is done in indent-according-to-mode and brings into view
> the whole multi-mode buffer, indeed.  Then MMM's indent-line-function is
> called, which narrows the buffer to the relevant chunk and then calls
> the submode's indent-line-function which hence only sees the part of the
> buffer it needs.

Ah, so this assumes that MMM is active in the buffer?  If so, can we
widen only if MMM is indeed active?  Doing so will I believe minimize
(or even eliminate) the effects on existing behaviors when MMM is not
used.

> It's a change that can have visible effects, of course, but these are
> fairly rare and I'm hard pressed to come up with scenarios where the
> effect is anything but beneficial.

Use being so deep into the pretest requires to be either 110% careful
not to change existing behaviors, or agree to postpone this to Emacs 27.

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: dgutov@yandex.ru,  emacs-devel@gnu.org
> Date: Mon, 04 Dec 2017 12:12:48 -0500
> 
> > The code which supports PREV-CHUNK is already in Emacs, so the
> > question IMO is "why remove it"?
> 
> ?Which code is that?

prog-indentation-context and its support in prog-widen.

> > If we want to try to get this into
> > Emacs 26, we should strive to make the code changes minimal, ideally
> > zero.  Once all we are left with are documentation changes, the
> > feature can land on emacs-26 right now.
> 
> Zero is not the intention: for the doc changes to be valid, we need to
> add a few `widen` calls in places like indent-according-to-mode.

If those calls are conditioned on MMM actually being active, then
existing behavior will remain unchanged, and we are good.

> But I'd be fine keeping the old prog-indentation-context if you want.

Actually, let me try to understand why any changes to the existing
code, apart of _adding_ those few 'widen' calls, are at all necessary.
Please bear with me.

What I see in the branch is this:

  1) the calls to prog-widen are replaced with calls to widen.
  2) some calls to widen are added
  3) prog-indentation-context is removed
  4) prog-first-column the function is removed, and its calls replaced
     with accessing the (existing) name-sake variable

For 4), we already agreed that keeping a function is better.

Since prog-widen already calls widen if prog-indentation-context is
nil (which it is by default), calling prog-widen without setting up
prog-indentation-context will just call widen; this magically takes
care of 1).

For 3), if we leave prog-indentation-context in the code, and also
allow direct calls to widen in modes which don't want to use the
context, we are not losing anything, while leaving the option of using
the context to those modes which will want that.  This currently
cannot be used by MMM (AFAIU), but other features which need embedded
code, such as ANTLR, could still use it, and even MMM will be able to
do that if it is extended.

2) can be taken care of as indicated above, thus avoiding any possible
effects on existing behaviors when MMM is not active.

If you agree with the above, then we can have the cake and eat it,
too.  Or am I missing something?

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

> That's true, but why would a developer object to including his/her
> package in Emacs?  It will get better integration, better maintenance,
> better publicity, etc.  Sounds like win-win to me.

Beats me as well, but I know that many people see it very differently.


        Stefan

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

>> > The code which supports PREV-CHUNK is already in Emacs, so the
>> > question IMO is "why remove it"?
>> ?Which code is that?
> prog-indentation-context and its support in prog-widen.

AFAICT prog-widen doesn't look at the PREVIOUS-CHUNK part of
prog-indentation-context.
So PREVIOUS-CHUNK only exists in the docstring of
prog-indentation-context and nowhere else.

>> Zero is not the intention: for the doc changes to be valid, we need to
>> add a few `widen` calls in places like indent-according-to-mode.
> If those calls are conditioned on MMM actually being active, then
> existing behavior will remain unchanged, and we are good.

How do you propose we detect when MMM is active?


        Stefan "and by the way, those widen calls are beneficial even
                when MMM is not active, but if you insist I'm OK
                delaying this side benefit to Emacs-27"

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Mon, 04 Dec 2017 12:52:39 -0500
> 
> >> > The code which supports PREV-CHUNK is already in Emacs, so the
> >> > question IMO is "why remove it"?
> >> ?Which code is that?
> > prog-indentation-context and its support in prog-widen.
> 
> AFAICT prog-widen doesn't look at the PREVIOUS-CHUNK part of
> prog-indentation-context.

But it looks at the other data in prog-indentation-context, and
PREVIOUS-CHUNK, if it's a function, could store there something.
Or a mode could use some other method to store the previous
restriction directly in prog-indentation-context.  Right?

> >> Zero is not the intention: for the doc changes to be valid, we need to
> >> add a few `widen` calls in places like indent-according-to-mode.
> > If those calls are conditioned on MMM actually being active, then
> > existing behavior will remain unchanged, and we are good.
> 
> How do you propose we detect when MMM is active?

I don't know (didn't look at its sources close enough), but I'd be
surprised if there wasn't some variable or some other method to tell.

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> Date: Mon, 04 Dec 2017 21:53:36 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> > From: Stefan Monnier <monnier@iro.umontreal.ca>
> > Date: Mon, 04 Dec 2017 12:52:39 -0500
> > 
> > How do you propose we detect when MMM is active?
> 
> I don't know (didn't look at its sources close enough)

I did now, and it sounds like the buffer-local variable mmm-mode
should be non-nil when MMM is active (of course!).

Does that fit the bill?

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

>> AFAICT prog-widen doesn't look at the PREVIOUS-CHUNK part of
>> prog-indentation-context.
> But it looks at the other data in prog-indentation-context, and
> PREVIOUS-CHUNK, if it's a function, could store there something.
> Or a mode could use some other method to store the previous
> restriction directly in prog-indentation-context.  Right?

My point is that currently, I can only find 3 bytes of code which
do anything with PREVIOUS-CHUNK, it's in mhtml-mode.el where they do:

    (let ((prog-indentation-context (list base-indent
                                          (cons (point-min) nil)
                                          nil)))

where the last `nil` is actually the PREVIOUS-CHUNK (we could remove
that `nil` and it would make no difference, tho).
It really only exists in the docstring.

>> >> Zero is not the intention: for the doc changes to be valid, we need to
>> >> add a few `widen` calls in places like indent-according-to-mode.
>> > If those calls are conditioned on MMM actually being active, then
>> > existing behavior will remain unchanged, and we are good.
>> How do you propose we detect when MMM is active?
> I don't know (didn't look at its sources close enough), but I'd be
> surprised if there wasn't some variable or some other method to tell.

I'd rather not hard code knowledge of mmm-mode as *the* MMM system.
Maybe we should just forget about this whole thing for emacs-26 and move
on to emacs-27.


        Stefan

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/4/17 7:12 PM, Stefan Monnier wrote:

>> If we want to try to get this into
>> Emacs 26, we should strive to make the code changes minimal, ideally
>> zero.  Once all we are left with are documentation changes, the
>> feature can land on emacs-26 right now.
> 
> Zero is not the intention: for the doc changes to be valid, we need to
> add a few `widen` calls in places like indent-according-to-mode.

We also need to *remove* widen calls in indentation functions, or the 
new convention is not followed, and thus useless.

> But I'd be fine keeping the old prog-indentation-context if you want.

How would that work? It's second element must be unused for the above 
protocol to work. So it will be a list with just one effective element?

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/4/17 9:53 PM, Eli Zaretskii wrote:

>>>> Zero is not the intention: for the doc changes to be valid, we need to
>>>> add a few `widen` calls in places like indent-according-to-mode.
>>> If those calls are conditioned on MMM actually being active, then
>>> existing behavior will remain unchanged, and we are good.
>>
>> How do you propose we detect when MMM is active?
> 
> I don't know (didn't look at its sources close enough), but I'd be
> surprised if there wasn't some variable or some other method to tell.

I'm sorry, but this is some nonsense you are discussing.

The change in question fixes an old problem, namely: some major modes 
perform indentation while taking the whole buffer into account, and some 
only look at the current restriction (which would be a bug, but nobody 
has reported it yet in those cases, apparently because there is some 
correlation between users of C and Python, and users of interactive 
narrowing).

So the fix makes *all* indentation code take into account the whole 
buffer, even when interactive narrowing is in place. Unless mmm-mode, on 
something like it, applies an extra restriction through its custom 
indent-line-function.

That part of the change seems obviously correct to me, no need to 
predicate it on mmm-mode being active.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/4/17 7:28 PM, Eli Zaretskii wrote:

>> The widening is done in indent-according-to-mode and brings into view
>> the whole multi-mode buffer, indeed.  Then MMM's indent-line-function is
>> called, which narrows the buffer to the relevant chunk and then calls
>> the submode's indent-line-function which hence only sees the part of the
>> buffer it needs.
> 
> Ah, so this assumes that MMM is active in the buffer?  If so, can we
> widen only if MMM is indeed active?  Doing so will I believe minimize
> (or even eliminate) the effects on existing behaviors when MMM is not
> used.

That widening is for correctness when working during an interactive 
narrowing. MMM mode being enabled or not is irrelevant (it's not the 
beneficiary).

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

>> But I'd be fine keeping the old prog-indentation-context if you want.
> How would that work? It's second element must be unused for the above
> protocol to work.

That's right.  Keep it and ignore it.


        Stefan

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/5/17 12:05 AM, Stefan Monnier wrote:
>>> But I'd be fine keeping the old prog-indentation-context if you want.
>> How would that work? It's second element must be unused for the above
>> protocol to work.
> 
> That's right.  Keep it and ignore it.

Then any real consumer of prog-indentation-context will also have to add 
a narrow-to-region to their code.

So backward compatibility will be poor anyway.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/4/17 6:56 PM, Eli Zaretskii wrote:

>> His proposal is reworking prog-indentation-context.
>> CC-mode doesn't support prog-indentation-context, so it's no surprise
>> that it doesn't support this replacement= and I don't see why suddenly
>> we should first get CC-mode to support it before the replacement can
>> be installed.
> 
> Not "first", "together with", or at least "soon enough after".

That still doesn't follow.

>> Apparently, this has not been enough motivation for the last decade, so
>> I'm not holding my breath.
> 
> Well, Alan just said he will try to make that happen, so I _am_
> holding my breath.

That's wonderful. But I'm still not going to hold my breath regarding 
the time estimates. It might turn out to be fairly time-consuming.

>> There isn't any.  PREV-CHUNK is probably a good idea somewhere in some
>> cases, but we haven't found those cases yet.
> 
> But it's already in Emacs.  So the question IMO is "is it such a bad
> idea that we need to get rid of it right away?"

It's pointless: PREV-CHUNK must be followed/used by a major mode. The 
majority of the important major modes are inside Emacs (the ones used by 
ANTLR are, at least). So there won't be any real benefit to keeping it in.

On the other hand, we might get a better idea how to solve that problem 
in a different way, during the next release cycle.

> AFAIU, prog-indentation-context could support a way to nest
> restrictions, because prev-chunk could be a function.

Nobody really knows how to use this.

>> You mean we should provide patches for those codes using
>> prog-indentation-context?  Doesn't Dmitry's branch already do that?
> 
> Not for ANTLR whose latest versions are maintained outside Emacs,
> AFAIU.

The diff will be like 4 lines. I can submit it whenever, wherever you want.

>>> I don't see the incompatibilities.  Basically, you replaced prog-widen
>>> with widen,
>>
>> Actually, no.
> 
> Again, I might be missing something, but "git diff" says I'm right.

Not really. There is one place where it looks like prog-widen is 
replaced by widen, but it's inside a function that's not part of the 
indentation engine (python-indent-guess-indent-offset, despite what its 
name suggests, is called from the major mode function, not through 
indent-line-function).

The other places, inside indentation engines, feature the removals of 
prog-widen and (in one case) widen calls.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/4/17 5:52 PM, Eli Zaretskii wrote:

> Your work on this is highly appreciated.

Thank you.

> But I have certain
> responsibilities, and I'm trying to do my job here.

True, and I fully expect to discuss API breakage, development cycles, 
and that sort of issues. Adding a big requirement, and it being 
something we've effectively given up on solving in the shorter term, has 
been very surprising.

> I'm also your fellow developer of the same project and its active
> user.  I think it means my opinions should be of some importance, not
> something to be ignored solely because I'm not coding this.

I kind of expected opinions in different directions.

> Just like
> I would never dream of ignoring your opinions about features I'm
> implementing.

Not opinions like "don't push this change, unless you make it work for 
my use case X too!", I expect. At least, not for all such opinions.

>> [C-like major modes] can be catered to, *as soon as* the authors of each individual major mode make the effort to support the feature.
> 
> Agreed.  What I'm trying to establish is whether the design supports
> those major modes, or does it ignore them.  The latter I would like to
> avoid as much as possible.

Whatever special requirements particular major modes turn out to have, 
can be added later. I've tried my best to make sure that improvements 
would be smooth. Hopefully just additive.

>> Eli, we already have a feature in Emacs (prog-indentation-context) that does not adhere to your (unreasonably high) requirements.
> 
> Emacs development is not only about not losing existing features.  It
> is mainly about gaining new important features.  Here you are adding a
> feature that I think is important enough for us to try make it support
> C-like languages.

So far, the discussion of supporting it in CC Mode, has not uncovered 
any extra requirements to the proposed protocol, from where I'm standing.

> I don't think this requirement is unreasonably
> high, as those languages are still pretty much the mainstay of the
> industry, and definitely used by many Emacs users.

The requirement seems a bit late, is all. Either you apply it to 
prog-indentation-context and vote to remove it, or you don't get to 
apply it to this proposal. That's my opinion, at least.

>> And that cache point to a position outside of the current restriction. Confusion ensues. I've seen errors originating from that (I think), but can't recall the exact sequence of calls.
> 
> Thanks, so the caches will have to be sensitive to restrictions as
> they are sensitive to deletion of text.

Right, and that's the responsibility of the major mode. So there's not 
much we can add in terms of code (I think), but we can add some more 
sentences to the manual.

> But given your example with CC Mode's caches, that's not all of it, is
> it?  You also need the mode to adapt any such caches to changes in
> buffer restrictions, right?

I think so. But only CC Mode, in my experience, includes such caches 
that lead to problems in MMM context.

>> We keep 'prog-first-column' from that proposal
> 
> But it was a function, and you made it a variable.  This will get in
> the way of compatibility, and also potentially will make future
> extensions harder.  Why not keep it a function?

To have a better API. If we keep it a function, do we also have a 
variable prog-first-column?

Then, some consumers might have doubt which ones to use, and might opt 
to reference the variable. Then, we lose all benefits of having it a 
function (like making its implementation somehow smarter later), and the 
only benefit remaining is backward compatibility of having a function 
with that name.

But! Like with PREV-CHUNKS, prog-first-column (and the later prog-widen) 
are supposed to be used by the major modes only. There are no references 
to either of these functions in the latest antlr-mode.el, and there 
shouldn't be.

And as long as all the calls to that function are inside Emacs, we're 
free to change it however, including turning it into a variable.

>> and instead of allowing MMM to indicate the chunk bounds through prog-indentation-context, we allow MMM to apply narrowing directly, and that modes honor it.
> 
> This simplification also took away some of the features, like the
> ability to nest restrictions.  I wonder why did you discard that.
> It's existing functionality, pretty lightweight, which was in Emacs
> for some time, and is reasonably well documented.  And it already
> satisfied your requirement of not allowing sub-modes to widen.  Why
> not leave it alone?  What am I missing?

It's just pointless. I've asked, in the original discussion 2 years ago, 
for at least one patch that would use it. None have arrived since.

And as long as major modes do not use it, MMM packages don't have any 
benefits from it.

>>          Transitioning from prog-indentation-context to the new approach is very
>>          easy.
>>
>>      That's what I thought.  And that's why we should do that
>>      simultaneously with landing the new approach.  There's no reason to do
>>      that earlier.
>>
>> Why confuse the language modes authors?
> 
> If they should be confused (I don't think so), they are already
> confused, as we've been having this for the last 2 years.

Not in a release version. And the confusion I'm talking about will come 
later, when we either try to keep both ways of doing this (I don't think 
we'll manage), or abruptly switch from one way to another.

The latter is easy code-wise, but by that time, whatever compatibility 
code they had, they might have removed it intending to support only 
Emacs 26+, for instance. And here we come saying Emacs 27 will be 
strictly incompatible.

>>      We cannot know that.  Once the code is that long with us, we cannot
>>      throw it away without a equivalent replacement.  And I see no reason
>>      to do that now, since the fact that this code will exist in Emacs 26
>>      doesn't hurt anyone, especially since the replacement will be easy, as
>>      you say.
>>
>> Well... you have a point there. The result will be pretty ugly, though. Adding an API in one version, and removing it in another.
> 
> We will have to discuss the "removal" part.  An alternative is to
> provide compatibility shims, or even leave (some or all of) the
> existing API intact.

Yeah, I cannot imagine how a compatibility shim would work. Hence my 
request to replace it as soon as possible.

>> They are incompatible with each other, so it's not like there can be a smooth transition.
> 
> I don't see the incompatibilities.  Basically, you replaced prog-widen
> with widen,

Nope, just removed prog-widen. And widen in one place.

Also replaced one prog-widen call with widen, but it was not in an 
indentation function (os it probably shouldn't have been there in the 
first place).

> made prog-first-column a variable instead of a function,

See (*).

> and tossed prog-indentation-context.  We could instead keep the
> existing API with minimal changes: prog-widen calls widen internally,
> and we could allow direct calls to widen as well.

That won't work. Either indentation functions call prog-widen (which 
fully widens in the absence of prog-indentation-context binding), or 
they don't widen at all. These are two different, incompatible ways of 
doing things.

> IOW, the transition could be much smoother than it will be if we
> actually remove that stuff, because I don't think there's any
> incompatibility here which would disallow direct calls to 'widen'
> and/or leaving prog-indentation-context at its default nil value.

Yes, there is. Please go back to one of the several descriptions, in 
this thread only, of how MMM uses narrowing to restrict indentation 
engine to the current chunk.

> Or
> maybe I'm missing something again.  But if I'm right, and if we can
> make the necessary changes limited only to the documentation, then it
> could well make it into Emacs 26.  So I think it is worth our while
> to make some effort in that direction, if at all possible.

Unfortunately, I don't imagine it's really possible.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/4/17 6:41 PM, Eli Zaretskii wrote:

>> Point is, putting mmm-mode in ELPA or not is the developer's choice.
> 
> That's true, but why would a developer object to including his/her
> package in Emacs?  It will get better integration, better maintenance,
> better publicity, etc.  Sounds like win-win to me.

Better control over releases, faster to get fixes to the users. That's 
if we're comparing "in Emacs only" with "in ELPA only".

If the package is released to both, I have no experience with that, but 
I imagine there needs to be some extra work synchronizing, as well as 
making sure the APIs are compatible, at least (to which set of versions 
of Emacs? I'm not sure...).

Overall, I'm in favor of "modular Emacs", and so I'd rather ask why 
include this particular package, and how it would benefit. Users being 
able to avoid using ELPA is a very minor difference, in my mind, 
especially compared to benefits above.

mmm-mode, in particular, is designed to work "outside in", with little 
support from the code inside Emacs, so it shouldn't need much support 
over the years, not much more than it would get being part of ELPA, at 
least.

Further, I'm not sure that its way of doing things (the user-facing 
part) is best. Maybe polymode has it thought out better, I have not 
studied it well enough. Or an extraction from mhtml-mode would turn out 
better still. Putting an MMM package into Emacs would seem like a choice 
of the better package, and I'd rather we have several equally supported 
ones, at least until one comes out decisively at the top.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

On December 4, 2017 11:52:26 PM GMT+02:00, Dmitry Gutov <dgutov@yandex.ru> wrote:
> On 12/4/17 7:28 PM, Eli Zaretskii wrote:
> 
> >> The widening is done in indent-according-to-mode and brings into
> view
> >> the whole multi-mode buffer, indeed.  Then MMM's
> indent-line-function is
> >> called, which narrows the buffer to the relevant chunk and then
> calls
> >> the submode's indent-line-function which hence only sees the part
> of the
> >> buffer it needs.
> > 
> > Ah, so this assumes that MMM is active in the buffer?  If so, can we
> > widen only if MMM is indeed active?  Doing so will I believe
> minimize
> > (or even eliminate) the effects on existing behaviors when MMM is
> not
> > used.
> 
> That widening is for correctness when working during an interactive 
> narrowing. MMM mode being enabled or not is irrelevant (it's not the 
> beneficiary).

I understand that you and Stefan think so, but doing that unconditionally
means these changes cannot be committed to emacs-26.  I was
trying to find a way of squeezing them into the release branch, but if
you think there's no way, it's fine with me to have this on master.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

On December 5, 2017 7:08:14 AM GMT+02:00, Eli Zaretskii <eliz@gnu.org> wrote:
> On December 4, 2017 11:52:26 PM GMT+02:00, Dmitry Gutov
> <dgutov@yandex.ru> wrote:
> > On 12/4/17 7:28 PM, Eli Zaretskii wrote:
> > 
> > >> The widening is done in indent-according-to-mode and brings into
> > view
> > >> the whole multi-mode buffer, indeed.  Then MMM's
> > indent-line-function is
> > >> called, which narrows the buffer to the relevant chunk and then
> > calls
> > >> the submode's indent-line-function which hence only sees the part
> > of the
> > >> buffer it needs.
> > > 
> > > Ah, so this assumes that MMM is active in the buffer?  If so, can
> we
> > > widen only if MMM is indeed active?  Doing so will I believe
> > minimize
> > > (or even eliminate) the effects on existing behaviors when MMM is
> > not
> > > used.
> > 
> > That widening is for correctness when working during an interactive 
> > narrowing. MMM mode being enabled or not is irrelevant (it's not the
> 
> > beneficiary).
> 
> I understand that you and Stefan think so, but doing that
> unconditionally
> means these changes cannot be committed to emacs-26.  I was
> trying to find a way of squeezing them into the release branch, but if
> you think there's no way, it's fine with me to have this on master.

And btw, I'm not really convinced the unconditional widening is a good idea in general, even on master.  I could probably agree that in most cases it is TRT, but why would we _force_ all modes to indent with restrictions lifted?  No exceptions?  Not even a fire escape for some specialized mode with weird needs and requirements?  I'm not sure.  It certainly should be discussed, regardless of the MMM issues.

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

On December 5, 2017 12:45:34 AM GMT+02:00, Dmitry Gutov <dgutov@yandex.ru> wrote:
> On 12/5/17 12:05 AM, Stefan Monnier wrote:
> >>> But I'd be fine keeping the old prog-indentation-context if you
> want.
> >> How would that work? It's second element must be unused for the
> above
> >> protocol to work.
> > 
> > That's right.  Keep it and ignore it.
> 
> Then any real consumer of prog-indentation-context will also have to
> add 
> a narrow-to-region to their code.
> 
> So backward compatibility will be poor anyway.

I think any users of prog-indentation-context already do narrow.
So we don't harm anything in that area by keeping it.

Once again, we are talking about removing _existing_ code, even
if unused.  So "unnecessary" or "useless" just don't cut it; please
explain how it does some real damage.

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/5/17 8:03 AM, Eli Zaretskii wrote:

>> Then any real consumer of prog-indentation-context will also have to
>> add
>> a narrow-to-region to their code.
>>
>> So backward compatibility will be poor anyway.
> 
> I think any users of prog-indentation-context already do narrow.

Not antlr-mode, no.

It specifically applies narrowing via prog-indentation-context as 
designed, because python-mode supports that (unlike other major modes).

> So we don't harm anything in that area by keeping it.
> 
> Once again, we are talking about removing _existing_ code, even
> if unused.  So "unnecessary" or "useless" just don't cut it; please
> explain how it does some real damage.

It solidifies an unused and unproved design which nobody can really use 
until the next release of Emacs anyway.

And also, you call it "existing", I call it "never having been in a 
release".

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/5/17 7:33 AM, Eli Zaretskii wrote:

> And btw, I'm not really convinced the unconditional widening is a good idea in general, even on master.  I could probably agree that in most cases it is TRT, but why would we _force_ all modes to indent with restrictions lifted?  No exceptions?  Not even a fire escape for some specialized mode with weird needs and requirements?  I'm not sure.  It certainly should be discussed, regardless of the MMM issues.

After indent-for-tab-command calls widen, indent-line-function is 
called. And it can apply all the narrowing it wants (inside 
save-restriction), it just shouldn't call 'widen'.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/5/17 7:08 AM, Eli Zaretskii wrote:

> I understand that you and Stefan think so, but doing that unconditionally
> means these changes cannot be committed to emacs-26.  I was
> trying to find a way of squeezing them into the release branch, but if
> you think there's no way, it's fine with me to have this on master.

Suppose the patch goes on master.

Do I actually remove all references to prog-indentation-context like 
shown on the branch? Despite that it will be present in Emacs 26?

What about NEWS? Will it say "prog-indentation-context was removed, and 
a different protocol for multi-major-mode buffers is established now"?

These questions are still unresolved, and there's no point in delaying 
those decisions.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

Hey Alan,

On 12/4/17 5:52 PM, Alan Mackenzie wrote:

> Any chance you could give me some basic details of how to get into this
> MMM mode?  Like which git branch (? scratch/widen-less ?) can I find the
> code in, and where is the best documentation to get started.

I was talking in abstract (MMM meaning any similar framework), but it's 
a good question. To debug the problems, a concrete example is needed.

The latest version of mmm-mode lives here: 
https://github.com/purcell/mmm-mode.git, you can check out the master 
branch, and it even has a manual that's reasonably up-to-date WRT to usage.

Here are a couple of related bug reports:

https://github.com/purcell/mmm-mode/issues/55
https://github.com/purcell/mmm-mode/issues/57

Both pertaining to the use of c-mode in the context of mmm-noweb.el.

Please let me know if you need more localized examples.

> I've no idea how long it will take to make MMM and
> CC Mode work with eachother.

If you manage to do it at all, that would be an achievement. But just 
figuring out the biggest problems should help us to design the feature.

> Not much, if at all.  I found those discussions to be lacking the
> context I would have needed to understand them.

I hope the context is apparent now.

> That's fair.  But the "syntax islands" proposal would be a lot of work,
> which I don't want to commit to before I see some sort of undertaking to
> take it seriously.

Then it should be a good idea to try the proposed, smaller approach, and 
collect the drawbacks you meet along the way.

> [ .... ]
> 
>> There are several ways to skin that cat.
> 
> That's the second cat in as many days, here!

Lots of cats around here in Cyprus. ;-)

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

On December 5, 2017 10:42:27 AM GMT+00:00, Dmitry Gutov <dgutov@yandex.ru> wrote:
> On 12/5/17 8:03 AM, Eli Zaretskii wrote:
> 
> >> Then any real consumer of prog-indentation-context will also have
> to
> >> add
> >> a narrow-to-region to their code.
> >>
> >> So backward compatibility will be poor anyway.
> > 
> > I think any users of prog-indentation-context already do narrow.
> 
> Not antlr-mode, no.
> 
> It specifically applies narrowing via prog-indentation-context as 
> designed, because python-mode supports that (unlike other major
> modes).
> 
> > So we don't harm anything in that area by keeping it.
> > 
> > Once again, we are talking about removing _existing_ code, even
> > if unused.  So "unnecessary" or "useless" just don't cut it; please
> > explain how it does some real damage.
> 
> It solidifies an unused and unproved design which nobody can really
> use 
> until the next release of Emacs anyway.
> 
> And also, you call it "existing", I call it "never having been in a 
> release".

"Solidifies unused design" doesn't sound convincing to me, sorry.  Especially since I still think we can accommodate MMM without breaking prog-indentation-context and its users, whether existing or future.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

On December 5, 2017 10:55:33 AM GMT+00:00, Dmitry Gutov <dgutov@yandex.ru> wrote:
> On 12/5/17 7:33 AM, Eli Zaretskii wrote:
> 
> > And btw, I'm not really convinced the unconditional widening is a
> good idea in general, even on master.  I could probably agree that in
> most cases it is TRT, but why would we _force_ all modes to indent
> with restrictions lifted?  No exceptions?  Not even a fire escape for
> some specialized mode with weird needs and requirements?  I'm not
> sure.  It certainly should be discussed, regardless of the MMM issues.
> 
> After indent-for-tab-command calls widen, indent-line-function is 
> called. And it can apply all the narrowing it wants (inside 
> save-restriction), it just shouldn't call 'widen'.

You assume that the indentation function has the same information about the higher-level context as its callers.  But that assumption is not necessarily true; narrowing exists precisely because higher-level code wants to set up lower-level code without explicitly passing it this information.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

On December 5, 2017 12:55:46 PM GMT+00:00, Dmitry Gutov <dgutov@yandex.ru> wrote:
> On 12/5/17 7:08 AM, Eli Zaretskii wrote:
> 
> > I understand that you and Stefan think so, but doing that
> unconditionally
> > means these changes cannot be committed to emacs-26.  I was
> > trying to find a way of squeezing them into the release branch, but
> if
> > you think there's no way, it's fine with me to have this on master.
> 
> Suppose the patch goes on master.
> 
> Do I actually remove all references to prog-indentation-context like 
> shown on the branch? Despite that it will be present in Emacs 26?
> 
> What about NEWS? Will it say "prog-indentation-context was removed,
> and 
> a different protocol for multi-major-mode buffers is established now"?
> 
> These questions are still unresolved, and there's no point in delaying
> 
> those decisions.

I posted a more or less complete proposal for how to make the changes you want without breaking previous facilities.  If you agree with that proposal (which IMO makes sense even for master), then most of the issues you raise disappear.  If you disagree, please respond to that message of mine explaining why, and let's discuss that first.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/5/17 7:53 PM, Eli Zaretskii wrote:

> You assume that the indentation function has the same information about the higher-level context as its callers.  But that assumption is not necessarily true; narrowing exists precisely because higher-level code wants to set up lower-level code without explicitly passing it this information.

Without MMM in the equation, the caller here is indent-for-tab-command.

It's an interactive function, and the only narrowing present before it 
is the one possibly applied by the user, interactively.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/5/17 7:57 PM, Eli Zaretskii wrote:

> I posted a more or less complete proposal for how to make the changes you want without breaking previous facilities.  If you agree with that proposal (which IMO makes sense even for master), then most of the issues you raise disappear.  If you disagree, please respond to that message of mine explaining why, and let's discuss that first.

Sorry, is it this message?

http://lists.gnu.org/archive/html/emacs-devel/2017-12/msg00158.html

I thought I've addresses every point in there (that I could make sense 
of) in other messages in this thread, but if you prefer, I can respond 
to it directly, to put the critique in one place.

Have you read the following messages, BTW? There were no responses.

http://lists.gnu.org/archive/html/emacs-devel/2017-12/msg00169.html
http://lists.gnu.org/archive/html/emacs-devel/2017-12/msg00170.html

CC Mode in MMM Mode(s). [Was: Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls]

[Alan Mackenzie]

Hello, Dmitry.

On Tue, Dec 05, 2017 at 15:08:02 +0200, Dmitry Gutov wrote:
> Hey Alan,

> On 12/4/17 5:52 PM, Alan Mackenzie wrote:

> > Any chance you could give me some basic details of how to get into this
> > MMM mode?  Like which git branch (? scratch/widen-less ?) can I find the
> > code in, and where is the best documentation to get started.

> I was talking in abstract (MMM meaning any similar framework), but it's 
> a good question. To debug the problems, a concrete example is needed.

> The latest version of mmm-mode lives here: 
> https://github.com/purcell/mmm-mode.git, you can check out the master 
> branch, and it even has a manual that's reasonably up-to-date WRT to usage.

Yes, thanks.  Eli gave me that URL yesterday, so I've had a chance to
skim the manual and some of the code.

> Here are a couple of related bug reports:

> https://github.com/purcell/mmm-mode/issues/55
> https://github.com/purcell/mmm-mode/issues/57

> Both pertaining to the use of c-mode in the context of mmm-noweb.el.

> Please let me know if you need more localized examples.

> > I've no idea how long it will take to make MMM and
> > CC Mode work with eachother.

As a first approximation, dealing with CC Mode's need to widen might
help.  (I don't yet know how that will work for when several CC Mode
chunks are embedded in a single main buffer (?local variables), but that
should become clearer as time goes by.)

In my quick skim of MMM Mode's code, I didn't see any way of accessing
all the various pertinent "point-min"s: there's the point-min set by the
user, the point-min set by MMM Mode to restrict operations to the
current chunk, and there's the point-min set by (e.g.) CC Mode for its
own purposes.  The same applies (perhaps a bit less so) to all the
"point-max"es.

Wasn't `prog-widen' supposed to get the MMM Mode's chunk boundaries?
What are the recognised ways in MMM Mode of getting these three pairs of
(point-min point-max)?

> If you manage to do it at all, that would be an achievement. But just 
> figuring out the biggest problems should help us to design the feature.

I'm thinking about first adapting all the various CC Mode caches which
implicitly or explicitly start at 1, each to have a "cache-min" position
which would coincide with the chunk-start.  Maybe I could use some
mechanism to avoid invalidating them when a buffer change before the CC
Mode chunk changes this "cache-min" position.

> > Not much, if at all.  I found those discussions to be lacking the
> > context I would have needed to understand them.

> I hope the context is apparent now.

Much more so, yes.

> > That's fair.  But the "syntax islands" proposal would be a lot of work,
> > which I don't want to commit to before I see some sort of undertaking to
> > take it seriously.

> Then it should be a good idea to try the proposed, smaller approach, and 
> collect the drawbacks you meet along the way.

That sounds like the way to go, yes.

[ .... ]

> Lots of cats around here in Cyprus. ;-)

Ah, Cyprus!  I'll bet it's a deal warmer than Germany at the moment!

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

On December 5, 2017 6:54:59 PM GMT+00:00, Dmitry Gutov <dgutov@yandex.ru> wrote:
> On 12/5/17 7:57 PM, Eli Zaretskii wrote:
> 
> > I posted a more or less complete proposal for how to make the
> changes you want without breaking previous facilities.  If you agree
> with that proposal (which IMO makes sense even for master), then most
> of the issues you raise disappear.  If you disagree, please respond to
> that message of mine explaining why, and let's discuss that first.
> 
> Sorry, is it this message?
> 
> http://lists.gnu.org/archive/html/emacs-devel/2017-12/msg00158.html
> 
> I thought I've addresses every point in there (that I could make sense
> 
> of) in other messages in this thread, but if you prefer, I can respond
> 
> to it directly, to put the critique in one place.
> 
> Have you read the following messages, BTW? There were no responses.
> 
> http://lists.gnu.org/archive/html/emacs-devel/2017-12/msg00169.html
> http://lists.gnu.org/archive/html/emacs-devel/2017-12/msg00170.html

Yes, please respond to that message directly so that we could have all the points together.
To make myself clear: I'd like to see your changes land in a way that as much as possible doesn't remove prog-indentation-context except where there's a clear and present contradiction.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

On December 5, 2017 6:40:00 PM GMT+00:00, Dmitry Gutov <dgutov@yandex.ru> wrote:
> On 12/5/17 7:53 PM, Eli Zaretskii wrote:
> 
> > You assume that the indentation function has the same information
> about the higher-level context as its callers.  But that assumption is
> not necessarily true; narrowing exists precisely because higher-level
> code wants to set up lower-level code without explicitly passing it
> this information.
> 
> Without MMM in the equation, the caller here is
> indent-for-tab-command.
> 
> It's an interactive function, and the only narrowing present before it
> 
> is the one possibly applied by the user, interactively.

Interactive functions can also be called from Lisp.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Ingo Lohmar]

On Tue, Dec 05 2017 20:48 (+0000), Eli Zaretskii wrote:

> To make myself clear: I'd like to see your changes land in a way that as much as possible doesn't remove prog-indentation-context except where there's a clear and present contradiction.

Previously I dabbled with mmm-mode and even maintained an unpublished
fork to fix some shortcomings.  Therefore I followed this and the
related threads with quite some interest.

Apologies if I misunderstood or if this is just noise (in which case
simply ignore my message), but I have only seen this point briefly
raised in passing, and it appears to be very important:

1. I gathered that from what is known now, prog-indentation-context
   cannot provide all that which multiple major mode frameworks
   need/want from Emacs core.

2. Also, it is not orthogonal, but rather incompatible with all other
   proposals (be they concrete or vague) discussed thus far.

So if we really want to facilitate mmm-like frameworks (which is
wonderful and surely a great thing to do), prog-indentation-context will
have to be removed whenever that happens.

Since it's not in any released version yet, it seems obvious to me that
it should be removed right now, or else it will be "in the wild".  And
then it will be much more awkward to remove it in the next release (and
I would assume that you will strongly object to removing it then).

What am I missing here?

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/5/17 10:49 PM, Eli Zaretskii wrote:

>> It's an interactive function, and the only narrowing present before it
>>
>> is the one possibly applied by the user, interactively.
> 
> Interactive functions can also be called from Lisp.

True, and that can be a reason to worry about stability (emacs-26 vs 
master).

I don't think it should affect the eventual design here, though.

Re: CC Mode in MMM Mode(s). [Was: Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls]

[Dmitry Gutov]

On 12/5/17 9:01 PM, Alan Mackenzie wrote:

> As a first approximation, dealing with CC Mode's need to widen might
> help.  (I don't yet know how that will work for when several CC Mode
> chunks are embedded in a single main buffer (?local variables), but that
> should become clearer as time goes by.)

As far as I'm conncerned, CC Mode can consider every chunk a separate 
buffer. It might be suboptimal for some features, but let's take one 
step at a time.

> In my quick skim of MMM Mode's code, I didn't see any way of accessing
> all the various pertinent "point-min"s: there's the point-min set by the
> user, the point-min set by MMM Mode to restrict operations to the
> current chunk, and there's the point-min set by (e.g.) CC Mode for its
> own purposes.  The same applies (perhaps a bit less so) to all the
> "point-max"es.

You use the current restriction as set when indent-line-function is 
called. Or font-lock-keywords are applied. You can stash those point-min 
and point-max dynamically for your convenience, too.

> Wasn't `prog-widen' supposed to get the MMM Mode's chunk boundaries?

Kind of. It only returned those boundaries when inside 
indent-line-function, too.

> What are the recognised ways in MMM Mode of getting these three pairs of
> (point-min point-max)?

For the ones "set by MMM Mode", you can call point-min and point-max at 
the beginning of indent-line-function. Again, you can stash it for later.

Of restrictions set by CC Mode I know nothing about.

And as for point-min set by the user, why do you want to know? 
indent-for-tab-command should have taken care of widening by that time.

>> If you manage to do it at all, that would be an achievement. But just
>> figuring out the biggest problems should help us to design the feature.
> 
> I'm thinking about first adapting all the various CC Mode caches which
> implicitly or explicitly start at 1, each to have a "cache-min" position
> which would coincide with the chunk-start.

That sounds like a plan. Maybe also avoid using some of the caches when 
the length of a chunk is smaller than a predefined value.

> Maybe I could use some
> mechanism to avoid invalidating them when a buffer change before the CC
> Mode chunk changes this "cache-min" position.

The overhead of such solution would probably be N(number of chunks), 
wouldn't it?

> Ah, Cyprus!  I'll bet it's a deal warmer than Germany at the moment!

Indeed. About 15 degrees warmer. :-)

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

On December 5, 2017 9:08:56 PM GMT+00:00, Ingo Lohmar <i.lohmar@gmail.com> wrote:
> On Tue, Dec 05 2017 20:48 (+0000), Eli Zaretskii wrote:
> 
> > To make myself clear: I'd like to see your changes land in a way
> that as much as possible doesn't remove prog-indentation-context
> except where there's a clear and present contradiction.
> 
> Previously I dabbled with mmm-mode and even maintained an unpublished
> fork to fix some shortcomings.  Therefore I followed this and the
> related threads with quite some interest.
> 
> Apologies if I misunderstood or if this is just noise (in which case
> simply ignore my message), but I have only seen this point briefly
> raised in passing, and it appears to be very important:
> 
> 1. I gathered that from what is known now, prog-indentation-context
>    cannot provide all that which multiple major mode frameworks
>    need/want from Emacs core.
> 
> 2. Also, it is not orthogonal, but rather incompatible with all other
>    proposals (be they concrete or vague) discussed thus far.
> 
> So if we really want to facilitate mmm-like frameworks (which is
> wonderful and surely a great thing to do), prog-indentation-context
> will
> have to be removed whenever that happens.
> 
> Since it's not in any released version yet, it seems obvious to me
> that
> it should be removed right now, or else it will be "in the wild".  And
> then it will be much more awkward to remove it in the next release
> (and
> I would assume that you will strongly object to removing it then).
> 
> What am I missing here?

My conclusions from reviewing the discussions and the proposed code changes, as they appear on Dmitry's branch, were that prog-indentation-context is a super-set of what is needed for MMM and similar frameworks.  As I understand, Dmitry submits that the extra features are not needed and have never been used, but I haven't yet heard any arguments to the effect that prog-indentation-context _contradicts_ MMM.  Which is why I still think it'd be prudent not to remove it, at least not yet, because the extra features it provides might be useful, and because the code is part of the development sources for more than 2 years.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

On December 5, 2017 11:16:08 PM GMT+00:00, Dmitry Gutov <dgutov@yandex.ru> wrote:
> On 12/5/17 10:49 PM, Eli Zaretskii wrote:
> 
> >> It's an interactive function, and the only narrowing present before
> it
> >>
> >> is the one possibly applied by the user, interactively.
> > 
> > Interactive functions can also be called from Lisp.
> 
> True, and that can be a reason to worry about stability (emacs-26 vs 
> master).
> 
> I don't think it should affect the eventual design here, though.

My point is that this aspect of the design should be discussed before we make the code changes.  (I think this aspect is separate from the rest of MMM related discussions.)

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/6/17 11:28 AM, Eli Zaretskii wrote:

> My point is that this aspect of the design should be discussed before we make the code changes.

Let's go ahead and discuss it then?

Short overview:

- We have indentation functions that widen (some major modes), and 
indentation function that don't (where the authors didn't bother, or, in 
one place, where I personally removed it, from js-mode, after a 
MMM-related discussion).

- IME, whenever the question of narrowing comes up, it's usually from a 
user who wants command X to (save-restriction (widen) ...) before doing 
what it always did. Never to avoid calling widen, which makes sense from 
a perspective of a user who uses interactive narrowing.

Hence, it seems to make sense that indent-for-tab-command should always 
operate of a widened buffer.

Your turn.

 > (I think this aspect is separate from the rest of MMM related 
discussions.)

Not exactly separate. My proposal hinges on it somewhat. In particular, 
on the idea that all major modes should be treated the same in this respect.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/5/17 10:48 PM, Eli Zaretskii wrote:

> Yes, please respond to that message directly so that we could have all the points together.
> To make myself clear: I'd like to see your changes land in a way that as much as possible doesn't remove prog-indentation-context except where there's a clear and present contradiction.

Right. IMO, that's impossible to do in any useful way.

I'll respond to that message in full today.

Re: CC Mode in MMM Mode(s).

[Alan Mackenzie]

Hello, Dmitry.

On Wed, Dec 06, 2017 at 01:34:40 +0200, Dmitry Gutov wrote:
> On 12/5/17 9:01 PM, Alan Mackenzie wrote:

> > As a first approximation, dealing with CC Mode's need to widen might
> > help.  (I don't yet know how that will work for when several CC Mode
> > chunks are embedded in a single main buffer (?local variables), but that
> > should become clearer as time goes by.)

> As far as I'm conncerned, CC Mode can consider every chunk a separate 
> buffer. It might be suboptimal for some features, but let's take one 
> step at a time.

I think I now understand.  Each chunk is a separate overlay, and each
overlay maintains its own list of local variables.

> > In my quick skim of MMM Mode's code, I didn't see any way of accessing
> > all the various pertinent "point-min"s: there's the point-min set by the
> > user, the point-min set by MMM Mode to restrict operations to the
> > current chunk, and there's the point-min set by (e.g.) CC Mode for its
> > own purposes.  The same applies (perhaps a bit less so) to all the
> > "point-max"es.

> You use the current restriction as set when indent-line-function is 
> called. Or font-lock-keywords are applied. You can stash those point-min 
> and point-max dynamically for your convenience, too.

As a matter of interest, how does MMM mode handle its chunks for
commands which aren't indentation (such as beginning-of-defun)?

> > Wasn't `prog-widen' supposed to get the MMM Mode's chunk boundaries?

> Kind of. It only returned those boundaries when inside 
> indent-line-function, too.

OK.

> > What are the recognised ways in MMM Mode of getting these three pairs of
> > (point-min point-max)?

> For the ones "set by MMM Mode", you can call point-min and point-max at 
> the beginning of indent-line-function. Again, you can stash it for later.

> Of restrictions set by CC Mode I know nothing about.

> And as for point-min set by the user, why do you want to know? 

Because CC Mode's "state cache" (bad name, I know), which records open
parens/braces/brackets preceding point, records those things only in the
accessible portion of the buffer.  If that portion started within a CC
Mode chunk, things might/would go wrong if adjustments weren't made.

> indent-for-tab-command should have taken care of widening by that time.

Perhaps a function to return (user-point-min . user-point-max) might be
useful.

> >> If you manage to do it at all, that would be an achievement. But just
> >> figuring out the biggest problems should help us to design the feature.

> > I'm thinking about first adapting all the various CC Mode caches which
> > implicitly or explicitly start at 1, each to have a "cache-min" position
> > which would coincide with the chunk-start.

> That sounds like a plan. Maybe also avoid using some of the caches when 
> the length of a chunk is smaller than a predefined value.

That's an idea, but it would add (a little) complexity.

> > Maybe I could use some
> > mechanism to avoid invalidating them when a buffer change before the CC
> > Mode chunk changes this "cache-min" position.

> The overhead of such solution would probably be N(number of chunks), 
> wouldn't it?

Yes.  But it would be less than the overhead of recalculating the caches
after a buffer change within the CC Mode chunk, since it would just need
to add a constant offset onto each buffer position in the caches.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/4/17 7:40 PM, Eli Zaretskii wrote:
>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Cc: dgutov@yandex.ru,  emacs-devel@gnu.org
>> Date: Mon, 04 Dec 2017 12:12:48 -0500
>>
>>> The code which supports PREV-CHUNK is already in Emacs, so the
>>> question IMO is "why remove it"?
>>
>> ?Which code is that?
> 
> prog-indentation-context and its support in prog-widen.

To reiterate:

Yes, this code has been in Emacs source tree for two years. But:

It's only used is one line in antlr-mode.el, which wasn't supposed to 
start using it before Emacs 26 was released anyway. But it's easy to change.

On the flip side, prog-indentation-context is *only* well-supported in 
python-mode. Not in css-mode, js-mode, ruby-mode or others. In those, 
you still have to combine its binding with narrow-to-region, essentially 
obviating the need for the second element in the list. In short: we have 
the code, and even Emacs, after the said 2 years, isn't supporting it 
properly.

PREVIOUS-CHUNKS isn't supported anywhere either. And, in that case, 
there's no code corresponding to it. Just a few pieces of documentation.

>>> If we want to try to get this into
>>> Emacs 26, we should strive to make the code changes minimal, ideally
>>> zero.  Once all we are left with are documentation changes, the
>>> feature can land on emacs-26 right now.
>>
>> Zero is not the intention: for the doc changes to be valid, we need to
>> add a few `widen` calls in places like indent-according-to-mode.
> 
> If those calls are conditioned on MMM actually being active, then
> existing behavior will remain unchanged, and we are good.

No, the widen calls are conditioned on whether we want all 
indent-line-function calls to start in fully widened state. Which seems 
like an obvious improvement. Point is, they won't be allowed to call 
'widen' themselves, so the addition of 'widen' to indent-for-tab-command 
and indent-according-to-mode makes them act on the whole buffer in the 
simple case.

> What I see in the branch is this:
> 
>    1) the calls to prog-widen are replaced with calls to widen.

No, they are not. The calls to prog-widen (or widen) made in indentation 
related code are removed.

The only place in the diff where prog-widen was replaced with widen, is 
where it probably shouldn't have been in the first place (that code is 
not inside indent-line-function; only tangentially related).

>    2) some calls to widen are added

In code that later either calls indent-line-function or 
beginning-of-defun-function.

>    3) prog-indentation-context is removed

Yup.

>    4) prog-first-column the function is removed, and its calls replaced
>       with accessing the (existing) name-sake variable

Yes. It's not a hard requirement, but there doesn't seem much benefit in 
keeping it a function. And if it's a function, what will it return?

> For 4), we already agreed that keeping a function is better.

I did not. Allow me to post a quote from my other message:

If we keep it a function, do we also have a variable prog-first-column?

Then, some consumers might have doubt which ones to use, and might opt 
to reference the variable. In that case, we lose all benefits of having 
it a function (like making its implementation somehow smarter later), 
and the only benefit remaining is backward compatibility of having a 
function with that name.

But! Like with PREV-CHUNKS, prog-first-column (and the later prog-widen) 
are supposed to be used by the major modes only. There are no references 
to either of these functions in the latest antlr-mode.el, and there 
shouldn't be.

And as long as all the calls to that function are inside Emacs, we're 
free to change it however, including turning it into a variable.

> Since prog-widen already calls widen if prog-indentation-context is
> nil (which it is by default), calling prog-widen without setting up
> prog-indentation-context will just call widen; this magically takes
> care of 1).
> 
> For 3), if we leave prog-indentation-context in the code, and also
> allow direct calls to widen in modes which don't want to use the
> context, we are not losing anything, while leaving the option of using
> the context to those modes which will want that.  This currently
> cannot be used by MMM (AFAIU), but other features which need embedded
> code, such as ANTLR, could still use it, and even MMM will be able to
> do that if it is extended.

As I hopefully described, the above does not make sense, unfortunately.

> 2) can be taken care of as indicated above, thus avoiding any possible
> effects on existing behaviors when MMM is not active.

Not sure I understand you here.

Re: CC Mode in MMM Mode(s).

[Dmitry Gutov]

On 12/6/17 8:19 PM, Alan Mackenzie wrote:

> I think I now understand.  Each chunk is a separate overlay, and each
> overlay maintains its own list of local variables.

Pretty much. With the exception that overlays can nest, and so hunk can 
be only a part of an overlay.

For font-lock, that is (because it's usually the least 
context-dependent, so that seems the least error-prone).

For indentation, however, we might narrow to the bounds of the current 
overlay, even if there are some nested ones inside it. We don't actually 
do that in the general case (random combination of modes), though. I've 
only really worked on custom indentation for html-erb-mode (defined in 
mmm-erb.el).

> As a matter of interest, how does MMM mode handle its chunks for
> commands which aren't indentation (such as beginning-of-defun)?

It doesn't. For beginning-of-defun, it should likewise define its own 
beginning-of-defun-function which narrows and delegates (like usual), 
but that never actually came up in user requests so far.

The other, non-customizable commands (for example major mode specific 
ones) work on best effort basis. It's not an exact science. :-)

More importantly, mmm-mode also applies font-lock-keywords also by 
narrowing, like mentioned above. IIRC, some code called in 
font-lock-keywords in CC Mode widens, so that would have to be taken 
care of. And it's even more important than indentation, I'd say.

>> And as for point-min set by the user, why do you want to know?
> 
> Because CC Mode's "state cache" (bad name, I know), which records open
> parens/braces/brackets preceding point, records those things only in the
> accessible portion of the buffer.  If that portion started within a CC
> Mode chunk, things might/would go wrong if adjustments weren't made.

That's, um, why? Sounds like a very niche optimization.

Still, that cache might also be keyed on point-min, can't it?

>> indent-for-tab-command should have taken care of widening by that time.
> 
> Perhaps a function to return (user-point-min . user-point-max) might be
> useful.

I... don't really want to do that, but we might add that to the next 
version of the multi-mode API. Can you do without it in the meantime?

>> That sounds like a plan. Maybe also avoid using some of the caches when
>> the length of a chunk is smaller than a predefined value.
> 
> That's an idea, but it would add (a little) complexity.

It might not be helpful at all anyway. syntax-ppss derives some benefit 
from it, but that's because parse-partial-sexp is faster than running a 
bunch of Lisp code, for smaller chunks of text.

>>> Maybe I could use some
>>> mechanism to avoid invalidating them when a buffer change before the CC
>>> Mode chunk changes this "cache-min" position.
> 
>> The overhead of such solution would probably be N(number of chunks),
>> wouldn't it?
> 
> Yes.  But it would be less than the overhead of recalculating the caches
> after a buffer change within the CC Mode chunk, since it would just need
> to add a constant offset onto each buffer position in the caches.

Tradeoffs. I've worked on files with *lots* of Ruby chunks, but the 
situations where CC Mode will be used for inner chunks might be 
sufficiently different that it becomes less of a problem.

Another approach might be to put the caches into chunk-local variables, 
but I'm not sure if we handle before/after-change-function correctly 
when a change crosses chunk boundaries. Not sure how to implement this 
properly either.

Anyway, I'd devote as little time as possible to performance 
optimization right now, and see if we can get it working at all.

Re: CC Mode in MMM Mode(s).

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Why does MMM use overlays, rather than text properties?

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: CC Mode in MMM Mode(s).

[Dmitry Gutov]

On 12/7/17 9:49 PM, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
> Why does MMM use overlays, rather than text properties?

- No need to worry about putting the buffer into a modified state.

- Overlays expand when you insert some text inside. By default, mmm-mode 
doesn't re-parse the buffer into submode chunks automatically (probably 
because of performance concerns, more apparent back in the day), so that 
was probably useful.

- Overlays have properties. It's a more meaningful data structure, I 
guess. Easier to build with.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Wed, 6 Dec 2017 15:36:16 +0200
> 
> On 12/6/17 11:28 AM, Eli Zaretskii wrote:
> 
> > My point is that this aspect of the design should be discussed before we make the code changes.
> 
> Let's go ahead and discuss it then?

Not with me, with everyone else.  I already described my doubts.

I will start a new thread about this, I think this issue all but
drowned in this long discussion, and I'd expect some people to avoid
reading it.

>  > (I think this aspect is separate from the rest of MMM related 
> discussions.)
> 
> Not exactly separate. My proposal hinges on it somewhat. In particular, 
> on the idea that all major modes should be treated the same in this respect.

The separate issue is to widen unconditionally.  I could probably
agree to have these functions widen when MMM-like mode is active.

Re: CC Mode in MMM Mode(s).

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > - No need to worry about putting the buffer into a modified state.

I understand that.  Text properties are designed to be considered
part of the contents.  But font-lock uses them, so it is possible
to use them for things that reflect the contents and are not contents.

  > - Overlays expand when you insert some text inside. By default, mmm-mode 
  > doesn't re-parse the buffer into submode chunks automatically (probably 
  > because of performance concerns, more apparent back in the day), so that 
  > was probably useful.

Text properties expand, too.

  > - Overlays have properties. It's a more meaningful data structure, I 
  > guess. Easier to build with.

Do you mean, it is useful in the code to have a single overlay object
to put the properties on, rather than putting them directly on the
characters in the buffer?

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

Sorry for the delay in responding: 3 days of absence took their toll.

> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Wed, 6 Dec 2017 20:41:21 +0200
> 
> Yes, this code has been in Emacs source tree for two years. But:
> 
> It's only used is one line in antlr-mode.el, which wasn't supposed to 
> start using it before Emacs 26 was released anyway. But it's easy to change.
> 
> On the flip side, prog-indentation-context is *only* well-supported in 
> python-mode. Not in css-mode, js-mode, ruby-mode or others. In those, 
> you still have to combine its binding with narrow-to-region, essentially 
> obviating the need for the second element in the list. In short: we have 
> the code, and even Emacs, after the said 2 years, isn't supporting it 
> properly.
> 
> PREVIOUS-CHUNKS isn't supported anywhere either. And, in that case, 
> there's no code corresponding to it. Just a few pieces of documentation.

I think we are failing to communicate.  "Not supported anywhere" is
not a reason good enough in my book to remove something.  We have
there a feature that is a super-set of what you need for the MMM
related support, so a much better way forward is to leave that stuff,
since it should transparently support your use case, or almost so.
Arguing that there's a good reason for removing
prog-indentation-context would need to show how keeping it _precludes_
some of what you are trying to do, or at least makes it unreasonably
hard.

And please keep in mind that 2 years without any additional users is
not long enough to prove that this feature is useless.  Let's also
keep in mind that this feature was reviewed at the time and admitted
into Emacs, which means Stefan did think at least back then that it
could be useful in practice.  We should give people a chance before
deciding it is completely useless and worthy of removal.

> No, the widen calls are conditioned on whether we want all 
> indent-line-function calls to start in fully widened state. Which seems 
> like an obvious improvement. Point is, they won't be allowed to call 
> 'widen' themselves, so the addition of 'widen' to indent-for-tab-command 
> and indent-according-to-mode makes them act on the whole buffer in the 
> simple case.

Whether we should unconditionally call 'widen' there is now a subject
of a separate discussion.  For the purposes of this discussion, the
only issue that should matter is whether using 'prog-widen' instead of
'widen' in those places will get in the way of the MMM support.  I
think it doesn't.

> > What I see in the branch is this:
> > 
> >    1) the calls to prog-widen are replaced with calls to widen.
> 
> No, they are not. The calls to prog-widen (or widen) made in indentation 
> related code are removed.

I don't think this distinction is important.  I presumed that you are
removing those widen calls because you added similar calls on higher
levels, and didn't mention those removals.  If that's not so, please
point to specific parts of the patch where the widen calls are removed
without that assumption.

> The only place in the diff where prog-widen was replaced with widen, is 
> where it probably shouldn't have been in the first place (that code is 
> not inside indent-line-function; only tangentially related).

And I'm proposing to leave that prog-widen call intact, because by
default it's the same as calling 'widen' in the first place.  By
keeping the prog-widen call there, we make the change run lower risk
of being backward incompatible.

> >    2) some calls to widen are added
> 
> In code that later either calls indent-line-function or 
> beginning-of-defun-function.

This is about widening unconditionally, so it's now unrelated to the
current discussion.

> >    3) prog-indentation-context is removed
> 
> Yup.

And I see no reason for removing it, because if not set to any non-nil
value, it is harmless.  If you can explain why leaving it contradicts
support for MMM, please do.

> >    4) prog-first-column the function is removed, and its calls replaced
> >       with accessing the (existing) name-sake variable
> 
> Yes. It's not a hard requirement, but there doesn't seem much benefit in 
> keeping it a function. And if it's a function, what will it return?

I don't think it matters what it will return.  What matters is that
(a) keeping a function doesn't interfere with the MMM support in any
way I could see; (b) keeping a function makes the changes fully
backward-compatible; and (c) keeping a function will allow future
extensions where the value returned is not trivially taken from a
variable.

> > For 4), we already agreed that keeping a function is better.
> 
> I did not. Allow me to post a quote from my other message:
> 
> If we keep it a function, do we also have a variable prog-first-column?

We need to store the value somewhere, or have a way of recomputing it.
Whether that is done by keeping a variable, and whether that variable
is called the same as the function, are secondary issues; the primary
issue IMO is why remove a function that is already there, and is the
documented interface for accessing the value.  I'd again prefer
keeping the stuff unchanged, if just for the reason that this is how
the code was submitted 2 years ago, and this is how it was approved.
It's not the first time that we have a variable named the same as a
function.  But if there are good reason to keep the function, while
renaming the variable, I will probably agree to renaming the variable
much easier than I'd agree to removing the function.

> Then, some consumers might have doubt which ones to use, and might opt 
> to reference the variable. In that case, we lose all benefits of having 
> it a function (like making its implementation somehow smarter later), 
> and the only benefit remaining is backward compatibility of having a 
> function with that name.
> 
> But! Like with PREV-CHUNKS, prog-first-column (and the later prog-widen) 
> are supposed to be used by the major modes only. There are no references 
> to either of these functions in the latest antlr-mode.el, and there 
> shouldn't be.
> 
> And as long as all the calls to that function are inside Emacs, we're 
> free to change it however, including turning it into a variable.

These all are very weak reasons in my eyes.  Keeping the documented
interfaces stable and backward-compatible is a much stronger argument,
and IMO in this case it easily overcomes the above considerations.

> > Since prog-widen already calls widen if prog-indentation-context is
> > nil (which it is by default), calling prog-widen without setting up
> > prog-indentation-context will just call widen; this magically takes
> > care of 1).
> > 
> > For 3), if we leave prog-indentation-context in the code, and also
> > allow direct calls to widen in modes which don't want to use the
> > context, we are not losing anything, while leaving the option of using
> > the context to those modes which will want that.  This currently
> > cannot be used by MMM (AFAIU), but other features which need embedded
> > code, such as ANTLR, could still use it, and even MMM will be able to
> > do that if it is extended.
> 
> As I hopefully described, the above does not make sense, unfortunately.

You explained that you consider the design and implementation of
prog-indentation-context unnecessarily complex, and that a simpler
design and implementation would be more elegant, given the current
practices in related modes.  You have _not_ explained why keeping the
prog-indentation-context stuff would prevent us from supporting MMM
and similar features as easily as under your proposed changes.  If
there are such aspects of prog-indentation-context, please point them
out, because those are _the_only_ ones which would convince me to
remove prog-indentation-context.

> > 2) can be taken care of as indicated above, thus avoiding any possible
> > effects on existing behaviors when MMM is not active.
> 
> Not sure I understand you here.

That's the "should we unconditionally widen" issue, which I now took
to a separate thread.  It was also my attempt to keep the changes safe
enough to land them on the release branch.  So we can forget about
this part for the purposes of this discussion.

Thanks.

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> Date: Sat, 09 Dec 2017 12:47:04 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> > Then, some consumers might have doubt which ones to use, and might opt 
> > to reference the variable. In that case, we lose all benefits of having 
> > it a function (like making its implementation somehow smarter later), 
> > and the only benefit remaining is backward compatibility of having a 
> > function with that name.
> > 
> > But! Like with PREV-CHUNKS, prog-first-column (and the later prog-widen) 
> > are supposed to be used by the major modes only. There are no references 
> > to either of these functions in the latest antlr-mode.el, and there 
> > shouldn't be.
> > 
> > And as long as all the calls to that function are inside Emacs, we're 
> > free to change it however, including turning it into a variable.
> 
> These all are very weak reasons in my eyes.

Just to make myself clear: these reasons would have been very relevant
and serious if we were now considering the prog-indentation-context
stuff for inclusion in Emacs.  But we are 2 years past that stage, so
other considerations take precedence, I think.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/8/17 6:41 PM, Eli Zaretskii wrote:

> The separate issue is to widen unconditionally.  I could probably
> agree to have these functions widen when MMM-like mode is active.

While all indent-line-functions stop calling 'widen'? Or will then also 
have to do that conditionally?

Re: CC Mode in MMM Mode(s).

[Dmitry Gutov]

On 12/8/17 11:36 PM, Richard Stallman wrote:

> I understand that.  Text properties are designed to be considered
> part of the contents.  But font-lock uses them, so it is possible
> to use them for things that reflect the contents and are not contents.

Yes. I'm not saying it's not possible, just describing the probable 
reasons for why this feature has been implemented this way.

> Text properties expand, too.

Oh. Okay.

> Do you mean, it is useful in the code to have a single overlay object
> to put the properties on, rather than putting them directly on the
> characters in the buffer?

I think so. But it's not something I've discussed before, or considered 
for a long time.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 9 Dec 2017 17:17:34 +0200
> 
> On 12/8/17 6:41 PM, Eli Zaretskii wrote:
> 
> > The separate issue is to widen unconditionally.  I could probably
> > agree to have these functions widen when MMM-like mode is active.
> 
> While all indent-line-functions stop calling 'widen'? Or will then also 
> have to do that conditionally?

If a particular indent-line-function needs to widen, then yes, it will
need to do that by itself, and if that gets in the way of MMM, then
the widening will have to be conditional.

(I thought that prog-widen and prog-indentation-context were a means
to solve this, isn't that so?)

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[John Wiegley]

>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:

EZ> And please keep in mind that 2 years without any additional users is not
EZ> long enough to prove that this feature is useless. Let's also keep in mind
EZ> that this feature was reviewed at the time and admitted into Emacs, which
EZ> means Stefan did think at least back then that it could be useful in
EZ> practice. We should give people a chance before deciding it is completely
EZ> useless and worthy of removal.

I agree with Eli here. It's very hard for any one person, with their own
particular way of using Emacs, and communication channels they pay attention
to, to know for certain what other people are using a feature for -- or if
they're using it at all. Lest we forget:

    https://xkcd.com/1172/

So unless there's a more compelling reason, like that it's getting in your
way, we'd need a stronger argument to reverse an agreed-upon decision that's
only 2 years old.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

> Just to make myself clear: these reasons would have been very relevant
> and serious if we were now considering the prog-indentation-context
> stuff for inclusion in Emacs.  But we are 2 years past that stage, so
> other considerations take precedence, I think.

Could we just mark them as obsolete in Emacs-26?
That would be good enough for me.


        Stefan

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: dgutov@yandex.ru,  emacs-devel@gnu.org
> Date: Sun, 10 Dec 2017 00:01:32 -0500
> 
> > Just to make myself clear: these reasons would have been very relevant
> > and serious if we were now considering the prog-indentation-context
> > stuff for inclusion in Emacs.  But we are 2 years past that stage, so
> > other considerations take precedence, I think.
> 
> Could we just mark them as obsolete in Emacs-26?

Doesn't sound like a good idea to me: something that was introduced
recently and barely had any chance to be used doesn't fall under the
"obsolete" label for me.

What's the damage of having it with us for a while, and using it in
our own sources?

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/9/17 5:43 PM, Eli Zaretskii wrote:

> If a particular indent-line-function needs to widen, then yes, it will
> need to do that by itself, and if that gets in the way of MMM, then
> the widening will have to be conditional.

That sounds surprisingly more complex than the whole solution that I'm 
proposing.

> (I thought that prog-widen and prog-indentation-context were a means
> to solve this, isn't that so?)

It was one of the goals, but that approach comes with certain inherent, 
as well as certain incidental, drawbacks.

Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 10 Dec 2017 21:59:45 +0200
> 
> On 12/9/17 5:43 PM, Eli Zaretskii wrote:
> 
> > If a particular indent-line-function needs to widen, then yes, it will
> > need to do that by itself, and if that gets in the way of MMM, then
> > the widening will have to be conditional.
> 
> That sounds surprisingly more complex than the whole solution that I'm 
> proposing.

I'm surprised that you think this to be so much more complex.

But in any case, this is the subject of a separate thread now, so I
suggest to discuss there, to avoid people who are interested missing
this.

> > (I thought that prog-widen and prog-indentation-context were a means
> > to solve this, isn't that so?)
> 
> It was one of the goals, but that approach comes with certain inherent, 
> as well as certain incidental, drawbacks.

It looks sufficiently lightweight to me, so I hope the drawbacks are
not unbearable.

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

> What's the damage of having it with us for a while, and using it in
> our own sources?

Because using narrowing instead of using a variable will make things
simpler, and having 2 competing standards to support MMM is a bad idea.


        Stefan

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

Hi John!

Glad you could join us.

On 12/9/17 7:56 PM, John Wiegley wrote:
>>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
> 
> EZ> And please keep in mind that 2 years without any additional users is not
> EZ> long enough to prove that this feature is useless. Let's also keep in mind
> EZ> that this feature was reviewed at the time and admitted into Emacs, which
> EZ> means Stefan did think at least back then that it could be useful in
> EZ> practice. We should give people a chance before deciding it is completely
> EZ> useless and worthy of removal.
> 
> I agree with Eli here. It's very hard for any one person, with their own
> particular way of using Emacs, and communication channels they pay attention
> to, to know for certain what other people are using a feature for -- or if
> they're using it at all.

What "one person"? The expected uptick in support and users I'm talking 
about would have to be inside Emacs. Because that's where the most 
popular and important major modes are.

And yet, nothing happened since the original feature's addition.

> Lest we forget:
> 
>      https://xkcd.com/1172/
> 
> So unless there's a more compelling reason, like that it's getting in your
> way, we'd need a stronger argument to reverse an agreed-upon decision that's
> only 2 years old.

Let me remind you of the previous (shorter) discussion on this subject:

http://lists.gnu.org/archive/html/emacs-devel/2016-03/msg01425.html

A "better answer", if I do say so myself, has been proposed. Even 
Stefan, who objected to the original removal, is onboard. And yet, it's 
not being accepted.

So tell me, please, which of these "agreed-upon decisions" are we going 
to reverse? One that Stefan made and would now prefer to go back on, or 
the one that you made yourself?

(Sorry for getting all political, but the amount of messages in this 
thread is getting ridiculous. I did not sign up for this much wasted time.)

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

> And yet, nothing happened since the original feature's addition.

Actually, something *did* happen: Emacs did integrate such
a multi-major-mode mode (mhtml-mode) in the mean time, and what did
it do?

It didn't use the prog-indentation-context approach but relied on the
narrowing approach suggested by Dmitry.


        Stefan

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 10 Dec 2017 22:19:25 +0200
> 
> http://lists.gnu.org/archive/html/emacs-devel/2016-03/msg01425.html
> 
> A "better answer", if I do say so myself, has been proposed. Even 
> Stefan, who objected to the original removal, is onboard. And yet, it's 
> not being accepted.
> 
> So tell me, please, which of these "agreed-upon decisions" are we going 
> to reverse? One that Stefan made and would now prefer to go back on, or 
> the one that you made yourself?
> 
> (Sorry for getting all political, but the amount of messages in this 
> thread is getting ridiculous. I did not sign up for this much wasted time.)

Forget all the messages.  Forget the politics.  There are only 2
questions that need to be answered: Can you modify your branch such
that it doesn't remove prog-indentation-context, doesn't replace
prog-widen with widen, and leaves prog-first-column as a function?
And if so, can the resulting changeset allow reasonable MMM-style
multi-mode support in Emacs?

If the answer to both questions is YES, then we have a clear way
forward, and can stop talking.

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ab846: Replace prog-widen with consolidating widen calls

[John Wiegley]

>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:

EZ> Forget all the messages. Forget the politics. There are only 2 questions
EZ> that need to be answered: Can you modify your branch such that it doesn't
EZ> remove prog-indentation-context, doesn't replace prog-widen with widen,
EZ> and leaves prog-first-column as a function? And if so, can the resulting
EZ> changeset allow reasonable MMM-style multi-mode support in Emacs?

Thanks for the pointer to past discussions Dmitry, that's very helpful. In
this case, though, I'd like to second Eli and have us just move forward. I'm a
bit less opposed to "putting it out there" as I was when I wrote the message
you linked to.

--
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/10/17 10:38 PM, Stefan Monnier wrote:

> It didn't use the prog-indentation-context approach but relied on the
> narrowing approach suggested by Dmitry.

To be precise, it did use prog-indentation-context for its first element 
(prog-first-column), but used "my" approach in the contented area.

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ab846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/10/17 11:39 PM, John Wiegley wrote:
>>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
> 
> EZ> Forget all the messages. Forget the politics. There are only 2 questions
> EZ> that need to be answered: Can you modify your branch such that it doesn't
> EZ> remove prog-indentation-context, doesn't replace prog-widen with widen,
> EZ> and leaves prog-first-column as a function? And if so, can the resulting
> EZ> changeset allow reasonable MMM-style multi-mode support in Emacs?
> 
> Thanks for the pointer to past discussions Dmitry, that's very helpful.

Helpful for what?

> In
> this case, though, I'd like to second Eli and have us just move forward.

"move forward" sounds nice, except here it means "keep whatever code we 
already have and hope for the best". Without, you know, any planned 
steps to perform that forward movement.

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/10/17 10:39 PM, Eli Zaretskii wrote:

> Forget all the messages.  Forget the politics.  There are only 2
> questions that need to be answered: Can you modify your branch such
> that it doesn't remove prog-indentation-context, doesn't replace
> prog-widen with widen, and leaves prog-first-column as a function?

You seem to be asking to "modify your branch to a nil changeset".

I repeat: the branch does not "replace prog-widen with widen". It 
removes prog-widen, and all but one of its invocations, entirely.

prog-first-column a function, sure. But it has to be backed by some 
data. prog-indentation-context, remaining purely for its first element? 
We can do that, if you'd insist, though I hardly see the point from 
either perspective.

So, yes and no, I'm not sure how to keep prog-indentation-context around 
in a meaningful way (without changing it incompatibly).

> And if so, can the resulting changeset allow reasonable MMM-style
> multi-mode support in Emacs?

Having said the above, not sure how to respond here.

> If the answer to both questions is YES, then we have a clear way
> forward, and can stop talking.

I think you have asked these questions before, and I answered them, too.

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/9/17 12:47 PM, Eli Zaretskii wrote:

> I think we are failing to communicate.  "Not supported anywhere" is
> not a reason good enough in my book to remove something.

Allow me to clarify:

"Not supported anywhere" is, in my mind, a reason to disregard the "2 
years" qualifier. It's important to keep around code that's likely to be 
already used by someone, and it's barely the case here.

And the reasons to remove are different ones, and more technical, most 
of which have been mentioned in this thread already. And we discussed 
PREVIOUS-CHUNKS before, e.g. in the older thread I just linked to in a 
reply to John's email.

In addition to that, "not supported anywhere" can be a reason to remove 
PREVIOUS-CHUNKS in particular, because in the original discussion where 
they've been added, it was understood that the author will follow up 
with corresponding support for them (to prove the design), which he 
never did.

> We have
> there a feature that is a super-set of what you need for the MMM
> related support,

Not really, no.

> Arguing that there's a good reason for removing
> prog-indentation-context would need to show how keeping it _precludes_
> some of what you are trying to do, or at least makes it unreasonably
> hard.

Yes, it precludes us from removing prog-widen and widen calls from 
indentation code.

> And please keep in mind that 2 years without any additional users is
> not long enough to prove that this feature is useless.  Let's also
> keep in mind that this feature was reviewed at the time and admitted
> into Emacs, which means Stefan did think at least back then that it
> could be useful in practice.

It was predicated on some support coming later. Which, again, never did.

 > We should give people a chance before
 > deciding it is completely useless and worthy of removal.

What kind of chance? The support for it needed to be added *inside* Emacs.

It's not "completely" useless. prog-first-column was a good idea, but we 
have a better idea about the other parts.

> Whether we should unconditionally call 'widen' there is now a subject
> of a separate discussion. 

And why is that "separate discussion" not considered finished and 
concluded yet? I think the result is a fairly solid "yes, we can".

> For the purposes of this discussion, the
> only issue that should matter is whether using 'prog-widen' instead of
> 'widen' in those places will get in the way of the MMM support.

Unfortunately, that's not how the issue stands. Not calling widen in 
indentation functions matters.

>> No, they are not. The calls to prog-widen (or widen) made in indentation
>> related code are removed.
> 
> I don't think this distinction is important.  I presumed that you are
> removing those widen calls because you added similar calls on higher
> levels, and didn't mention those removals.  If that's not so, please
> point to specific parts of the patch where the widen calls are removed
> without that assumption.

Sorry, didn't mention where?

Indeed, widen calls are moved to a higher level above the 
indent-line-function funcall. That's a design which 
prog-indentation-context cannot support because prog-indentation-context 
is supposed to be bound inside MMM-specific indentation function.

> And I'm proposing to leave that prog-widen call intact, because by
> default it's the same as calling 'widen' in the first place.

Which means you're actively arguing against the core change proposed in 
the scratch/widen-less branch.

>>>     2) some calls to widen are added
>>
>> In code that later either calls indent-line-function or
>> beginning-of-defun-function.
> 
> This is about widening unconditionally, so it's now unrelated to the
> current discussion.

If those 'widen' calls are not added, interactive narrowing by the user 
will affect indentation results in the usual case. Which is probably not 
what we want to do.

>>>     3) prog-indentation-context is removed
>>
>> Yup.
> 
> And I see no reason for removing it, because if not set to any non-nil
> value, it is harmless.  If you can explain why leaving it contradicts
> support for MMM, please do.

Using prog-widen contradicts it. And if we can't promise that it will be 
used in all indentation functions (or at least those that purport to 
support MMM), prog-indentation-context, as documented, will be confusing 
and fairly useless.

>>>     4) prog-first-column the function is removed, and its calls replaced
>>>        with accessing the (existing) name-sake variable
>>
>> Yes. It's not a hard requirement, but there doesn't seem much benefit in
>> keeping it a function. And if it's a function, what will it return?
> 
> I don't think it matters what it will return.

If matters for me, because I want to know exactly what to write to make 
you happy on this issue. prog-first-column can be a function, but the 
choice of its backing needs to be made, and as I could see, my train of 
thought on that issue has not arrived at an any particular conclusion.

 > What matters is that
 > (a) keeping a function doesn't interfere with the MMM support in any
 > way I could see; (b) keeping a function makes the changes fully
 > backward-compatible; and (c) keeping a function will allow future
 > extensions where the value returned is not trivially taken from a
 > variable.

I concede points a and c.

>> If we keep it a function, do we also have a variable prog-first-column?
> 
> > But if there are good reason to keep the function, while
> renaming the variable, I will probably agree to renaming the variable
> much easier than I'd agree to removing the function.

It won't be just a rename, though, right? The prog-first-column will 
store only the first element of what was prog-indentation-context.

>> And as long as all the calls to that function are inside Emacs, we're
>> free to change it however, including turning it into a variable.
> 
> These all are very weak reasons in my eyes.  Keeping the documented
> interfaces stable and backward-compatible is a much stronger argument,
> and IMO in this case it easily overcomes the above considerations.

Even the last one? I though it was pretty solid. But no matter, see above.

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/9/17 12:47 PM, Eli Zaretskii wrote:

> You explained that you consider the design and implementation of
> prog-indentation-context unnecessarily complex, and that a simpler
> design and implementation would be more elegant, given the current
> practices in related modes.  You have _not_ explained why keeping the
> prog-indentation-context stuff would prevent us from supporting MMM
> and similar features as easily as under your proposed changes.

I'm honestly might be forgetting some things, given the brain-numbing 
effects of discussions like this, but:

- We need to expand the scope of use of all elements of 
prog-indentation-context but the first to other facilities as well, such 
as font-lock, beginning-of-defun-function, etc, at which point the name 
of the variable kind of ceases to make sense.

Out of all these, font-lock is the most apparent to the user, and thus, 
arguably, the most important. It is, luckily, not an issue in most major 
modes, but it *is* an issue in CC Mode, which you said you wanted to see 
supported.

- PREVIOUS-CHUNKS, given how it's permissively documented now, will 
likely prevent some useful caching mechanisms inside major modes that 
will decide to use it (in particular the first option will, "A string 
containing text"). I also fail to see how one would write support for it 
without having to program two fairly different code paths.

- In case the discussion about moving 'widen' calls to 
indent-for-tab-command and indent-according-to-mode concludes 
affirmatively, we won't be able to use prog-widen there because 
prog-indentation-context gets set "later".

This issue can probably be avoided by dropping the backing variable and 
moving to a similar-but-different design with hooks like 
prog-first-column-function, prog-current-chunk-function, etc, but these 
details still need to be finalized, and for that I'm waiting for Alan's 
feedback on adapting CC Mode to MMM.

- Then, we'll go onto solving the less well-examined issues with our 
designer hands tied more strongly than they could have been, because 
we've committed to one particular API, one that's harder to smoothly 
migrate over from.

Next, WHAT HAPPENS IF WE SUPPORT PROG-INDENTATION-CONTEXT PROPERLY:

- All the major modes will have to be adapted to use prog-widen in 
indentation code, and maybe some other places they use 'widen' at now. 
This, in turn, will break their uses in mmm-mode, polymode, and whatever 
other MMM packages are out there (including mhtml-mode, but it's easy to 
fix).

- Thus, MMM frameworks will need to be updated, changing over from the 
current practice. Compatibility shims will need to be added in them, to 
work with both older and new Emacs.

So if you were worried that removing prog-indentation-context would 
cause inconvenience to antlr-mode (its last released version which 
jumped the gun with supporting prog-indentation-context), properly 
supporting prog-indentation-context will cause more breakage overall, in 
said MMM frameworks. Until they all get updated and all users install 
the new versions, by various means those updates are delivered.

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/11/17 1:30 AM, Dmitry Gutov wrote:

> Next, WHAT HAPPENS IF WE SUPPORT PROG-INDENTATION-CONTEXT PROPERLY:

"PROPERLY" meaning "merely adequately", of course: still without support 
for PREVIOUS-CHUNKS.

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Tom Tromey]

>>>>> "Dmitry" == Dmitry Gutov <dgutov@yandex.ru> writes:

Dmitry> On 12/10/17 10:38 PM, Stefan Monnier wrote:
>> It didn't use the prog-indentation-context approach but relied on the
>> narrowing approach suggested by Dmitry.

Dmitry> To be precise, it did use prog-indentation-context for its first
Dmitry> element (prog-first-column), but used "my" approach in the contented
Dmitry> area.

FWIW, I think I misunderstood the prog-indentation-context documentation
when doing this work.  But also, narrowing in mhtml seems mildly more
convenient; most likely the other approach would have been to narrow in
each sub-mode's indent-line-function.

Tom

Getting rid of prog-indentation-context

[Stefan Monnier]

>> Could we just mark them as obsolete in Emacs-26?
> Doesn't sound like a good idea to me: something that was introduced
> recently and barely had any chance to be used doesn't fall under the
> "obsolete" label for me.

Eli, I fail to understand what you expect prog-indentation-context will
be used for.

Here's what I'd like to see happen for Emacs-26:
- document the way major modes should behave in order to work properly
  in MMM contexts like mhtml-mode and mmm-mode.  Part of this doc says
  that indent-line-functions can assume that the buffer is already
  widened (to the current chunk or the whole buffer).
- change indent-according-to-mode to widen before calling
  indent-line-function (such that the above documentation is actually
  valid).
- get rid of prog-indentation-context or at least mark it as obsolete,
  since its 2nd element competes with the above changes and its 3rd
  element, while potentially useful, lacks even a proof-of-concept
  demonstration that it does what it purports to do.
- get rid of prog-widen.

To be honest, I could live with a version of prog-indentation-context
stripped down to its first element (which means: change its docstring
to say something like:

      "When non-nil, provides context for indenting embedded code chunks.
    
    There are languages where part of the code is actually written in
    a sub language, e.g., a Yacc/Bison or ANTLR grammar also consists
    of plain C code.  This variable enables the major mode of the
    main language to use the indentation engine of the sub-mode for
    lines in code chunks written in the sub-mode's language.
    
    When a major mode of such a main language decides to delegate the
    indentation of a line/region to the indentation engine of the sub
    mode, it should bind this variable to non-nil around the call.
    
    The non-nil value should be a list of the form:
    
       (FIRST-COLUMN . REST)
    
    FIRST-COLUMN is the column the indentation engine of the sub-mode
    should use for top-level language constructs inside the code
    chunk (instead of 0).
    
    REST is currently unused.")

We could also keep prog-widen as

    (defalias prog-widen #'ignore)

or a more verbose version of the same, if you insist.


        Stefan

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

> FWIW, I think I misunderstood the prog-indentation-context documentation
> when doing this work.  But also, narrowing in mhtml seems mildly more
> convenient; most likely the other approach would have been to narrow in
> each sub-mode's indent-line-function.

That's the idea: rather than have every indentation function do the
narrowing itself, it's done once and for all in the MMM (mhtml in this
case).  This way, most major modes will "magically" work without any
MMM-specific code.


        Stefan

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 10 Dec 2017 23:53:01 +0200
> 
> On 12/10/17 10:39 PM, Eli Zaretskii wrote:
> 
> > Forget all the messages.  Forget the politics.  There are only 2
> > questions that need to be answered: Can you modify your branch such
> > that it doesn't remove prog-indentation-context, doesn't replace
> > prog-widen with widen, and leaves prog-first-column as a function?
> 
> You seem to be asking to "modify your branch to a nil changeset".

I hope not.  I thought the goal of the branch was to allow Emacs
better support MMM and similar modes, and that removing/replacing
prog-indentation-context was just a means to that end.  But maybe I
was confused, and the primary goal was to remove
prog-indentation-context, while MMM support was just a nice side
effect, a bonus.  In that case, everything else you said makes perfect
sense.  I don't support such a goal, but at least I will understand
what this argument is about, so it would be a kind of progress from my
POV.

Re: Getting rid of prog-indentation-context

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Mon, 11 Dec 2017 09:18:42 -0500
> 
> >> Could we just mark them as obsolete in Emacs-26?
> > Doesn't sound like a good idea to me: something that was introduced
> > recently and barely had any chance to be used doesn't fall under the
> > "obsolete" label for me.
> 
> Eli, I fail to understand what you expect prog-indentation-context will
> be used for.

Stefan, you admitted this stuff into Emacs 2 years ago, so why are you
asking me these questions?  I'm sure you asked that yourself back
then, and I'm sure you had good answers.

> Here's what I'd like to see happen for Emacs-26:

How is this different from what began this thread quite some time ago?
Do we want to start all over again?

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/11/17 6:04 PM, Eli Zaretskii wrote:

> I hope not.  I thought the goal of the branch was to allow Emacs
> better support MMM and similar modes, and that removing/replacing
> prog-indentation-context was just a means to that end.

Improving MMM support while retaining prog-indentation-context is, of 
course, possible, though it won't reach all the goals I have set for 
this endeavor.

But that would be a very different branch, with a different name. I have 
described the required work and its consequences in a separate email 
already.

> But maybe I
> was confused, and the primary goal was to remove
> prog-indentation-context, while MMM support was just a nice side
> effect, a bonus.

I hope you are not being deliberately obtuse.

But sure, whatever, I also eat puppies for dinner.

Re: Getting rid of prog-indentation-context

[Stefan Monnier]

>> >> Could we just mark them as obsolete in Emacs-26?
>> > Doesn't sound like a good idea to me: something that was introduced
>> > recently and barely had any chance to be used doesn't fall under the
>> > "obsolete" label for me.
>> Eli, I fail to understand what you expect prog-indentation-context will
>> be used for.
> Stefan, you admitted this stuff into Emacs 2 years ago, so why are you
> asking me these questions?

Because you clearly think it's important to have this new variable in
Emacs-26.  The important word in my question is "will".
I don't expect an answer like "it will be used for mmm-support", but
rather, which specific packages do you expect will make use of it?

> I'm sure you asked that yourself back then, and I'm sure you had
> good answers.

I thought it would be used by updating "all" major modes to obey it,
because I thought that relying on narrowing was unworkable (because of
the usual ambiguities linked to narrowing's intentions).

I now understand that narrowing's ambiguities are resolved by locally
overriding the narrowing so it has a precise meaning.  and I also know
from experience that changing "all" major modes to obey
prog-indentation-context is not going to happen.

So, yes, we can keep prog-indentation-context in Emacs-26, but I still
expect that antlr-mode (the only current package which sets
prog-indentation-context) will sooner or later be changed to use
narrowing instead, so prog-indentation-context will be just a weird wart
whose only effect will be to confuse some people.

Note also antlr-mode will work just as well without Emacs defining
prog-indentation-context (it currently tests (boundp
'prog-indentation-context), but it only ever let-binds it, so the
boundp test is not needed).

>> Here's what I'd like to see happen for Emacs-26:
> How is this different from what began this thread quite some time ago?
> Do we want to start all over again?

I'm not sure how the past discussions have changed people's positions on
those different points.  So I don't want to "start over", but rather
I want to see what do we stand now.

Also past discussion have changed my position to some extent which is
reflected in my above presentation of the issues (e.g. mentioning that
we could keep the first element of prog-indentation-context).


        Stefan

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

> I hope not.  I thought the goal of the branch was to allow Emacs
> better support MMM and similar modes, and that removing/replacing
> prog-indentation-context was just a means to that end.

There are may ways to be "better".  The claim is that using narrowing
instead of (nth 1 prog-indentation-context) will make mmm support
applicable in more circumstances because it reduces the amount of
changes needed in each major mode (in many cases the required changes
are reduced to nothing).

I don't think anybody claims that prog-indentation-context makes it
hard/impossible to get as good mmm-support as what Dmitry
is suggesting.

They are just 2 alternative (and mostly equivalent) ways to get the
same result.  One of the two requires changes to every major mode and
is currently only used by antlr+python.  The other is used by
mhtml-mode, mmm-mode and generally requires few if any changes to
major modes.  Also changing antlr+python to use the other approach is
easy and leads to marginally simpler code.


        Stefan

Re: Getting rid of prog-indentation-context

[Stefan Monnier]

> I want to see what do we stand now.
                ^^^^^^^
                 where

-- Stefan

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 11 Dec 2017 18:25:52 +0200
> 
> > But maybe I
> > was confused, and the primary goal was to remove
> > prog-indentation-context, while MMM support was just a nice side
> > effect, a bonus.
> 
> I hope you are not being deliberately obtuse.
> 
> But sure, whatever, I also eat puppies for dinner.

Whatever did I do to deserve such a harsh treatment from you?

Re: Getting rid of prog-indentation-context

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Mon, 11 Dec 2017 12:08:28 -0500
> 
> > Stefan, you admitted this stuff into Emacs 2 years ago, so why are you
> > asking me these questions?
> 
> Because you clearly think it's important to have this new variable in
> Emacs-26.

Actually, I rather think it's important that we stick to our own
decisions, especially ones made not so long ago.

This particular decision wasn't made out of ignorance, or slipped
through the cracks.  It was discussed at length, reviewed, and updated
according to discussions.  It in effect set a standard for doing this
stuff, so we should expect others to follow the standard.  We could
also extend and improve the standard, as we gain experience.  But we
shouldn't throw it away, just because it is marginally more complex in
some cases than some simplistic alternative.  Throwing it away without
a very good reason sends a message to the community that we don't
uphold our own development decisions, and are quite ready and capable
of overturning them at will soon after they were made in good faith.
That's a bad message for a veteran project with a vibrant,
decentralized community.

But it's clear at this point that no one shares these views, at least
of those who voiced their opinions.  Too bad, but what else is new?

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: Dmitry Gutov <dgutov@yandex.ru>, emacs-devel@gnu.org
> Date: Mon, 11 Dec 2017 12:21:14 -0500
> 
> I don't think anybody claims that prog-indentation-context makes it
> hard/impossible to get as good mmm-support as what Dmitry
> is suggesting.

AFAIU, Dmitry claimed precisely that.  Otherwise, I cannot understand
the nature of his resistance to a simple request to keep backward
compatibility, something that is standard Emacs development policy.

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Eli Zaretskii]

> Date: Mon, 11 Dec 2017 19:43:09 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> > Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> > From: Dmitry Gutov <dgutov@yandex.ru>
> > Date: Mon, 11 Dec 2017 18:25:52 +0200
> > 
> > > But maybe I
> > > was confused, and the primary goal was to remove
> > > prog-indentation-context, while MMM support was just a nice side
> > > effect, a bonus.
> > 
> > I hope you are not being deliberately obtuse.
> > 
> > But sure, whatever, I also eat puppies for dinner.
> 
> Whatever did I do to deserve such a harsh treatment from you?

John, I'm clearly not helping to reach some kind of consensus here.
So I'm taking myself out of the equation.  Please decide what to do
with this branch, where to merge it, and if it will be merged to
master, what, if anything, to do with prog-indentation-context and
related stuff on the release branch.

Re: Getting rid of prog-indentation-context

[Ingo Lohmar]

On Mon, Dec 11 2017 20:02 (+0200), Eli Zaretskii wrote:

> Actually, I rather think it's important that we stick to our own
> decisions, especially ones made not so long ago.
>
> This particular decision wasn't made out of ignorance, or slipped
> through the cracks.  It was discussed at length, reviewed, and updated
> according to discussions.  It in effect set a standard for doing this
> stuff, so we should expect others to follow the standard.  We could
> also extend and improve the standard, as we gain experience.  But we
> shouldn't throw it away, just because it is marginally more complex in
> some cases than some simplistic alternative.  Throwing it away without
> a very good reason sends a message to the community that we don't
> uphold our own development decisions, and are quite ready and capable
> of overturning them at will soon after they were made in good faith.
> That's a bad message for a veteran project with a vibrant,
> decentralized community.

Without getting into the technical stuff: I have made (albeit small)
contributions to several Emacs- and non-Emacs-related free software
projects, and I strongly disagree with your description of what kind of
message would be sent.

[Note: I am honestly convinced that everybody participating in the
discussion has Emacs' best interests close to his/her heart.]

1) If the discussed, agreed-upon, reviewed and committed changes would
already have been released, and now reverted, that would send a very bad
message, namely pretty much what you describe above.

But these are *unreleased* changes, ie, we're still in the same
development cycle.

2) If the [...] changes from earlier in the cycle would be reverted w/o
wide consensus (on the *technical* desirability to revert them) of those
who actually discussed, coded, reviewed and committed them, that would
/also/ send a very bad message to contributors, namely that their hard
work might easily be discarded later, on a whim.

But (AFAIU) those who actually worked on that stuff have changed their
mind (again, for technical reasons) and now *want* to revert the
changes.

3) In such a situation, reverting the changes sends a *positive* message
to me: That people working on Emacs are capable and willing to
re-evaluate situations, to change their opinions, to learn and to find
better solutions.

OTOH, keeping the changes under the above circumstances, for formal
reasons, would suggest to me that Emacs development is like some big
blob that keeps moving mostly because of its huge inertial mass.

To my mind, that's the opposite of the message that the Emacs community
wants to send out.

Not even $0.02...

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 1521 bytes --]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> John, I'm clearly not helping to reach some kind of consensus here. So I'm
> taking myself out of the equation. Please decide what to do with this
> branch, where to merge it, and if it will be merged to master, what, if
> anything, to do with prog-indentation-context and related stuff on the
> release branch.

Ok.

Dmitry, I do want to say that I'm disappointed this thread has gotten so out
of hand. Even when frustrated, we have a responsibility to subordinate our
emotions to the matter at hand, which is the relatively straightforward
question: should we abandon prog-indentation-context and prog-widen or not?
Though Eli has opinions on this matter, he's not unreasonable, and we even
have Stefan available to comment on past maintainer decisions.

With that said, I've spoken to Eli directly, and we've agreed that it's OK to
merge your branch and eliminate these features that were introduced two years
ago. I'm not entirely sure why they were added, or why Stefan has changed his
mind so greatly since, but it appears the present consensus is that they're no
longer needed. Simplicity wins.

So, you have my go ahead to merge, but before you do so, please clarify for me
(who hasn't been following this discussion as carefully), exactly what impact
your merge will have on master and on the status quo.

Thank you,
--
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 658 bytes --]

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/11/17 10:47 PM, John Wiegley wrote:

> Dmitry, I do want to say that I'm disappointed this thread has gotten so out
> of hand. Even when frustrated, we have a responsibility to subordinate our
> emotions to the matter at hand,

My emotions are okay. And I've apologized in private. Still, I believe I 
had a cause to get offended at the grand-grandparent message.

> which is the relatively straightforward
> question: should we abandon prog-indentation-context and prog-widen or not?
> Though Eli has opinions on this matter, he's not unreasonable, and we even
> have Stefan available to comment on past maintainer decisions.

We're all reasonable people, who apparently disagree on some 
fundamentals, like what kind of "stability" and "maintainer 
responsibility" we're looking for.

I'm not happy about having this argument, but as someone closer to the 
problem, also feel a responsibility, to make my position heard, and for 
the better technical approach to be chosen.

> With that said, I've spoken to Eli directly, and we've agreed that it's OK to
> merge your branch and eliminate these features that were introduced two years
> ago. I'm not entirely sure why they were added, or why Stefan has changed his
> mind so greatly since, but it appears the present consensus is that they're no
> longer needed. Simplicity wins.

To clarify, we're not exactly "eliminating" features so much as changing 
the way programmers will go about obtaining them.

> So, you have my go ahead to merge, but before you do so, please clarify for me
> (who hasn't been following this discussion as carefully), exactly what impact
> your merge will have on master and on the status quo.

Is it okay to merge to emacs-26, then? If yes, the effect on master will 
be the same.

Not sure I understand your question. Do you want a full description of 
the changes? I can rebase the branch first before merging, and add 
proper change log messages.

Re: Getting rid of prog-indentation-context

[Dmitry Gutov]

On 12/11/17 8:53 PM, Ingo Lohmar wrote:
> But (AFAIU) those who actually worked on that stuff have changed their
> mind (again, for technical reasons) and now *want* to revert the
> changes.

Not exactly. The author of the patch (who produced it under Stefan's 
guidance) has not been following these discussions. Although I Cc'd him 
on the previous one.

I don't think he will object much, considering the change to be made in 
antlr-mode.el will be trivial. And his main stated priority IIRC was to 
get some improvement in the current situation, which is still going to 
happen.

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[John Wiegley]

>>>>> "DG" == Dmitry Gutov <dgutov@yandex.ru> writes:

GD> Is it okay to merge to emacs-26, then? If yes, the effect on master will
GD> be the same.

Yes.

GD> Not sure I understand your question. Do you want a full description of the
DG> changes? I can rebase the branch first before merging, and add proper
DG> change log messages.

Please. :)

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

> AFAIU, Dmitry claimed precisely that.  Otherwise, I cannot understand
> the nature of his resistance to a simple request to keep backward
> compatibility, something that is standard Emacs development policy.

Given that prog-indentation-context has never been part of a release,
this kind of "backward compatibility" is something we (until now) have
never really tried to preserve.

As for why we want prog-indentation-context gone rather than keep it for
backward-compatibility, it's because it requires major modes to follow
a convention different from the one that Dmitry suggests, so we end up
with two contradictory recommendations in terms of what major modes
should do.

Hence my question: what's the concrete benefit of keeping in
prog-indentation-context, i.e. of introducing in Emacs-26 a new variable
which according to the evidence I see is destined to being left unused.


        Stefan

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/11/17 11:43 PM, John Wiegley wrote:

> GD> Not sure I understand your question. Do you want a full description of the
> DG> changes? I can rebase the branch first before merging, and add proper
> DG> change log messages.
> 
> Please. :)

I've pushed it to widen-less (without scratch/ now). Comments welcome!

It keeps prog-indentation-context for its first element (for 
change-minimization and possibly emotional reasons; but who knows, 
someday we might add some, specific only to indentation). The added 
bonus is that calling prog-first-column is easier than digging out the 
first element of the list.

What is not included, but probably should be:

- Manual changes describing that font-lock matchers shouldn't call 
'widen' either. It's an important thing, but so far it's been only an 
issue with CC Mode, which isn't supported anyway (so far). So: should 
that info be included, and where should I put it?

- Similar info for syntax-propertize-function, though no violators have 
been observed so far.

These will be documentation-only changes.

What is not included, and maybe should wait until Emacs 27:

- Similar consolidations of 'widen' calls for add-log-current-defun, 
eldoc-print-current-symbol-info, beginning-of-defun-function and others. 
And accompanying documentation (again, where should it go?).

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/14/17 11:32 AM, Dmitry Gutov wrote:

> What is not included, but probably should be:

One more thing: the corresponding manual still mentions C and 
c-indent-line. It probably shouldn't, right?

Because c-indent-line doesn't work in multi-mode context.

Would someone like to suggest a correction?

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

> It keeps prog-indentation-context for its first element (for
> change-minimization and possibly emotional reasons; but who knows, someday
> we might add some, specific only to indentation). The added bonus is that
> calling prog-first-column is easier than digging out the first element of
> the list.

Looks good.

> - Manual changes describing that font-lock matchers shouldn't call 'widen'
> either.  It's an important thing, but so far it's been only an issue with CC
> Mode, which isn't supported anyway (so far). So: should that info be
> included, and where should I put it?

font-lock has always widened (except when font-lock-dont-widen is set,
which AFAICT is only the case in rmail.el), so that's probably why no
major mode widens (CC mode probably widens during font-lock because it
uses functions which are also used in other circumstances where they
need to widen).

So I don't think we need to make any changes on that front.

> - Similar info for syntax-propertize-function, though no violators have been
> observed so far.

Since there are no current violators, we can fix this part of the doc
when we fix the corresponding code.

> What is not included, and maybe should wait until Emacs 27:
> - Similar consolidations of 'widen' calls for add-log-current-defun,
> eldoc-print-current-symbol-info, beginning-of-defun-function and others. And
> accompanying documentation (again, where should it go?).

Maybe some of the doc should be in the narrowing section, where we can
document the contexts where the narrowing is "guaranteed"
(e.g. font-lock and indentation).


        Stefan

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/14/17 4:01 PM, Stefan Monnier wrote:

> font-lock has always widened (except when font-lock-dont-widen is set,
> which AFAICT is only the case in rmail.el), so that's probably why no
> major mode widens (CC mode probably widens during font-lock because it
> uses functions which are also used in other circumstances where they
> need to widen).
> 
> So I don't think we need to make any changes on that front.

So, nothing to add to the manual?

Code changes would only be needed in CC Mode, of course.

>> - Similar info for syntax-propertize-function, though no violators have been
>> observed so far.
> 
> Since there are no current violators, we can fix this part of the doc
> when we fix the corresponding code.

OK. The recent change in spss cache also depends on 
synax-propertize-function not calling widen, BTW.

>> What is not included, and maybe should wait until Emacs 27:
>> - Similar consolidations of 'widen' calls for add-log-current-defun,
>> eldoc-print-current-symbol-info, beginning-of-defun-function and others. And
>> accompanying documentation (again, where should it go?).
> 
> Maybe some of the doc should be in the narrowing section, where we can
> document the contexts where the narrowing is "guaranteed"
> (e.g. font-lock and indentation).

Hmm. Currently, that page doesn't exactly describe when one should call 
'widen', and when shouldn't.

Also, the piece of documentation I'm thinking about will probably need 
to reference, again, "Some major modes need to support embedded regions 
of text whose
syntax belongs to a different major mode.  Examples include “literate
programming” source files that combine documentation and snippets of
source code..."; maybe that introduction should leave the Indentation 
Controlled by Major Mode section, and get a section of its own.

So we can refer to it from several places, at least. Or maybe put all 
mixed-major-mode considerations in it together.

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

>> Maybe some of the doc should be in the narrowing section, where we can
>> document the contexts where the narrowing is "guaranteed"
>> (e.g. font-lock and indentation).
> Hmm. Currently, that page doesn't exactly describe when one should call
> 'widen', and when shouldn't.

[ I don't think in those terms: with the new convention,
  an indent-line-function still can call widen as much as it wants,
  it just doesn't need to and probably doesn't want to.  ]

No, indeed, it doesn't talk about the different uses of narrowing.
It probably should say something, tho, especially w.r.t this new convention.

> Also, the piece of documentation I'm thinking about will probably need to
> reference, again, "Some major modes need to support embedded regions of text
> whose
> syntax belongs to a different major mode.  Examples include “literate
> programming” source files that combine documentation and snippets of
> source code..."; maybe that introduction should leave the Indentation
> Controlled by Major Mode section, and get a section of its own.

Makes sense.  And then refer to it from the narrowing section,
indentation section, etc...


        Stefan

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Robert Weiner]

[-- Attachment #1: Type: text/plain, Size: 264 bytes --]

I suggest the Major Mode Conventions node of the Elisp Manual
include a section on this topic and possibly reference to
another new node that references the mmm-mode package and
documents how to utilize multiple major modes in a single
buffer.

​Bob
​

[-- Attachment #2: Type: text/html, Size: 894 bytes --]

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/14/17 4:32 PM, Stefan Monnier wrote:

> [ I don't think in those terms: with the new convention,
>    an indent-line-function still can call widen as much as it wants,
>    it just doesn't need to and probably doesn't want to.  ]

How should we word it, then? "shouldn't" is not a far cry from "should 
avoid narrowing" which I used in the indentation section. Any better way 
to say that, as well?

> No, indeed, it doesn't talk about the different uses of narrowing.
> It probably should say something, tho, especially w.r.t this new convention.

Suggestions welcome, if any. If you want to see it in Emacs 26, at least.

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/14/17 5:17 PM, Robert Weiner wrote:
> I suggest the Major Mode Conventions node of the Elisp Manual
> include a section on this topic and possibly reference to
> another new node that references the mmm-mode package and
> documents how to utilize multiple major modes in a single
> buffer.

Not sure we can refer to a package that's not in the core there.

It'll hopefully get into ELPA, though.

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

Anybody? I don't know a lot about literal programming or Antlr, so not 
100% sure what will be appropriate substitutes there.

On 12/14/17 3:20 PM, Dmitry Gutov wrote:
> On 12/14/17 11:32 AM, Dmitry Gutov wrote:
> 
>> What is not included, but probably should be:
> 
> One more thing: the corresponding manual still mentions C and 
> c-indent-line. It probably shouldn't, right?
> 
> Because c-indent-line doesn't work in multi-mode context.
> 
> Would someone like to suggest a correction?

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

Hi again!

On 12/14/17 11:32 AM, Dmitry Gutov wrote:
> On 12/11/17 11:43 PM, John Wiegley wrote:
> 
>> GD> Not sure I understand your question. Do you want a full 
>> description of the
>> DG> changes? I can rebase the branch first before merging, and add proper
>> DG> change log messages.
>>
>> Please. :)
> 
> I've pushed it to widen-less (without scratch/ now). Comments welcome!

The current state looks good to me. Okay to merge?

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Stefan Monnier]

> The current state looks good to me. Okay to merge?

LGTM,


        Stefan

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[John Wiegley]

>>>>> "DG" == Dmitry Gutov <dgutov@yandex.ru> writes:

>> I've pushed it to widen-less (without scratch/ now). Comments welcome!

DG> The current state looks good to me. Okay to merge?

It looks fine to me. Maybe indent--funcall-widened should be a macro or
defsubst?

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/20/17 9:13 PM, John Wiegley wrote:
>>>>>> "DG" == Dmitry Gutov <dgutov@yandex.ru> writes:
> 
>>> I've pushed it to widen-less (without scratch/ now). Comments welcome!
> 
> DG> The current state looks good to me. Okay to merge?
> 
> It looks fine to me.

Thanks! Merging.

> Maybe indent--funcall-widened should be a macro or
> defsubst?

I'd rather avoid premature optimization (and there's a couple of way we 
could choose to rewrite indent-for-tab-command that this function is not 
needed), but I wouldn't argue if someone else did this.

FWIW, I'm told define-inline is a better choice these days.

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[John Wiegley]

>>>>> "DG" == Dmitry Gutov <dgutov@yandex.ru> writes:

DG> FWIW, I'm told define-inline is a better choice these days.

Hmm... no docstring, although it's in the Elisp ref manual. Quoting from
there:

   Functions defined via ‘define-inline’ have several advantages with
   respect to macros defined by ‘defsubst’ or ‘defmacro’:

   − They can be passed to ‘mapcar’ (*note Mapping Functions::).
   − They are more efficient.
   − They can be used as “place forms” to store values (*note
     Generalized Variables::).
   − They behave in a more predictable way than ‘cl-defsubst’ (*note
     (cl)Argument Lists::).

So, yeah. And I wouldn't really call using define-inline instead of defun for
tiny functions "premature optimization" in cases like these. You're not
changing the semantics of the function, just indicating it's small enough that
it should be inlined when byte-compiling. And you know the number of call
sites too, so it won't cause code bloat. When it's that harmless, I usually do
it as a matter of convention in my own code.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/21/17 3:33 AM, John Wiegley wrote:

> And I wouldn't really call using define-inline instead of defun for
> tiny functions "premature optimization" in cases like these. You're not
> changing the semantics of the function, just indicating it's small enough that
> it should be inlined when byte-compiling. And you know the number of call
> sites too, so it won't cause code bloat. When it's that harmless, I usually do
> it as a matter of convention in my own code.

It's simply extra work. Also see what the manual says: 
https://www.gnu.org/software/emacs/manual/html_node/elisp/Inline-Functions.html, 
"you should not make a function inline...".

Re: [SUSPECTED SPAM] Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[John Wiegley]

>>>>> "DG" == Dmitry Gutov <dgutov@yandex.ru> writes:

DG> It's simply extra work. Also see what the manual says:
DG> https://www.gnu.org/software/emacs/manual/html_node/elisp/Inline-Functions.html,"you
DG> should not make a function inline...".

Great point. Noted.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2