antlr-mode.el - need some support by python.el

antlr-mode.el - need some support by python.el

[Wedler, Christoph]

As the author of lisp/progmodes/antlr-mode.el, I finally found some time
to update this Emacs mode according to the changes of ANTLR (a tool for
language recognition, i.e. in the same area as Flex and Bison).  And
ANTLR has changed quite a bit...

For the changes I have done so far, see the README or the Changelog at
    http://sourceforge.net/projects/antlr-mode/
More changes (and some cleanup) is needed.

This brings me to 2 questions (I am currently not on the emacs-devel
mailing list, I just skim the archive once a month, i.e. please keep
my email address on replies / follow-ups - if useful, I could join the
mailing list for a month or two).

 1. How am I supposed to integrate this into the main Emacs branch?  My
    current idea is to do this after some cleanup.

 2. ANTLR (v3) supports that more languages can be used for actions
    inside the grammar - I support Java and Cpp as before, and now also
    C, ObjC, Delphi, JavaScript, Python and Ruby

    Syntax-highlighting support for these works quite well, indentation
    for Python actions inside the grammar needs support by that mode
    (support by opascal-mode and ruby-mode would also be appreciated,
    but is not entirely necessary).

    Q: Does the maintainer of python.el has some time to do some (small)
    change there, or am I supposed to send a patch?

There are three potential issues when I call the indentation engine of
the "sub-mode" (if it is not based on cc-mode) - see below for an
example.

 a. The indentation engine of the sub-mode might get confused by the
    grammar code around the code snippet.  Not an issue for JavaScript
    (has a Cish syntax).  I have avoided that for opascal and ruby via
    `narrow-to-region', but Python's indentation calls `widen'.

 b. The indentation engine might want to parse a correct program, not
    just some code snippet (not an issue so far).

 c. The indentation engine might want to indent the first statement
    inside the grammar action at column 0 (because it is the first
    statement it sees at all), but that might not be nice for the first
    statement inside a grammar action.

    A workaround for all modes except python was to do the indentation
    myself (or to be honest by cc-mode) if the submode indents the line
    at column 0.  For Python, this is no option as it cycles through the
    possible indentation columns.

Here is an extract from an example grammar for ANTLR v3:

//-------------------------------------------------------
grammar C;
options { language=Python; }

@members {
    def isTypeName(self, name):
        for scope in reversed(self.Symbols_stack):
            if name in scope.types:
                return True                                # line 09

        return False
}

direct_declarator
    :   (   IDENTIFIER
            {
                if len($declaration) > 0 and $declaration::isTypedef:
                    $Symbols::types.add($IDENTIFIER.text)
                    print "define type "+$IDENTIFIER.text  # line 19
            }
        |   '(' declarator ')'
        )
        declarator_suffix*
    ;
//-------------------------------------------------------

The indentation points for line 09 should be columns 16, 12,8 and 4, but
not 0 - for line 19 the columns should be 20 and 16, but not 12, 8, 4
and 0.

Therefore, before I call `python-indent-line, I need a possibility to
specify the region of the action, and the "minimal column position".

Regards,
Christoph

Re: antlr-mode.el - need some support by python.el

[Stefan Monnier]

Hi Christoph,

>  1. How am I supposed to integrate this into the main Emacs branch?  My
>     current idea is to do this after some cleanup.

You can send a patch, or I can provide you with write access.  You can
do it "all at once in the end" or in smaller chunks, as you wish.

>     Q: Does the maintainer of python.el has some time to do some (small)
>     change there, or am I supposed to send a patch?

I believe he reads this mailing-list, so he'll probably answer, but
otherwise you can contact him directly.

> There are three potential issues when I call the indentation engine of
> the "sub-mode" (if it is not based on cc-mode) - see below for an
> example.

These problems seem to be fairly general (apply to any multi-mode
situation), so do not hesitate to suggest introducing general guidelines
or new functionality to make it easier for major modes to behave well
for your use-case.


        Stefan

RE: antlr-mode.el - need some support by python.el

[Wedler, Christoph]

Hi Stefan,

>>  1. How am I supposed to integrate this into the main Emacs branch?  My
>>     current idea is to do this after some cleanup.

> You can send a patch, or I can provide you with write access.  You can
> do it "all at once in the end" or in smaller chunks, as you wish.

OK, thanks. Then I wait until I have finished the `defcustom's for the
different ANTLR flavors and the different sublanguages.

>     Q: Does the maintainer of python.el has some time to do some (small)
>     change there, or am I supposed to send a patch?

> I believe he reads this mailing-list, so he'll probably answer, but
> otherwise you can contact him directly.

For my last minor antlr-mode release, I also provided a patch proposal
for python.el

Ok, I'll send a specific followup, put Fabián also directly into the
"To" header and use a catchier subject (starting with python.el).

>> There are three potential issues when I call the indentation engine of
>> the "sub-mode" (if it is not based on cc-mode) - see below for an
>> example.

> These problems seem to be fairly general (apply to any multi-mode
> situation), so do not hesitate to suggest introducing general guidelines
> or new functionality to make it easier for major modes to behave well
> for your use-case.

You are right - for a start, I started with a variable specifically for
Python, but in the end, I would see it as part of prog-mode (s/th like
xml-mode would not be covered, but to make it part of fundamental-mode
is probably too generic?).

Here is the variable I defined in my python.el patch proposal:

+ (defvar python-submode-indentation-context nil
+   "((MIN . MAX) LEFTMOST-COL . <future>).")

Ok, here is some real doc:

   Non-nil while indenting lines considered as belonging to a sub mode.

   There are languages where part of the code is actually written in a sub
   language, e.g. a Yacc/Bison grammar also consists of plain C code.  This
   variables enables the major mode of the main language to use the
   indentation function of the sub language for lines belonging the code
   of the sub language.

   When a major mode of such a main language decides to propagate the
   indentation of a line/region to the indentation engine of the sub
   mode, it is supposed to bind this variable to non-nil around the call.
   The non-nil value looks as follows
      ((MIN . MAX) LEFTMOST-COL . SUBMODE-SPECIFIC)

   MIN to MAX is the region consisting of code of the sub mode.
   LEFTMOST-COL is the minimum column the indentation engine of the sub
   mode should choose (instead of 0).  SUBMODE-SPECIFIC can be used to
   allow the indentation engine of the submode work well on code
   snippets (not complete programs), e.g. by providing
   `defun-block-intro' (instead `topmost-intro') to the indentation
   engine of cc-mode.

Is this fine for you?  Should I send a corresponding patch to
progmodes/prog-mode.el ?

   Christoph

Re: antlr-mode.el - need some support by python.el

[Stefan Monnier]

> You are right - for a start, I started with a variable specifically for
> Python, but in the end, I would see it as part of prog-mode (s/th like
> xml-mode would not be covered, but to make it part of fundamental-mode
> is probably too generic?).

prog-mode sounds good.  That doesn't prevent non-prog modes from using
it (and really, modes like xml-mode or latex-mode should inherit from
both text-mode and prog-mode).

> + (defvar python-submode-indentation-context nil
> +   "((MIN . MAX) LEFTMOST-COL . <future>).")
> Ok, here is some real doc:

Sounds like a good start.  Two questions:
- how/who provides the <future>?  Is it the outer mode (which hence
  needs "internal" knowledge of the inner mode)?
- for some outer modes (think of a literate programming mode, for
  example), the chunk specified by MIN/MAX should really be considered
  as being a continuation of a previous chunk.  So maybe the inner mode
  should be provided with some way to find the "text before MIN" (if any).

I guess for your use case, the second point doesn't apply, tho, so maybe
we can skip this part for now.

> Is this fine for you?  Should I send a corresponding patch to
> progmodes/prog-mode.el ?

Sure,


        Stefan

RE: antlr-mode.el - need some support by python.el

[Wedler, Christoph]

> prog-mode sounds good.  That doesn't prevent non-prog modes from using
> it (and really, modes like xml-mode or latex-mode should inherit from
> both text-mode and prog-mode).

Ok, how should I name the variable then?  `submode-indentation-context' ?

>> + (defvar python-submode-indentation-context nil
>> +   "((MIN . MAX) LEFTMOST-COL . <future>).")
>> Ok, here is some real doc:

> Sounds like a good start.  Two questions:
> - how/who provides the <future>?  Is it the outer mode (which hence
>   needs "internal" knowledge of the inner mode)?
> - for some outer modes (think of a literate programming mode, for
>   example), the chunk specified by MIN/MAX should really be considered
>   as being a continuation of a previous chunk.  So maybe the inner mode
>   should be provided with some way to find the "text before MIN" (if any).

The values are provided by the outer mode.  But the inner mode decides
what it expects in <future> (that's why I called it SUBMODE-SPECIFIC in
the "real doc").  Reason: the outer mode wants support from the inner
mode, thus it should be easy for the inner mode... (and there are more
potential inner than outer modes).

While it is true that the chunk specified by MIN/MAX should really be
considered begin a continuation of a "chunk series", most modes won't
probably like the idea of having to react to the "text before MIN",
because that would require them to create a temporary buffer for the
indent calculation.

Providing a possibility for some initial syntactic context is probably
easier.  E.g. cc-mode can decide that it can accept some initial value
for c-syntactic-context...

   Christoph

Re: antlr-mode.el - need some support by python.el

[Stefan Monnier]

>> prog-mode sounds good.  That doesn't prevent non-prog modes from using
>> it (and really, modes like xml-mode or latex-mode should inherit from
>> both text-mode and prog-mode).
> Ok, how should I name the variable then?  `submode-indentation-context' ?

I'd use prog-indentation-context.

> The values are provided by the outer mode.  But the inner mode decides
> what it expects in <future> (that's why I called it SUBMODE-SPECIFIC in
> the "real doc").

That requires some cross-knowledge of the two modes.  It's not ideal,
but I guess it's OK.

> While it is true that the chunk specified by MIN/MAX should really be
> considered begin a continuation of a "chunk series", most modes won't
> probably like the idea of having to react to the "text before MIN",
> because that would require them to create a temporary buffer for the
> indent calculation.

Not necessarily: the submode can just go to the end of the previous
chunk, compute some state at that position and use that state as initial
context at MIN.

> Providing a possibility for some initial syntactic context is probably
> easier.  E.g. cc-mode can decide that it can accept some initial value
> for c-syntactic-context...

Providing the location of the previous chunk(s) is all that's needed and
the submode can do that "initial context" dance if that's what it likes.
This is better because the outer mode doesn't need to know about those
"initial contexts".

But we can start without support for that kind of "previous chunks" info
and add it later.  It's often better to add such support only once you
have an actual concrete need for it, so you can check that it really
does satisfy the need.


        Stefan

RE: antlr-mode.el - need some support by python.el

[Wedler, Christoph]

> I'd use prog-indentation-context.

OK, I append a patch for prog-mode.el below.

 >> The values are provided by the outer mode.  But the inner mode decides
 >> what it expects in <future> (that's why I called it SUBMODE-SPECIFIC in
 >> the "real doc").

 > That requires some cross-knowledge of the two modes.  It's not ideal,
 > but I guess it's OK.

You are right, it is not ideal.  I do not think it requires cross-knowledge,
just some knowledge by the outer mode about the inner (bad for me ;-))

What I deduce from your text is that you do not like to have this
variable "polluted" by some sub-mode specific stuff - and I do agree.

In the patch below I have now deleted SUBMODE-SPECIFIC.  It has the
advantage that we could extend the variable with some future generic
context info in a compatible manner.

Instead, I added a paragraph in the doc string that sub modes can define
their own `SUB-extra-indentation-context', which would have the
additional advantage that the variable content is documented.

 >> [...] that would require them to create a temporary buffer for the
 >> indent calculation.

 > Not necessarily: the submode can just go to the end of the previous
 > chunk, compute some state at that position and use that state as initial
 > context at MIN.

... if scan-lists, mode-specific skip-comments etc are prepared for this.

It would also not help me as the previous chunks are not part of the
ANTLR grammar buffer (but of the ANTLR code generation code) - I could
just fake some context _string_. (Providing the context as a string
would be actually the easiest for me.)

        Christoph

diff -c prog-mode.el.\~24.4\~ prog-mode.el
*** prog-mode.el.~24.4~	2014-04-27 19:41:16.000000000 +0000
--- prog-mode.el	2015-02-13 09:19:26.285457000 +0000
***************
*** 41,46 ****
--- 41,76 ----
      map)
    "Keymap used for programming modes.")
  
+ (defvar prog-indentation-context nil
+   "Non-nil while indenting lines considered as belonging to a sub mode.
+ There are languages where part of the code is actually written in
+ a sub language, e.g., a Yacc/Bison or ANTLR grammar also consists
+ of plain C code.  This variables enables the major mode of the
+ main language to use the indentation function of the sub language
+ for lines belonging the code of the sub language.
+ 
+ When a major mode of such a main language decides to propagate the
+ indentation of a line/region to the indentation engine of the sub
+ mode, it is supposed to bind this variable to non-nil around the call.
+ 
+ The non-nil value looks as follows
+    ((MIN . MAX) LEFTMOST-COL)
+ 
+ MIN to MAX is the region consisting of code of the sub mode.
+ LEFTMOST-COL is the minimum column the indentation engine of the
+ sub mode should choose (instead of 0).
+ 
+ It is the task of the calling main mode to (temporary) set the
+ syntax table and related variables to values suitable for the sub
+ mode, and to call `syntax-ppss-flush-cache' if necessary.
+ 
+ If the indentation engine of the sub mode SUB depends on some
+ global context, it may introduce a variable named
+ `SUB-extra-indentation-context', which is to be bound like this
+ variable by the main major mode.  For example, cc-mode might
+ define `c-extra-indentation-context' which is to be bound to a
+ list of SYNTACTIC-SYMBOL as explained in `c-offsets-alist'.")
+ 
  (defun prog-indent-sexp (&optional defun)
    "Indent the expression after point.
  When interactively called with prefix, indent the enclosing defun

Diff finished.  Fri Feb 13 10:19:56 2015

Re: antlr-mode.el - need some support by python.el

[Stefan Monnier]

> OK, I append a patch for prog-mode.el below.

Thanks, see comments below.

>> That requires some cross-knowledge of the two modes.  It's not ideal,
>> but I guess it's OK.
> You are right, it is not ideal.  I do not think it requires cross-knowledge,
> just some knowledge by the outer mode about the inner (bad for me ;-))

That's right.

> What I deduce from your text is that you do not like to have this
> variable "polluted" by some sub-mode specific stuff - and I do agree.

Well, I don't mind about having the info in this particular place, I'm
just worried about needing such knowledge, but indeed in your case you
probably do have that knowledge (ANTLR does specify that the embedded
C code should be a sequence of statements rather than a sequence of
declarations, say).

> Instead, I added a paragraph in the doc string that sub modes can define
> their own `SUB-extra-indentation-context', which would have the
> additional advantage that the variable content is documented.

Since this is a specific to a particular submode and both the inner-mode
and the outer-mode will need to agree on it, there's no need to document
it in the generic part of the interface.

>>> [...] that would require them to create a temporary buffer for the
>>> indent calculation.
>> Not necessarily: the submode can just go to the end of the previous
>> chunk, compute some state at that position and use that state as initial
>> context at MIN.
> ... if scan-lists, mode-specific skip-comments etc are prepared for this.

That's the responsibility of the sub-mode.

> It would also not help me

Indeed, in your case, there's no such "previous chunk" in the current buffer.
I was thinking of other multi-mode cases such as literate programming or
HTML+Javascript.

> as the previous chunks are not part of the ANTLR grammar buffer (but
> of the ANTLR code generation code) - I could just fake some context
> _string_. (Providing the context as a string would be actually the
> easiest for me.)

Hmm.. you mean you'd specify the c-extra-indentation-context as
a "pseudo previous chunk of code"?  That's a nice idea, in that it would
not encode specific knowledge about CC-mode, only specific knowledge
about the C language, so it'd be fairly generic.  So maybe
prog-indentation-context could come with a PREVIOUS value which could
either be a string (so the indentation should pretend that the code was
preceded by this string) or some other value (e.g. a function, or a list
of buffer positions) which gives access to the previous chunks.

> + When a major mode of such a main language decides to propagate the
                                                         ^^^^^^^^^
                                                         delegate

> + indentation of a line/region to the indentation engine of the sub
> + mode, it is supposed to bind this variable to non-nil around the call.
> +
> + The non-nil value looks as follows
> +    ((MIN . MAX) LEFTMOST-COL)
> +
> + MIN to MAX is the region consisting of code of the sub mode.
> + LEFTMOST-COL is the minimum column the indentation engine of the
> + sub mode should choose (instead of 0).

I think (BEG . END) would better match the names we use in other
contexts (on a naive/superficial reading, I assumed MIN/MAX were column
positions).

> + It is the task of the calling main mode to (temporary) set the
                                                ^^^^^^^^^
                                                temporarily

-- Stefan

RE: antlr-mode.el - need some support by python.el

[Wedler, Christoph]

> Thanks, see comments below.

Thanks for the comments - below the updated patch for prog-mode.el.  I
use START/END as in `narrow-to-region'.

>> Instead, I added a paragraph in the doc string that sub modes can define
>> their own `SUB-extra-indentation-context', which would have the
>> additional advantage that the variable content is documented.

> Since this is a specific to a particular submode and both the inner-mode
> and the outer-mode will need to agree on it, there's no need to document
> it in the generic part of the interface.

Hm, I actually like these kind of cross-links in comments.  Otherwise,
someone could read this docstring and ask themselves "...and where can I
specify that my code snippet is a sequence of statements and not a seq
of declarations?".  With the comment they know that the sub mode might
specify such a possibility if they are lucky.

But it is up to you whether you include this paragraph.

>>>> [...] that would require them to create a temporary buffer for the
>>>> indent calculation.
>>> Not necessarily: the submode can just go to the end of the previous
>>> chunk, compute some state at that position and use that state as initial
>>> context at MIN.
>> ... if scan-lists, mode-specific skip-comments etc are prepared for this.

> That's the responsibility of the sub-mode.

To get support by the authors of potential sub-modes, I would really
like to keep the effort for the sub-modes small.
Given that the indentation code of many prog-modes is probably the most
complex part, I would understand if the authors do not want to change it
considerably just to support some special-topic major mode.

>> as the previous chunks are not part of the ANTLR grammar buffer (but
>> of the ANTLR code generation code) - I could just fake some context
>> _string_. (Providing the context as a string would be actually the
>> easiest for me.)

> Hmm.. you mean you'd specify the c-extra-indentation-context as
> a "pseudo previous chunk of code"?  That's a nice idea, in that it would
> not encode specific knowledge about CC-mode, only specific knowledge
> about the C language, so it'd be fairly generic.  So maybe
> prog-indentation-context could come with a PREVIOUS value which could
> either be a string (so the indentation should pretend that the code was
> preceded by this string) or some other value (e.g. a function, or a list
> of buffer positions) which gives access to the previous chunks.

Yes, I like to see some generic support in the future, both for literate
programming and my use case.  To get it, the path to that future should
be easy for the authors of the sub-mode.  Therefore, I think that it
will involve some mode-specific `SUB-extra-indentation-context', whose
exact format will just be of lesser importance in later stages.

I would see the following stages of support by the major mode SUB:

 1. SUB supports START/END and LEFTMOST-COL in 'prog-indentation-context'

 2. SUB provides a function which sets the syntax table and all
    variables which are necessary to properly run its indentation code and
    functions used as font-lock highlight functions (and does not set
    variables which are not necessary for these tasks)

 3. SUB supports `SUB-extra-indentation-context' (with a format which is
    most easily to consume by SUBs indentation code)

 4. SUB provides a function which creates a valid value for
    `SUB-extra-indentation-context' - valid arguments for that function
    would either be a string, or a list of buffer ranges.

    For C (assuming that a valid value for `c-extra-indentation-context'
    would look like the return value of `c-guess-basic-syntax'), this
    function could simply put the string / buffer chunks into a
    temporary buffer and call `c-guess-basic-syntax' at eob.

    In my use case, I would run this function once - in the literate
    programming case, you would run it whenever you delegate the
    indentation to the sub mode.

 5. If SUB decides that it can support the "multi-chunk case" directly
    in the indentation code (e.g. for performance improvements), it can
    do so without any change in the call by the main mode - the function
    introduced in stage 4 would then simply return the buffer-positions.

    Christoph





diff -c prog-mode.el.\~24.4\~ prog-mode.el
*** prog-mode.el.~24.4~	2014-04-27 19:41:16.000000000 +0000
--- prog-mode.el	2015-02-16 14:26:58.376762000 +0000
***************
*** 41,46 ****
--- 41,76 ----
      map)
    "Keymap used for programming modes.")
  
+ (defvar prog-indentation-context nil
+   "Non-nil while indenting lines considered as belonging to a sub mode.
+ There are languages where part of the code is actually written in
+ a sub language, e.g., a Yacc/Bison or ANTLR grammar also consists
+ of plain C code.  This variables enables the major mode of the
+ main language to use the indentation function of the sub language
+ for lines belonging the code of the sub language.
+ 
+ When a major mode of such a main language decides to delegate the
+ indentation of a line/region to the indentation engine of the sub
+ mode, it is supposed to bind this variable to non-nil around the call.
+ 
+ The non-nil value looks as follows
+    ((START . END) LEFTMOST-COL)
+ 
+ START to END is the region consisting of code of the sub mode.
+ LEFTMOST-COL is the minimum column the indentation engine of the
+ sub mode should choose (instead of 0).
+ 
+ It is the task of the calling main mode to (temporarily) set the
+ syntax table and related variables to values suitable for the sub
+ mode, and to call `syntax-ppss-flush-cache' if necessary.
+ 
+ If the indentation engine of the sub mode SUB depends on some
+ global context, it may introduce a variable named
+ `SUB-extra-indentation-context', which is to be bound like this
+ variable by the main major mode.  For example, cc-mode might
+ define `c-extra-indentation-context' which is to be bound to a
+ list of SYNTACTIC-SYMBOL as explained in `c-offsets-alist'.")
+ 
  (defun prog-indent-sexp (&optional defun)
    "Indent the expression after point.
  When interactively called with prefix, indent the enclosing defun

Re: antlr-mode.el - need some support by python.el

[Stefan Monnier]

>>>> Not necessarily: the submode can just go to the end of the previous
>>>> chunk, compute some state at that position and use that state as initial
>>>> context at MIN.
>>> ... if scan-lists, mode-specific skip-comments etc are prepared for this.
>> That's the responsibility of the sub-mode.
> To get support by the authors of potential sub-modes, I would really
> like to keep the effort for the sub-modes small.
> Given that the indentation code of many prog-modes is probably the most
> complex part, I would understand if the authors do not want to change it
> considerably just to support some special-topic major mode.

But it's not like there are many other options.  The options are either:
- the submode doesn't know how to use previous chunks, so it doesn't
  work correctly when the code is split into related chunks.
- the submode does know how to parse and connect the various chunks.
In either case, the outer mode can't do anything more than provide the
boundaries and relationships between the various chunks (e.g. in the
case of antlr, I guess the relationship is that each chunk for an action
is independent) because the outer mode can't know what info the submode
might need to propagate from one chunk to the other.

IOW, what I stated is not a design choice, it's just a fact of life.

>  4. SUB provides a function which creates a valid value for
>     `SUB-extra-indentation-context' - valid arguments for that function
>     would either be a string, or a list of buffer ranges.

>     For C (assuming that a valid value for `c-extra-indentation-context'
>     would look like the return value of `c-guess-basic-syntax'), this
>     function could simply put the string / buffer chunks into a
>     temporary buffer and call `c-guess-basic-syntax' at eob.

>     In my use case, I would run this function once - in the literate
>     programming case, you would run it whenever you delegate the
>     indentation to the sub mode.

>  5. If SUB decides that it can support the "multi-chunk case" directly
>     in the indentation code (e.g. for performance improvements), it can
>     do so without any change in the call by the main mode - the function
>     introduced in stage 4 would then simply return the buffer-positions.

I don't see much difference between 4 and 5: basically, the difference
is between putting the code in the outer-mode or in the submode, but
it'll be the same code.  And since this code is submode-specific, I'd
rather skip the step 4.  If the submode can't be modified (e.g. the
author doesn't want to cooperate or you want your outer mode to work
with older versions), then we can use with-eval-after-load, add-hook,
add-advice and friends to get the effect of 4 without having explicit
support for it.

In any case your patch looks good, feel free to install it.


        Stefan

RE: antlr-mode.el - need some support by python.el

[Wedler, Christoph]

>> To get support by the authors of potential sub-modes, I would really
>> like to keep the effort for the sub-modes small.
>> Given that the indentation code of many prog-modes is probably the most
>> complex part, I would understand if the authors do not want to change it
>> considerably just to support some special-topic major mode.

> But it's not like there are many other options.  The options are either:
> - the submode doesn't know how to use previous chunks, so it doesn't
>   work correctly when the code is split into related chunks.
> - the submode does know how to parse and connect the various chunks.

That is not the difference we have:
 - You want to connect the chunks on-the-fly during the indent
   calculation.

 - I want to first connect the chunks and then do the indent
   calculation, or to be more exact:
   I want to first connect the previous chunks, get the parse state at
   the end of these chunks, and use this parse state as the initial one
   when indenting the current chunk.

This is exactly the difference between 5 (yours) and 4 (mine) below - and
the nice thing is that the submode can decide for itself whether to do it
like 4 or 5.

>>  4. SUB provides a function which creates a valid value for
>>     `SUB-extra-indentation-context' - valid arguments for that function
>>     would either be a string, or a list of buffer ranges.

>>     For C (assuming that a valid value for `c-extra-indentation-context'
>>     would look like the return value of `c-guess-basic-syntax'), this
>>     function could simply put the string / buffer chunks into a
>>     temporary buffer and call `c-guess-basic-syntax' at eob.

>>     In my use case, I would run this function once - in the literate
>>     programming case, you would run it whenever you delegate the
>>     indentation to the sub mode.

>>  5. If SUB decides that it can support the "multi-chunk case" directly
>>     in the indentation code (e.g. for performance improvements), it can
>>     do so without any change in the call by the main mode - the function
>>     introduced in stage 4 would then simply return the buffer-positions.

> I don't see much difference between 4 and 5: basically, the difference
> is between putting the code in the outer-mode or in the submode, but
> it'll be the same code.

Actually not.  In both cases, the submode provides the code.

In 4, the sub mode provides a function to compute the initial parse
state given the string / chunks.  It can do that quite simply by
inserting it/them into a temporary buffer and using a function which
computes the syntactic context at the end of that buffer - most major
mode already have such a function.  In other words: the effort for the
submode is small.

In 5, they follow your idea:
>>>>>> [...] that would require them to create a temporary buffer for the
>>>>>> indent calculation.
>>>>> Not necessarily: the submode can just go to the end of the previous
>>>>> chunk, compute some state at that position and use that state as initial
>>>>> context at MIN.
>>>> ... if scan-lists, mode-specific skip-comments etc are prepared for this.
>>> That's the responsibility of the sub-mode.

To me, that seems to be more work.  And it would not be enough for me,
as my previous chunk is not part of the buffer, but provided by a
string.
Performance-wise, this could be better in the literate programming use
case, but not in my use case.

> In any case your patch looks good, feel free to install it.

If I might come back to your offer from last month (concerning
antlr-mode.el, actually) - it would be excellent if you could apply the
prog-mode.el.patch to the Emacs repository...

        Christoph

Re: antlr-mode.el - need some support by python.el

[Stefan Monnier]

> This is exactly the difference between 5 (yours) and 4 (mine) below

I think we're misunderstanding each other.

> Actually not.  In both cases, the submode provides the code.

> In 4, the sub mode provides a function to compute the initial parse
> state given the string / chunks.  It can do that quite simply by
> inserting it/them into a temporary buffer and using a function which
> computes the syntactic context at the end of that buffer - most major
> mode already have such a function.  In other words: the effort for the
> submode is small.

IIUC you're suggesting that the submode provides a function "get-state"
to the outer mode, and the outer mode calls this "get-state" with some
"previous-chunk-descriptor" (either buffer positions or a string)
on the previous chunk and then passes the result to the submode.

> In 5, they follow your idea:
[...]
> To me, that seems to be more work.

The only difference is that the submode doesn't provide "get-state" to
the outer mode, instead the outer mode provides the
"previous-chunk-descriptor" and the submode calls "get-state" on it if
it wants to.

> Performance-wise, this could be better in the literate programming use
> case, but not in my use case.

I think the two are pretty much identical.

>> In any case your patch looks good, feel free to install it.
> If I might come back to your offer from last month (concerning
> antlr-mode.el, actually) - it would be excellent if you could apply the
> prog-mode.el.patch to the Emacs repository...

OK, will do,


        Stefan

RE: antlr-mode.el - need some support by python.el

[Wedler, Christoph]

> The only difference is that the submode doesn't provide "get-state" to
> the outer mode, instead the outer mode provides the
> "previous-chunk-descriptor" and the submode calls "get-state" on it if
> it wants to.

OK, that is fine with me.  And this PREVIOUS-CHUNK-DESCRIPTOR could then
be even a part of `indent-multi-mode-context' (or whatever the name is).

(I was too much on "adopting forward-sexpr" vs "use temporary buffer",
but both implementations are also possible with your API).

        Christoph

Re: antlr-mode.el - need some support by python.el

[Dmitry Gutov]

I have to say I don't like this:

On 02/16/2015 04:38 PM, Wedler, Christoph wrote:

> + (defvar prog-indentation-context nil

Submodes could just as well be in text-based modes (XML comes to mind). 
And the primary mode is quite often that (html-mode, for instance).

Since multi-mode machinery, in the general case, will require support 
from the "primary" major modes, too, I suppose prog-mode it's not the 
proper place for the facilitating code.

The variable itself looks, to me, like it's solving a non-problem:

> + The non-nil value looks as follows
> +    ((START . END) LEFTMOST-COL)

The first element tries to re-implement what's currently being handled 
with narrowing, successfully. Why?

Like you said: "I have avoided that for opascal and ruby via
`narrow-to-region', but Python's indentation calls `widen'." Then Python 
should avoid calling `widen'. And/or someone could try and solve the 
long-standing problem of separating "visibility" narrowing and 
"functional" narrowing. Indentation code, AFAIK, only ever intends to 
temporarily undo any interactive narrowing by the user.

LEFTMOST-COL, likewise, can by added by the calling function. No need to 
make the submode's indentation function be aware of it.

And overall, this change looks very specific to this one use case (Antlr 
and literate programming modes, I guess). On the other hand, it's not 
going to be of much help with the other trickier situation, quite common 
in web programming, when the primary mode indentation depends on the 
submode (and vice versa). Examples:

ERB:

<div id='flash'>
   <% flash.each do |key, value| -%>
     <div id='flash_<%= key %>'><%=h value %></div>
   <% end %>
</div>

Handlebars:

<div class="entry">
   {{#if author}}
     <h1>{{firstName}} {{lastName}}</h1>
   {{else}}
     <h1>Unknown Author</h1>
   {{/if}}
</div>

> + It is the task of the calling main mode to (temporarily) set the
> + syntax table and related variables to values suitable for the sub
> + mode, and to call `syntax-ppss-flush-cache' if necessary.

These instructions would be better suited for a doc for writing 
multi-mode-support facilities.

> + If the indentation engine of the sub mode SUB depends on some
> + global context, it may introduce a variable named
> + `SUB-extra-indentation-context', which is to be bound like this
> + variable by the main major mode.  For example, cc-mode might
> + define `c-extra-indentation-context' which is to be bound to a
> + list of SYNTACTIC-SYMBOL as explained in `c-offsets-alist'.")

This, I guess, might be useful. But until many modes intend to use it, 
what stops you from just keeping a small alist of (MAJOR-MODE . 
OFFSETS-ALIST-VAR)?

Re: antlr-mode.el - need some support by python.el

[Stefan Monnier]

> Since multi-mode machinery, in the general case, will require support from
> the "primary" major modes, too, I suppose prog-mode it's not the proper
> place for the facilitating code.

We could place it elsewhere.  E.g. indent.el.

> The first element tries to re-implement what's currently being handled with
> narrowing, successfully. Why?

Narrowing sucks, so no it's not a viable alternative.

> And/or someone could try and solve the long-standing problem of
> separating "visibility" narrowing and "functional" narrowing.

If/when someone solves this, we could try and make sure that this
solution can also be used to solve this problem.  But until that happens
we need something else.

> LEFTMOST-COL, likewise, can by added by the calling function.  No need to
> make the submode's indentation function be aware of it.

Yes and no: currently, the standard API to indentation is just
indent-line-function which performs the indentation directly, rather
than returning an indentation column.  So implementing LEFTMOST-COL
"outside" means re-indenting after you called the indentation function.

Furthermore, I can imagine some cases where the inner mode may still
want to indent to a column smaller than LEFTMOST-COL for good reasons.

> And overall, this change looks very specific to this one use case (Antlr and
> literate programming modes, I guess).

It's definitely not the end of the story, indeed.  But I don't think
anyone knows what a good general solution looks like, right now
(especially if "good" includes "can be made to work OK without
re-implementing the indentation algorithm of all submodes").

We've been waiting for such a magic bullet for a long time, and I don't
expect it to come any time soon.  This proposed prog-indentation-context
is just a very small step along the way, but at least it's making
some progress.

> On the other hand, it's not going to be of much help with the other
> trickier situation, quite common in web programming, when the primary
> mode indentation depends on the submode (and vice versa).

We can try to handle this case as a second step.  There are many other
problems to solve (e.g. interaction with syntax-ppss, font-lock, eldoc,
...), tho, so time will tell which step will come second.

>> + It is the task of the calling main mode to (temporarily) set the
>> + syntax table and related variables to values suitable for the sub
>> + mode, and to call `syntax-ppss-flush-cache' if necessary.
> These instructions would be better suited for a doc for writing
> multi-mode-support facilities.

Agreed.

> This, I guess, might be useful.  But until many modes intend to use it, what
> stops you from just keeping a small alist of (MAJOR-MODE
> . OFFSETS-ALIST-VAR)?

As discussed earlier in the thread, I'm not really sure what this "extra
context" should/will ideally look like.  I think we need more experience
before we can usefully give guidance.


        Stefan

RE: antlr-mode.el - need some support by python.el

[Wedler, Christoph]

>> Since multi-mode machinery, in the general case, will require support from
>> the "primary" major modes, too, I suppose prog-mode it's not the proper
>> place for the facilitating code.

> We could place it elsewhere.  E.g. indent.el.

It sounds so obvious after you have said it...
How should we call the variable then?  Just `indent-context' ?  Or
`indent-multi-mode-context'?

>> And overall, this change looks very specific to this one use case
>> (Antlr and literate programming modes, I guess).

> It's definitely not the end of the story, indeed.  But I don't think
> anyone knows what a good general solution looks like, right now
> (especially if "good" includes "can be made to work OK without
> re-implementing the indentation algorithm of all submodes").

> We've been waiting for such a magic bullet for a long time, and I don't
> expect it to come any time soon.  This proposed prog-indentation-context
> is just a very small step along the way, but at least it's making
> some progress.

I fully agree.  It can happen that a problem becomes easier when you
generalize it extensively.  But quite often, it becomes more complex.

> [...]  There are many other problems to solve (e.g. interaction with
> syntax-ppss, font-lock, eldoc, ...), tho, so time will tell which step
> will come second.

Exactly.  Already this specific problem is quite complex.
Indentation is the first step for me, because doing it wrongly could
uglify your code as file content, whereas non-perfect syntax
highlighting just looks non-ideal in your Emacs buffer.
(Of course, I also checked syntax-highlighting - it looks pretty good
even with the current hands-on approach).

        Christoph

Re: antlr-mode.el - need some support by python.el

[Dmitry Gutov]

On 02/18/2015 07:48 AM, Stefan Monnier wrote:

 > Narrowing sucks, so no it's not a viable alternative.

Do you suppose the submodes will implement prog-indentation-context 
support using something other than narrowing? It's a powerful approach.

>> LEFTMOST-COL, likewise, can by added by the calling function.  No need to
>> make the submode's indentation function be aware of it.
>
> Yes and no: currently, the standard API to indentation is just
> indent-line-function which performs the indentation directly, rather
> than returning an indentation column.  So implementing LEFTMOST-COL
> "outside" means re-indenting after you called the indentation function.

I think I'd rather change the indent-line-function API, to return the 
column offset. Or add a new one, like `line-indentation-function'.

> Furthermore, I can imagine some cases where the inner mode may still
> want to indent to a column smaller than LEFTMOST-COL for good reasons.

I wonder what that is. But anyway, line-indentation-function would 
return a negative value in that case.

RE: antlr-mode.el - need some support by python.el

[Wedler, Christoph]

>> Narrowing sucks, so no it's not a viable alternative.

> Do you suppose the submodes will implement prog-indentation-context 
> support using something other than narrowing? It's a powerful approach.

Indeed, in my proposed patch for python.el (1st version), I changed the
call of `widen' by `python-widen' which is defined as follows

(defun python-widen ()
  (widen)
  (when (car python-submode-indentation-context)
    (narrow-to-region (caar python-submode-indentation-context)
                      (or (cdar python-submode-indentation-context)
                          (point-max)))))

> I think I'd rather change the indent-line-function API, to return the 
> column offset. Or add a new one, like `line-indentation-function'.

This would be useful, yes.

Re: antlr-mode.el - need some support by python.el

[Stefan Monnier]

>> Narrowing sucks, so no it's not a viable alternative.
> Do you suppose the submodes will implement prog-indentation-context support
> using something other than narrowing? It's a powerful approach.

They can choose to do so if it doesn't break some of the things they do,
of course, but forcing it upon them will lead to trouble.

> I think I'd rather change the indent-line-function API, to return the column
> offset.  Or add a new one, like `line-indentation-function'.

Agreed, but someone's got to do it.  FWIW I've appended code I've had in
my prog-mode.el for many years (e.g. long before add-function came
along ;-).

BTW, I consider any mode which needs auto-indentation to be
a "programming mode".  That includes LaTeX, XML, ... (which are also
text-modes, but I don't think the two are mutually exclusive).

>> Furthermore, I can imagine some cases where the inner mode may still
>> want to indent to a column smaller than LEFTMOST-COL for good reasons.
> I wonder what that is.

E.g. in latex-mode, a "\begin{verbatim}" has to go to column 0, no
matter what the context.

> But anyway, line-indentation-function would
> return a negative value in that case.

How would it know it has to do that (and how negative should it be) if
it's not told about LEFTMOST-COL?


        Stefan


;; Generic indentation support ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

(defun prog-indent-line (&optional arg)
  "Reindent current line.
If ARG is non-nil, indent any additional lines of the same expression rigidly
along with this one."
  (interactive)
  (if arg
      ;; We could use it to reindent the sexp at point:
      ;; (indent-region (point)
      ;;                (progn (forward-sexp 1) (point)))
      ;; But Lisp-mode by default does "indent any additional lines of the
      ;; same expression rigidly along with this one.", so let's
      ;; reproduce that.
      (let ((old-indent (current-indentation)))
        (if (eq (indent-according-to-mode) 'noindent)
            (funcall (default-value 'indent-line-function)))
        (let ((new-indent (current-indentation))
              (beg (line-beginning-position 2))
              (end (save-excursion (beginning-of-line)
                                   (forward-sexp 1)
                                   (point))))
          (when (and (> end beg) (/= new-indent old-indent))
            (indent-code-rigidly beg end (- new-indent old-indent)))))
    ;; Normal case.
    (let* ((savep (point))
           (indent (condition-case nil
                       (save-excursion
                         (forward-line 0)
                         (skip-chars-forward " \t")
                         (if (>= (point) savep) (setq savep nil))
                         (or (prog-indent-calculate) 0))
                     (error 0))))
      (if (not (numberp indent))
          ;; If something funny is used (e.g. `noindent'), return it.
          indent
        (if (< indent 0) (setq indent 0)) ;Just in case.
        (if savep
            (save-excursion (indent-line-to indent))
          (indent-line-to indent))))))

(defvar prog-indent-calculate-functions
  '(prog-indent-calculate-in-string
    prog-indent-calculate-fixindent
    prog-indent-calculate-comment
    prog-indent-calculate-close-sexp)
  "Functions to indent the current line.
Each function is called in turn with no argument until one of them returns
  a non-nil value, which is then used as the column number to which point
  should be indented.
Each function is responsible for restoring point to its correct location
  for the next function to work properly.
Functions should expect to be sometimes called with point in
  the middle of a line, in which case they should pretend that the
  preceding text is on another line.")

(defun prog-indent-calculate ()
  "Compute the column at which point should be indented.
Point may be in the middle of a line, in which case this returns the column
at which point should be indented if it were at the beginning of a line.
May return `noindent' to indicate that point's indentation should not be
touched, typically because it is inside a string."
  (run-hook-with-args-until-success 'prog-indent-calculate-functions))

(defun prog-indent-calculate-fixindent ()
  "Don't indent the line if the magic string is there.
The magic string is \"fixindent\" alone inside a comment or at the end
of the line (preceded by a TAB) if no comment style is defined."
  (when (looking-at (concat ".*" (regexp-quote (or comment-start "\t"))
			    "[ \t]*fixindent[ \t]*"
			    (regexp-quote (or comment-end "\n"))))
    (current-indentation)))

(defun prog-indent-calculate-comment ()
  "Indentation of a comment.
The indentation used is the same as would be used for the line following
the comment, unless `comment-indent-function' says otherwise."
  ;; FIXME: This should be moved to newcomment.el.
  (let ((pos (point)))
    (when (and (not (looking-at "\\s-\\|\n")) (forward-comment 1))
      (or (save-excursion
            (goto-char pos)
            (funcall comment-indent-function))
          (save-excursion
            (skip-chars-forward " \t")
            (prog-indent-calculate))))))

(defun prog-indent-calculate-in-string ()
  "Indentation inside a string.  Don't touch it."
  (if (nth 3 (syntax-ppss)) 'noindent))

(defun prog-bolp ()
  (save-excursion (skip-chars-backward " \t") (bolp)))

(defun prog-indent-hanging-p ()
  "Non-nil if token at point is hanging.
A hanging token is a token that starts an sexp (e.g. an open parenthesis),
is not at the beginning of a line, and is at the end of a line."
  (and (not (prog-bolp))
       (looking-at "\\s(+[ \t]*$")))

(defun prog-indent-current ()
  "Current indentation at point.
Return the indentation currently used at point."
  (if (prog-indent-hanging-p)
      (prog-indent-calculate)
    (current-column)))

(defun prog-indent-calculate-close-sexp ()
  "Indentation of the closing token of an sexp."
  (and (looking-at "\\s)+")
       (save-excursion
         (goto-char (match-end 0))
         (forward-sexp -1)
         ;; Align with the matching opening token.
         (prog-indent-current))))

Re: antlr-mode.el - need some support by python.el

[Dmitry Gutov]

On 02/18/2015 07:56 PM, Stefan Monnier wrote:

> They can choose to do so if it doesn't break some of the things they do,
> of course, but forcing it upon them will lead to trouble.

Can you think of a situation when _actually_ forcing it (with an 
un-widen-able narrowing) can lead to trouble here?

>> I think I'd rather change the indent-line-function API, to return the column
>> offset.  Or add a new one, like `line-indentation-function'.
>
> Agreed, but someone's got to do it.  FWIW I've appended code I've had in
> my prog-mode.el for many years (e.g. long before add-function came
> along ;-).

This looks appealing, but way beyond my suggestion. :)

I'd omit `prog-indent-calculate-functions' for now, but otherwise 
`prog-indent-line' is mostly in line with what I'd imagined.

Anyway, I'll look into it if nobody else does, before 25 release.

> BTW, I consider any mode which needs auto-indentation to be
> a "programming mode".  That includes LaTeX, XML, ... (which are also
> text-modes, but I don't think the two are mutually exclusive).

Okay, that's a reasonable interpretation. We don't have multiple 
inheritance, though. But if prog-mode inherited from text-mode...

>>> Furthermore, I can imagine some cases where the inner mode may still
>>> want to indent to a column smaller than LEFTMOST-COL for good reasons.
>> I wonder what that is.
>
> E.g. in latex-mode, a "\begin{verbatim}" has to go to column 0, no
> matter what the context.

Are there important cases like this where we can't "flip" the modes and 
consider latex the "primary" mode instead?

>> But anyway, line-indentation-function would
>> return a negative value in that case.
>
> How would it know it has to do that (and how negative should it be) if
> it's not told about LEFTMOST-COL?

Hmm, maybe it would return nil when at the top level. And when it 
returns non-nil, that would be interpreted as the indentation was 
calculated using some preceding buffer text.

So the multi-mode package would replace nil with LEFTMOST-COL, but 
otherwise obey the returned value. The value 0 will be obeyed, too.

Re: antlr-mode.el - need some support by python.el

[Stefan Monnier]

> Can you think of a situation when _actually_ forcing it (with an
> un-widen-able narrowing) can lead to trouble here?

I can't give you such a situation off-hand, but I've seen plenty of
problems when executing innocent code in a narrowed buffer.
I've gone and rewrote all kinds of codes (e.g. filling) to avoid using
narrowing specifically for that reason.

>> E.g. in latex-mode, a "\begin{verbatim}" has to go to column 0, no
>> matter what the context.
> Are there important cases like this where we can't "flip" the modes and
> consider latex the "primary" mode instead?

All this just to avoid passing LEFTMOST-COL?


        Stefan

Re: antlr-mode.el - need some support by python.el

[Dmitry Gutov]

On 02/19/2015 05:20 AM, Stefan Monnier wrote:

> I can't give you such a situation off-hand, but I've seen plenty of
> problems when executing innocent code in a narrowed buffer.
> I've gone and rewrote all kinds of codes (e.g. filling) to avoid using
> narrowing specifically for that reason.

I've also seen problems with narrowing, but maybe classfying them and 
adding some rules would turn out a useful result.

>>> E.g. in latex-mode, a "\begin{verbatim}" has to go to column 0, no
>>> matter what the context.
>> Are there important cases like this where we can't "flip" the modes and
>> consider latex the "primary" mode instead?
>
> All this just to avoid passing LEFTMOST-COL?

Not really. The leftmost-col avoidance scheme I described further on 
would deal with latex as submode just fine.

Rather, it's a general observation: flipping the modes can simplify 
things. For instance, whenever we talk about indentation in a submode 
being dependent on the previous chunks of that submode, but not the 
primary mode contents between then, it sounds like the roles should be 
reversed.

Re: antlr-mode.el - need some support by python.el

[Stefan Monnier]

> I've also seen problems with narrowing, but maybe classfying them and adding
> some rules would turn out a useful result.

Could be.  But using `widen' is a simple solution to those problems
(until you bump into problematic cases like Rmail and Info where
`widen' ends up widening too much).

BTW, a good start might be for someone to sit down and design a sane
way for narrow and syntax-ppss to interact.  If that includes a cleanish
way to support multiple modes, then it's even better.

>>>> E.g. in latex-mode, a "\begin{verbatim}" has to go to column 0, no
>>>> matter what the context.
>>> Are there important cases like this where we can't "flip" the modes and
>>> consider latex the "primary" mode instead?
>> All this just to avoid passing LEFTMOST-COL?
> Not really. The leftmost-col avoidance scheme I described further on would
> deal with latex as submode just fine.

Still: why do you want to avoid LEFTMOST-COL at all?

> Rather, it's a general observation: flipping the modes can simplify
> things. For instance, whenever we talk about indentation in a submode being
> dependent on the previous chunks of that submode, but not the primary mode
> contents between then, it sounds like the roles should be reversed.

I don't see how that influences the usefulness (or not) for
LEFTMOST-COL.


        Stefan

Re: antlr-mode.el - need some support by python.el

[Dmitry Gutov]

On 02/19/2015 03:18 PM, Stefan Monnier wrote:

> Could be.  But using `widen' is a simple solution to those problems
> (until you bump into problematic cases like Rmail and Info where
> `widen' ends up widening too much).

Maybe, whenever `widen' is required, the original caller is doing 
something wrong. The only valid use for `widen' is to undo the "visual" 
sort of narrowing, enacted by the user interactively. If that feature 
ever gets removed, I won't shed a tear.

> BTW, a good start might be for someone to sit down and design a sane
> way for narrow and syntax-ppss to interact.  If that includes a cleanish
> way to support multiple modes, then it's even better.

My first reaction to this: invalidate the cache more often. Like if the 
cached entry is for the position above point-min, don't use it, and 
calculate the state starting from point-min. This way `syntax-ppss' will 
remain a good substitute for `parse-partial-sexp'.

But then, do we allow it to add entries to cache (and save 
syntax-ppss-last), when narrowed? What will happen if syntax-pps is 
called without narrowing, after these entries are added? To keep ideal 
results, we'd have to save syntax-ppss-point-min-last, as well as 
include point-min in the cache entries, and invalidate the latter 
whenever it doesn't match.

And I'm pretty sure, in this scenario, the cached entries from submode 
regions won't ever be useful in the primary region, nor a cache entry 
from one submode region will be useful in another. So we could as well 
discard them, and discard syntax-ppss-last after we leave (in a way 
dependent on the task we've using syntax-ppss for) the subregion.

On the other hand, as long as narrowing is only used locally (within, 
say, `save-restriction'), the same function might as well bind 
`syntax-ppss-last' and `syntax-ppss-cache' to nil, so that multiple 
calls to `syntax-ppss' within the narrowing are still quick (that's 
important if the function in question is 
font-lock-fontify-region-function; a "composite" one, if we're 
discussing the multiple-mode case), but after we're done, nothing is 
remembered.

The previously mentioned idea of adding a variable like 
syntax-ppss-function will help a lot more in the situation when there's 
no narrowing, and we need to compute the accurate level of nesting in 
the primary mode (or, to complicate everything by an order of magnitude, 
when we're in a region that itself has subregions, but I digress). To 
ignore several regions within, for the purposes of syntax-ppss.

`add-function' could even be used instead of the new variable.

To sum up:

- Strong transient narrowing solves a class of problems.
- Anyone doing it might have to take care of syntax-ppss cache (binding 
two vars to nil isn't hard).
- The effect of persistent narrowing (one that lives between command 
invocations) should be only visual (the rest of the buffer contents are 
not displayed). Alternatively, if we're worried about editing commands 
changing text beyond the visible area, it should be made read-only. Or 
we just keep the current narrowing mechanism for visual purposes, and 
avoid using it outside of interactive commands. Or remove it entirely, 
like mentioned above.

>> Not really. The leftmost-col avoidance scheme I described further on would
>> deal with latex as submode just fine.
>
> Still: why do you want to avoid LEFTMOST-COL at all?

I don't mind it that much, rather I'd like to avoid the variable 
submode-indentation-context. If prog-indent-calculate-function gets 
passed LEFTMOST-COL instead of nothing, that can work well.

Benefits:

- Since the value is passed to the function, the implementors are more 
likely to pay attention and include it in the calculation.

- It's somewhat more likely that they get it right. That nil is returned 
in all top-level places, is much harder to test interactively.

The one con, I guess, is a bit more complicated API.

> I don't see how that influences the usefulness (or not) for
> LEFTMOST-COL.

Um yeah, it's irrelevant.

RE: antlr-mode.el - need some support by python.el

[Wedler, Christoph]

>> Could be.  But using `widen' is a simple solution to those problems
>> (until you bump into problematic cases like Rmail and Info where
>> `widen' ends up widening too much).

It might be a good idea to disallow widen and narrw-xxx in these modes.

> Maybe, whenever `widen' is required, the original caller is doing 
> something wrong. The only valid use for `widen' is to undo the "visual" 
> sort of narrowing, enacted by the user interactively. If that feature 
> ever gets removed, I won't shed a tear.

I tend to agree, now that many commands like query-replace also work on
the highlighted region.  Non-friends of transient-mark-mode probably
won't agree.

>> Not really. The leftmost-col avoidance scheme I described further on would
>> deal with latex as submode just fine.
>
> Still: why do you want to avoid LEFTMOST-COL at all?

> I don't mind it that much, rather I'd like to avoid the variable
> submode-indentation-context.  If prog-indent-calculate-function gets
> passed LEFTMOST-COL instead of nothing, that can work well.

I would be happy with it.  Additionally to that function (and its
parameter LEFTMOST-COL), necessary changes would be:

 1. indent-for-tab-command and friends must call widen (inside a
    save-restriction of course) to allow the indentation of all modes
    work correctly with visible narrowing.

    (BTW, it could be even made that indentation of the modes work
    correctly with intra-defun narrowing - currently, EmacsLisp mode
    indents at col 0 here, C++ mode dumps, ...)

 2. prog-indent-calculate-function (or whatever it is called) AND ALL
    CALLED FUNCTIONS must not call widen (in order to work in the
    multi-mode case).  This could actually induce a bit more work than
    the change induced by submode-indentation-context.

> Benefits:

> - Since the value is passed to the function, the implementors are more 
> likely to pay attention and include it in the calculation.

True

> - It's somewhat more likely that they get it right. That nil is returned 
>  in all top-level places, is much harder to test interactively.

If LEFTMOST-COL is passed to the function in
prog-indent-calculate-function (and respected), there is no need to
distinguish between 0 (indentation offset calculated relatively to some
previous line) and nil (indentation offset calculated absolutely,
e.g. because the language construct is top-level).

     Christoph

Re: antlr-mode.el - need some support by python.el

[Stefan Monnier]

>> Could be.  But using `widen' is a simple solution to those problems
>> (until you bump into problematic cases like Rmail and Info where
>> `widen' ends up widening too much).
> Maybe, whenever `widen' is required, the original caller is doing something
> wrong. The only valid use for `widen' is to undo the "visual" sort of
> narrowing, enacted by the user interactively.

Then there are many invalid uses of narrow-to-region, such as those used
to operate on a particular region of the buffer (and where you can use
markers instead).


        Stefan

Re: antlr-mode.el - need some support by python.el

[Dmitry Gutov]

On 03/01/2015 07:04 PM, Stefan Monnier wrote:

> Then there are many invalid uses of narrow-to-region, such as those used
> to operate on a particular region of the buffer (and where you can use
> markers instead).

Maybe we need examples: narrow-to-region to operate on a particular 
region of the buffer is its primary use case.

As long as it's wrapped in `save-restriction', no `widen' calls are 
needed, unless the operating code somehow requires it. But then we're 
not just operating on that particular region anymore.

Re: antlr-mode.el - need some support by python.el

[Stefan Monnier]

> As long as it's wrapped in `save-restriction', no `widen' calls are needed,

Why?  This "code which operates on this region" can very easily want to
call indent-according-to-mode or syntax-ppss to do its job.


        Stefan

Re: antlr-mode.el - need some support by python.el

[Dmitry Gutov]

On 03/02/2015 07:23 AM, Stefan Monnier wrote:

> Why?  This "code which operates on this region" can very easily want to
> call indent-according-to-mode or syntax-ppss to do its job.

Maybe we can say that code inside narrowing is only allowed to call 
`generic' features, which may or may not perform widening themselves.

For indent-according-to-mode, see my previous message.

The uses of `syntax-ppss' are harder to classify, but for font-lock we 
at least have font-lock-dont-widen. It's hard for the "inner" code to 
know whether it's allowed to widen, so when in doubt, it probably shouldn't.

But to keep the current code everywhere more or less unchanged, we can 
come from the other side and use an extra-flexible solution: introduce 
`widen-function' or `widen-bounds'. The former would be the function 
`widen' would delegate to from now on (and the interested parties would 
be able to bind it to #'ignore), and the latter would be the maximum 
bounds the new implementation of `widen' would widen to.

Re: antlr-mode.el - need some support by python.el

[Stefan Monnier]

> Maybe we can say that code inside narrowing is only allowed to call
> `generic' features, which may or may not perform widening themselves.

I'm not sure what that buys us.

> The uses of `syntax-ppss' are harder to classify, but for font-lock we at
> least have font-lock-dont-widen.

But that's very crude.  It only accounts for those (rather rare) cases
where narrowing is used to focus on a "sub-buffer" (i.e. Info mode, and
old Rmail).  If something uses narrowing in those modes, then the result
is plain breakage.

> It's hard for the "inner" code to know whether it's allowed to widen,
> so when in doubt, it probably shouldn't.

Most uses of narrowing should be widened, so the rule goes rather in
the opposite direction (which is why font-lock-dont-widen defaults to
nil, for example).

> But to keep the current code everywhere more or less unchanged, we can come
> from the other side and use an extra-flexible solution: introduce
> `widen-function' or `widen-bounds'.

Right, that's basically attacking the problem of distinguishing between
different kinds of narrowing.

But in any case, I think that relying on narrowing for the multi-mode
support is a bad idea.  It's OK if the API *allows* the use of
narrowing, but requiring it would be a mistake.


        Stefan

Re: antlr-mode.el - need some support by python.el

[Dmitry Gutov]

On 03/02/2015 06:48 PM, Stefan Monnier wrote:
>> Maybe we can say that code inside narrowing is only allowed to call
>> `generic' features, which may or may not perform widening themselves.
>
> I'm not sure what that buys us.

The multi-mode facilities can override the generic features via 
variables and/or advices, and disable widening, or make it somehow smarter.

>> The uses of `syntax-ppss' are harder to classify, but for font-lock we at
>> least have font-lock-dont-widen.
>
> But that's very crude.  It only accounts for those (rather rare) cases
> where narrowing is used to focus on a "sub-buffer" (i.e. Info mode, and
> old Rmail).  If something uses narrowing in those modes, then the result
> is plain breakage.

Indeed. But a multi-mode font-lock-fontify-region-function could perform 
the "smarter" widening, which wouldn't suffer from this exact problem.

> Most uses of narrowing should be widened, so the rule goes rather in
> the opposite direction (which is why font-lock-dont-widen defaults to
> nil, for example).

That wouldn't be my preference, but either way, there could be a way to 
disable or alter the behavior of `widen'.

While `widen-function' should be enough for this, maybe it's too 
flexible after all: that variable would never be used (correctly) with a 
buffer-local value. Only with local dynamic binding.

> Right, that's basically attacking the problem of distinguishing between
> different kinds of narrowing.
>
> But in any case, I think that relying on narrowing for the multi-mode
> support is a bad idea.

Maybe it's unfortunate that the main approaches to multi-mode 
implementation ended up using one of the two bad ideas (indirect buffers 
and narrowing), but they did, and we still don't have a good model from 
implementing the needed features in a different way.

 > It's OK if the API *allows* the use of
> narrowing, but requiring it would be a mistake.

I don't think any version of the API requires it. But in the multi-mode 
context, I indeed expect it to be used.

Anyway, I think doing `widen' in prog-indent-line and prohibiting (in 
the docstring) prog-indent-calculate-functions from doing that should 
work just as well as prog-indentation-context (and better, from my 
standpoint).

It doesn't require any changes to the narrowing/widening logic, so the 
question of whether narrowing is a good approach can again be deferred 
until later.

Re: antlr-mode.el - need some support by python.el

[Stefan Monnier]

> While `widen-function' should be enough for this, maybe it's too flexible
> after all: that variable would never be used (correctly) with a buffer-local
> value. Only with local dynamic binding.

Actually, for Info-mode, it wouldn't be "scoped", so it would need to be
buffer-local (and even if it's scoped, it needs to say to which buffer
it applies).

>> It's OK if the API *allows* the use of
>> narrowing, but requiring it would be a mistake.
> I don't think any version of the API requires it.

IIUC, you want to remove the (START . END) data:

   > + The non-nil value looks as follows
   > +    ((START . END) LEFTMOST-COL)
   The first element tries to re-implement what's currently being handled with
   narrowing, successfully. Why?

I understood this as saying you want to enforce the use of narrowing to
pass to the sub-mode to bounds of its chunk.

> It doesn't require any changes to the narrowing/widening logic, so the
> question of whether narrowing is a good approach can again be deferred
> until later.

If you remove (START . END), then you decide that narrowing is the only
way to go.


        Stefan

Re: antlr-mode.el - need some support by python.el

[Dmitry Gutov]

On 03/02/2015 08:51 PM, Stefan Monnier wrote:

> Actually, for Info-mode, it wouldn't be "scoped", so it would need to be
> buffer-local (and even if it's scoped, it needs to say to which buffer
> it applies).

Sorry, can't follow you here. Not really familiar with Info-mode (even 
though I've tried to get into it a few times). But it's good if 
widen-function can have normal uses.

> IIUC, you want to remove the (START . END) data:
>
>     > + The non-nil value looks as follows
>     > +    ((START . END) LEFTMOST-COL)
>     The first element tries to re-implement what's currently being handled with
>     narrowing, successfully. Why?
>
> I understood this as saying you want to enforce the use of narrowing to
> pass to the sub-mode to bounds of its chunk.

First of all, we've already agreed (I think?) to move LEFTMOST-COL from 
that variable to an argument of the indent-calculate function.

Thus, removing (START . END) will amount to not introducing the 
aforementioned variable. Maybe never, maybe not just yet.

While LEFTMOST-COL is quite useful, (START . END) is less so, since it 
would replace what already works in e.g. mmm-mode is doing it now, *and* 
it would require more changes to existing modes.

>> It doesn't require any changes to the narrowing/widening logic, so the
>> question of whether narrowing is a good approach can again be deferred
>> until later.
>
> If you remove (START . END), then you decide that narrowing is the only
> way to go.

It can be added later, although I don't see a lot of value in it.

Honestly, if we had such variable, after we have prog-indent-line 
delegating to indent-calculate functions, I'd be very tempted to 
implement its support in prog-indent-line itself, instead of leaving it 
up to individual modes. And yes, using narrowing.

Re: antlr-mode.el - need some support by python.el

[Stefan Monnier]

>> Actually, for Info-mode, it wouldn't be "scoped", so it would need to be
>> buffer-local (and even if it's scoped, it needs to say to which buffer
>> it applies).
> Sorry, can't follow you here.

Info-mode's buffers (i.e. *.info files) contain several Info pages at
the same time, so when you "go to an Info page" what really happens is
that the mode narrows the buffer to the corresponding chunk in
the buffer.  Clearly this is not scoped inside a `let': the narrowing
will simply be in effect until you jump to another page, at which point
the narrowing will be changed to select another chunk.

> But it's good if widen-function can have normal uses.

Indeed, I don't see any problem there (I was just pointing out that
assuming let-like scoping is a bad idea).

> First of all, we've already agreed (I think?) to move LEFTMOST-COL from that
> variable to an argument of the indent-calculate function.

I can't remember agreeing to this level of detail, but I'm not
necessarily opposed to that.

> Thus, removing (START . END) will amount to not introducing the
> aforementioned variable. Maybe never, maybe not just yet.

The issue is simply: how to tell the submode what are the bounds of its chunk.
If you don't do it by passing START/END, then you have to do it via side
channels such as by narrowing.

> While LEFTMOST-COL is quite useful, (START . END) is less so,

The information of the bounds of the chunk is indispensable (the END
part much less so, but the START part very much so).

> modes. And yes, using narrowing.

And I'm firmly opposed to imposing such an API.  I much prefer passing
START/END via dynamically scoped vars or via explicit arguments, and
then let the submode use a little wrapper that sets up narrowing and
calls the "same old" code (if the submode prefers using narrowing).


        Stefan

RE: antlr-mode.el - need some support by python.el

[Wedler, Christoph]

>> First of all, we've already agreed (I think?) to move LEFTMOST-COL from that
>> variable to an argument of the indent-calculate function.

> I can't remember agreeing to this level of detail, but I'm not
> necessarily opposed to that.

I just remember the following: if we would introduce a
prog-indent-calculate-function, we could as well pass that value as
argument to that function (with a benefit that the value will be more
likely be respected).

But this would not be enough: we need to do something similar to
`indent-region-function', i.e. the different indent region functions
should also respect LEFTMOST-COL.

In my opinion, this would only worth the trouble if we can get
completely rid of `prog-indentation-context'.

>> While LEFTMOST-COL is quite useful, (START . END) is less so,

> The information of the bounds of the chunk is indispensable (the END
> part much less so, but the START part very much so).

I fully agree (END is just necessary for the theoretical case that the
indentation calculation checks lines after the current one).

>> modes. And yes, using narrowing.

> And I'm firmly opposed to imposing such an API.  I much prefer passing
> START/END via dynamically scoped vars or via explicit arguments, and
> then let the submode use a little wrapper that sets up narrowing and
> calls the "same old" code (if the submode prefers using narrowing).

I agree.  If we do the START/END boundaries via narrowing, we probably
induce too much work in major modes.  As I wrote earlier: all
functions which are called by the indent calculation must not call widen
anymore - with the dynamically scoped variable, they can just replace
(widen) by a call of a widen-like function which respects START/END - we
could actually introduce a function `prog-widen' in prog-mode.el which
does that.

In summary: as we do not get rid of `prog-indentation-context', we could
as well stick to the original proposal.

Re: antlr-mode.el - need some support by python.el

[Dmitry Gutov]

On 03/04/2015 06:53 PM, Wedler, Christoph wrote:

> I just remember the following: if we would introduce a
> prog-indent-calculate-function, we could as well pass that value as
> argument to that function (with a benefit that the value will be more
> likely be respected).

That's a considerable benefit, IMO.

> But this would not be enough: we need to do something similar to
> `indent-region-function', i.e. the different indent region functions
> should also respect LEFTMOST-COL.

IME, most of the time that variable's value is nil. But of course, we 
could pass an extra argument to it too (and fall back to the current 
calling convention if the call signals `wrong-number-of-arguments'.

> In my opinion, this would only worth the trouble if we can get
> completely rid of `prog-indentation-context'.

Yes, probably.

> If we do the START/END boundaries via narrowing, we probably
> induce too much work in major modes.  As I wrote earlier: all
> functions which are called by the indent calculation must not call widen
> anymore

How is it "too much work"? I've written my share of indentation code, 
and the only related function that I found useful were 
`save-restriction' and `narrow-to-region'.

RE: antlr-mode.el - need some support by python.el

[Wedler, Christoph]

>> If we do the START/END boundaries via narrowing, we probably
>> induce too much work in major modes.  As I wrote earlier: all
>> functions which are called by the indent calculation must not call widen
>> anymore

> How is it "too much work"? I've written my share of indentation code, 
> and the only related function that I found useful were 
> `save-restriction' and `narrow-to-region'.

save-restriction: sure, every use of narrow/widen in indentation code
should be in a save-restriction.

narrow-to-region: for indentation code, this is probably only multi-mode
related

widen: If you grep for `widen' in EMACS/lisp/progmodes, you find quite a
few matches.  Probably half of them are in indentation functions or
functions which are used by indentation functions.

And actually, they uses are _correct_, as the indentation commands
should also work when the user has used narrowing (via C-x n n and
friends).
(As I wrote earlier, quite a few major modes do not work correctly
or at least strange with intra-defun narrowing: C++, EmacsLisp, ...).


If we do the START/END boundaries only via narrowing, only the outer
indentation command should use widen (which the multi-mode code won't
call), all called function are not allowed to use widen anymore.  In
other words, the widen calls must be moved to the outer commands.

If we do the START/END boundaries via the dynamically scoped variable,
the major modes just have to replace their (widen) call by (prog-widen),
a widen function which respects prog-indentation-context.
If the multi-mode indentation function additionally narrows to
START/END, our approach also works for sub modes which do not call widen.

Yes, relying on dynamically scoped variable is not completely nice, but
there had been no programming guideline that indentation code should
only call widen only once directly at the beginning (as I said before:
inside save-restriction, of course).
I would not introduce such a guideline after decades...

Re: antlr-mode.el - need some support by python.el

[Dmitry Gutov]

On 03/05/2015 11:46 AM, Wedler, Christoph wrote:

> narrow-to-region: for indentation code, this is probably only multi-mode
> related

Yeah, maybe. The old indentation engine uses it in one place, but that's 
probably just a bad decision.

> widen: If you grep for `widen' in EMACS/lisp/progmodes, you find quite a
> few matches.  Probably half of them are in indentation functions or
> functions which are used by indentation functions.

True. But I'd expect this to change, after either of the proposed 
schemes is implemented.

Even if the major mode indentation function does the widening itself, 
that should happen in one place, right? Or otherwise centralized, as you 
suggest below.

> If we do the START/END boundaries only via narrowing, only the outer
> indentation command should use widen (which the multi-mode code won't
> call), all called function are not allowed to use widen anymore.  In
> other words, the widen calls must be moved to the outer commands.

Right. In this case, indent-according-to-mode (for instance) could call 
`widen'. It's a rather straightforward change.

> If we do the START/END boundaries via the dynamically scoped variable,
> the major modes just have to replace their (widen) call by (prog-widen),
> a widen function which respects prog-indentation-context.

That sounds good to me too, but then we should call it something like 
`prog-submode-widen', right? And then maybe introduce 
`prog-submode-narrow-to-region' and `prog-submode-save-restriction'.

Et voilà, we now have a "new kind of narrowing", an approach Stefan, 
AFAICT, is still on the fence about.

The above is also a bit less flexible than having a `widen-function'.

> If the multi-mode indentation function additionally narrows to
> START/END, our approach also works for sub modes which do not call widen.

Yes, that's a nice bonus either way.

> Yes, relying on dynamically scoped variable is not completely nice, but
> there had been no programming guideline that indentation code should
> only call widen only once directly at the beginning (as I said before:
> inside save-restriction, of course).
> I would not introduce such a guideline after decades...

I believe it's totally fine to introduce. It's not like the currently 
written modes will suddenly break. Only if they don't follow the 
guideline, they'll work worse in a multi-mode context.

Note that your `prog-widen' suggestion (as well as 
`prog-indentation-context') also requires changes in those modes' 
implementations.

Re: antlr-mode.el - need some support by python.el

[Dmitry Gutov]

On 03/05/2015 02:29 PM, Dmitry Gutov wrote:

>> narrow-to-region: for indentation code, this is probably only multi-mode
>> related
>
> Yeah, maybe. The old   indentation engine uses it in one place, but that's
                        ^ ruby-mode
> probably just a bad decision.

RE: antlr-mode.el - need some support by python.el

[Wedler, Christoph]

What is the status of the status of the discussion / my patch which has
addressed the following issue:

If multi-modes (for grammars with actions, literate programming, ...)
want to delegate the indentation of sub-mode specific code to the
corresponding functions of the sub-mode, they face the following issues

  a) the indentation engine of some major modes do a `widen', because
  they want to do the right thing if the user has used narrowing (but
  then it gets confused by the surrounding grammar code)

  b) if the indentation engine just looks at the visible part of the
  code, it will indent top-level lines at column 0.  This might not be
  the right thing for code snippets (grammar actions).

  There might be a workaround, but not for sub-languages with
  indentation-based block structures (Python, Haskell, ...), where the
  indentation works by cycling through all possible indentation
  positions.

  c) for the indentation of code-snippets to work properly, the
  indentation engine might need to know the context, e.g. as
  externally provided string (for grammars) or by a function which
  provides (previous) code blocks of the current buffer.


Potential solutions:

  1. no nothing

  2. (my approach/patch): dynamically bind a variable in the outer mode
  which specifies

    - the region for any potential `widen' in the sub-mode's indentation
      engine (the submode can just replace all their call of `widen' by
      `prog-widen')
    - the leftmost column for the indentation
    - the context for issue c above - this is optional for the sub-mode
      to support as it would require more substantial support by it

  3. require that all indentation engines to not call widen, do that in
  `indent-according-to-mode' and friends; passing the leftmost-column as
  parameter to the major-mode specific indentation functions, ...

I would vote for (2) and would like to hear suggestions for my
corresponding patch.
I fear that wanting (3) ends up BEING just the solution (1).

Regards,
Christoph

Re: antlr-mode.el - need some support by python.el

[Stefan Monnier]

>   2. (my approach/patch): dynamically bind a variable in the outer mode
>   which specifies

Yes, go ahead with this one.


        Stefan

RE: antlr-mode.el - need some support by python.el

[Wedler, Christoph]

>>   2. (my approach/patch): dynamically bind a variable in the outer mode
>>   which specifies

> Yes, go ahead with this one.

OK, fine.
What was the issue with the patch I've sent you?  (one or more of...):

 1. the patch got lost during the final stages of the 24.5 release...

 2. it should be a patch of indent.el, not prog-mode.el

 3. function `prog-widen' should be included

 4. more explanation of CONTEXT

 5. other - which?

For a new patch: diff versus the master or emacs-24 git branch?

- Christoph

Re: antlr-mode.el - need some support by python.el

[Stefan Monnier]

> What was the issue with the patch I've sent you?  (one or more of...):

Let's see...

>  1. the patch got lost during the final stages of the 24.5 release...

Yes.

>  2. it should be a patch of indent.el, not prog-mode.el

I think either way is fine.

>  3. function `prog-widen' should be included

No, that's largely independent.

> For a new patch: diff versus the master or emacs-24 git branch?

Against "master", please.


        Stefan

RE: antlr-mode.el - need some support by python.el

[Wedler, Christoph]

Hi Stefan,

finally, I prepared a patch (vs master fetched this morning ca 06:10 UTC).
I incorporated the things we had in the mail discussions earlier

Remarks / corrections welcome.
I hope the format is correct.

Regards,
Christoph

5102d1d513d82d7a34851411494c29624c50ae47 HEAD master
Author: Christoph Wedler <christoph.wedler@sap.com>
Date:   Wed Jun 3 13:54:31 2015 +0000

    Some generic support for multi-mode indentation.

2 files changed, 63 insertions(+)
 ChangeLog.2                 |  7 ++++++
 lisp/progmodes/prog-mode.el | 56 +++++++++++++++++++++++++++++++++++++++++++++

	Modified   ChangeLog.2
diff --git a/ChangeLog.2 b/ChangeLog.2
index 115ccda..e110ea0 100644
--- a/ChangeLog.2
+++ b/ChangeLog.2
@@ -1,3 +1,10 @@
+2015-06-03  Christoph Wedler  <christoph.wedler@sap.com>
+
+	Some generic support for multi-mode indentation.
+	* lisp/progmodes/prog-mode.el (prog-indentation-context): New
+	variable.
+	(prog-first-column, prog-widen): New convenience functions.
+
 2015-05-30  Dmitry Gutov  <dgutov@yandex.ru>
 
 	Make sure there's no explicit tag name
	Modified   lisp/progmodes/prog-mode.el
diff --git a/lisp/progmodes/prog-mode.el b/lisp/progmodes/prog-mode.el
index 0d9fabd..33abb6d 100644
--- a/lisp/progmodes/prog-mode.el
+++ b/lisp/progmodes/prog-mode.el
@@ -48,6 +48,43 @@
     map)
   "Keymap used for programming modes.")
 
+(defvar prog-indentation-context nil
+  "Non-nil while indenting embedded code chunks.
+There are languages where part of the code is actually written in
+a sub language, e.g., a Yacc/Bison or ANTLR grammar also consists
+of plain C code.  This variable enables the major mode of the
+main language to use the indentation engine of the sub mode for
+lines in code chunks written in the sub language.
+
+When a major mode of such a main language decides to delegate the
+indentation of a line/region to the indentation engine of the sub
+mode, it is supposed to bind this variable to non-nil around the call.
+
+The non-nil value looks as follows
+   \(FIRST-COLUMN (START . END) PREVIOUS-CHUNKS)
+
+FIRST-COLUMN is the column the indentation engine of the sub mode
+should usually choose for top-level language constructs inside
+the code chunk (instead of 0).
+
+START to END is the region of the code chunk.  See function
+`prog-widen' for additional info.
+
+PREVIOUS-CHUNKS, if non-nil, provides the indentation engine of
+the sub mode with the virtual context of the code chunk.  Valid
+values are:
+
+ - A string containing code which the indentation engine can
+   consider as standing in front of the code chunk.  For example,
+   it can contain a function header to make the code chunk
+   being correctly indented as a function body.
+
+ - A function called with the start position of the current
+   chunk.  It will return either the region of the previous chunk
+   as \(PREV-START . PREV-END) or nil if there is no further
+   previous chunk.")
+
+
 (defun prog-indent-sexp (&optional defun)
   "Indent the expression after point.
 When interactively called with prefix, indent the enclosing defun
@@ -61,6 +98,25 @@ instead."
 	  (end (progn (forward-sexp 1) (point))))
       (indent-region start end nil))))
 
+(defun prog-first-column ()
+  "Return the indentation column normally used for top-level constructs."
+  (or (car prog-indentation-context) 0))
+
+(defun prog-widen ()
+  "Remove restrictions (narrowing) from current code chunk or buffer.
+This function should be used instead of `widen' in any function
+used by the indentation engine to make it respect the value
+`prog-indentation-context'.
+
+This function (like 'widen') is useful inside a
+`save-restriction' to make the indentation correctly work when
+narrowing is in effect."
+  (widen)
+  (let ((chunk (cadr prog-indentation-context)))
+    (when chunk
+      (narrow-to-region (car chunk) (or (cdr chunk) (point-max))))))
+
+
 (defvar-local prettify-symbols-alist nil
   "Alist of symbol prettifications.
 Each element looks like (SYMBOL . CHARACTER), where the symbol

Re: antlr-mode.el - need some support by python.el

[Stefan Monnier]

> +FIRST-COLUMN is the column the indentation engine of the sub mode
> +should usually choose for top-level language constructs inside
> +the code chunk (instead of 0).
> +
> +START to END is the region of the code chunk.  See function
> +`prog-widen' for additional info.

These depends on the surrounding outer mode, so that's fine.

> +PREVIOUS-CHUNKS, if non-nil, provides the indentation engine of
> +the sub mode with the virtual context of the code chunk.  Valid
> +values are:
> +
> + - A string containing code which the indentation engine can
> +   consider as standing in front of the code chunk.  For example,
> +   it can contain a function header to make the code chunk
> +   being correctly indented as a function body.
> +
> + - A function called with the start position of the current
> +   chunk.  It will return either the region of the previous chunk
> +   as \(PREV-START . PREV-END) or nil if there is no further
> +   previous chunk.")

This is very unclear, so I'm not sure how an inner-mode could make use
of it.  Who (and how) decides what the values will be?  The outer-mode
or the inner-mode?

> +(defun prog-widen ()
> +  "Remove restrictions (narrowing) from current code chunk or buffer.
> +This function should be used instead of `widen' in any function
> +used by the indentation engine to make it respect the value
> +`prog-indentation-context'.

I don't see a need for "should" here.

> +  (widen)
> +  (let ((chunk (cadr prog-indentation-context)))
> +    (when chunk
> +      (narrow-to-region (car chunk) (or (cdr chunk) (point-max))))))

Better do

     (let ((chunk (cadr prog-indentation-context)))
       (if chunk
           (narrow-to-region (car chunk) (or (cdr chunk) (point-max)))
         (widen))))


-- Stefan

RE: antlr-mode.el - need some support by python.el

[Wedler, Christoph]

>> +PREVIOUS-CHUNKS, if non-nil, provides the indentation engine of
>> +the sub mode with the virtual context of the code chunk.  Valid
>> +values are:
>> +
>> + - A string containing code which the indentation engine can
>> +   consider as standing in front of the code chunk.  For example,
>> +   it can contain a function header to make the code chunk
>> +   being correctly indented as a function body.
>> +
>> + - A function called with the start position of the current
>> +   chunk.  It will return either the region of the previous chunk
>> +   as \(PREV-START . PREV-END) or nil if there is no further
>> +   previous chunk.")

> This is very unclear, so I'm not sure how an inner-mode could make use
> of it.

Many modes use more than the current line / code chunk in their
indentation calculations.  For example, in `c-guess-basic-syntax', the
outer language constructs are also considered.  If the indentation
engine of the inner mode just see the code chunks, there is a potential
problem...

Code chunks in grammars (like ANTLR) are valid sub constructs (basically
like a function bodies), but are not necessarily valid top/root constructs
according to the grammar of the inner mode.  This might affect the
indentation calculation - and if so, the inner mode hopefully uses the
PREVIOUS-CHUNKS specification, e.g. `c-guess-basic-syntax' could use it
to compute an initial value for `c-syntactic-context'.

In the literate programming case (which I'm less interested in, but you
and Fabián seemed to be), the code chunks are not necessary even valid
sub constructs, e.g. a code chunk could just be the "else" clause.  Here,
it might be even essential that the indentation engine of the inner mode
knows the previous code chunks.

> Who (and how) decides what the values will be?  The outer-mode
> or the inner-mode?

The outer mode (like the rest of the value) - after all, the preceding
part of the docstring was "the major mode of such a main language [...]
is supposed to bind this variable"

>> +(defun prog-widen ()
>> +  "Remove restrictions (narrowing) from current code chunk or buffer.
>> +This function should be used instead of `widen' in any function
>> +used by the indentation engine to make it respect the value
>> +`prog-indentation-context'.

> I don't see a need for "should" here.

What else - a "must" (This function is to be used)?

>> +  (widen)
>> +  (let ((chunk (cadr prog-indentation-context)))
>> +    (when chunk
>> +      (narrow-to-region (car chunk) (or (cdr chunk) (point-max))))))

> Better do

>     (let ((chunk (cadr prog-indentation-context)))
>       (if chunk
>           (narrow-to-region (car chunk) (or (cdr chunk) (point-max)))
>         (widen))))

OK, I didn't know whether I should rely on the behavior (defined in
src/editfns.c) that narrow-to-region might also `widen'...

This behavior could be mentioned in the docstring  - other possibilties
would be:
 - (when (or (> (point-min) start) (< (point-max) end)) (error ...))
 - (narrow--internal (max (point-min) start) (min (point-max) end)

-- Christoph

Re: antlr-mode.el - need some support by python.el

[Dmitry Gutov]

On 06/05/2015 05:17 PM, Wedler, Christoph wrote:

> Many modes use more than the current line / code chunk in their
> indentation calculations.  For example, in `c-guess-basic-syntax', the
> outer language constructs are also considered.  If the indentation
> engine of the inner mode just see the code chunks, there is a potential
> problem...

I think the string value is not ideal, because then the inner mode can't 
cache syntactic information, and reuse it between chunks. This is 
something we probably want to facilitate.

Adapting the current major mode indentation code to work with either 
kind of PREVIOUS-CHUNKS value, looks non-trivial as well.

I don't currently have a better suggestion, though. But this proposal 
could use a corresponding patch to cc-mode.

RE: antlr-mode.el - need some support by python.el

[Wedler, Christoph]

>> Many modes use more than the current line / code chunk in their
>> indentation calculations.  For example, in `c-guess-basic-syntax', the
>> outer language constructs are also considered.  If the indentation
>> engine of the inner mode just see the code chunks, there is a potential
>> problem...

> I think the string value is not ideal, because then the inner mode can't 
> cache syntactic information, and reuse it between chunks. This is 
> something we probably want to facilitate.

Originally, I had some inner mode dependent value here.  Stefan had
objections, because mode-specific stuff in a generic value is not that
nice (agreed) and the outer mode would need to know inner mode internals.

For caching, I can imagine the following:

 - the inner mode does at least cache the syntactic info for the last
   string (for the grammar case, you would almost ever have a cache hit)

 - the inner mode is encouraged to change PREVIOUS-CHUNKS inside
   `prog-indentation-context' to the inner mode specific value.  The
   outer mode can retrieve the value after the call and use it for
   further calls.

> Adapting the current major mode indentation code to work with either 
> kind of PREVIOUS-CHUNKS value, looks non-trivial as well.

For the string value (for grammars = the case I am interested in):
looks manageable

 - I could later add a corresponding patch to cc-mode, see below

 - in general, the inner mode could insert the string at START inside a
   `call-with-transparent-undo' (XEmacs, whatever the Emacs equivalent
   is)

The get-previous-chunk function case (for literate programming = the
case Stefan and you, Dmitry, were interested in): I agree, this is
non-trivial (see my remarks back in Feb/Mar).

> I don't currently have a better suggestion, though. But this proposal 
> could use a corresponding patch to cc-mode.

Well, I really like to see something included in Emacs before I spend
more time on this.

My suggestion:

 1.  Stefan tells me what he had against the "should" in the docstring
    of `prog-widen' (and/or sends me a corrected docstring).

    I reduce the patch to NOT include PREVIOUS-CHUNKS (and use the
    widen-feature of narrow-to-region ;-))

 2. If that is included, I reintroduce PREVIOUS-CHUNKS and include a
    corresponding patch to cc-mode - but only for the "string/grammar
    case"

-- Christoph

Re: antlr-mode.el - need some support by python.el

[Stefan Monnier]

>> I think the string value is not ideal, because then the inner mode can't
>> cache syntactic information, and reuse it between chunks.  This is
>> something we probably want to facilitate.

Maybe we should simply document that it's valid&expected for the inner
mode to add text-properties to the string.

>  - the inner mode is encouraged to change PREVIOUS-CHUNKS inside
>    `prog-indentation-context' to the inner mode specific value.  The
>    outer mode can retrieve the value after the call and use it for
>    further calls.

This looks fiddly, and at least requires changing the doc of
PREVIOUS-CHUNKS to mention that it can contain yet some other data in
some "internal" format.

> The get-previous-chunk function case (for literate programming = the
> case Stefan and you, Dmitry, were interested in): I agree, this is
> non-trivial (see my remarks back in Feb/Mar).

I can't imagine it being significantly more complex than the string case.
In the worst case, you can fairly easily turn the function into a string
by extracting&concatenating the corresponding chunks.


        Stefan

Re: antlr-mode.el - need some support by python.el

[Dmitry Gutov]

On 06/08/2015 04:26 PM, Stefan Monnier wrote:

> Maybe we should simply document that it's valid&expected for the inner
> mode to add text-properties to the string.

Which string? Suppose an inner chunk A is followed by an "external" 
region, and then an inner chunk B. When we work on the contents of the 
chunk A, the new variable clearly won't include the contents of that 
chunk in the PREVIOUS-CHUNKS field. Thus, it's not possible for the code 
working in A to cache some state in a way accessible to the chunk B (to 
anny code running when it's current).

Making PREVIOUS-CHUNKS store a list of position pairs (regions), or 
making it function, both sound better to me.

> I can't imagine it being significantly more complex than the string case.
> In the worst case, you can fairly easily turn the function into a string
> by extracting&concatenating the corresponding chunks.

I agree.

Re: antlr-mode.el - need some support by python.el

[Stefan Monnier]

>> Maybe we should simply document that it's valid&expected for the inner
>> mode to add text-properties to the string.
> Which string?

IIUC we were talking about the case where PREVIOUS-CHUNKS is a string.
So: *the* string.

> Suppose an inner chunk A is followed by an "external" region,
> and then an inner chunk B. When we work on the contents of the chunk A, the
> new variable clearly won't include the contents of that chunk in the
> PREVIOUS-CHUNKS field.

What new variable?  You mean prog-indentation-context?

I think in the case you describe, the outer mode wouldn't pass a string
in PREVIOUS-CHUNKS but would instead pass a function that gives access
to the chunk A.

> Thus, it's not possible for the code working in A to
> cache some state in a way accessible to the chunk B (to anny code running
> when it's current).

The inner mode can keep track of its state via text-properties applied
to the actual buffer text.  So while working on chunk A, it could add
properties there, and while working on chunk B it would get access to
those properties via the region-bounds returned by the
PREVIOUS-CHUNKS function.

> Making PREVIOUS-CHUNKS store a list of position pairs (regions), or making
> it function, both sound better to me.

The intended use case where PREVIOUS-CHUNKS is a string is when this
"context" is a fixed short string, e.g. in a tool like yacc/bison, each
C chunk would use a PREVIOUS-CHUNKS along the lines of "void
pseudofunction () {" so as to make it clear that the chunk's content is
expected to be a function body rather than a top-level declaration.


        Stefan

Re: antlr-mode.el - need some support by python.el

[Dmitry Gutov]

On 06/08/2015 11:50 PM, Stefan Monnier wrote:

> What new variable?  You mean prog-indentation-context?

Yes.

> The intended use case where PREVIOUS-CHUNKS is a string is when this
> "context" is a fixed short string, e.g. in a tool like yacc/bison, each
> C chunk would use a PREVIOUS-CHUNKS along the lines of "void
> pseudofunction () {" so as to make it clear that the chunk's content is
> expected to be a function body rather than a top-level declaration.

Okay, thanks for the explanation. Indeed, that sounds fine, even if 
it'll require more smarts on the part of the inner major mode.

RE: antlr-mode.el - need some support by python.el

[Wedler, Christoph]

Here as now a new version - changes to previous one:

 - fetch/rebase this morning

 - mention text-properties (good idea) and typical use cases in the
   docstring of `prog-indentation-context'

 - prog-widen: no widen for narrow-to-region case, docstring
   "should"->"can" (you are right, "should" was meant for "make it
   respect the value `prog-indentation-context'"


a1bd75f57b6d2c726a9c214536da9b63c9c67672 HEAD master
Author: Christoph Wedler <christoph.wedler@sap.com>
Date:   Wed Jun 3 13:54:31 2015 +0000

    Some generic support for multi-mode indentation.

2 files changed, 73 insertions(+)
 ChangeLog.2                 |  7 +++++
 lisp/progmodes/prog-mode.el | 66 +++++++++++++++++++++++++++++++++++++++++++++

	Modified   ChangeLog.2
diff --git a/ChangeLog.2 b/ChangeLog.2
index 4d59b8f..22c4684 100644
--- a/ChangeLog.2
+++ b/ChangeLog.2
@@ -1,3 +1,10 @@
+2015-06-09  Christoph Wedler  <christoph.wedler@sap.com>
+
+	Some generic support for multi-mode indentation.
+	* lisp/progmodes/prog-mode.el (prog-indentation-context): New
+	variable.
+	(prog-first-column, prog-widen): New convenience functions.
+
 2015-06-06  Paul Eggert  <eggert@cs.ucla.edu>
 
 	Merge from gnulib
	Modified   lisp/progmodes/prog-mode.el
diff --git a/lisp/progmodes/prog-mode.el b/lisp/progmodes/prog-mode.el
index 0d9fabd..cb8aaad 100644
--- a/lisp/progmodes/prog-mode.el
+++ b/lisp/progmodes/prog-mode.el
@@ -48,6 +48,51 @@
     map)
   "Keymap used for programming modes.")
 
+(defvar prog-indentation-context nil
+  "Non-nil while indenting embedded code chunks.
+There are languages where part of the code is actually written in
+a sub language, e.g., a Yacc/Bison or ANTLR grammar also consists
+of plain C code.  This variable enables the major mode of the
+main language to use the indentation engine of the sub mode for
+lines in code chunks written in the sub language.
+
+When a major mode of such a main language decides to delegate the
+indentation of a line/region to the indentation engine of the sub
+mode, it is supposed to bind this variable to non-nil around the call.
+
+The non-nil value looks as follows
+   \(FIRST-COLUMN (START . END) PREVIOUS-CHUNKS)
+
+FIRST-COLUMN is the column the indentation engine of the sub mode
+should usually choose for top-level language constructs inside
+the code chunk (instead of 0).
+
+START to END is the region of the code chunk.  See function
+`prog-widen' for additional info.
+
+PREVIOUS-CHUNKS, if non-nil, provides the indentation engine of
+the sub mode with the virtual context of the code chunk.  Valid
+values are:
+
+ - A string containing code which the indentation engine can
+   consider as standing in front of the code chunk.  To cache the
+   string's calculated syntactic information for repeated calls
+   with the same string, it is valid and expected for the inner
+   mode to add text-properties to the string.
+
+   A typical use case is for grammars with code chunks which are
+   to be indented like function bodies - the string would contain
+   a corresponding function header.
+
+ - A function called with the start position of the current
+   chunk.  It will return either the region of the previous chunk
+   as \(PREV-START . PREV-END) or nil if there is no further
+   previous chunk.
+
+   A typical use case are literate programming sources - the
+   function would successively return the code chunks of the
+   previous macro definitions for the same name.")
+
 (defun prog-indent-sexp (&optional defun)
   "Indent the expression after point.
 When interactively called with prefix, indent the enclosing defun
@@ -61,6 +106,27 @@ instead."
 	  (end (progn (forward-sexp 1) (point))))
       (indent-region start end nil))))
 
+(defun prog-first-column ()
+  "Return the indentation column normally used for top-level constructs."
+  (or (car prog-indentation-context) 0))
+
+(defun prog-widen ()
+  "Remove restrictions (narrowing) from current code chunk or buffer.
+This function can be used instead of `widen' in any function used
+by the indentation engine to make it respect the value
+`prog-indentation-context'.
+
+This function (like 'widen') is useful inside a
+`save-restriction' to make the indentation correctly work when
+narrowing is in effect."
+  (let ((chunk (cadr prog-indentation-context)))
+    (if chunk
+        ;; no widen necessary here, as narrow-to-region changes (not
+        ;; just narrows) existing restrictions
+        (narrow-to-region (car chunk) (or (cdr chunk) (point-max)))
+      (widen))))
+
+
 (defvar-local prettify-symbols-alist nil
   "Alist of symbol prettifications.
 Each element looks like (SYMBOL . CHARACTER), where the symbol

Re: antlr-mode.el - need some support by python.el

[Stefan Monnier]

Looks good to me, please install,


        Stefan


>>>>> "Wedler," == Wedler, Christoph <christoph.wedler@sap.com> writes:

> Here as now a new version - changes to previous one:
>  - fetch/rebase this morning

>  - mention text-properties (good idea) and typical use cases in the
>    docstring of `prog-indentation-context'

>  - prog-widen: no widen for narrow-to-region case, docstring
>    "should"->"can" (you are right, "should" was meant for "make it
>    respect the value `prog-indentation-context'"


> a1bd75f57b6d2c726a9c214536da9b63c9c67672 HEAD master
> Author: Christoph Wedler <christoph.wedler@sap.com>
> Date:   Wed Jun 3 13:54:31 2015 +0000

>     Some generic support for multi-mode indentation.

> 2 files changed, 73 insertions(+)
>  ChangeLog.2                 |  7 +++++
>  lisp/progmodes/prog-mode.el | 66 +++++++++++++++++++++++++++++++++++++++++++++

> 	Modified   ChangeLog.2
> diff --git a/ChangeLog.2 b/ChangeLog.2
> index 4d59b8f..22c4684 100644
> --- a/ChangeLog.2
> +++ b/ChangeLog.2
> @@ -1,3 +1,10 @@
> +2015-06-09  Christoph Wedler  <christoph.wedler@sap.com>
> +
> +	Some generic support for multi-mode indentation.
> +	* lisp/progmodes/prog-mode.el (prog-indentation-context): New
> +	variable.
> +	(prog-first-column, prog-widen): New convenience functions.
> +
>  2015-06-06  Paul Eggert  <eggert@cs.ucla.edu>
 
>  	Merge from gnulib
> 	Modified   lisp/progmodes/prog-mode.el
> diff --git a/lisp/progmodes/prog-mode.el b/lisp/progmodes/prog-mode.el
> index 0d9fabd..cb8aaad 100644
> --- a/lisp/progmodes/prog-mode.el
> +++ b/lisp/progmodes/prog-mode.el
> @@ -48,6 +48,51 @@
>      map)
>    "Keymap used for programming modes.")
 
> +(defvar prog-indentation-context nil
> +  "Non-nil while indenting embedded code chunks.
> +There are languages where part of the code is actually written in
> +a sub language, e.g., a Yacc/Bison or ANTLR grammar also consists
> +of plain C code.  This variable enables the major mode of the
> +main language to use the indentation engine of the sub mode for
> +lines in code chunks written in the sub language.
> +
> +When a major mode of such a main language decides to delegate the
> +indentation of a line/region to the indentation engine of the sub
> +mode, it is supposed to bind this variable to non-nil around the call.
> +
> +The non-nil value looks as follows
> +   \(FIRST-COLUMN (START . END) PREVIOUS-CHUNKS)
> +
> +FIRST-COLUMN is the column the indentation engine of the sub mode
> +should usually choose for top-level language constructs inside
> +the code chunk (instead of 0).
> +
> +START to END is the region of the code chunk.  See function
> +`prog-widen' for additional info.
> +
> +PREVIOUS-CHUNKS, if non-nil, provides the indentation engine of
> +the sub mode with the virtual context of the code chunk.  Valid
> +values are:
> +
> + - A string containing code which the indentation engine can
> +   consider as standing in front of the code chunk.  To cache the
> +   string's calculated syntactic information for repeated calls
> +   with the same string, it is valid and expected for the inner
> +   mode to add text-properties to the string.
> +
> +   A typical use case is for grammars with code chunks which are
> +   to be indented like function bodies - the string would contain
> +   a corresponding function header.
> +
> + - A function called with the start position of the current
> +   chunk.  It will return either the region of the previous chunk
> +   as \(PREV-START . PREV-END) or nil if there is no further
> +   previous chunk.
> +
> +   A typical use case are literate programming sources - the
> +   function would successively return the code chunks of the
> +   previous macro definitions for the same name.")
> +
>  (defun prog-indent-sexp (&optional defun)
>    "Indent the expression after point.
>  When interactively called with prefix, indent the enclosing defun
> @@ -61,6 +106,27 @@ instead."
>  	  (end (progn (forward-sexp 1) (point))))
>        (indent-region start end nil))))
 
> +(defun prog-first-column ()
> +  "Return the indentation column normally used for top-level constructs."
> +  (or (car prog-indentation-context) 0))
> +
> +(defun prog-widen ()
> +  "Remove restrictions (narrowing) from current code chunk or buffer.
> +This function can be used instead of `widen' in any function used
> +by the indentation engine to make it respect the value
> +`prog-indentation-context'.
> +
> +This function (like 'widen') is useful inside a
> +`save-restriction' to make the indentation correctly work when
> +narrowing is in effect."
> +  (let ((chunk (cadr prog-indentation-context)))
> +    (if chunk
> +        ;; no widen necessary here, as narrow-to-region changes (not
> +        ;; just narrows) existing restrictions
> +        (narrow-to-region (car chunk) (or (cdr chunk) (point-max)))
> +      (widen))))
> +
> +
>  (defvar-local prettify-symbols-alist nil
>    "Alist of symbol prettifications.
>  Each element looks like (SYMBOL . CHARACTER), where the symbol

Re: antlr-mode.el - need some support by python.el

[Wedler, Christoph]

Hi Stefan
Can you install it for me, please?  I do not have push rights...

Thanks
-- Christoph

Re: antlr-mode.el - need some support by python.el

[Dmitry Gutov]

On 06/08/2015 12:12 PM, Wedler, Christoph wrote:

> Well, I really like to see something included in Emacs before I spend
> more time on this.

I think that's a wrong position to take. It would be better to include 
the agreed-upon parts, but defer the more questionable ones until later. 
And include a corresponding patch for at least one mode.

>   2. If that is included, I reintroduce PREVIOUS-CHUNKS and include a
>      corresponding patch to cc-mode - but only for the "string/grammar
>      case"

While PREVIOUS-CHUNKS, as it's described now, makes sense, it's not 
obvious to me how to implement its support in js-indent-line, for instance.

Re: antlr-mode.el - need some support by python.el

[Stefan Monnier]

>>> +PREVIOUS-CHUNKS, if non-nil, provides the indentation engine of
>>> +the sub mode with the virtual context of the code chunk.  Valid
>>> +values are:
>>> +
>>> + - A string containing code which the indentation engine can
>>> +   consider as standing in front of the code chunk.  For example,
>>> +   it can contain a function header to make the code chunk
>>> +   being correctly indented as a function body.
>>> +
>>> + - A function called with the start position of the current
>>> +   chunk.  It will return either the region of the previous chunk
>>> +   as \(PREV-START . PREV-END) or nil if there is no further
>>> +   previous chunk.")
>> This is very unclear, so I'm not sure how an inner-mode could make use
>> of it.

Sorry, I re-read it and thought about it some more and realized that it
actually makes perfect sense and is not unclear.

>>> +(defun prog-widen ()
>>> +  "Remove restrictions (narrowing) from current code chunk or buffer.
>>> +This function should be used instead of `widen' in any function
>>> +used by the indentation engine to make it respect the value
>>> +`prog-indentation-context'.
>> I don't see a need for "should" here.
> What else - a "must" (This function is to be used)?

No, it's perfectly valid for the indentation code to use `widen', or to
manually use narrow-to-region based on prog-indentation-context, if it
feels like it, or to use neither widen nor prog-widen.  So rather than
"should", I'd expect "can" or something like that.

> OK, I didn't know whether I should rely on the behavior (defined in
> src/editfns.c) that narrow-to-region might also `widen'...

I think a lot of existing code relies on it, so I can't imagine changing this.


        Stefan

Re: antlr-mode.el - need some support by python.el

[Dmitry Gutov]

On 03/03/2015 06:32 PM, Stefan Monnier wrote:

> Info-mode's buffers (i.e. *.info files) contain several Info pages at
> the same time, so when you "go to an Info page" what really happens is
> that the mode narrows the buffer to the corresponding chunk in
> the buffer.  Clearly this is not scoped inside a `let': the narrowing
> will simply be in effect until you jump to another page, at which point
> the narrowing will be changed to select another chunk.

That is interesting. Thanks. But considering this, we're unlikely to 
have `Info-mode' as a submode somewhere.

>> But it's good if widen-function can have normal uses.
>
> Indeed, I don't see any problem there (I was just pointing out that
> assuming let-like scoping is a bad idea).

If widen-function is technically feasible, I'd rather go in this 
direction rather than introduce a variable everyone can ignore.

> The issue is simply: how to tell the submode what are the bounds of its chunk.
> If you don't do it by passing START/END, then you have to do it via side
> channels such as by narrowing.

That is true. However, as long as we don't have solutions for the hard 
problems in the multiple-mode space, we might as well hold of on making 
the change that only solves the relatively easy one (and let 
multiple-mode solutions use `widen' for the time being).

It doesn't seem to me like it moves toward solutions for the hard 
problems, in any way.

> And I'm firmly opposed to imposing such an API.  I much prefer passing
> START/END via dynamically scoped vars or via explicit arguments, and
> then let the submode use a little wrapper that sets up narrowing and
> calls the "same old" code (if the submode prefers using narrowing).

Do you have a submode in mind that will benefit from this distinction 
(external vs internal narrowing) in practice, right now?

Re: antlr-mode.el - need some support by python.el

[Stefan Monnier]

> Do you have a submode in mind that will benefit from this distinction
> (external vs internal narrowing) in practice, right now?

I don't understand the question.  I'm just firmly opposed to an API
that *relies* on narrowing, because I've been burned by narrowing too
often already.


        Stefan

Re: antlr-mode.el - need some support by python.el

[Dmitry Gutov]

On 03/05/2015 12:26 AM, Stefan Monnier wrote:
>> Do you have a submode in mind that will benefit from this distinction
>> (external vs internal narrowing) in practice, right now?
>
> I don't understand the question.

The question is, do you really expect to be burned by unexpected 
narrowing/widening in indentation code. That problem, in isolation, 
seems easily solvable.

 > I'm just firmly opposed to an API
> that *relies* on narrowing, because I've been burned by narrowing too
> often already.

I'm softly opposed to the color chosen for this bike shed. And since the 
plans for the power plant are non-existing, maybe we should avoid 
ordering the bikes just yet.

Or to put it in a different way, the lack of an API (which encourages 
people use some unseemly approach) is not the same as an API that relies 
on it. Because we won't have to support it for ever and ever.

Re: antlr-mode.el - need some support by python.el

[Stefan Monnier]

> The question is, do you really expect to be burned by unexpected
> narrowing/widening in indentation code.

I'm not sure what you mean by that question: if the outer mode uses
narrowing to specify the bounds of the chunk, then indentation code
can't use `widen'.  Use of narrowing inside the indentation code will be
fine, but use of widen will not.  Yet, "grep widen lisp/progmodes/*.el"
suggests that it's probably used by several indenters.

So, yes, I expect problems.

> Or to put it in a different way, the lack of an API (which encourages people
> use some unseemly approach) is not the same as an API that relies on
> it. Because we won't have to support it for ever and ever.

I think specifying the chunk bounds via some dynamically-scoped var is
a pretty safe option.  Maybe it will be supplanted in the future by some
other alternative, but it doesn't impose any particular implementation
constraint, contrary to the use of narrowing.


        Stefan

Re: antlr-mode.el - need some support by python.el

[Dmitry Gutov]

On 03/10/2015 03:16 AM, Stefan Monnier wrote:

> I'm not sure what you mean by that question: if the outer mode uses
> narrowing to specify the bounds of the chunk, then indentation code
> can't use `widen'.  Use of narrowing inside the indentation code will be
> fine, but use of widen will not.  Yet, "grep widen lisp/progmodes/*.el"
> suggests that it's probably used by several indenters.

The same places that use `widen' now would have to be adapted anyway, 
whether not to use it at all, or to behave according to the new variable.

The former change sounds simpler to me.

> I think specifying the chunk bounds via some dynamically-scoped var is
> a pretty safe option.  Maybe it will be supplanted in the future by some
> other alternative, but it doesn't impose any particular implementation
> constraint, contrary to the use of narrowing.

What do you think of `prog-narrow-to-region' and `prog-widen'?

Just having a dynamically-scoped var by itself seems a more verbose and 
error-prone approach to me.

Re: antlr-mode.el - need some support by python.el

[Stefan Monnier]

> Just having a dynamically-scoped var by itself seems a more verbose and
> error-prone approach to me.

The narrowing state is just the same as a dynamically-scoped var, it's
just not exposed in the same way to Elisp.


        Stefan

Re: antlr-mode.el - need some support by python.el

[Dmitry Gutov]

On 03/21/2015 07:08 PM, Stefan Monnier wrote:
>> Just having a dynamically-scoped var by itself seems a more verbose and
>> error-prone approach to me.
>
> The narrowing state is just the same as a dynamically-scoped var, it's
> just not exposed in the same way to Elisp.

I'm not sure I understand your point. It's the same, yet not the same: 
it's a special var which many built-in function respect, so the user 
respects it automatically, most of the time, without even trying to.

Re: antlr-mode.el - need some support by python.el

[Stefan Monnier]

>>> Just having a dynamically-scoped var by itself seems a more verbose and
>>> error-prone approach to me.
>> The narrowing state is just the same as a dynamically-scoped var, it's
>> just not exposed in the same way to Elisp.
> I'm not sure I understand your point.

That's just because I misunderstood yours ;-)

> it's a special var which many built-in function respect, so the user
> respects it automatically, most of the time, without even trying to.

The fact that it's only obeyed by using it explicitly is a feature: as
explained, setting narrowing has too far reaching effects which are not
always desirable.


        Stefan

Re: antlr-mode.el - need some support by python.el

[Dmitry Gutov]

On 03/22/2015 03:54 PM, Stefan Monnier wrote:

>> it's a special var which many built-in function respect, so the user
>> respects it automatically, most of the time, without even trying to.
>
> The fact that it's only obeyed by using it explicitly is a feature: as
> explained, setting narrowing has too far reaching effects which are not
> always desirable.

But obeying something automatically is nice for correctness, even if the 
client come has to opt-in, as long as the latter is easy to do.

You haven't really commented on introduction of another kind of 
narrowing. As I understand, you don't want the new kind to be stronger 
than the current one. Okay, what about `prog-submode-narrow-to-region', 
`prog-submode-widen' and `prog-submode-save-restriction'? They would use 
the new dynamic var to contain the bounds. The indentation facility 
documentation would simply indicate not to use `widen' and friends, but 
use their `prog-submode-' counterparts.

And the extra indentation column, I think, would still be better to pass 
in as an argument to the indent-calculate function.

Re: antlr-mode.el - need some support by python.el

[Stefan Monnier]

> But obeying something automatically is nice for correctness, even if the
> client come has to opt-in, as long as the latter is easy to do.

It's very easy to do: take the dyn-bound var's data and narrow accordingly.


        Stefan

Re: antlr-mode.el - need some support by python.el

[Dmitry Gutov]

On 03/22/2015 11:59 PM, Stefan Monnier wrote:
>> But obeying something automatically is nice for correctness, even if the
>> client come has to opt-in, as long as the latter is easy to do.
>
> It's very easy to do: take the dyn-bound var's data and narrow accordingly.

What would you replace all (most?) `widen' occurrences in cc-engine 
with, then?

(widen)
(when prog-submode-region
   (narrow-to-region (car prog-submode-region)
                     (cdr prog-submode-region)))

?

Re: antlr-mode.el - need some support by python.el

[Stefan Monnier]

> What would you replace all (most?) `widen' occurrences in cc-engine
> with, then?

Emails to Alan,


        Stefan

RE: antlr-mode.el - need some support by python.el

[Wedler, Christoph]

> While LEFTMOST-COL is quite useful, (START . END) is less so, since it 
> would replace what already works in e.g. mmm-mode is doing it now, *and* 
> it would require more changes to existing modes.

Huh?  Indentation for python (sub) lines inside a C-like language (main)
already works in mmm-mode?  I just checked mmm-mode 0.5.2, but I do not
see how it is able to support that...

Re: antlr-mode.el - need some support by python.el

[Dmitry Gutov]

On 03/04/2015 06:29 PM, Wedler, Christoph wrote:

> Huh?  Indentation for python (sub) lines inside a C-like language (main)
> already works in mmm-mode?  I just checked mmm-mode 0.5.2, but I do not
> see how it is able to support that...

I've implemented that for HTML-CSS, HTML-JS and HTML-Ruby, in 
mmm-erb.el, which were the language combinations that interested me at 
the time.

Since the (C, Python) pair doesn't unambiguously define the indentation 
logic of mixed-mode buffers, someone should write support for the actual 
template languages they're used together in.

Re: antlr-mode.el - need some support by python.el

[Dmitry Gutov]

Anyway, if prog-indent-line calls `save-restriction' and `widen', the 
indent calculation functions won't have to, and we can disrecommend them 
doing that (and change the modes in Emacs not to do that).

Then multi-mode packages can use `narrow' to calculate indentation in 
submode regions without problem, as long as they know what to do about 
syntax-ppss cache.

RE: antlr-mode.el - need some support by python.el

[Wedler, Christoph]

>> + The non-nil value looks as follows
>> +    ((START . END) LEFTMOST-COL)

> The first element tries to re-implement what's currently being handled 
> with narrowing, successfully. Why?

> Like you said: "I have avoided that for opascal and ruby via
> `narrow-to-region', but Python's indentation calls `widen'." Then Python 
> should avoid calling `widen'.

Well, if the indentation calculation in python-mode needs some context
information which might be outside the visible part of the buffer, it
must call `widen', right?  Of course, Python does that inside a
`save-restriction'.

> And/or someone could try and solve the long-standing problem of
> separating "visibility" narrowing and "functional" narrowing.

To be honest, I do not fully understand what you mean here, or how it
would solve the problem.

Cross-command-wise, there is one narrowing.  It is used for restricting
some editing operations ("functional"?) and the visible part of the
buffer.  If we would have two narrowings, then the "functional" should
always be a subset of the "visible" narrowing (and the "functional"
should be made somehow detectable for the user by a different background
color).  Otherwise, things would get very confusing for the user.

Then, for python-mode not to call widen itself, the "functional" region
must be widened, which would imply a "visible" widen with the condition
above.  In other words, pyhon users could not use narrowing when they
want to get a correct indentation.

Most commands do not do any non-temporarily narrowing, they just use a
temporary narrowing, which is only "functional" anyway.

> LEFTMOST-COL, likewise, can by added by the calling function. No need to 
> make the submode's indentation function be aware of it.

As Stefan has mention, it would first require that the mode not just
indents the line, but also has a function which returns the correct
indentation.

Even if we have this, things are not that easy - we start with the
non-Python case:
 - some indent calculation return the correct indentation based on the
   indentation of previous lines - here, I won't have to add anything 
   (given that I calculate the correct index of the first SUB-line myself, 
   which I can do), except sometimes for top-level constructs
 - some indent calculation return the correct indentation based on the
   syntactic context - here, I would have to add LEFTMOST-COL

In other words, I would have to know the kind of indent calculation (and
Whether the current construct is a top-level one).

In Python, there is more than one correct indentation offset.  And since
the python mode does not want look at the complete syntax context (and
use the indentation offsets of all opening statements and similar), it
simply calculates the most inner valid indentation and cycles up to
column 0 - and in my case it should stop earlier.

Well, if I do the cycling myself, I would of course reject any positions
smaller than LEFTMOST-COL.  But then I am starting to implement SUB-mode
specific indentation code.

>> + If the indentation engine of the sub mode SUB depends on some
>> + global context, it may introduce a variable named
>> + `SUB-extra-indentation-context', which is to be bound like this
>> + variable by the main major mode.  For example, cc-mode might
>> + define `c-extra-indentation-context' which is to be bound to a
>> + list of SYNTACTIC-SYMBOL as explained in `c-offsets-alist'.")

> This, I guess, might be useful. But until many modes intend to use it, 
> what stops you from just keeping a small alist of (MAJOR-MODE . 
> OFFSETS-ALIST-VAR)?

What I mean is that `c-guess-basic-offset' should add some
SYNTACTIC-SYMBOLs at the end of the function.  This function is used
inside the indentation code - no own alist is of help here.

   Christoph

Re: antlr-mode.el - need some support by python.el

[Dmitry Gutov]

On 02/18/2015 02:22 PM, Wedler, Christoph wrote:

> Well, if the indentation calculation in python-mode needs some context
> information which might be outside the visible part of the buffer, it
> must call `widen', right?  Of course, Python does that inside a
> `save-restriction'.

Well, I say it shouldn't, until we have a better `widen'.

>> And/or someone could try and solve the long-standing problem of
>> separating "visibility" narrowing and "functional" narrowing.
>
> To be honest, I do not fully understand what you mean here, or how it
> would solve the problem.

Some `widen' calls mean `don't go beyond this region', and some mean 
`don't show me anything beyond this region'. Indentation code would undo 
the latter kind, and the multi-mode code would use first kind.

> Cross-command-wise, there is one narrowing.  It is used for restricting
> some editing operations ("functional"?) and the visible part of the
> buffer.

Yes, the problem is it does both.

> If we would have two narrowings, then the "functional" should
> always be a subset of the "visible" narrowing (and the "functional"
> should be made somehow detectable for the user by a different background
> color).  Otherwise, things would get very confusing for the user.
>
> Then, for python-mode not to call widen itself, the "functional" region
> must be widened, which would imply a "visible" widen with the condition
> above.  In other words, pyhon users could not use narrowing when they
> want to get a correct indentation.

I believe it will be in reverse. Though many editing commands will have 
to respect the "visible" narrowing, instead of "functional".

>> LEFTMOST-COL, likewise, can by added by the calling function. No need to
>> make the submode's indentation function be aware of it.
>
> As Stefan has mention, it would first require that the mode not just
> indents the line, but also has a function which returns the correct
> indentation.

Not necessarily: you also can adjust the indentation after the submode 
applies it.

> Even if we have this, things are not that easy - we start with the
> non-Python case:
>   - some indent calculation return the correct indentation based on the
>     indentation of previous lines - here, I won't have to add anything
>     (given that I calculate the correct index of the first SUB-line myself,
>     which I can do), except sometimes for top-level constructs
>   - some indent calculation return the correct indentation based on the
>     syntactic context - here, I would have to add LEFTMOST-COL

Currently, I'm using this trick: narrow, then add LEFTMOST-COL at the 
beginning of the region. In the end, remove them.

But using `line-indentation-function' would avoid it, too.

RE: antlr-mode.el - need some support by python.el

[Wedler, Christoph]

>> If we would have two narrowings, then the "functional" should
>> always be a subset of the "visible" narrowing (and the "functional"
>> should be made somehow detectable for the user by a different background
>> color).  Otherwise, things would get very confusing for the user.

> I believe it will be in reverse. Though many editing commands will have 
> to respect the "visible" narrowing, instead of "functional".

Well, if we think that the advantage of a 2nd narrowing is greater then
the disadvantage (complexity) -I am not convinced-, then we should at
least reduce the disadvantage.  In this case, this means that the
"functional" narrowing should be visible - I am not sure how to do it if
the "visible" narrowing is smaller...

>> LEFTMOST-COL, likewise, can by added by the calling function. No need to
>> make the submode's indentation function be aware of it.
>
> As Stefan has mention, it would first require that the mode not just
> indents the line, but also has a function which returns the correct
> indentation.

> Not necessarily: you also can adjust the indentation after the submode 
> applies it.

I do exactly that, but it does not work for python due to its cycling
functionality.  (It is also not ideal buffer-undo-list -wise.)

>> Even if we have this, things are not that easy - we start with the
>> non-Python case:
>>   - some indent calculation return the correct indentation based on the
>>     indentation of previous lines - here, I won't have to add anything
>>     (given that I calculate the correct index of the first SUB-line myself,
>>     which I can do), except sometimes for top-level constructs
>>>   - some indent calculation return the correct indentation based on the
>     syntactic context - here, I would have to add LEFTMOST-COL

> Currently, I'm using this trick: narrow, then add LEFTMOST-COL at the 
> beginning of the region. In the end, remove them.

Well, this does not work always.  Let us assume that ANTLR would support
EmacsLisp code generation (there is actually some experimental plugin
somewhere) - in this case, the context of the actions would be some
implicit progn in some function.  A rule would look like

direct_declarator
    :   (   IDENTIFIER
            {
                (when (and (> (len $declaration)
                      (isTypedef $declaration)))
                  (print "define type etc"))
                (do-something)
                (do-something-more)
            }
        |   '(' declarator ')'
        )
        declarator_suffix*
    ;

The action would be only almost correctly indented - but not the last
two actions like (do-something).  Reason: these sexps are top-level in
the code chunk, and Emacs Lisp indents it not relatively to some
previous line, but at column 0.

We could again do some workaround if the indentation of the sub mode
indents at column 0 (that is exactly what I am currently doing), but it
again does not work for Python due to its indentation cycling.

   Christoph

Re: antlr-mode.el - need some support by python.el

[Dmitry Gutov]

On 02/18/2015 06:10 PM, Wedler, Christoph wrote:

> Well, if we think that the advantage of a 2nd narrowing is greater then
> the disadvantage (complexity) -I am not convinced-, then we should at
> least reduce the disadvantage.  In this case, this means that the
> "functional" narrowing should be visible - I am not sure how to do it if
> the "visible" narrowing is smaller...

I believe, if this distinction will sometime come into life, the 
"functional" kind will only be applied programmatically, and won't stay 
on between commands. So there'll be nothing to highlight.

>> Not necessarily: you also can adjust the indentation after the submode
>> applies it.
>
> I do exactly that, but it does not work for python due to its cycling
> functionality.  (It is also not ideal buffer-undo-list -wise.)

I don't know about cycling. Sounds like something that could be handled 
in a different way.

buffer-undo-list should be fine: several changes performed by one 
command invocation will be treated as one by undo.

> The action would be only almost correctly indented - but not the last
> two actions like (do-something).  Reason: these sexps are top-level in
> the code chunk, and Emacs Lisp indents it not relatively to some
> previous line, but at column 0.
>
> We could again do some workaround if the indentation of the sub mode
> indents at column 0 (that is exactly what I am currently doing), but it
> again does not work for Python due to its indentation cycling.

Yes, sorry. I'm actually doing the same as you: 0 column means top-level 
(and top-level means add N columns). The whitespace-padding is only 
performed in a certain, special case.

Looking at `python-indent-line-function', it seems like it's going to 
cycle only under certain conditions (this-command is the same as 
last-command, and it's in the special list). From what I understand, it 
might even be what the user expects. If not, why not bind `this-command' 
to some odd value?

RE: antlr-mode.el - need some support by python.el

[Wedler, Christoph]

>>> Not necessarily: you also can adjust the indentation after the submode
>>> applies it.

>> I do exactly that, but it does not work for python due to its cycling
>> functionality.  (It is also not ideal buffer-undo-list -wise.)

> I don't know about cycling. Sounds like something that could be handled 
> in a different way.

As you guessed below: it is what the user expects (indentation in
Haskell also works that way).

> buffer-undo-list should be fine: several changes performed by one 
> command invocation will be treated as one by undo.

And if the resulting changes is a non-change (insertion, then deletion)?
Or if it should have been a non-change, but the several changes result
in TAB/space differences?

> Looking at `python-indent-line-function', it seems like it's going to 
> cycle only under certain conditions (this-command is the same as 
> last-command, and it's in the special list). From what I understand, it 
> might even be what the user expects. If not, why not bind `this-command' 
> to some odd value?

...or temporary binding that special list to some other value = this is
what I do.

   Christoph

Re: antlr-mode.el - need some support by python.el

[Dmitry Gutov]

On 02/13/2015 09:40 PM, Stefan Monnier wrote:

 > Indeed, in your case, there's no such "previous chunk" in the current 
buffer.
> I was thinking of other multi-mode cases such as literate programming or
> HTML+Javascript.

In all cases of multi-mode conbinations in web programming that I'm 
aware of, the inner chunk doesn't really have to know about the previous 
occurrence of the chunk of the same type.

Even template chunks like <%>}<%> only need to return calculate offset 
based on the contents (curly means -1), and add it to outer mode's 
indentation.

And JavaScript chunks inside <script> tags are the simplest case: they 
only care about the indentation of the <script> tag surrounding them.

Re: antlr-mode.el - need some support by python.el

[Andreas Röhler]

On 16.01.2015 20:50, Wedler, Christoph wrote:

[ ... ]

Hi,

what about sending the stuff in question to a temporary buffer? Then widen should not go into the way.

BTW it might help to have a real example-case where it fails currently.
Can't see this at tickets.

Cheers,

Andreas