Honesty with parse-partial-sexp

Honesty with parse-partial-sexp

[Alan Mackenzie]

Hi, Emacs!

It has become evident that certain files.el in Emacs use (car
(nth  9 ppss-state)) to obtain the position of the none-nested open
paren containing point.

However, syntax.texi says that this is secret data, not to be accessed
by the ordinary plebian Emacs user.  This doesn't seem entirely
consistent with the principles of free software.

So I propose documenting this properly.  Such as:


2006-12-07  Alan Mackenzie  <acm@muc.de>

	* syntax.texi (Parsing Balanced Expressions): Document the car of
	the ninth element returned by parse-partial-sexp.


Index: syntax.texi
===================================================================
RCS file: /cvsroot/emacs/emacs/lispref/syntax.texi,v
retrieving revision 1.53
diff -c -r1.53 syntax.texi
*** syntax.texi	18 Jul 2006 00:06:43 -0000	1.53
--- syntax.texi	7 Dec 2006 17:37:00 -0000
***************
*** 716,724 ****
  this element is @code{nil}.
  
  @item
! Internal data for continuing the parsing.  The meaning of this
! data is subject to change; it is used if you pass this list
! as the @var{state} argument to another call.
  @end enumerate
  
  Elements 1, 2, and 6 are ignored in the argument @var{state}.  Element
--- 716,727 ----
  this element is @code{nil}.
  
  @item
! This element is a list.  Its car is the character position of the
! start of the outermost parenthetical grouping containing the stopping
! point; @code{nil} if none.  The rest of the list is internal data for
! continuing the parsing.  The meaning of this data is subject to
! change; it is used if you pass the list as the @var{state} argument to
! another call.
  @end enumerate
  
  Elements 1, 2, and 6 are ignored in the argument @var{state}.  Element


-- 
Alan Mackenzie (Ittersbach, Germany).

Re: Honesty with parse-partial-sexp

[Richard Stallman]

    2006-12-07  Alan Mackenzie  <acm@muc.de>

	    * syntax.texi (Parsing Balanced Expressions): Document the car of
	    the ninth element returned by parse-partial-sexp.

I think this is a good idea.  The meaning of this is simple and useful.
And if we ever want to change the data passed along from one call to
parse-partial-sexp to the next, we still can.

However, I think this text is not exactly right:

    ! This element is a list.  Its car is the character position of the
    ! start of the outermost parenthetical grouping containing the stopping
    ! point; @code{nil} if none.

Isn't it really the outermost parenthetical grouping that begins in
the text that was parsed?

Re: Honesty with parse-partial-sexp

[Stefan Monnier]

>     ! This element is a list.  Its car is the character position of the
>     ! start of the outermost parenthetical grouping containing the stopping
>     ! point; @code{nil} if none.

> Isn't it really the outermost parenthetical grouping that begins in
> the text that was parsed?

I believe Alan's text is more correct, although it depends on the state
argument you passed into parse-partial-sexp.


        Stefan

Re: Honesty with parse-partial-sexp

[martin rudalics]

In fact, the Elisp doc for `parse-partial-sexp' reading

      Element 9 is used only to set element 1
      of the return value, in trivial cases where parsing starts and
      stops within the same pair of parentheses.

should become

      Element 9 is used only to set elements 1 and 9
      of the return value, in trivial cases where parsing starts and
      stops within the same pair of parentheses.

and I'm in doubt whether all these cases are "trivial".

Finally, if you really want to document the 9th return value you should
talk about all its elements.

Re: Honesty with parse-partial-sexp

[Miles Bader]

martin rudalics <rudalics@gmx.at> writes:
> Finally, if you really want to document the 9th return value you should
> talk about all its elements.

Wouldn't someone's earlier suggestion of providing an accessor macro to
extract 9th value be a tad more maintainable and make for more readable
code?

-Miles

-- 
Come now, if we were really planning to harm you, would we be waiting here,
 beside the path, in the very darkest part of the forest?

Re: Honesty with parse-partial-sexp

[Richard Stallman]

    >     ! This element is a list.  Its car is the character position of the
    >     ! start of the outermost parenthetical grouping containing the stopping
    >     ! point; @code{nil} if none.

    > Isn't it really the outermost parenthetical grouping that begins in
    > the text that was parsed?

    I believe Alan's text is more correct, although it depends on the state
    argument you passed into parse-partial-sexp.

His text is correct if you pass a STATE argument that reflects parsing
since the beginning of the buffer.  I think my text is always correct,
with the understanding that "the text that was parsed" includes
previous calls carried over in the STATE argument.

Re: Honesty with parse-partial-sexp

[Richard Stallman]

    Finally, if you really want to document the 9th return value you should
    talk about all its elements.

Not necessarily.

The really clean thing to do would be to store ONLY the position
of the outermost paren in elt 9, and move the rest to a new elt 10.
But I'd rather not do that at this late stage.

For now, let's define a defsubst to access it.  Would someone please do so?

Re: Honesty with parse-partial-sexp

[Stefan Monnier]

>> ! This element is a list.  Its car is the character position of the
>> ! start of the outermost parenthetical grouping containing the stopping
>> ! point; @code{nil} if none.

>> Isn't it really the outermost parenthetical grouping that begins in
>> the text that was parsed?

>     I believe Alan's text is more correct, although it depends on the state
>     argument you passed into parse-partial-sexp.

> His text is correct if you pass a STATE argument that reflects parsing
> since the beginning of the buffer.  I think my text is always correct,
> with the understanding that "the text that was parsed" includes
> previous calls carried over in the STATE argument.

Indeed.  And I'm glad I'm not the one trying to explain this in the doc ;-)


        Stefan

Re: Honesty with parse-partial-sexp

[Stefan Monnier]

>     Finally, if you really want to document the 9th return value you should
>     talk about all its elements.

> Not necessarily.

> The really clean thing to do would be to store ONLY the position
> of the outermost paren in elt 9, and move the rest to a new elt 10.
> But I'd rather not do that at this late stage.

> For now, let's define a defsubst to access it.  Would someone please do so?

Would the definition below be good?


        Stefan


--- syntax.el	07 mai 2006 10:09:22 -0400	1.18
+++ syntax.el	09 déc 2006 15:28:50 -0500	
@@ -52,6 +52,14 @@
 (defsubst syntax-ppss-depth (ppss)
   (nth 0 ppss))
 
+(defun syntax-ppss-toplevel-pos (ppss)
+  "Return the closest preceding position at toplevel.
+\"At toplevel\" means that it is outside of any syntactic entity:
+outside of any parentheses, or comments or strings.
+Returns nil iff PPSS itself corresponds to a toplevel position."
+  (or (car (nth 9 ppss))
+      (nth 8 ppss)))
+
 (defsubst syntax-ppss-context (ppss)
   (cond
    ((nth 3 ppss) 'string)

Re: Honesty with parse-partial-sexp

[Richard Stallman]

Please install this.  Then I will try to document it in the manual.

Re: Honesty with parse-partial-sexp

[Stefan Monnier]

> Please install this.  Then I will try to document it in the manual.

Installed.  Please people (I guess I mean Alan ;-) tell me if it provides
the right feature for you.

I'll update syntax.el later to use this,


        Stefan

Re: Honesty with parse-partial-sexp

[Alan Mackenzie]

Hi, Stefan and Emacs!

On Mon, Dec 11, 2006 at 09:41:12PM -0500, Stefan Monnier wrote:
> > Please install this.  Then I will try to document it in the manual.

> Installed.  Please people (I guess I mean Alan ;-) tell me if it provides
> the right feature for you.

I think it's almost right, but not quite.  Sorry I didn't reply earlier
before you committed it.  Here, for reference, is the current version:

> (defun syntax-ppss-toplevel-pos (ppss)
>   "Return the last preceding position at toplevel.
> \"At toplevel\" means that it is outside of any syntactic entity:
> outside of any parentheses, or comments, or strings.
> Returns nil iff PPSS itself corresponds to a toplevel position."
>   (or (car (nth 9 ppss))
>       (nth 8 ppss)))

I don't think a special case should be made for point already being at
top level:  It's surely more consistent just to return (point) in this
case.  To return nil would make callers of the function have to take
evasive action, such as:

   (let ((base-pos (or (syntax-ppss-toplevel-pos ppss)
                       (point))))

instead of the more natural

   (let ((base-pos (syntax-ppss-toplevel-pos ppss)))
		       
Also the doc-string is suboptimal; It doesn't say what PPSS is, and
isn't quite accurate about "toplevel": If we start a (parse-partial-sexp
..) when we're already inside an open paren, we'll (probably) get a
position at this nested level, not the top level.  Won't we?

How about this amended version?  I know the first line of the
doc-string's a bit long, but maybe that's OK, or somebody handier with
words (Richard?) could shorten it.

(defun syntax-ppss-toplevel-pos (ppss)
  "Get the last position at outermost syntactic level found in a syntactic scan.
PPSS is \(or looks like) the return value of the scanning function
`partial-parse-sexp' \(or `syntactic-ppss').

\"At outermost level\" means that it is outside of any syntactic
entity encountered in the scan: outside of any parentheses, or comments, or
strings."
  (or (car (nth 9 ppss))
      (nth 8 ppss)
      (point)))

Other than those small points, I thing the function is bang on target.

Thanks!

> I'll update syntax.el later to use this,
 
>         Stefan

-- 
Alan.

Re: Honesty with parse-partial-sexp

[Stefan Monnier]

> I don't think a special case should be made for point already being at
> top level:  It's surely more consistent just to return (point) in this
> case.  To return nil would make callers of the function have to take
> evasive action, such as:

>    (let ((base-pos (or (syntax-ppss-toplevel-pos ppss)
>                        (point))))

> instead of the more natural

>    (let ((base-pos (syntax-ppss-toplevel-pos ppss)))

But there can be other uses.  E.g. testing "toplevel-p" becomes just
(null (syntax-ppss-toplevel-pos (syntax-ppss))) rather than having to check
equality with (point).

More to the point, all places in syntax.el where I'd want to use this new
function currently use
		       
   (or (car (nth 9 ppss))
       (nth 8 ppss)
       (nth 2 ppss))

So with your function I'd end up having to do:

  (let ((pos (syntax-ppss-toplevel-pos ppss)))
    (if (eq pos (point)) (nth 2 ppss) pos))

Nothin terrible, of course, but it seems that it's marginally easier to add
(or ... (point)) than to detect the special case where the value returned in
(point).

There's a tension between making this function just an accessor to the raw
data, and making it a solution to a particular problem.

I was actually expecting someone to complain that she wants the raw
(car (nth 9 ppss)) data for her particular application.

> (or (car (nth 9 ppss))
>     (nth 8 ppss)
>     (point)))

Of course, the other problem with this code is that there's no guarantee
that (point) is the position of the end of the scan (we may even have
switched to some other buffer in the mean time).


        Stefan

Re: Honesty with parse-partial-sexp

[Alan Mackenzie]

Hi, Stephan,

That was quick!

On Tue, Dec 12, 2006 at 02:29:44PM -0500, Stefan Monnier wrote:
> > I don't think a special case should be made for point already being at
> > top level:  It's surely more consistent just to return (point) in this
> > case.  To return nil would make callers of the function have to take
> > evasive action, such as:

> >    (let ((base-pos (or (syntax-ppss-toplevel-pos ppss)
> >                        (point))))

> > instead of the more natural

> >    (let ((base-pos (syntax-ppss-toplevel-pos ppss)))

> But there can be other uses.  E.g. testing "toplevel-p" becomes just
> (null (syntax-ppss-toplevel-pos (syntax-ppss))) rather than having to check
> equality with (point).

I would argue for a coherent definition rather than optimising it for
particular uses.  However, ....

[ snippety snip ]

> Of course, the other problem with this code is that there's no guarantee
> that (point) is the position of the end of the scan (we may even have
> switched to some other buffer in the mean time).

This is the bit that I missed - ppss doesn't record the stopping point at
all.  So I now agree with you.  How about this:

(defun syntax-ppss-toplevel-pos (ppss)
  "Get the latest syntactically outermost position found in a syntactic scan.

PPSS is a record of the scan; it is \(or looks like) the return
value of the function `partial-parse-sexp' \(or
`syntactic-ppss').  If no such position is recorded in PPSS
\(because the end of the scan was itself at the outermost level),
return NIL.

An \"outermost position\" means one that it is outside of any
syntactic entity: outside of any parentheses encountered in the
scan, or comments, or strings."
  (or (car (nth 9 ppss))
      (nth 8 ppss)))

?

>         Stefan

-- 
Alan.