From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Stephen Leake Newsgroups: gmane.emacs.devel Subject: Re: smie-next-sexp vs associative operators Date: Sat, 20 Oct 2012 05:15:02 -0400 Message-ID: <85ehktijxl.fsf@stephe-leake.org> References: <85lif9e7m8.fsf@member.fsf.org> <851uh0x59u.fsf@stephe-leake.org> NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Trace: ger.gmane.org 1350724513 10418 80.91.229.3 (20 Oct 2012 09:15:13 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Sat, 20 Oct 2012 09:15:13 +0000 (UTC) To: emacs-devel@gnu.org Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Sat Oct 20 11:15:21 2012 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1TPV9L-0006dq-Su for ged-emacs-devel@m.gmane.org; Sat, 20 Oct 2012 11:15:20 +0200 Original-Received: from localhost ([::1]:60098 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TPV9E-0002FR-Dw for ged-emacs-devel@m.gmane.org; Sat, 20 Oct 2012 05:15:12 -0400 Original-Received: from eggs.gnu.org ([208.118.235.92]:60338) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TPV9A-0002Dy-KC for emacs-devel@gnu.org; Sat, 20 Oct 2012 05:15:10 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1TPV98-0008E6-TJ for emacs-devel@gnu.org; Sat, 20 Oct 2012 05:15:08 -0400 Original-Received: from qmta09.westchester.pa.mail.comcast.net ([76.96.62.96]:59815) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TPV98-0008A1-Mw for emacs-devel@gnu.org; Sat, 20 Oct 2012 05:15:06 -0400 Original-Received: from omta09.westchester.pa.mail.comcast.net ([76.96.62.20]) by qmta09.westchester.pa.mail.comcast.net with comcast id DMEX1k0050SCNGk59MF97g; Sat, 20 Oct 2012 09:15:09 +0000 Original-Received: from TAKVER ([69.140.67.196]) by omta09.westchester.pa.mail.comcast.net with comcast id DMFZ1k00P4E4Fsd3VMFZQ6; Sat, 20 Oct 2012 09:15:34 +0000 User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.2 (windows-nt) X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 76.96.62.96 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.devel:154462 Archived-At: --=-=-= Content-Type: text/plain Stefan Monnier writes: >> if A then >> B; >> elsif C then >> D; >> elsif >> E then >> F; >> else >> G; >> end if; > > I'm not completely sure how to interpret the above. > Is it an example of good indentation, or does it present a problem > case? This is the prefered indentation. I'm using it as an example of a case when calling smie-backward-sexp is necessary, when computing the indentation of F. That's what modula-2-mode does as well; smie-indent-after-keyword calls smie-indent-virtual to compute the indentation for "then"; that calls smie-indent-keyword, which calls (smie-backward-sexp token), with `token' = "then". This parses all the way back to "if". > In which way is it related to the select...do case? smie-backward-sexp ought to treat them similarly, as you agree below. But it does not, because smie-backward-sexp treats associative keywords specially. >> Here the internal keywords "then", "elsif" are not associative; "else" is >> associative. > > That doesn't sound right. "then" should not be associative, > but "elseif" should be. There's a comment on smie--associative-p that says tokens have the same left and right levels if they are optional; that makes sense, and applies in this case. "else" is optional, "elsif then" is optional, but "elsif" on its own is not optional. > More specifically, with point right after an "elsif", I'd expect C-M-b > to stop right after the previous "elsif". How is that useful for indentation? What about C-M-b after "else"; should that stop on the preceding "then"? or "elsif"? (In fact, it goes nowhere). I don't understand the general rule you are expressing here. The attached example_grammars.el, has several grammars for "if/then/else/endif", and in the comments, I show what smie-backward-sexp does for several points in the "if" statement, with different values of halfsexp. C-M-b calls (smie-backward-sexp 'halfsexp). In most cases, it does move to after the previous keyword. But not in all, so I don't see how it is generally useful. On the other hand, the behavior of (smie-backward-sexp token) is quite reliable. >> In the analogous situation with "select or", (smie-backward-sexp "or") >> stops at the previous "or". > > The "or" in select should largely behave like the "elsif" above. That is precisely what I'd like to have, but it is not happening. You are saying it is the "elsif" that's behaving incorrectly, but the main point is that they are different. In addition, the way I've defined "elsif" in Ada mode is the same as "elsif" in modula-2-mode. > You might like to use rules like: > > (exp ("if" expthenexp "end") > ("if" expthenexp "else" exp "end") > ("if" expthenexp "elsif" expthenexp "end") > ("if" expthenexp "elsif" expthenexp "end") > (expthenexp (exp "then" exp)) I tried that (see the attached example_grammars.el, grammar-3). It does change the levels assigned by the grammar compiler. But "then" does not have equal left and right levels, and the behavior of smie-backward-sexp is no more useful, as far as I can tell. >> Which suggests another way to state my core complaint; since >> smie-next-sexp will skip an arbitrary number of "transitive pairs", > > That's annoying. > >> it should also skip an arbitrary number of "transitive singles". Or at >> least, provide an option to do so. > > No, you got it backwards: you want indentation to walk back as little > as possible. E.g. you don't want to walk all the way back to "if" if > you can indent relative to the closest "elsif". But walking back to "if" is what modula-2-mode does, via smie-indent-keyword. Looking at the various grammars in example_grammars.el, I don't see any consistent patterns, except that (smie-backward-sexp token) always goes to "if" when token is one of the statement keywords. That's why I use it, and I assume that's why smie-indent-keyword uses it. I could fuss with the grammar of each statement to get slightly shorter parses in some cases. I don't have the time, and that would be premature optimization. It comes down to trading indentation code complexity & programmer time vs CPU time. The modula-2 grammar has 63 keywords, only a few refined, including many operators. So far, my Ada grammar has 124 keywords, most refined, no operators. modula2.el is 617 lines for the entire mode. ada-indent.el is 3153 lines and counting just for indentation (mostly refinement code); Ada is a much bigger language. So I need reliable, simple patterns to make this practical. (smie-backward-sexp token) is very reliable, except for associative keywords. That behaviour should be controlled by the language implementor, not dictated by smie. I've verified that the patch shown below allows me to make smie-backward-sexp behave the same for the Ada "select/or" and "if/elsif" statements. -- -- Stephe --- /Apps/emacs-24.2/lisp/emacs-lisp/smie.el 2012-09-14 04:44:57.521167100 -0400 +++ smie.el 2012-10-17 20:33:06.616977700 -0400 @@ -672,6 +672,8 @@ ;; has to be careful to distinguish those different cases. (eq (smie-op-left toklevels) (smie-op-right toklevels))) +(defvar smie-skip-associative nil) + (defun smie-next-sexp (next-token next-sexp op-forw op-back halfsexp) "Skip over one sexp. NEXT-TOKEN is a function of no argument that moves forward by one @@ -770,7 +772,8 @@ ;; The new operator is associative. Two cases: ;; - it's really just an associative operator (like + or ;) ;; in which case we should have stopped right before. - ((and lastlevels + ((and (not smie-skip-associative) + lastlevels (smie--associative-p (car lastlevels))) (throw 'return (prog1 (list (or (car toklevels) t) (point) token) --=-=-= Content-Type: application/emacs-lisp Content-Disposition: attachment; filename=example_grammars.el Content-Transfer-Encoding: quoted-printable ;; Exploring different ways of declaring Ada statements, to see how that ;; affects smie-backward-sexp behavior (require 'smie) ;; the comments give the result of calling smie-backward-sexp with ;; various args (via the keystrokes defined below) at various points ;; in the following Ada statement: ;; ;; procedure Debug is ;; begin ;; if A then -- then-1 ;; B; ;; ;; elsif C then -- elsif-1, then-2 ;; D; ;; ;; elsif -- elsif-2 ;; E then -- then-3 ;; F; ;; ;; else ;; G; ;; end fi; ;; ;; select ;; accept A1; ;; B1; ;; or ;; accept A2; ;; B2; ;; or ;; accept A3; ;; ;; else ;; B4; ;; end tceles; ;; end; (defconst grammar-1 (smie-prec2->grammar (smie-bnf->prec2 '( (identifier) (expression (identifier) (expression "+" identifier)) (statements (statement) (statement ";" statement)) (statement (if_statement)) (if_statement ;; Ada syntax requires "end if;". We use "end fi;" here, to ;; avoid the need for refine-if code in this simple example. ("if" expression "then" statements "end" "fi") ("if" expression "then" statements "else" statements "end" "fi") ("if" expression "then" statements ;; Ada syntax allows indefinite repeat of "elseif then". This ;; does not express that explicitly, but it is allowed by ;; smie-backward-sexp "elsif" expression "then" statements "else" statements "end" "fi") ) )))) ;;("if" (26) 0) ("then" 0 1) ("elsif" 1 0) ("else" 1 1) ("end" 1 25) ("fi" = 25 (27)) (";" 13 12) ;; "else" is optional, so it can't change levels; left =3D right. Therefore= "else" can also be repeated. ;; "elsif ... then" is optional; left elsif =3D right then. Therefore "elsi= f ... then" can be repeated. (defun use-grammar-1 () (interactive) (example-setup grammar-1)) ;; calling smie-backward-sexp ;; ;; from arg result point ;; after fi nil if before if ;; 'halfsexp if before if ;; ; nil bob ;; ;; before end nil ; before end ;; 'halfsexp else before G ;; end if before if ;; ;; before else nil ; before else ;; 'halfsexp then before F ;; else if before if ;; ;; before F nil then before F ;; 'halfsexp if before if ;; F error ;; ;; before then-3 nil nil before E ;; 'halfsexp nil before E ;; then if before if (defconst grammar-1a ;; add select (smie-prec2->grammar (smie-bnf->prec2 '( (identifier) (expression (identifier) (expression "+" identifier)) (statements (statement) (statement ";" statement)) (statement (if_statement) (select_statement)) (if_statement ;; Ada syntax requires "end if;". We use "end fi;" here, to ;; avoid the need for refine-if code in this simple example. ("if" expression "then" statements "end" "fi") ("if" expression "then" statements "else" statements "end" "fi") ("if" expression "then" statements ;; Ada syntax allows indefinite repeat of "elseif then". This ;; does not express that explicitly, but it is allowed by ;; smie-backward-sexp "elsif" expression "then" statements "else" statements "end" "fi") ) (select_statement ;; accept_statement, delay_statement are covered here by ;; "statements". "terminate" looks like a procedure call, so ;; we leave it as an identifier. ("select" statements "or" statements "else" statements "end" "tceles") ("select" statements "else" statements "end" "tceles") ) )))) ;; ("if" (28) 1) ("then" 1 0) ("elsif" 0 1) ("else" 0 0) ("end" 0 25) ("fi"= 25 (29)) ;; same structure as grammar-1 ;; ;; ("select" (27) 0) ("or" 0 0) ("else" 0 0) ("end" 0 25) ("tceles" 25 (28)) ;; "or" is optional and repeater, as is "else" ;; ;; (";" 13 12) ("+" 14 30) (defun use-grammar-1a () (interactive) (example-setup grammar-1a)) ;; calling smie-backward-sexp on select statement, smie-skip-associative nil ;; ;; from arg result point ;; after tceles nil select before select ;; 'halfsexp select before select ;; ; select before select ;; ;; before end nil ; before end ;; 'halfsexp else before B4 ;; end select before select ;; ;; before else nil ; before else ;; 'halfsexp or before accept A3 ;; else or before accept A3 ;; ;; before accept A3 nil or before accept A3 ;; 'halfsexp or A2 before accept A2 ;; accept error ;; ;; before or A3 nil nil before or ;; 'halfsexp ; A2 before B2 ;; or or A2 before accept ;; ;; before or A2 nil nil before or ;; 'halfsexp ; A1 before B1 ;; or select before select ;; calling smie-backward-sexp on select statement, smie-skip-associative t ;; ;; from arg result point ;; after tceles nil select before select ;; 'halfsexp select before select ;; ; select before select ;; ;; before end nil ; before end ;; 'halfsexp else before B4 ;; end select before select ;; ;; before else nil ; before else ;; 'halfsexp or before accept A3 ;; else select before accept select ;; ;; before accept A3 nil or before accept A3 ;; 'halfsexp select before select ;; accept error ;; ;; before or A3 nil ; B2 before or ;; 'halfsexp ; A2 before B2 ;; or select before select ;; ;; before or A2 nil ; B1 before or ;; 'halfsexp ; A1 before B1 ;; or select before select (defconst grammar-2 (smie-prec2->grammar (smie-bnf->prec2 '( (identifier) (expression (identifier) (expression "+" identifier)) (statements (statement) (statement ";" statement)) (statement ;; variant suggested by Stefan Monnier; does not explicitly ;; express allowed repeats of "elsif exp then", but changes the ;; levels assigned by the compiler, thus changing the behavior ;; of smie-backward-sexp. ("if" exp-then-stmt "end" "fi") ("if" exp-then-stmt "else" statements "end" "fi") ("if" exp-then-stmt "elsif" exp-then-stmt "end" "fi") ("if" exp-then-stmt "elsif" exp-then-stmt "else" statements "end" "f= i")) (exp-then-stmt (expression "then" statements)) )))) ; ("elsif" 0 0) ("else" 0 0) ("end" 0 36) ("then" 12 11) ("if" (37) 0) (";"= 25 13) ("fi" 36 (38)) ("+" 14 39) ; grammar compiler sees "elsif" as optional, even though it is actually "el= sif then" that is optional (defun use-grammar-2 () (interactive) (example-setup grammar-2)) ;; calling smie-backward-sexp ;; ;; from arg result point ;; after fi nil if before if ;; 'halfsexp if before if ;; ; nil bob ;; ;; before end nil ; before end ;; 'halfsexp else before G ;; end if before if ;; ;; before else nil ; before else ;; 'halfsexp then before F ;; else elsif-1 before E ;; ;; before F nil then before F ;; 'halfsexp elsif-2 before E ;; F error ;; ;; before then-3 nil nil before E ;; 'halfsexp nil before E ;; then elsif-1 before E (defconst grammar-3 (smie-prec2->grammar (smie-bnf->prec2 '( (identifier) (expression (identifier) (expression "+" identifier)) (statements (statement) (statement ";" statement)) (statement (if_statement)) (if_statement ;; This explicitly expresses the allowed repeats of elsif_clause ("if" exp-then-stmt-s "end" "fi") ("if" exp-then-stmt-s "else" statements "end" "fi")) (exp-then-stmt-s (exp-then-stmt) (exp-then-stmt-s "elsif" exp-then-stmt)) (exp-then-stmt (expression "then" statements)) )))) ;; ("else" 0 0) ("end" 0 48) ("elsif" 11 23) ("then" 35 22) ("if" (49) 0) (= ";" 36 24) ("fi" 48 (50)) ("+" 37 51) ;; only "else" is a repeater ;; even though "then" is optional, its levels are not equal, and they also = don't match elsif or any other token (defun use-grammar-3 () (interactive) (example-setup grammar-3)) ;; calling smie-backward-sexp ;; ;; from arg result point ;; after fi nil if before if ;; 'halfsexp if before if ;; ; nil bob ;; ;; before end nil ; before end ;; 'halfsexp else before G ;; end if before if ;; ;; before else nil ; before else ;; 'halfsexp then before F ;; else if before if ;; ;; before F nil then before F ;; 'halfsexp elsif-1 before E ;; F error ;; ;; before then-3 nil nil before E ;; 'halfsexp nil before E ;; then elsif-1 before E ;; modula-2 grammar fragment for "if"; ;; ("IF" exp "THEN" insts "END") ;; ("IF" exp "THEN" insts "ELSE" insts "END") ;; ("IF" exp "THEN" insts ;; "ELSIF" exp "THEN" insts "ELSE" insts "END") ;; ("IF" exp "THEN" insts ;; "ELSIF" exp "THEN" insts ;; "ELSIF" exp "THEN" insts "ELSE" insts "END")) ;; ;; ("ELSIF" 0 1) ("THEN" 1 0) ("ELSE" 0 0) ("IF" (172) 1) (";" 45 45) ("END= " 0 (187)) ;; "else", "elsif ... then" optional, repeater ;; ;; (define-key m2-mode-map "\M-o" 'example-show-sexp-nil) ;; (define-key m2-mode-map "\M-p" 'example-show-sexp-halfsexp) ;; (define-key m2-mode-map "\M-[" 'example-show-sexp-token) ;; ;; calling smie-backward-sexp ;; ;; from arg result point ;; after end nil if before if ;; 'halfsexp if before if ;; ; begin before if ;; ;; before end nil ; before end ;; 'halfsexp else before G ;; end if before if ;; ;; before else nil ; before else ;; 'halfsexp then before F ;; else if before if ;; ;; before F nil then before F ;; 'halfsexp if before if ;; F error ;; ;; before then-3 nil nil before E ;; 'halfsexp nil before E ;; then if before if ;; Ada mode 5.0 grammar fragment for "if" ;; ;; ("if-open" expression "then-if" statements "end-if" "if-end") ;; ("if-open" expression "then-if" statements "else-other" statements "end-= if" "if-end") ;; ("if-open" expression "then-if" statements ;; "elsif" expression "then-if" statements ;; "else-other" statements "end-if" "if-end") ;; ;; ("if-open" (184) 3) ("then-if" 3 25) ("elsif" 25 3) ("else-other" 25 25)= ("end-if" 25 127) ("if-end" 127 (174)) ;; (";" 37 36) ;; "else-other" is optional and repeater ;; "then-if .. elsif" is optional and repeater ;; ;; (define-key ada-mode-map "\M-o" 'example-show-sexp-nil) ;; (define-key ada-mode-map "\M-p" 'example-show-sexp-halfsexp) ;; (define-key ada-mode-map "\M-[" 'ada-indent-show-sexp-token) ;; ;; calling smie-backward-sexp ;; ;; from arg result point ;; after end nil if before if ;; 'halfsexp if before if ;; ; begin before if ;; ;; before end nil ; before end ;; 'halfsexp else before G ;; end if before if ;; ;; before else nil ; before else ;; 'halfsexp then before F ;; else if before if ;; ;; before F nil then before F ;; 'halfsexp if before if ;; F error ;; ;; before then-3 nil nil before E ;; 'halfsexp nil before E ;; then if before if (defun example-rules (method arg) ;; just meets the rules API, never intended to be called nil) (defun example-show-keyword-backward () "Show the smie grammar info for word preceding point, and move across it." (interactive) (message "%s" (assoc (smie-default-backward-token) smie-grammar))) (defun example-show-sexp-nil () (interactive) (message "%s" (smie-backward-sexp nil))) (defun example-show-sexp-halfsexp () (interactive) (message "%s" (smie-backward-sexp 'halfsexp))) (defun example-show-sexp-token () (interactive) (message "%s" (smie-backward-sexp (save-excursion (smie-default-forward-token))))) (defvar example-mode-map (make-sparse-keymap) "Local keymap used for Example mode.") (define-key example-mode-map "\M-i" 'example-show-keyword-backward) (define-key example-mode-map "\M-o" 'example-show-sexp-nil) (define-key example-mode-map "\M-p" 'example-show-sexp-halfsexp) (define-key example-mode-map "\M-[" 'example-show-sexp-token) (defun example-setup (grammar) (smie-setup grammar #'example-rules :forward-token #'smie-default-forward-token :backward-token #'smie-default-backward-token) (setq major-mode 'example-mode mode-name "Example") (set (make-variable-buffer-local 'smie-skip-associative) t) (use-local-map example-mode-map) ) ;; end of file --=-=-=--