SMIE questions

[Thibaut Verron <thibaut.verron@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 2570 bytes --]

Dear all,

I am trying to integrate support for a syntactic construct into the
SMIE setup of a major mode I maintain. The construct is somewhat
unconventional, and I might be at a loss as to where to even start.

In a simplified form the construct looks like this:

begin <identifier> ( <arguments> ) -> <returntype>
> " <docstring> "
> <instructions>
> end


Instructions are separated by semicolons, and both arguments and return
types can be comma-separated lists. The complexity is far from ending
there, but the point at which I cannot get enough control starts here.

A test file and an attempt at defining a corresponding mode are attached,
but it's really not much more than an initial attempt. I'm not even sure
that the grammar is the most appropriate one.

I have the following questions:

- What is/should be the parent of a token? For example, should the grammar
try to ensure that the parents of "->" and the docstring's opening quote
are "begin", or should the rules function take care of navigating to the
"real" syntactic parent?

Also, the documentation makes a difference between the parent and the
opener of a token. Is there a way to find the opener? Can we for example
trust that `backward-up-list` takes us there?

- Is it possible to understand the meaning of the variable `smie--parent`?
Its 2nd and 3rd items are the position and the token corresponding to the
parent, I believe, but what is the car?
And is there a difference between cases where `smie--parent` is nil and
cases where it is unbound?

- How are the calls to the rules function determined? For example, if I
indent before the quote (point 42), the function is called 2 or 3 times:
once with :list-intro, one with :elem args, and once (if the previous was
nil) with :elem basic (incidentally, there seems to be a small mistake in
the documentation there).

I was so far assuming that those cases were just meant for base settings
which are used everywhere, but does this behaviour mean that I should have
the rules function return something different also for :elem, based on
context? Actually that would also be a good reason for not simply having
variables.

Even more confusing, if I indent the line immediately after (point 55), it
seems that the rules function is not called at all. It shouldn't be a
problem for my case, the indentation after the docstring should always be
the one for the docstring, but why does it happen?

Sorry for the possibly naive questions! It's very possible that I have been
digging too far in the wrong direction...

Thanks, best wishes!
Thibaut

[-- Attachment #2: file.test --]
[-- Type: application/octet-stream, Size: 138 bytes --]

toto;

begin toto(a,
	   b)
    -> c,d
	 " string "
	 toto;
    tata;
    titi;
end;


/* Local Variables: */
/* mode: test */
/* End: */

[-- Attachment #3: test-smie.el --]
[-- Type: text/x-emacs-lisp, Size: 2122 bytes --]

(defconst test-grammar
  (smie-prec2->grammar
   (smie-bnf->prec2
    '((id)

      (inst (id)
	    ("begin" bblock "end"))

      (bblock ;;(id "(" funargs ")" "\"" id "\"" insts)
	      (id "(" funargs ")" "->" funargs "\"" id "\"" insts)
	      )
      
      (insts (inst)
	     (insts ";" insts))

      (funargs (id)
               (funargs "," funargs)))

    '((assoc ";" ",")
      ;(assoc ",")
      ))))

(defun test-smie-rules-verbose (kind token)
  (message (format "point=%s kind=%s token=%s parent=%s"
			 (point) kind token
			 (if (boundp 'smie--parent)
			     smie--parent
			   'none)))
  (let ((res (test-smie-rules kind token)))
    (message (format "-> %s" res))
    res))

(defun test-smie-rules (kind token)
  (pcase (cons kind token)
    (`(:elem . basic)
     4)
    (`(:elem . args)
     0)
   (`(:before . "->")
     (smie-rule-parent 4))
    (`(:before . "\"")
     (save-excursion
       (backward-up-list)
       ('column . (+ (current-column) 4))))
    ;; (`(:after . "\"")
    ;;  0)

    (`(:after . ")")
     (smie-rule-parent 4))
    (`(:after . ";")
     0)
    (`(:before . ";")
     (smie-rule-parent))))

(defconst test-keywords-regexp
  (regexp-opt '(";" "," "::" "begin" "end")))

(defun test-smie-forward-token ()
  (forward-comment (point-max))
  (cond
   ((looking-at test-keywords-regexp)
    (goto-char (match-end 0))
    (match-string-no-properties 0))
   (t (buffer-substring-no-properties
       (point)
       (progn (skip-syntax-forward "w_")
	      (point))))))

(defun test-smie-backward-token ()
  (forward-comment (- (point)))
  (cond
   ((looking-back test-keywords-regexp (- (point) 25) t)
    (goto-char (match-beginning 0))
    (match-string-no-properties 0))
   (t (buffer-substring-no-properties
       (point)
       (progn (skip-syntax-backward "w_")
	      (point))))))

(define-derived-mode test-mode
  prog-mode
  "Test"
  "Test mode"
  (setq-local comment-start "/* ")
  (setq-local comment-end " */")
  (smie-setup
   test-grammar
   #'test-smie-rules-verbose
   :forward-token #'test-smie-forward-token
   :backward-token #'test-smie-backward-token))