Tree-sitter documentation

Tree-sitter documentation

[Yuan Fu]

> This function takes a series of @var{query-spec}s, which are triplets
> @w{@code{@var{:keyword} @var{value} @dots{} @var{query}}}.  Each
> @var{query} is a tree-sitter query in either the string, s-expression
> or compiled form.
> 
Is is ok to use “triplets” here? Because there can be more than one pair of :keyword and values before a query, eg,

:keyword value
:keyword value
query


Yuan

Re: Tree-sitter documentation

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Mon, 7 Nov 2022 02:13:20 -0800
> 
> > This function takes a series of @var{query-spec}s, which are triplets
> > @w{@code{@var{:keyword} @var{value} @dots{} @var{query}}}.  Each
> > @var{query} is a tree-sitter query in either the string, s-expression
> > or compiled form.
> > 
> Is is ok to use “triplets” here? Because there can be more than one pair of :keyword and values before a query, eg,
> 
> :keyword value
> :keyword value
> query

In that case, "triplets" is definitely incorrect, but I had no way of
understanding that this is possible.

It should be possible top describe this kind of argument list, but
does it really have to be so complicated?  These are not internal
functions, so Lisp programmers will have to battle with this signature
all the time.  Can we make the function's signature easier to
document, understand, and use?

For example, what about accepting an alist as the argument, where each
alist element specifies a query and its keyword/value pairs that
customize the query?

Re: Tree-sitter documentation

[Yuan Fu]

> On Nov 7, 2022, at 4:11 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Mon, 7 Nov 2022 02:13:20 -0800
>> 
>>> This function takes a series of @var{query-spec}s, which are triplets
>>> @w{@code{@var{:keyword} @var{value} @dots{} @var{query}}}.  Each
>>> @var{query} is a tree-sitter query in either the string, s-expression
>>> or compiled form.
>>> 
>> Is is ok to use “triplets” here? Because there can be more than one pair of :keyword and values before a query, eg,
>> 
>> :keyword value
>> :keyword value
>> query
> 
> In that case, "triplets" is definitely incorrect, but I had no way of
> understanding that this is possible.
> 
> It should be possible top describe this kind of argument list, but
> does it really have to be so complicated?  These are not internal
> functions, so Lisp programmers will have to battle with this signature
> all the time.  Can we make the function's signature easier to
> document, understand, and use?
> 
> For example, what about accepting an alist as the argument, where each
> alist element specifies a query and its keyword/value pairs that
> customize the query?

Alists has too much layers of parenthesizes that is verbose and easy to get wrong. Compare:

(treesit-font-lock-rules
 :language 'python
 :override t
 :feature 'string
 '((string :anchor "\"" @python--treesit-fontify-string)
   (string) @contextual)

 :feature 'string-interpolation
 :language 'python
 :override t
 '((interpolation (identifier) @font-lock-variable-name-face)))

With

(treesit-font-lock-rules
 '((((string :anchor "\"" @python--treesit-fontify-string)
     (string) @contextual)
    (language . python)
    (override . t)
    (feature . string))

   (((interpolation (identifier) @font-lock-variable-name-face))
    (language . python)
    (override . t)
    (feature . string-interpolation))))


Yuan

Re: Tree-sitter documentation

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Mon, 7 Nov 2022 12:47:30 -0800
> Cc: emacs-devel@gnu.org
> 
> > In that case, "triplets" is definitely incorrect, but I had no way of
> > understanding that this is possible.
> > 
> > It should be possible top describe this kind of argument list, but
> > does it really have to be so complicated?  These are not internal
> > functions, so Lisp programmers will have to battle with this signature
> > all the time.  Can we make the function's signature easier to
> > document, understand, and use?
> > 
> > For example, what about accepting an alist as the argument, where each
> > alist element specifies a query and its keyword/value pairs that
> > customize the query?
> 
> Alists has too much layers of parenthesizes that is verbose and easy to get wrong. Compare:

I don't share your pessimism about alists.  And the way the functions
are defined now are also very error-prone and complicate the code,
which needs to distinguish between several very different signatures.

How about making the query itself the value of a keyword/value pair?
Like this:

   :language 'python
   :override t
   :feature 'string
   :query '((string :anchor "\"" @python--treesit-fontify-string)
            (string) @contextual)

Re: Tree-sitter documentation

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

>> Alists has too much layers of parenthesizes that is verbose and easy to get wrong. Compare:

(BTW, they are much easier to edit with something like paredit.
I guess you already use it.)

> I don't share your pessimism about alists.  And the way the functions
> are defined now are also very error-prone and complicate the code,
> which needs to distinguish between several very different signatures.
>
> How about making the query itself the value of a keyword/value pair?
> Like this:
>
>    :language 'python
>    :override t
>    :feature 'string
>    :query '((string :anchor "\"" @python--treesit-fontify-string)
>             (string) @contextual)

I don't mind if it's not a well-formed plist myself, if the last element
is always the query.  It saves some typing.

On that note, I'd probably prefer it if we had something like this:

(treesit-font-lock-rules
 '(;; To avoid repetitive typing:
   (:default t
    :language 'python
    :override t)
   (:feature 'string
    '((string :anchor "\"" @python--treesit-fontify-string)
      (string) @contextual))
   (:feature 'string-interpolation
    '((interpolation (identifier) @font-lock-variable-name-face))))))

Re: Tree-sitter documentation

[Yuan Fu]

> On Nov 8, 2022, at 6:40 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Mon, 7 Nov 2022 12:47:30 -0800
>> Cc: emacs-devel@gnu.org
>> 
>>> In that case, "triplets" is definitely incorrect, but I had no way of
>>> understanding that this is possible.
>>> 
>>> It should be possible top describe this kind of argument list, but
>>> does it really have to be so complicated?  These are not internal
>>> functions, so Lisp programmers will have to battle with this signature
>>> all the time.  Can we make the function's signature easier to
>>> document, understand, and use?
>>> 
>>> For example, what about accepting an alist as the argument, where each
>>> alist element specifies a query and its keyword/value pairs that
>>> customize the query?
>> 
>> Alists has too much layers of parenthesizes that is verbose and easy to get wrong. Compare:
> 
> I don't share your pessimism about alists.  And the way the functions
> are defined now are also very error-prone and complicate the code,
> which needs to distinguish between several very different signatures.

Ah, I’m not saying alist per se has too much layers of parenthesizes, but if we use alists rather than keywords in treesit-font-lock-rules, there really are a lot of parenthesizes. The current signature might be harder to formally explain, but should be pretty intuitive, it’s just a series of queries, and queries can have keyword value pairs preceding it that adds meta information to the query.

I might help to think of treesit-font-lock-rules not as a function but as a macro. 

The code to handle this signature isn’t particularly error-prone, the signal forms you see are just nice type-checks that I added for developer’s aid. They check, for example, that the value of a :language keyword is a symbol, the value of an :override keyword is one of the five possible values, etc. These checks are not related to the signature.

> 
> How about making the query itself the value of a keyword/value pair?
> Like this:
> 
>   :language 'python
>   :override t
>   :feature 'string
>   :query '((string :anchor "\"" @python--treesit-fontify-string)
>            (string) @contextual)

It’s ok, but query isn’t the same as other things. Query is the subject and other keywords adds information to it. I tried to improve the documentation of treesit-font-lock-rules, so hopefully changing the signature is not needed. Is the new version ok to you?

Yuan

Re: Tree-sitter documentation

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Tue, 8 Nov 2022 10:44:29 -0800
> Cc: emacs-devel@gnu.org
> 
> I tried to improve the documentation of treesit-font-lock-rules, so hopefully changing the signature is not needed. Is the new version ok to you?

Thanks.