Re: How to add pseudo vector types

[Yuan Fu <casouri@gmail.com>]

> On Jul 22, 2021, at 1:00 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Thu, 22 Jul 2021 09:47:45 -0400
>> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,
>> Clément Pit-Claudel <cpitclaudel@gmail.com>,
>> emacs-devel@gnu.org
>> 
>> Yes, I meant to discuss this. The problem with respecting narrowing is that, a user can freely narrow and widen arbitrarily, and Emacs needs to translate them into insertion & deletion of the buffer text for tree-sitter, every time a user narrows or widens the buffer. Plus, if tree-sitter respects narrowing, it could happen where a user narrows the buffer, the font-locking changes and is not correct anymore. Maybe that’s not the user want. Also, if someone narrows and widens often, maybe narrow to a function for better focus, tree-sitter needs to constantly re-parse most of the buffer. These are not significant disadvantages, but what do we get from respecting narrowing that justifies code complexity and these small annoyances?
> 
> But that's how the current font-lock and indentation work: they never
> look beyond the narrowing limits.  So why should the TS-based features
> behave differently?
> 
> As for temporary narrowing: if we record the changes, but don't send
> them to TS until we actually need re-parsing, then we could eliminate
> the temporary narrowing when we report the changes to TS, leaving only
> the narrowing that exists at the time of the re-parse.  At least for
> fontifications, that time is redisplay time, and users do expect to
> see the text fontified according to the current narrowing.



> 
>>>>  *bytes_read = (uint32_t) len;
>>> 
>>> Is using uint32_t the restriction of tree-sitter?  Doesn't it support
>>> reading more than 2 gigabytes?
>> 
>> I’m not sure why it asks for uint32 specifically, but that’s what it asks for its api. I don’t think you are supposed to use tree-sitter on files of size of gigabytes, because the author mentioned that tree-sitter uses over 10x as much memory as the size of the source file [1]. On files larger than a couple of megabytes, I think we better turn off tree-sitter. Normally those files are not regular source files, anyway, and we don’t need a parse tree for a log.
> 
> I don't necessarily agree with the "not regular source files" part.
> For example, JSON files can be quite large.  And there are also log
> files, which are even larger -- did no one adapt TS to fontifying
> those yet?

There is a JSON parser, but I don’t think there is one for log files.

> 
> More generally: is the problem real?  If you make a file that is 1000
> copies of xdisp.c, and then submit it to TS, do you really get 10GB of
> memory consumption?  This is something that is good to know up front,
> so we'd know what to expect down the road.

Yes. I concatenated 100 xdisp.c together, and parsed them with my simple C program. It used 1.8 G. I didn’t test for 1000 together, but I think the trend is linear.

time -l ./main-large-c
       16.48 real        15.32 user         0.81 sys
          1883959296  maximum resident set size
                   0  average shared memory size
                   0  average unshared data size
                   0  average unshared stack size
              459951  page reclaims
                  22  page faults
                   0  swaps
                   0  block input operations
                   0  block output operations
                   0  messages sent
                   0  messages received
                   0  signals received
                   6  voluntary context switches
                1653  involuntary context switches
        107310143182  instructions retired
         58561420060  cycles elapsed
          1883095040  peak memory footprint

> 
>> That leads to another point. I suspect the memory limit will come before the speed limit, i.e., as the file size increases, the memory consumption will become unacceptable before the speed does. So it is possible that we want to outright disable tree-sitter for larger files, then we don’t need to do much to improve the responsiveness of tree-sitter on large files. And we might want to delete the parse tree if a buffer has been idle for a while. Of course, that’s just my superstition, we’ll see once we can measure the performance.
> 
> See above: IMO, we should benchmark both the CPU and memory
> performance of TS for such large files, before we decide on the course
> of action.

That’s my thought, too. I should have reserved my suspicion until I have benchmark measurements.

> 
>>>> +DEFUN ("tree-sitter-node-type",
>>>> +       Ftree_sitter_node_type, Stree_sitter_node_type, 1, 1, 0,
>>>> +       doc: /* Return the NODE's type as a symbol.  */)
>>>> +  (Lisp_Object node)
>>>> +{
>>>> +  CHECK_TS_NODE (node);
>>>> +  TSNode ts_node = XTS_NODE (node)->node;
>>>> +  const char *type = ts_node_type(ts_node);
>>>> +  return intern_c_string (type);
>>> 
>>> Why do we need to intern the string each time? can't we store the
>>> interned symbol there, instead of a C string, in the first place?
>> 
>> I’m not sure what do you mean by “store the interned symbol there”, where do I store the interned symbol?
> 
> In the struct that ts_node_type accesses, instead of the 'char *'
> string you store there now.

The struct that ts_node_type accesses is a TSNode, which is defined by tree-sitter. ts_node_type is an API provided by tree-sitter, I’m just exposing it to lisp. I could return strings instead of symbols, but I thought symbols might be more appropriate and more convenient for users of this function. 

>> (BTW, If you see something wrong, that’s probably because I don’t know the right way to do it, and grepping only got me that far.)
> 
> Do what? feel free to ask questions when you aren't sure how to
> accomplish something on the C level.

Thanks. Is below the correct way to set a buffer-local variable? (I’m setting tree-sitter-parser-list.)

struct buffer *old_buffer = current_buffer;
  set_buffer_internal (XBUFFER (buffer));

  Fset (Qtree_sitter_parser_list,
	Fcons (lisp_parser, Fsymbol_value (Qtree_sitter_parser_list)));

  set_buffer_internal (old_buffer);

Also, we don’t call change hooks in replace_range_2, why? Should I update tree-sitter trees in that function, or should I not?

Yuan