bug#69952: [PATCH] Support pdumping compiled queries by dumping their source

bug#69952: [PATCH] Support pdumping compiled queries by dumping their source

[Sergey Vinokurov]

[-- Attachment #1: Type: text/plain, Size: 540 bytes --]

Hello,

This patch aims to allow more things to be dumped. In particular, 
compiled treesitter queries can be easily dumped by storing their source 
so that they will be recompiled on load.

I noticed that in my config compiled quires are created somewhere which 
prevents me dumping with standard Emacs build. But with this patch I can 
dump successfully and not bother finding out who produced the queries.

It seems like there should be no drawbacks in allowing more things to be 
dumped, please correct me if I'm wrong.

Regards,
Sergey

[-- Attachment #2: pdump.patch --]
[-- Type: text/x-patch, Size: 2016 bytes --]

From 8b7263181ed358fe88a358355c2269b93379703c Mon Sep 17 00:00:00 2001
From: Sergey Vinokurov <serg.foo@gmail.com>
Date: Sun, 6 Aug 2023 16:24:29 +0100
Subject: [PATCH] Support compiled queries in pdump by dumping source

* src/pdumper.c (dump_vectorlike): Dump compiled queries
* src/pdumper.c (dump_treesit_compiled_query): New function
---
 src/pdumper.c | 23 ++++++++++++++++++++++-
 1 file changed, 22 insertions(+), 1 deletion(-)

diff --git a/src/pdumper.c b/src/pdumper.c
index c7ebb38dea5..f2dd24c56a6 100644
--- a/src/pdumper.c
+++ b/src/pdumper.c
@@ -44,6 +44,7 @@
 #include "systime.h"
 #include "thread.h"
 #include "bignum.h"
+#include "treesit.h"
 
 #ifdef CHECK_STRUCTS
 # include "dmpstruct.h"
@@ -2215,6 +2216,21 @@ dump_finalizer (struct dump_context *ctx,
   return finish_dump_pvec (ctx, &out->header);
 }
 
+#ifdef HAVE_TREE_SITTER
+static dump_off
+dump_treesit_compiled_query (struct dump_context *ctx,
+			     struct Lisp_TS_Query *query)
+{
+  START_DUMP_PVEC (ctx, &query->header, struct Lisp_TS_Query, out);
+  dump_field_lv (ctx, &out->language, query, &query->language, WEIGHT_STRONG);
+  dump_field_lv (ctx, &out->source, query, &query->source, WEIGHT_STRONG);
+  /* Recompile these after load */
+  out->query = NULL;
+  out->cursor = NULL;
+  return finish_dump_pvec (ctx, &out->header);
+}
+#endif
+
 struct bignum_reload_info
 {
   dump_off data_location;
@@ -3108,6 +3124,12 @@ dump_vectorlike (struct dump_context *ctx,
           return DUMP_OBJECT_IS_RUNTIME_MAGIC;
         }
       break;
+#ifdef HAVE_TREE_SITTER
+    case PVEC_TS_COMPILED_QUERY:
+      return dump_treesit_compiled_query (ctx, XTS_COMPILED_QUERY (lv));
+#else
+    case PVEC_TS_COMPILED_QUERY:
+#endif
     case PVEC_WINDOW_CONFIGURATION:
     case PVEC_OTHER:
     case PVEC_XWIDGET:
@@ -3122,7 +3144,6 @@ dump_vectorlike (struct dump_context *ctx,
     case PVEC_FREE:
     case PVEC_TS_PARSER:
     case PVEC_TS_NODE:
-    case PVEC_TS_COMPILED_QUERY:
       break;
     }
   char msg[60];
-- 
2.43.1

bug#69952: [PATCH] Support pdumping compiled queries by dumping their source

[Eli Zaretskii]

> Date: Sat, 23 Mar 2024 03:27:18 +0000
> From: Sergey Vinokurov <serg.foo@gmail.com>
> 
> This patch aims to allow more things to be dumped. In particular, 
> compiled treesitter queries can be easily dumped by storing their source 
> so that they will be recompiled on load.
> 
> I noticed that in my config compiled quires are created somewhere which 
> prevents me dumping with standard Emacs build. But with this patch I can 
> dump successfully and not bother finding out who produced the queries.
> 
> It seems like there should be no drawbacks in allowing more things to be 
> dumped, please correct me if I'm wrong.

Thanks.  I added Daniel and Yuan to the discussion, in case they have
comments.

And I have a question: what happens if the pdumper file dumped with
tree-sitter available is loaded by an Emacs session in which
tree-sitter is not available?  That can happen on Windows, for
example, if the tree-sitter library or the grammar library required
for recomputing the query is not available.  I think we need some code
to prevent Emacs from crashing on startup in that case.

bug#69952: [PATCH] Support pdumping compiled queries by dumping their source

[Sergey Vinokurov]

On 23/03/2024 07:08, Eli Zaretskii wrote:
>> Date: Sat, 23 Mar 2024 03:27:18 +0000
>> From: Sergey Vinokurov <serg.foo@gmail.com>
>>
>> This patch aims to allow more things to be dumped. In particular,
>> compiled treesitter queries can be easily dumped by storing their source
>> so that they will be recompiled on load.
>>
>> I noticed that in my config compiled quires are created somewhere which
>> prevents me dumping with standard Emacs build. But with this patch I can
>> dump successfully and not bother finding out who produced the queries.
>>
>> It seems like there should be no drawbacks in allowing more things to be
>> dumped, please correct me if I'm wrong.
> 
> Thanks.  I added Daniel and Yuan to the discussion, in case they have
> comments.
> 
> And I have a question: what happens if the pdumper file dumped with
> tree-sitter available is loaded by an Emacs session in which
> tree-sitter is not available?  That can happen on Windows, for
> example, if the tree-sitter library or the grammar library required
> for recomputing the query is not available.  I think we need some code
> to prevent Emacs from crashing on startup in that case.

According to c2ecb08775dc24618de507d2d1ce0f9b0debe17e, treesitter 
queries are compiled lazily on fist use. When loading dump file in an 
Emacs with missing grammar library the user should encounter an error on 
first use of the query.

Quick testing shows that dump file can be loaded even with grammar 
definitions removed. Compiled queries are still there in elisp and 
there's no error during load.

bug#69952: [PATCH] Support pdumping compiled queries by dumping their source

[Eli Zaretskii]

> Date: Sat, 23 Mar 2024 12:53:37 +0000
> Cc: 69952@debbugs.gnu.org, Daniel Colascione <dancol@dancol.org>,
>  Yuan Fu <casouri@gmail.com>
> From: Sergey Vinokurov <serg.foo@gmail.com>
> 
> On 23/03/2024 07:08, Eli Zaretskii wrote:
> >> Date: Sat, 23 Mar 2024 03:27:18 +0000
> >> From: Sergey Vinokurov <serg.foo@gmail.com>
> >>
> >> This patch aims to allow more things to be dumped. In particular,
> >> compiled treesitter queries can be easily dumped by storing their source
> >> so that they will be recompiled on load.
> >>
> >> I noticed that in my config compiled quires are created somewhere which
> >> prevents me dumping with standard Emacs build. But with this patch I can
> >> dump successfully and not bother finding out who produced the queries.
> >>
> >> It seems like there should be no drawbacks in allowing more things to be
> >> dumped, please correct me if I'm wrong.
> > 
> > Thanks.  I added Daniel and Yuan to the discussion, in case they have
> > comments.
> > 
> > And I have a question: what happens if the pdumper file dumped with
> > tree-sitter available is loaded by an Emacs session in which
> > tree-sitter is not available?  That can happen on Windows, for
> > example, if the tree-sitter library or the grammar library required
> > for recomputing the query is not available.  I think we need some code
> > to prevent Emacs from crashing on startup in that case.
> 
> According to c2ecb08775dc24618de507d2d1ce0f9b0debe17e, treesitter 
> queries are compiled lazily on fist use. When loading dump file in an 
> Emacs with missing grammar library the user should encounter an error on 
> first use of the query.
> 
> Quick testing shows that dump file can be loaded even with grammar 
> definitions removed. Compiled queries are still there in elisp and 
> there's no error during load.

OK, thanks.

bug#69952: [PATCH] Support pdumping compiled queries by dumping their source

[Eli Zaretskii]

Ping!  Daniel and Yuan, any comments on this proposal?

> Cc: 69952@debbugs.gnu.org, Daniel Colascione <dancol@dancol.org>,
>  Yuan Fu <casouri@gmail.com>
> Date: Sat, 23 Mar 2024 09:08:21 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > Date: Sat, 23 Mar 2024 03:27:18 +0000
> > From: Sergey Vinokurov <serg.foo@gmail.com>
> > 
> > This patch aims to allow more things to be dumped. In particular, 
> > compiled treesitter queries can be easily dumped by storing their source 
> > so that they will be recompiled on load.
> > 
> > I noticed that in my config compiled quires are created somewhere which 
> > prevents me dumping with standard Emacs build. But with this patch I can 
> > dump successfully and not bother finding out who produced the queries.
> > 
> > It seems like there should be no drawbacks in allowing more things to be 
> > dumped, please correct me if I'm wrong.
> 
> Thanks.  I added Daniel and Yuan to the discussion, in case they have
> comments.
> 
> And I have a question: what happens if the pdumper file dumped with
> tree-sitter available is loaded by an Emacs session in which
> tree-sitter is not available?  That can happen on Windows, for
> example, if the tree-sitter library or the grammar library required
> for recomputing the query is not available.  I think we need some code
> to prevent Emacs from crashing on startup in that case.
> 
> 
> 
> 

bug#69952: [PATCH] Support pdumping compiled queries by dumping their source

[Yuan Fu]

> On Apr 13, 2024, at 12:41 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> Ping!  Daniel and Yuan, any comments on this proposal?
> 
>> Cc: 69952@debbugs.gnu.org, Daniel Colascione <dancol@dancol.org>,
>> Yuan Fu <casouri@gmail.com>
>> Date: Sat, 23 Mar 2024 09:08:21 +0200
>> From: Eli Zaretskii <eliz@gnu.org>
>> 
>>> Date: Sat, 23 Mar 2024 03:27:18 +0000
>>> From: Sergey Vinokurov <serg.foo@gmail.com>
>>> 
>>> This patch aims to allow more things to be dumped. In particular, 
>>> compiled treesitter queries can be easily dumped by storing their source 
>>> so that they will be recompiled on load.
>>> 
>>> I noticed that in my config compiled quires are created somewhere which 
>>> prevents me dumping with standard Emacs build. But with this patch I can 
>>> dump successfully and not bother finding out who produced the queries.
>>> 
>>> It seems like there should be no drawbacks in allowing more things to be 
>>> dumped, please correct me if I'm wrong.
>> 
>> Thanks.  I added Daniel and Yuan to the discussion, in case they have
>> comments.
>> 
>> And I have a question: what happens if the pdumper file dumped with
>> tree-sitter available is loaded by an Emacs session in which
>> tree-sitter is not available?  That can happen on Windows, for
>> example, if the tree-sitter library or the grammar library required
>> for recomputing the query is not available.  I think we need some code
>> to prevent Emacs from crashing on startup in that case.

Yeah, being able to dump queries is certainly nice. There will be problems if the Emacs session that later loads the query either have a different grammar version loaded, or a difference grammar library for the same language, or outright doesn’t have tree-sitter like Eli mentioned. I don’t know if they are big enough problems to give up dumping queries, though.

Yuan

bug#69952: [PATCH] Support pdumping compiled queries by dumping their source

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Sat, 20 Apr 2024 01:08:15 -0700
> Cc: dancol@dancol.org,
>  serg.foo@gmail.com,
>  69952@debbugs.gnu.org
> 
> 
> 
> > On Apr 13, 2024, at 12:41 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> >> 
> >> And I have a question: what happens if the pdumper file dumped with
> >> tree-sitter available is loaded by an Emacs session in which
> >> tree-sitter is not available?  That can happen on Windows, for
> >> example, if the tree-sitter library or the grammar library required
> >> for recomputing the query is not available.  I think we need some code
> >> to prevent Emacs from crashing on startup in that case.
> 
> Yeah, being able to dump queries is certainly nice. There will be problems if the Emacs session that later loads the query either have a different grammar version loaded, or a difference grammar library for the same language, or outright doesn’t have tree-sitter like Eli mentioned. I don’t know if they are big enough problems to give up dumping queries, though.

Giving up: no.  But I think the code which loads the queries from the
pdumper file should be protected from crashing in those cases.  Can
you suggest how to do that?

bug#69952: [PATCH] Support pdumping compiled queries by dumping their source

[Yuan Fu]

> On Apr 20, 2024, at 2:32 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Sat, 20 Apr 2024 01:08:15 -0700
>> Cc: dancol@dancol.org,
>> serg.foo@gmail.com,
>> 69952@debbugs.gnu.org
>> 
>> 
>> 
>>> On Apr 13, 2024, at 12:41 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>>>> 
>>>> And I have a question: what happens if the pdumper file dumped with
>>>> tree-sitter available is loaded by an Emacs session in which
>>>> tree-sitter is not available?  That can happen on Windows, for
>>>> example, if the tree-sitter library or the grammar library required
>>>> for recomputing the query is not available.  I think we need some code
>>>> to prevent Emacs from crashing on startup in that case.
>> 
>> Yeah, being able to dump queries is certainly nice. There will be problems if the Emacs session that later loads the query either have a different grammar version loaded, or a difference grammar library for the same language, or outright doesn’t have tree-sitter like Eli mentioned. I don’t know if they are big enough problems to give up dumping queries, though.
> 
> Giving up: no.  But I think the code which loads the queries from the
> pdumper file should be protected from crashing in those cases.  Can
> you suggest how to do that?

Would Emacs crash? If the dump file just contains the query (which is a string), then Emacs would just compile the query with treesit-query-compile, it could fail (due to the reasons I mentioned), but shouldn’t crash Emacs.

Yuan

bug#69952: [PATCH] Support pdumping compiled queries by dumping their source

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Sat, 20 Apr 2024 15:50:36 -0700
> Cc: Daniel Colascione <dancol@dancol.org>,
>  serg.foo@gmail.com,
>  69952@debbugs.gnu.org
> 
> >>> On Apr 13, 2024, at 12:41 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> >>>> 
> >>>> And I have a question: what happens if the pdumper file dumped with
> >>>> tree-sitter available is loaded by an Emacs session in which
> >>>> tree-sitter is not available?  That can happen on Windows, for
> >>>> example, if the tree-sitter library or the grammar library required
> >>>> for recomputing the query is not available.  I think we need some code
> >>>> to prevent Emacs from crashing on startup in that case.
> >> 
> >> Yeah, being able to dump queries is certainly nice. There will be problems if the Emacs session that later loads the query either have a different grammar version loaded, or a difference grammar library for the same language, or outright doesn’t have tree-sitter like Eli mentioned. I don’t know if they are big enough problems to give up dumping queries, though.
> > 
> > Giving up: no.  But I think the code which loads the queries from the
> > pdumper file should be protected from crashing in those cases.  Can
> > you suggest how to do that?
> 
> Would Emacs crash? If the dump file just contains the query (which is a string), then Emacs would just compile the query with treesit-query-compile, it could fail (due to the reasons I mentioned), but shouldn’t crash Emacs.

Can Emacs compile a query if the tree-sitter shared library and/or the
grammar library is not available?

bug#69952: [PATCH] Support pdumping compiled queries by dumping their source

[Yuan Fu]

> On Apr 20, 2024, at 9:53 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Sat, 20 Apr 2024 15:50:36 -0700
>> Cc: Daniel Colascione <dancol@dancol.org>,
>> serg.foo@gmail.com,
>> 69952@debbugs.gnu.org
>> 
>>>>> On Apr 13, 2024, at 12:41 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>>>>>> 
>>>>>> And I have a question: what happens if the pdumper file dumped with
>>>>>> tree-sitter available is loaded by an Emacs session in which
>>>>>> tree-sitter is not available?  That can happen on Windows, for
>>>>>> example, if the tree-sitter library or the grammar library required
>>>>>> for recomputing the query is not available.  I think we need some code
>>>>>> to prevent Emacs from crashing on startup in that case.
>>>> 
>>>> Yeah, being able to dump queries is certainly nice. There will be problems if the Emacs session that later loads the query either have a different grammar version loaded, or a difference grammar library for the same language, or outright doesn’t have tree-sitter like Eli mentioned. I don’t know if they are big enough problems to give up dumping queries, though.
>>> 
>>> Giving up: no.  But I think the code which loads the queries from the
>>> pdumper file should be protected from crashing in those cases.  Can
>>> you suggest how to do that?
>> 
>> Would Emacs crash? If the dump file just contains the query (which is a string), then Emacs would just compile the query with treesit-query-compile, it could fail (due to the reasons I mentioned), but shouldn’t crash Emacs.
> 
> Can Emacs compile a query if the tree-sitter shared library and/or the
> grammar library is not available?

No. You need both tree-sitter library and the grammar library.

Yuan

bug#69952: [PATCH] Support pdumping compiled queries by dumping their source

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Sun, 21 Apr 2024 16:41:54 -0700
> Cc: Daniel Colascione <dancol@dancol.org>,
>  serg.foo@gmail.com,
>  69952@debbugs.gnu.org
> 
> 
> 
> > On Apr 20, 2024, at 9:53 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> > 
> >> From: Yuan Fu <casouri@gmail.com>
> >> Date: Sat, 20 Apr 2024 15:50:36 -0700
> >> Cc: Daniel Colascione <dancol@dancol.org>,
> >> serg.foo@gmail.com,
> >> 69952@debbugs.gnu.org
> >> 
> >>>>> On Apr 13, 2024, at 12:41 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> >>>>>> 
> >>>>>> And I have a question: what happens if the pdumper file dumped with
> >>>>>> tree-sitter available is loaded by an Emacs session in which
> >>>>>> tree-sitter is not available?  That can happen on Windows, for
> >>>>>> example, if the tree-sitter library or the grammar library required
> >>>>>> for recomputing the query is not available.  I think we need some code
> >>>>>> to prevent Emacs from crashing on startup in that case.
> >>>> 
> >>>> Yeah, being able to dump queries is certainly nice. There will be problems if the Emacs session that later loads the query either have a different grammar version loaded, or a difference grammar library for the same language, or outright doesn’t have tree-sitter like Eli mentioned. I don’t know if they are big enough problems to give up dumping queries, though.
> >>> 
> >>> Giving up: no.  But I think the code which loads the queries from the
> >>> pdumper file should be protected from crashing in those cases.  Can
> >>> you suggest how to do that?
> >> 
> >> Would Emacs crash? If the dump file just contains the query (which is a string), then Emacs would just compile the query with treesit-query-compile, it could fail (due to the reasons I mentioned), but shouldn’t crash Emacs.
> > 
> > Can Emacs compile a query if the tree-sitter shared library and/or the
> > grammar library is not available?
> 
> No. You need both tree-sitter library and the grammar library.

That's what I thought.  So starting Emacs in that case will attempt to
call a function from the tree-sitter library, and will segfault,
right?  If so, we should have some protection in the code in
pdumper.c that loads queries, which tests that tree-sitter is
availabale, and if not does something to prevent the segfault, like
not loading the query, perhaps?

bug#69952: [PATCH] Support pdumping compiled queries by dumping their source

[Yuan Fu]

> On Apr 21, 2024, at 10:47 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Sun, 21 Apr 2024 16:41:54 -0700
>> Cc: Daniel Colascione <dancol@dancol.org>,
>> serg.foo@gmail.com,
>> 69952@debbugs.gnu.org
>> 
>> 
>> 
>>> On Apr 20, 2024, at 9:53 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>>> 
>>>> From: Yuan Fu <casouri@gmail.com>
>>>> Date: Sat, 20 Apr 2024 15:50:36 -0700
>>>> Cc: Daniel Colascione <dancol@dancol.org>,
>>>> serg.foo@gmail.com,
>>>> 69952@debbugs.gnu.org
>>>> 
>>>>>>> On Apr 13, 2024, at 12:41 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>>>>>>>> 
>>>>>>>> And I have a question: what happens if the pdumper file dumped with
>>>>>>>> tree-sitter available is loaded by an Emacs session in which
>>>>>>>> tree-sitter is not available?  That can happen on Windows, for
>>>>>>>> example, if the tree-sitter library or the grammar library required
>>>>>>>> for recomputing the query is not available.  I think we need some code
>>>>>>>> to prevent Emacs from crashing on startup in that case.
>>>>>> 
>>>>>> Yeah, being able to dump queries is certainly nice. There will be problems if the Emacs session that later loads the query either have a different grammar version loaded, or a difference grammar library for the same language, or outright doesn’t have tree-sitter like Eli mentioned. I don’t know if they are big enough problems to give up dumping queries, though.
>>>>> 
>>>>> Giving up: no.  But I think the code which loads the queries from the
>>>>> pdumper file should be protected from crashing in those cases.  Can
>>>>> you suggest how to do that?
>>>> 
>>>> Would Emacs crash? If the dump file just contains the query (which is a string), then Emacs would just compile the query with treesit-query-compile, it could fail (due to the reasons I mentioned), but shouldn’t crash Emacs.
>>> 
>>> Can Emacs compile a query if the tree-sitter shared library and/or the
>>> grammar library is not available?
>> 
>> No. You need both tree-sitter library and the grammar library.
> 
> That's what I thought.  So starting Emacs in that case will attempt to
> call a function from the tree-sitter library, and will segfault,
> right?  If so, we should have some protection in the code in
> pdumper.c that loads queries, which tests that tree-sitter is
> availabale, and if not does something to prevent the segfault, like
> not loading the query, perhaps?

Right, it can use Ftreesit_available_p (we can define a C equivalent if pdumper can’t call lisp), and decide whether to compile the query or just keep the query as a string, or signal an error?

Yuan

bug#69952: [PATCH] Support pdumping compiled queries by dumping their source

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Sun, 21 Apr 2024 23:04:47 -0700
> Cc: Daniel Colascione <dancol@dancol.org>,
>  serg.foo@gmail.com,
>  69952@debbugs.gnu.org
> 
> >> No. You need both tree-sitter library and the grammar library.
> > 
> > That's what I thought.  So starting Emacs in that case will attempt to
> > call a function from the tree-sitter library, and will segfault,
> > right?  If so, we should have some protection in the code in
> > pdumper.c that loads queries, which tests that tree-sitter is
> > availabale, and if not does something to prevent the segfault, like
> > not loading the query, perhaps?
> 
> Right, it can use Ftreesit_available_p (we can define a C equivalent if pdumper can’t call lisp), and decide whether to compile the query or just keep the query as a string, or signal an error?

We cannot safely signal an error at that point, I think, so just
keeping the query as a string should do, as it will then signal an
error at run time when that query is used, is that right?

bug#69952: [PATCH] Support pdumping compiled queries by dumping their source

[Yuan Fu]

> On Apr 21, 2024, at 11:18 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Sun, 21 Apr 2024 23:04:47 -0700
>> Cc: Daniel Colascione <dancol@dancol.org>,
>> serg.foo@gmail.com,
>> 69952@debbugs.gnu.org
>> 
>>>> No. You need both tree-sitter library and the grammar library.
>>> 
>>> That's what I thought.  So starting Emacs in that case will attempt to
>>> call a function from the tree-sitter library, and will segfault,
>>> right?  If so, we should have some protection in the code in
>>> pdumper.c that loads queries, which tests that tree-sitter is
>>> availabale, and if not does something to prevent the segfault, like
>>> not loading the query, perhaps?
>> 
>> Right, it can use Ftreesit_available_p (we can define a C equivalent if pdumper can’t call lisp), and decide whether to compile the query or just keep the query as a string, or signal an error?
> 
> We cannot safely signal an error at that point, I think, so just
> keeping the query as a string should do, as it will then signal an
> error at run time when that query is used, is that right?

Yes, most likely a function-undefined signal, since all the treesit.c functions like treesit-query-capture or treesit-query-compile will be nonexistent. And usually the Lisp program trying to use the query would check for tree-sitter availability with treesit-available-p before trying to use any tree-sitter functions; so that signal will be usually avoided as well.

Yuan

bug#69952: [PATCH] Support pdumping compiled queries by dumping their source

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Sun, 21 Apr 2024 23:25:10 -0700
> Cc: dancol@dancol.org,
>  serg.foo@gmail.com,
>  69952@debbugs.gnu.org
> 
> > On Apr 21, 2024, at 11:18 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> > 
> > 
> > We cannot safely signal an error at that point, I think, so just
> > keeping the query as a string should do, as it will then signal an
> > error at run time when that query is used, is that right?
> 
> Yes, most likely a function-undefined signal, since all the treesit.c functions like treesit-query-capture or treesit-query-compile will be nonexistent. And usually the Lisp program trying to use the query would check for tree-sitter availability with treesit-available-p before trying to use any tree-sitter functions; so that signal will be usually avoided as well.

Can you suggest such an addition to the patch?

bug#69952: [PATCH] Support pdumping compiled queries by dumping their source

[Yuan Fu]

> On May 4, 2024, at 2:39 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Sun, 21 Apr 2024 23:25:10 -0700
>> Cc: dancol@dancol.org,
>> serg.foo@gmail.com,
>> 69952@debbugs.gnu.org
>> 
>>> On Apr 21, 2024, at 11:18 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>>> 
>>> 
>>> We cannot safely signal an error at that point, I think, so just
>>> keeping the query as a string should do, as it will then signal an
>>> error at run time when that query is used, is that right?
>> 
>> Yes, most likely a function-undefined signal, since all the treesit.c functions like treesit-query-capture or treesit-query-compile will be nonexistent. And usually the Lisp program trying to use the query would check for tree-sitter availability with treesit-available-p before trying to use any tree-sitter functions; so that signal will be usually avoided as well.
> 
> Can you suggest such an addition to the patch?

Let me take a look.

Yuan

bug#69952: [PATCH] Support pdumping compiled queries by dumping their source

[Eli Zaretskii]

Ping!

> From: Yuan Fu <casouri@gmail.com>
> Date: Sat, 4 May 2024 14:29:06 -0700
> Cc: serg.foo@gmail.com,
>  dancol@dancol.org,
>  69952@debbugs.gnu.org
> 
> 
> 
> > On May 4, 2024, at 2:39 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> > 
> >> From: Yuan Fu <casouri@gmail.com>
> >> Date: Sun, 21 Apr 2024 23:25:10 -0700
> >> Cc: dancol@dancol.org,
> >> serg.foo@gmail.com,
> >> 69952@debbugs.gnu.org
> >> 
> >>> On Apr 21, 2024, at 11:18 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> >>> 
> >>> 
> >>> We cannot safely signal an error at that point, I think, so just
> >>> keeping the query as a string should do, as it will then signal an
> >>> error at run time when that query is used, is that right?
> >> 
> >> Yes, most likely a function-undefined signal, since all the treesit.c functions like treesit-query-capture or treesit-query-compile will be nonexistent. And usually the Lisp program trying to use the query would check for tree-sitter availability with treesit-available-p before trying to use any tree-sitter functions; so that signal will be usually avoided as well.
> > 
> > Can you suggest such an addition to the patch?
> 
> Let me take a look.
> 
> Yuan

bug#69952: [PATCH] Support pdumping compiled queries by dumping their source

[Yuan Fu]

> On May 18, 2024, at 1:38 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> Ping!
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Sat, 4 May 2024 14:29:06 -0700
>> Cc: serg.foo@gmail.com,
>> dancol@dancol.org,
>> 69952@debbugs.gnu.org
>> 
>> 
>> 
>>> On May 4, 2024, at 2:39 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>>> 
>>>> From: Yuan Fu <casouri@gmail.com>
>>>> Date: Sun, 21 Apr 2024 23:25:10 -0700
>>>> Cc: dancol@dancol.org,
>>>> serg.foo@gmail.com,
>>>> 69952@debbugs.gnu.org
>>>> 
>>>>> On Apr 21, 2024, at 11:18 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>>>>> 
>>>>> 
>>>>> We cannot safely signal an error at that point, I think, so just
>>>>> keeping the query as a string should do, as it will then signal an
>>>>> error at run time when that query is used, is that right?
>>>> 
>>>> Yes, most likely a function-undefined signal, since all the treesit.c functions like treesit-query-capture or treesit-query-compile will be nonexistent. And usually the Lisp program trying to use the query would check for tree-sitter availability with treesit-available-p before trying to use any tree-sitter functions; so that signal will be usually avoided as well.
>>> 
>>> Can you suggest such an addition to the patch?
>> 
>> Let me take a look.
>> 
>> Yuan

Am I missing something? It seems the patch doesn’t include anything about loading a dumped query? I guess that’s the addition you’re talking about? If I want to add a special loader, where should I start?

Yuan

bug#69952: [PATCH] Support pdumping compiled queries by dumping their source

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Tue, 21 May 2024 23:36:47 -0700
> Cc: serg.foo@gmail.com,
>  dancol@dancol.org,
>  69952@debbugs.gnu.org
> 
> >>>> Yes, most likely a function-undefined signal, since all the treesit.c functions like treesit-query-capture or treesit-query-compile will be nonexistent. And usually the Lisp program trying to use the query would check for tree-sitter availability with treesit-available-p before trying to use any tree-sitter functions; so that signal will be usually avoided as well.
> >>> 
> >>> Can you suggest such an addition to the patch?
> >> 
> >> Let me take a look.
> >> 
> >> Yuan
> 
> Am I missing something? It seems the patch doesn’t include anything about loading a dumped query?

Looks like that, yes.

> I guess that’s the addition you’re talking about? If I want to add a special loader, where should I start?

I guess in dump_do_emacs_relocation and/or dump_do_dump_relocation?

Alternatively, perhaps it's better to define a hook via
pdumper_do_now_and_after_load_impl or
pdumper_do_now_and_after_late_load_impl, in which case the hook will
be run after loading the dump file.  This sounds easier if you can
access all the loaded queries one by one, instead of catching them
as they are being loaded.

Andrea, can you help Yuan here?  You have dealt with some of this when
you added dumping and restoring of native-compiled CUs, so I guess you
know your way better in pdumper.c.

bug#69952: [PATCH] Support pdumping compiled queries by dumping their source

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Yuan Fu <casouri@gmail.com>
>> Date: Tue, 21 May 2024 23:36:47 -0700
>> Cc: serg.foo@gmail.com,
>>  dancol@dancol.org,
>>  69952@debbugs.gnu.org
>> 
>> >>>> Yes, most likely a function-undefined signal, since all the
>> >>> treesit.c functions like treesit-query-capture or
>> >>> treesit-query-compile will be nonexistent. And usually the Lisp
>> >>> program trying to use the query would check for tree-sitter
>> >>> availability with treesit-available-p before trying to use any
>> >>> tree-sitter functions; so that signal will be usually avoided as
>> >>> well.
>> >>> 
>> >>> Can you suggest such an addition to the patch?
>> >> 
>> >> Let me take a look.
>> >> 
>> >> Yuan
>> 
>> Am I missing something? It seems the patch doesn’t include anything about loading a dumped query?
>
> Looks like that, yes.
>
>> I guess that’s the addition you’re talking about? If I want to add a special loader, where should I start?
>
> I guess in dump_do_emacs_relocation and/or dump_do_dump_relocation?

Yes that's correct.

> Alternatively, perhaps it's better to define a hook via
> pdumper_do_now_and_after_load_impl or
> pdumper_do_now_and_after_late_load_impl, in which case the hook will
> be run after loading the dump file.  This sounds easier if you can
> access all the loaded queries one by one, instead of catching them
> as they are being loaded.

Note also that if one decides to execute code at LATE_RELOCS or
VERY_LATE_RELOCS time he has the lisp machinery at disposal (including
allocation), but native code becomes functional only after
VERY_LATE_RELOCS relocs are done (not sure this can impact you here).

On this might be necessary if Yuan goes for the hook and needs native
Lisp code to be working, to add
'pdumper_do_now_and_after_very_late_load_impl' (which I think I didn't
bothered at the time).

  Andrea

bug#69952: [PATCH] Support pdumping compiled queries by dumping their source

[Yuan Fu]

> On May 29, 2024, at 9:35 AM, Andrea Corallo <acorallo@gnu.org> wrote:
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
>>> From: Yuan Fu <casouri@gmail.com>
>>> Date: Tue, 21 May 2024 23:36:47 -0700
>>> Cc: serg.foo@gmail.com,
>>> dancol@dancol.org,
>>> 69952@debbugs.gnu.org
>>> 
>>>>>>> Yes, most likely a function-undefined signal, since all the
>>>>>> treesit.c functions like treesit-query-capture or
>>>>>> treesit-query-compile will be nonexistent. And usually the Lisp
>>>>>> program trying to use the query would check for tree-sitter
>>>>>> availability with treesit-available-p before trying to use any
>>>>>> tree-sitter functions; so that signal will be usually avoided as
>>>>>> well.
>>>>>> 
>>>>>> Can you suggest such an addition to the patch?
>>>>> 
>>>>> Let me take a look.
>>>>> 
>>>>> Yuan
>>> 
>>> Am I missing something? It seems the patch doesn’t include anything about loading a dumped query?
>> 
>> Looks like that, yes.
>> 
>>> I guess that’s the addition you’re talking about? If I want to add a special loader, where should I start?
>> 
>> I guess in dump_do_emacs_relocation and/or dump_do_dump_relocation?
> 
> Yes that's correct.
> 
>> Alternatively, perhaps it's better to define a hook via
>> pdumper_do_now_and_after_load_impl or
>> pdumper_do_now_and_after_late_load_impl, in which case the hook will
>> be run after loading the dump file.  This sounds easier if you can
>> access all the loaded queries one by one, instead of catching them
>> as they are being loaded.
> 
> Note also that if one decides to execute code at LATE_RELOCS or
> VERY_LATE_RELOCS time he has the lisp machinery at disposal (including
> allocation), but native code becomes functional only after
> VERY_LATE_RELOCS relocs are done (not sure this can impact you here).
> 
> On this might be necessary if Yuan goes for the hook and needs native
> Lisp code to be working, to add
> 'pdumper_do_now_and_after_very_late_load_impl' (which I think I didn't
> bothered at the time).
> 
>  Andrea

Thanks to you both! I’ll see what I can come up with.

Yuan

bug#69952: [PATCH] Support pdumping compiled queries by dumping their source

[Eli Zaretskii]

Ping!  Any progress there?

> From: Yuan Fu <casouri@gmail.com>
> Date: Sat, 1 Jun 2024 10:07:38 -0700
> Cc: Eli Zaretskii <eliz@gnu.org>,
>  serg.foo@gmail.com,
>  Daniel Colascione <dancol@dancol.org>,
>  69952@debbugs.gnu.org
> 
> 
> 
> > On May 29, 2024, at 9:35 AM, Andrea Corallo <acorallo@gnu.org> wrote:
> > 
> > Eli Zaretskii <eliz@gnu.org> writes:
> > 
> >>> From: Yuan Fu <casouri@gmail.com>
> >>> Date: Tue, 21 May 2024 23:36:47 -0700
> >>> Cc: serg.foo@gmail.com,
> >>> dancol@dancol.org,
> >>> 69952@debbugs.gnu.org
> >>> 
> >>>>>>> Yes, most likely a function-undefined signal, since all the
> >>>>>> treesit.c functions like treesit-query-capture or
> >>>>>> treesit-query-compile will be nonexistent. And usually the Lisp
> >>>>>> program trying to use the query would check for tree-sitter
> >>>>>> availability with treesit-available-p before trying to use any
> >>>>>> tree-sitter functions; so that signal will be usually avoided as
> >>>>>> well.
> >>>>>> 
> >>>>>> Can you suggest such an addition to the patch?
> >>>>> 
> >>>>> Let me take a look.
> >>>>> 
> >>>>> Yuan
> >>> 
> >>> Am I missing something? It seems the patch doesn’t include anything about loading a dumped query?
> >> 
> >> Looks like that, yes.
> >> 
> >>> I guess that’s the addition you’re talking about? If I want to add a special loader, where should I start?
> >> 
> >> I guess in dump_do_emacs_relocation and/or dump_do_dump_relocation?
> > 
> > Yes that's correct.
> > 
> >> Alternatively, perhaps it's better to define a hook via
> >> pdumper_do_now_and_after_load_impl or
> >> pdumper_do_now_and_after_late_load_impl, in which case the hook will
> >> be run after loading the dump file.  This sounds easier if you can
> >> access all the loaded queries one by one, instead of catching them
> >> as they are being loaded.
> > 
> > Note also that if one decides to execute code at LATE_RELOCS or
> > VERY_LATE_RELOCS time he has the lisp machinery at disposal (including
> > allocation), but native code becomes functional only after
> > VERY_LATE_RELOCS relocs are done (not sure this can impact you here).
> > 
> > On this might be necessary if Yuan goes for the hook and needs native
> > Lisp code to be working, to add
> > 'pdumper_do_now_and_after_very_late_load_impl' (which I think I didn't
> > bothered at the time).
> > 
> >  Andrea
> 
> Thanks to you both! I’ll see what I can come up with.
> 
> Yuan
> 
> 

bug#69952: [PATCH] Support pdumping compiled queries by dumping their source

[Yuan Fu]

> On Jun 15, 2024, at 12:59 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> Ping!  Any progress there?

Unfortunately I haven’t make much progress. I’ll report back when I do. Thanks for the reminder!

Yuan

bug#69952: [PATCH] Support pdumping compiled queries by dumping their source

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Tue, 18 Jun 2024 23:25:45 -0700
> Cc: Andrea Corallo <acorallo@gnu.org>,
>  serg.foo@gmail.com,
>  Daniel Colascione <dancol@dancol.org>,
>  69952@debbugs.gnu.org
> 
> 
> 
> > On Jun 15, 2024, at 12:59 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> > 
> > Ping!  Any progress there?
> 
> Unfortunately I haven’t make much progress. I’ll report back when I do. Thanks for the reminder!

Any news with this?

bug#69952: [PATCH] Support pdumping compiled queries by dumping their source

[Eli Zaretskii]

Ping!

> Cc: serg.foo@gmail.com, acorallo@gnu.org, dancol@dancol.org,
>  69952@debbugs.gnu.org
> Date: Sat, 06 Jul 2024 10:24:32 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > From: Yuan Fu <casouri@gmail.com>
> > Date: Tue, 18 Jun 2024 23:25:45 -0700
> > Cc: Andrea Corallo <acorallo@gnu.org>,
> >  serg.foo@gmail.com,
> >  Daniel Colascione <dancol@dancol.org>,
> >  69952@debbugs.gnu.org
> > 
> > 
> > 
> > > On Jun 15, 2024, at 12:59 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> > > 
> > > Ping!  Any progress there?
> > 
> > Unfortunately I haven’t make much progress. I’ll report back when I do. Thanks for the reminder!
> 
> Any news with this?
> 
> 
> 
> 

bug#69952: [PATCH] Support pdumping compiled queries by dumping their source

[Yuan Fu]

Hey, sorry for the delay. It took me several attempts, but it’s actually pretty simple once I understand it. I just pushed the original patch plus my patch to master. Dumping query and loading it back now works pretty well! 

Yuan

> On Jul 20, 2024, at 2:43 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> Ping!
> 
>> Cc: serg.foo@gmail.com, acorallo@gnu.org, dancol@dancol.org,
>> 69952@debbugs.gnu.org
>> Date: Sat, 06 Jul 2024 10:24:32 +0300
>> From: Eli Zaretskii <eliz@gnu.org>
>> 
>>> From: Yuan Fu <casouri@gmail.com>
>>> Date: Tue, 18 Jun 2024 23:25:45 -0700
>>> Cc: Andrea Corallo <acorallo@gnu.org>,
>>> serg.foo@gmail.com,
>>> Daniel Colascione <dancol@dancol.org>,
>>> 69952@debbugs.gnu.org
>>> 
>>> 
>>> 
>>>> On Jun 15, 2024, at 12:59 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>>>> 
>>>> Ping!  Any progress there?
>>> 
>>> Unfortunately I haven’t make much progress. I’ll report back when I do. Thanks for the reminder!
>> 
>> Any news with this?
>> 
>> 
>> 
>> 

bug#69952: [PATCH] Support pdumping compiled queries by dumping their source

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Sat, 20 Jul 2024 14:06:41 -0700
> Cc: serg.foo@gmail.com,
>  Andrea Corallo <acorallo@gnu.org>,
>  Daniel Colascione <dancol@dancol.org>,
>  69952-done@debbugs.gnu.org
> 
> Hey, sorry for the delay. It took me several attempts, but it’s actually pretty simple once I understand it. I just pushed the original patch plus my patch to master. Dumping query and loading it back now works pretty well! 

Great, thanks.