Rewriting make-docfile.c in Lisp?

Rewriting make-docfile.c in Lisp?

[sbaugh]

I was recently trawling through make-docfile.c and wondered:

Couldn't this be written in Lisp?

It seems like this complex parsing code would be much simpler in Lisp,
and there's no reason the make-docfile step couldn't run after Emacs is
built.  It would be a fair bit of work, but I think the resulting
make-docfile.el would be much smaller and easier to understand.

I'm not about to go do it, but it would be a good self-contained
project, I think, so maybe a good addition to TODO?

Does anyone have any objections or reasons why this shouldn't be done?

Re: Rewriting make-docfile.c in Lisp?

[Eli Zaretskii]

> From: sbaugh@catern.com
> Date: Wed, 05 May 2021 11:56:03 -0400
> 
> I was recently trawling through make-docfile.c and wondered:
> 
> Couldn't this be written in Lisp?

make-docfile does several jobs; did you mean to convert them all to
Lisp?  One of them is globals.h, which is needed for compiling the
Emacs C sources.

Re: Rewriting make-docfile.c in Lisp?

[Spencer Baugh]

Eli Zaretskii <eliz@gnu.org> writes:
>> Couldn't this be written in Lisp?
>
> make-docfile does several jobs; did you mean to convert them all to
> Lisp?  One of them is globals.h, which is needed for compiling the
> Emacs C sources.

Ah, I didn't see that globals.h was generated by make-docfile. I just
meant generating etc/DOC. Pluss any other jobs done by make-docfile that
aren't required to compile Emacs, if there are any.

Re: Rewriting make-docfile.c in Lisp?

[Alan Mackenzie]

Hello, Spencer.

On Wed, May 05, 2021 at 12:49:12 -0400, Spencer Baugh wrote:
> Eli Zaretskii <eliz@gnu.org> writes:
> >> Couldn't this be written in Lisp?
> >
> > make-docfile does several jobs; did you mean to convert them all to
> > Lisp?  One of them is globals.h, which is needed for compiling the
> > Emacs C sources.

> Ah, I didn't see that globals.h was generated by make-docfile. I just
> meant generating etc/DOC. Pluss any other jobs done by make-docfile that
> aren't required to compile Emacs, if there are any.

Why would one want to write it in Lisp anyway?

We have a working program in C which requires little maintenance, yet
which is used continually.  Rewriting it in Lisp would be _work_.

Also, the Lisp version would run more slowly than the C version, leading
to more irritation over build speeds than there currently is.  I think
this is also part of the build which holds things up in a single core,
thus making its speed more important than, say, a C or Lisp compilation.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Rewriting make-docfile.c in Lisp?

[Spencer Baugh]

Alan Mackenzie <acm@muc.de> writes:
> Why would one want to write it in Lisp anyway?
>
> We have a working program in C which requires little maintenance, yet
> which is used continually.  Rewriting it in Lisp would be _work_.

I was making a change and found it quite difficult.  I think it would be
much simpler to change if it was in Lisp.  I admit that since it's
rarely changed, it may not be worth it.

> Also, the Lisp version would run more slowly than the C version, leading
> to more irritation over build speeds than there currently is.  I think
> this is also part of the build which holds things up in a single core,
> thus making its speed more important than, say, a C or Lisp compilation.

A rewrite in Lisp could also take the opportunity to add parallelization
and speed it up, if indeed this is a bottleneck for the build.

Re: Rewriting make-docfile.c in Lisp?

[Stefan Monnier]

>> Why would one want to write it in Lisp anyway?
>> We have a working program in C which requires little maintenance, yet
>> which is used continually.  Rewriting it in Lisp would be _work_.

Agreed: I'm generally all for writing ELisp instead of C, but
I wouldn't bother rewriting this code in ELisp, personally.

> I was making a change and found it quite difficult.  I think it would
> be much simpler to change if it was in Lisp.  I admit that since it's
> rarely changed, it may not be worth it.

Maybe if you described the change that motivates your proposal, we'd be
less negative ;-)

>> Also, the Lisp version would run more slowly than the C version, leading
>> to more irritation over build speeds than there currently is.  I think
>> this is also part of the build which holds things up in a single core,
>> thus making its speed more important than, say, a C or Lisp compilation.
> A rewrite in Lisp could also take the opportunity to add parallelization
> and speed it up, if indeed this is a bottleneck for the build.

I don't think it can be sped up by writing it in ELisp (on the contrary).
It could be changed such that `make` can run it in parallel on each
C file, indeed, but it would have to be done well enough to make up for
the slowdown imposed by the ELisp version.


        Stefan

Re: Rewriting make-docfile.c in Lisp?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Alan Mackenzie <acm@muc.de>,  Eli Zaretskii <eliz@gnu.org>,
>   emacs-devel@gnu.org
> Date: Wed, 05 May 2021 15:08:05 -0400
> 
> It could be changed such that `make` can run it in parallel on each
> C file, indeed

But the same could be done with the C version, no?

Re: Rewriting make-docfile.c in Lisp?

[Stefan Monnier]

>> It could be changed such that `make` can run it in parallel on each
>> C file, indeed
>
> But the same could be done with the C version, no?

Of course (I think the argument was just that it might be easier to
make such a change to the code if it's written in ELisp than in C;
I don't know whether that's indeed the case, I haven't thought about it
nor refreshed my memory of how `make-docfile.c` currently works).


        Stefan

Re: Rewriting make-docfile.c in Lisp?

[Glenn Morris]

> Agreed: I'm generally all for writing ELisp instead of C, but
> I wouldn't bother rewriting this code in ELisp, personally.

Sounds like the etc/TODO entry that suggests doing just this should be
updated/removed then. It refers to
https://lists.gnu.org/r/emacs-devel/2012-06/msg00037.html

Re: Rewriting make-docfile.c in Lisp?

[Stefan Monnier]

>> Agreed: I'm generally all for writing ELisp instead of C, but
>> I wouldn't bother rewriting this code in ELisp, personally.
> Sounds like the etc/TODO entry that suggests doing just this should be
> updated/removed then. It refers to
> https://lists.gnu.org/r/emacs-devel/2012-06/msg00037.html

But there the idea was specifically for the scan of .el(c) files and not
for the scan of C files.  Also the idea was not to implement a new
scanning code but to do it from existing code that already looks at
the files (e.g. `autoloads.el` or the `bytecomp.el`).


        Stefan

Re: Rewriting make-docfile.c in Lisp?

[Stefan Monnier]

>>> Agreed: I'm generally all for writing ELisp instead of C, but
>>> I wouldn't bother rewriting this code in ELisp, personally.
>> Sounds like the etc/TODO entry that suggests doing just this should be
>> updated/removed then. It refers to
>> https://lists.gnu.org/r/emacs-devel/2012-06/msg00037.html
> But there the idea was specifically for the scan of .el(c) files and not
> for the scan of C files.  Also the idea was not to implement a new
> scanning code but to do it from existing code that already looks at
> the files (e.g. `autoloads.el` or the `bytecomp.el`).

BTW, personally, I think a better goal would be to try and eliminate the
DOC file altogether: for the docstrings coming from C code we should be
able to keep them in normal static variables (and move them to
a separate section with `_attribute__ ((section (".docstrings")))` or
something like that), and for the docstrings coming from preloaded ELisp
files we can fetch them from the .elc file like we do for
non-preloaded files.

The `make-docfile.c` would still be needed (probably under a new name)
for things like `global.h`, of course, and I don't see a convenient way
to replace it with a version written in ELisp for bootstrapping reasons.


        Stefan

Re: Rewriting make-docfile.c in Lisp?

[Spencer Baugh]

Stefan Monnier <monnier@iro.umontreal.ca> writes:
> Maybe if you described the change that motivates your proposal, we'd be
> less negative ;-)

Perhaps foolishly, I (experimentally) switched some existing uses of
DEFVAR_PER_BUFFER to a new variant with a new name, and then noticed
some test failures.  I tried to grep for "DEFVAR_PER_BUFFER" (and other
variations) to find what might be depending on the name of the macro but
didn't find anything, since make-docfile.c parses DEFVAR_PER_BUFFER one
character at a time with getc and so doesn't actually include the string
"DEFVAR_PER_BUFFER" anywhere.  Eventually I figured out that it was
make-docfile that wasn't picking up the new name for the
DEFVAR_PER_BUFFER macro.  So then I dug through scan_c_stream in
make-docfile.c to see what would be required to add support for the
DEFVAR_PER_BUFFER variant, but eventually gave up on understanding the
best way to inject my new macro name into the manually-unrolled tree of
getc-based parsing.

Re: Rewriting make-docfile.c in Lisp?

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@catern.com>
> Cc: Alan Mackenzie <acm@muc.de>, Eli Zaretskii <eliz@gnu.org>,
>  emacs-devel@gnu.org
> Date: Wed, 05 May 2021 17:11:08 -0400
> 
> Perhaps foolishly, I (experimentally) switched some existing uses of
> DEFVAR_PER_BUFFER to a new variant with a new name, and then noticed
> some test failures.  I tried to grep for "DEFVAR_PER_BUFFER" (and other
> variations) to find what might be depending on the name of the macro but
> didn't find anything, since make-docfile.c parses DEFVAR_PER_BUFFER one
> character at a time with getc and so doesn't actually include the string
> "DEFVAR_PER_BUFFER" anywhere.

We could improve discoverability of this by adding appropriate
comments to the source code there.

> Eventually I figured out that it was make-docfile that wasn't
> picking up the new name for the DEFVAR_PER_BUFFER macro.  So then I
> dug through scan_c_stream in make-docfile.c to see what would be
> required to add support for the DEFVAR_PER_BUFFER variant, but
> eventually gave up on understanding the best way to inject my new
> macro name into the manually-unrolled tree of getc-based parsing.

It's just a series of if/else if tests that examine the characters one
by one, so adding a new macro would mean some more tests like that.
Or am I missing something?

Re: Rewriting make-docfile.c in Lisp?

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@catern.com>
> Date: Wed, 05 May 2021 14:39:18 -0400
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> 
> A rewrite in Lisp could also take the opportunity to add parallelization
> and speed it up, if indeed this is a bottleneck for the build.

It isn't a bottleneck in its current form, AFAIK, but what did you
mean by "parallelization" in this context?

Re: Rewriting make-docfile.c in Lisp?

[Spencer Baugh]

Eli Zaretskii <eliz@gnu.org> writes:
>> From: Spencer Baugh <sbaugh@catern.com>
>> Date: Wed, 05 May 2021 14:39:18 -0400
>> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
>> 
>> A rewrite in Lisp could also take the opportunity to add parallelization
>> and speed it up, if indeed this is a bottleneck for the build.
>
> It isn't a bottleneck in its current form, AFAIK, but what did you
> mean by "parallelization" in this context?

I meant to run it in parallel on each individual file, one Emacs process
running make-docfile.el per file; and then merge the results together at
the end in some fashion.

Re: Rewriting make-docfile.c in Lisp?

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@catern.com>
> Cc: emacs-devel@gnu.org
> Date: Wed, 05 May 2021 12:49:12 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> >> Couldn't this be written in Lisp?
> >
> > make-docfile does several jobs; did you mean to convert them all to
> > Lisp?  One of them is globals.h, which is needed for compiling the
> > Emacs C sources.
> 
> Ah, I didn't see that globals.h was generated by make-docfile. I just
> meant generating etc/DOC. Pluss any other jobs done by make-docfile that
> aren't required to compile Emacs, if there are any.

But writing globals.h uses the same text-parsing code as the other
jobs, so I'm not sure it's a win to rewrite only the DOC parts in
Lisp, as we'd still need to maintain the globals part.

Does anyone else have an opinion?