Colorful line numbers

Colorful line numbers

[João Távora]

Hi,

I'd like to play with fontified line numbers.

As far as I can read from maybe_produce_line_number in xdisp.c the face
is one of a fixed set: line-number, line-number-current-line,
line-number-major-tick or line-number-minor-tick.

IOW, there is no Elisp mechanism for changing the line number on a
line-by-line basis, for example with some text property set on any
character of the given line.

Could we implement one such mechanism?  Was such a mechanism ever
considered useful/practical/impractical?

Thanks,
João

Re: Colorful line numbers

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Fri, 22 Jul 2022 08:50:21 +0100
> 
> As far as I can read from maybe_produce_line_number in xdisp.c the face
> is one of a fixed set: line-number, line-number-current-line,
> line-number-major-tick or line-number-minor-tick.

Yes.

> IOW, there is no Elisp mechanism for changing the line number on a
> line-by-line basis, for example with some text property set on any
> character of the given line.
> 
> Could we implement one such mechanism?

No.  The native line-number display is implemented inside the low
level of the display code, and I don't want to call to Lisp from
there.  If nothing else, it will slow down redisplay when line numbers
are used, whereas avoiding that slowdown was an explicit goal of
developing native line-number display.

Why aren't the existing faces enough?  And if they aren't enough, why
cannot you use line-number-mode?

Re: Colorful line numbers

[João Távora]

Eli Zaretskii <eliz@gnu.org> writes:

>> Could we implement one such mechanism?
> No.  The native line-number display is implemented inside the low
> level of the display code, and I don't want to call to Lisp from
> there.  If nothing else, it will slow down redisplay when line numbers
> are used, whereas avoiding that slowdown was an explicit goal of
> developing native line-number display.

Does checking for a given text property on some buffer text call into
Lisp?  If it isn't, and it's cheap, then this hypothetical feature could
have very low overhead when not used.

Even if you're opposed the idea, could you perhaps some pointers on how
to implement it, so that I try my hand at it and benchmark the
before/after?

> Why aren't the existing faces enough?  

I would like to propertize each line number's background with a
different color based on assembler information.

This would create a visually cue similar to the one used in Matt
Godbolt's "Compiler explorer" which works in the browser.  There, you
have two side-by-side windows of source code, you enter some C code on
the left and it starts compiling it immediately.  If compilation
succeeds, it immediately shows the assembly code on the right.  Finally,
it colorizes the lines with a unique and dynamically determined, color
based on the assembler code that each line produces when the file
compiled (according to gcc -g).  The reader can visually match these
colors in the left and right windeos to see, to some extent, how the
compiler proceeded.

I've recently discovered Jay Kamat's most excellent rmsbolt.el.  It is
is even better than godbolt.org in many aspects but is missing this last
feature, which I would like to add.  But instead of highlighting the
whole line, I thought it would be nicer and less distracting if I just
highlighted the line number's background.

> And if they aren't enough, why cannot you use line-number-mode?

I do use line-number-mode, but I don't understand how it can help me
here.  AFAIK it shows me one line number at a time in the modeline, not
besides the text.

João

Re: Colorful line numbers

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Fri, 22 Jul 2022 12:29:59 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Could we implement one such mechanism?
> > No.  The native line-number display is implemented inside the low
> > level of the display code, and I don't want to call to Lisp from
> > there.  If nothing else, it will slow down redisplay when line numbers
> > are used, whereas avoiding that slowdown was an explicit goal of
> > developing native line-number display.
> 
> Does checking for a given text property on some buffer text call into
> Lisp?  If it isn't, and it's cheap, then this hypothetical feature could
> have very low overhead when not used.

You want the line number to have the same face as the character
displayed immediately after it?  That can be done from C without
calling Lisp, but how would that work reliably?  Any change in the
buffer text will risk breaking the feature.

And even this is already a complication I'd like to avoid: line number
of a line is rendered before we examine the buffer text of the line
(and even know whether there is any text there).  Now we would need to
look forward in the buffer and get text properties of that text.
Which might not work correctly, btw, if jit-lock was not yet invoked
to produce the faces there.  Likewise if the line is hscrolled, so
that the first character displayed to the right of the line number is
not the first character of the physical line.

> Even if you're opposed the idea, could you perhaps some pointers on how
> to implement it, so that I try my hand at it and benchmark the
> before/after?

The implementation is in maybe_produce_line_number in xdisp.c.

> > Why aren't the existing faces enough?  
> 
> I would like to propertize each line number's background with a
> different color based on assembler information.
> 
> This would create a visually cue similar to the one used in Matt
> Godbolt's "Compiler explorer" which works in the browser.  There, you
> have two side-by-side windows of source code, you enter some C code on
> the left and it starts compiling it immediately.  If compilation
> succeeds, it immediately shows the assembly code on the right.  Finally,
> it colorizes the lines with a unique and dynamically determined, color
> based on the assembler code that each line produces when the file
> compiled (according to gcc -g).  The reader can visually match these
> colors in the left and right windeos to see, to some extent, how the
> compiler proceeded.

You can change the colors of the text without affecting the line
numbers.  Line numbers have a distinct face for a good reason.

> I've recently discovered Jay Kamat's most excellent rmsbolt.el.  It is
> is even better than godbolt.org in many aspects but is missing this last
> feature, which I would like to add.  But instead of highlighting the
> whole line, I thought it would be nicer and less distracting if I just
> highlighted the line number's background.

Why not put some indication on the fringe instead?

> > And if they aren't enough, why cannot you use line-number-mode?
> 
> I do use line-number-mode, but I don't understand how it can help me
> here.  AFAIK it shows me one line number at a time in the modeline, not
> besides the text.

Sorry, I meant linum-mode.

Re: Colorful line numbers

[João Távora]

Eli Zaretskii <eliz@gnu.org> writes:

>> Does checking for a given text property on some buffer text call into
>> Lisp?  If it isn't, and it's cheap, then this hypothetical feature could
>> have very low overhead when not used.
> You want the line number to have the same face as the character
> displayed immediately after it?

Not the "same face", rather some face which is the result of querying
text properties on that text. 

> That can be done from C without calling Lisp, but how would that work
> reliably?  Any change in the buffer text will risk breaking the
> feature.

It's up to the Lisp package that is interested in this functionality to
set or unset the correct text property. 

> And even this is already a complication I'd like to avoid: line number
> of a line is rendered before we examine the buffer text of the line
> (and even know whether there is any text there).  Now we would need to
> look forward in the buffer and get text properties of that text.
> Which might not work correctly, btw, if jit-lock was not yet invoked
> to produce the faces there.

See above. This wouldn't be a problem because I'm not interesting in
choosing the same face that jit-lock selected.  The indirection would be
some text property, say line-number-face

> Likewise if the line is hscrolled, so
> that the first character displayed to the right of the line number is
> not the first character of the physical line.

It would be up to the Lisp package to propertize the whole line with
'line-number-face'.

>> Even if you're opposed the idea, could you perhaps some pointers on how
>> to implement it, so that I try my hand at it and benchmark the
>> before/after?
>
> The implementation is in maybe_produce_line_number in xdisp.c.

Yes, I've seen it (this is would I discovered the closed set of faces).
But I'm not familiar with xdisp.c code and how to get to the buffer text
about to be displayed besides the line number (if it indeed exists) and
the associated text properties of that text.  The only argument passed
to maybe_produce_line_number is a 'struct it'.  How to go from there to
any valid buffer position of the same line that the 'struct it' refers
to?

> You can change the colors of the text without affecting the line
> numbers.  Line numbers have a distinct face for a good reason.

See above.  I'm not interested in having line number have exactly the
same face/color as any one of the characters in the buffer text.

>> I've recently discovered Jay Kamat's most excellent rmsbolt.el.  It is
>> is even better than godbolt.org in many aspects but is missing this last
>> feature, which I would like to add.  But instead of highlighting the
>> whole line, I thought it would be nicer and less distracting if I just
>> highlighted the line number's background.
>
> Why not put some indication on the fringe instead?

I'm experimenting with that, too.  Not the fringe, since it only works
on graphical displays, but the margin.

>> > And if they aren't enough, why cannot you use line-number-mode?
>> I do use line-number-mode, but I don't understand how it can help me
>> here.  AFAIK it shows me one line number at a time in the modeline, not
>> besides the text.
> Sorry, I meant linum-mode.

Ah.  I could indeed, but I would still like to experiment with the
display-line-numbers-mode.

João

Re: Colorful line numbers

[Dmitry Gutov]

On 22.07.2022 10:50, João Távora wrote:
> Was such a mechanism ever
> considered useful/practical/impractical?

Also see https://debbugs.gnu.org/36472.

Re: Colorful line numbers

[Stefan Monnier]

João Távora [2022-07-22 08:50:21] wrote:
> Could we implement one such mechanism?  Was such a mechanism ever
> considered useful/practical/impractical?

I suggest you first add it to `(n)linum-mode`, since it should be easy
(all ELisp).  That will provide a useful prototype.

Dmitry Gutov [2022-07-22 15:18:36] wrote:
> Also see https://debbugs.gnu.org/36472.

Have you tried to add such a feature for `(n)linum-mode`?


        Stefan

Re: Colorful line numbers

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Fri, 22 Jul 2022 13:02:01 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Does checking for a given text property on some buffer text call into
> >> Lisp?  If it isn't, and it's cheap, then this hypothetical feature could
> >> have very low overhead when not used.
> > You want the line number to have the same face as the character
> > displayed immediately after it?
> 
> Not the "same face", rather some face which is the result of querying
> text properties on that text. 

I don't understand what that means.  Querying the properties how?  And
doing what with the results of those queries?  Eventually, you need a
face.

> > That can be done from C without calling Lisp, but how would that work
> > reliably?  Any change in the buffer text will risk breaking the
> > feature.
> 
> It's up to the Lisp package that is interested in this functionality to
> set or unset the correct text property. 

Well, good luck with that!  Because I don't think this can be
reasonably implemented from Lisp: there are too many aspects involved
that are hidden from Lisp.

> > And even this is already a complication I'd like to avoid: line number
> > of a line is rendered before we examine the buffer text of the line
> > (and even know whether there is any text there).  Now we would need to
> > look forward in the buffer and get text properties of that text.
> > Which might not work correctly, btw, if jit-lock was not yet invoked
> > to produce the faces there.
> 
> See above. This wouldn't be a problem because I'm not interesting in
> choosing the same face that jit-lock selected.  The indirection would be
> some text property, say line-number-face

I guess this gets back to the question about "queries" and their
processing?

> > The implementation is in maybe_produce_line_number in xdisp.c.
> 
> Yes, I've seen it (this is would I discovered the closed set of faces).
> But I'm not familiar with xdisp.c code and how to get to the buffer text
> about to be displayed besides the line number (if it indeed exists) and
> the associated text properties of that text.  The only argument passed
> to maybe_produce_line_number is a 'struct it'.  How to go from there to
> any valid buffer position of the same line that the 'struct it' refers
> to?

The answer depends on the condition(s) which that "valid buffer
position" should satisfy.  In general, IT_CHARPOS(*it) gives you the
next buffer position to be processed.

Re: Colorful line numbers

[Dmitry Gutov]

On 22.07.2022 15:38, Stefan Monnier wrote:
>> Also seehttps://debbugs.gnu.org/36472.
> Have you tried to add such a feature for `(n)linum-mode`?

I didn't: that's not what the diff-hl user was asking for.

But I proposed several different technical approaches, every one of 
which has been rejected so far. And not because of the difficulty of 
implementation.

Not sure how implementing any of them in nlinum would help, given that 
the primary complaint was about running Lisp.

And if we take an alternative approach (display-line-numbers-mode looks 
up text properties in the buffer to determine which face to use), it 
also depends on the intricacies of its implementation. E.g. how to 
reliably redraw when such text properties change.

Re: Colorful line numbers

[João Távora]

Eli Zaretskii <eliz@gnu.org> writes:

>> >> Does checking for a given text property on some buffer text call into
>> >> Lisp?  If it isn't, and it's cheap, then this hypothetical feature could
>> >> have very low overhead when not used.
>> > You want the line number to have the same face as the character
>> > displayed immediately after it?
>> Not the "same face", rather some face which is the result of querying
>> text properties on that text. 
> I don't understand what that means.  Querying the properties how?

With the C equivalent of get-text-property.  textget() or
Ftext_properties_at(), or some callee of those.  The lightest possible
option.

> And doing what with the results of those queries? Eventually, you need a face.

If the query (or "lookup" if you prefer) produces something non-nil, the
value it returns is a LispObject which is a face.

>> > That can be done from C without calling Lisp, but how would that work
>> > reliably?  Any change in the buffer text will risk breaking the
>> > feature.
>> It's up to the Lisp package that is interested in this functionality to
>> set or unset the correct text property. 
> Well, good luck with that!  Because I don't think this can be
> reasonably implemented from Lisp: there are too many aspects involved
> that are hidden from Lisp.

I don't foresee any great difficulties in selecting a region of a buffer
and propertizing the text with some arbitrary property.

>> > And even this is already a complication I'd like to avoid: line number
>> > of a line is rendered before we examine the buffer text of the line
>> > (and even know whether there is any text there).  Now we would need to
>> > look forward in the buffer and get text properties of that text.
>> > Which might not work correctly, btw, if jit-lock was not yet invoked
>> > to produce the faces there.
>> 
>> See above. This wouldn't be a problem because I'm not interesting in
>> choosing the same face that jit-lock selected.  The indirection would be
>> some text property, say line-number-face
>
> I guess this gets back to the question about "queries" and their
> processing?

To try to clear up any doubts, I meant nothing fancy with the word
"query": by it I mean inspecting and retrieving the value of a text
property at a certain buffer position.

> The answer depends on the condition(s) which that "valid buffer
> position" should satisfy.

That buffer position should satisfy just one condition: it should exist
in the same line to which the line number about to be produced
(maybe...)  by 'maybe_produce_line_number()' refers to.

>In general, IT_CHARPOS(*it) gives you the next buffer position to be
>processed.

If I'm reading the code correctly, the particular 'it' argument to
maybe_produce_line_number _does not_ refer to a buffer position, rather
to a thing that may be displayed.  In that case, I have to advance (a
copy) of this iterator until I find such a buffer position. Otherwise
(if it _does_ refer to some buffer text) I think my question is closer
to being answered.

Jo~ao

Re: Colorful line numbers

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> And if we take an alternative approach (display-line-numbers-mode
> looks up text properties in the buffer to determine which face to
> use), it also depends on the intricacies of its
> implementation. E.g. how to reliably redraw when such text properties
> change.

Good point.  'redisplay'?

João

Re: Colorful line numbers

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Fri, 22 Jul 2022 14:53:12 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Not the "same face", rather some face which is the result of querying
> >> text properties on that text. 
> > I don't understand what that means.  Querying the properties how?
> 
> With the C equivalent of get-text-property.  textget() or
> Ftext_properties_at(), or some callee of those.  The lightest possible
> option.
> 
> > And doing what with the results of those queries? Eventually, you need a face.
> 
> If the query (or "lookup" if you prefer) produces something non-nil, the
> value it returns is a LispObject which is a face.

So the value of the text property will be a face?

If so, how will that property be set on the text, and what will
trigger whatever sets it?

> >> > That can be done from C without calling Lisp, but how would that work
> >> > reliably?  Any change in the buffer text will risk breaking the
> >> > feature.
> >> It's up to the Lisp package that is interested in this functionality to
> >> set or unset the correct text property. 
> > Well, good luck with that!  Because I don't think this can be
> > reasonably implemented from Lisp: there are too many aspects involved
> > that are hidden from Lisp.
> 
> I don't foresee any great difficulties in selecting a region of a buffer
> and propertizing the text with some arbitrary property.

You mean, the Lisp program will scan the entire buffer text and update
the properties upon each and every change to buffer text?  Like in
some post-command-hook or something?

> To try to clear up any doubts, I meant nothing fancy with the word
> "query": by it I mean inspecting and retrieving the value of a text
> property at a certain buffer position.

So each character of each physical line will have those text
properties set whose value is the face in which you want to show the
line number?  And some post-command-hook will update the values in the
entire buffer?

> > The answer depends on the condition(s) which that "valid buffer
> > position" should satisfy.
> 
> That buffer position should satisfy just one condition: it should exist
> in the same line to which the line number about to be produced
> (maybe...)  by 'maybe_produce_line_number()' refers to.

So, to know that no character on the line has this property, we'd need
to scan all of the line's characters?

> >In general, IT_CHARPOS(*it) gives you the next buffer position to be
> >processed.
> 
> If I'm reading the code correctly, the particular 'it' argument to
> maybe_produce_line_number _does not_ refer to a buffer position, rather
> to a thing that may be displayed.

'struct it' is an iterator through text.  It includes all the state
information needed for the iteration, and also the metrics of the last
"display element" (character glyph, image, stretch, etc.) that it
calculated.

> In that case, I have to advance (a copy) of this iterator until I
> find such a buffer position.

Which "such"?

And please note that copying an iterator is not just copying a
structure, there's a protocol for that (see the macros SAVE_IT and
RESTORE_IT).

> Otherwise (if it _does_ refer to some buffer text) I think my
> question is closer to being answered.

I don't think I understand what you mean by "refers to buffer text".

Re: Colorful line numbers

[João Távora]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: João Távora <joaotavora@gmail.com>
>> Cc: emacs-devel@gnu.org
>> Date: Fri, 22 Jul 2022 14:53:12 +0100
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> Not the "same face", rather some face which is the result of querying
>> >> text properties on that text. 
>> > I don't understand what that means.  Querying the properties how?
>> 
>> With the C equivalent of get-text-property.  textget() or
>> Ftext_properties_at(), or some callee of those.  The lightest possible
>> option.
>> 
>> > And doing what with the results of those queries? Eventually, you need a face.
>> 
>> If the query (or "lookup" if you prefer) produces something non-nil, the
>> value it returns is a LispObject which is a face.
>
> So the value of the text property will be a face?
>
> If so, how will that property be set on the text, and what will
> trigger whatever sets it?

A Lisp function that is called once in a while, either manually or via a
idle timer.

>> >> > That can be done from C without calling Lisp, but how would that work
>> >> > reliably?  Any change in the buffer text will risk breaking the
>> >> > feature.
>> >> It's up to the Lisp package that is interested in this functionality to
>> >> set or unset the correct text property. 
>> > Well, good luck with that!  Because I don't think this can be
>> > reasonably implemented from Lisp: there are too many aspects involved
>> > that are hidden from Lisp.
>> 
>> I don't foresee any great difficulties in selecting a region of a buffer
>> and propertizing the text with some arbitrary property.
>
> You mean, the Lisp program will scan the entire buffer text and update
> the properties upon each and every change to buffer text?  Like in
> some post-command-hook or something?

In some hook yes, but not "post-command-hook" and not on each and every
change.  And I don't think the "entire buffer text" has to be scanned.

>> To try to clear up any doubts, I meant nothing fancy with the word
>> "query": by it I mean inspecting and retrieving the value of a text
>> property at a certain buffer position.
>
> So each character of each physical line will have those text
> properties set whose value is the face in which you want to show the
> line number?  And some post-command-hook will update the values in the
> entire buffer?

See above.  Not "each character" and not the entire buffer and not in the
hook.  At least for _my_ application.  Some other application might want
to do that, but not the one I was thinking to build with this feature.

Also my application typically works with small files.  And it already
suffers from significant lag from calling an external process.

>> > The answer depends on the condition(s) which that "valid buffer
>> > position" should satisfy.
>> 
>> That buffer position should satisfy just one condition: it should exist
>> in the same line to which the line number about to be produced
>> (maybe...)  by 'maybe_produce_line_number()' refers to.
>
> So, to know that no character on the line has this property, we'd need
> to scan all of the line's characters?

To know that piece of information, yes.  But only if that information is
useful.  It would be essential if the hypothetical feature advertised
itself like "put a property on any character and I will take care of the
rest".

Instead, if it advertises itself as: "put this text property on only
this one (first/last) character of the line..." or even "put this text
property on _every character_ of the line...", then no such scan need
exist, I think.

>> In that case, I have to advance (a copy) of this iterator until I
>> find such a buffer position.
> Which "such"?

I employed the word "such" because I was not sure that the iterator
referred to (or "iterated through") buffer positions.  I used it to
express conjecture/doubt.

> And please note that copying an iterator is not just copying a
> structure, there's a protocol for that (see the macros SAVE_IT and
> RESTORE_IT).

OK.

>> Otherwise (if it _does_ refer to some buffer text) I think my
>> question is closer to being answered.
> I don't think I understand what you mean by "refers to buffer text".

I don't know how else to express it.  But I perhaps your phrase "an
iterator through text" answered by question.  Given one iterator such as
the the one passed to maybe_produce_line_number, it seems I can get some
value that I can eventually use to call Ftext_properties_at.

João

Re: Colorful line numbers

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Fri, 22 Jul 2022 16:10:54 +0100
> 
> > You mean, the Lisp program will scan the entire buffer text and update
> > the properties upon each and every change to buffer text?  Like in
> > some post-command-hook or something?
> 
> In some hook yes, but not "post-command-hook" and not on each and every
> change.  And I don't think the "entire buffer text" has to be scanned.

How else would you know where such re-scanning is needed?  An Emacs
command could change buffer text in several places and in different
ways, and you must ensure all the characters of a line have the
correct properties.

rmsbolt.el [Was: Colorful line numbers]

[João Távora]

[-- Attachment #1.1: Type: text/plain, Size: 1366 bytes --]

On Fri, Jul 22, 2022 at 4:38 PM Eli Zaretskii <eliz@gnu.org> wrote:

> How else would you know where such re-scanning is needed?  An Emacs
> command could change buffer text in several places and in different
> ways, and you must ensure all the characters of a line have the
> correct properties.
>

Depends on the application.  Each application just sets properties where
and when it sees fit.  It can keep track of where it set them using
markers,
for example.   Also many commands such as movement don't change text.

Aaanyway, I think I'm happy with the current implementation that _doesn't_
use line numbers at all. I think this discussion has been productive (to
some
degree).  If I ever need this again (or someone else, like Dmitry), I'll be
sure
to ping here again.

For now, my itch has been scratched with a less complicated
back-scratcher. And what a scratch! I attach an animated gif, which
incidentally shows the how the rmsbolt.el package I was describing
earlier works.

Changing the subject completely, I wonder if there's any interest in
adding rmsbolt.el to GNU Elpa or even to core. It's not just for C/C++,
it works for a bunch of languages.

https://gitlab.com/jgkamat/rmsbolt

Apologies if it's old news and everyone's been using this, but
it's the best package I've seen appear in some time.

João Távora

[-- Attachment #1.2: Type: text/html, Size: 2073 bytes --]

[-- Attachment #2: rmsbolt-is-great.gif --]
[-- Type: image/gif, Size: 200311 bytes --]

Re: rmsbolt.el [Was: Colorful line numbers]

[Stefan Monnier]

> Changing the subject completely, I wonder if there's any interest in
> adding rmsbolt.el to GNU Elpa or even to core. It's not just for C/C++,
> it works for a bunch of languages.

I'd be happy to see it in (Non)GNU ELPA (I don't see any reason to add
it to core, OTOH).


        Stefan

Re: Colorful line numbers

[Stefan Monnier]

>>> Also seehttps://debbugs.gnu.org/36472.
>> Have you tried to add such a feature for `(n)linum-mode`?
> I didn't: that's not what the diff-hl user was asking for.

But you could add that feature to `diff-hl` for `(n)linum-mode` so
people can at least try it out.

> But I proposed several different technical approaches, every one of which
> has been rejected so far. And not because of the difficulty
> of implementation.
>
> Not sure how implementing any of them in nlinum would help, given that the
> primary complaint was about running Lisp.

I suspect the problem is not "running Lisp" but "running Lisp from the
code that implements `display-line-numbers-mode`", because that code is
deep in the redisplay where running Lisp code can be tricky.

That same problem does not apply to `(n)linum-mode` because their code
is not "deep in redisplay".

> And if we take an alternative approach (display-line-numbers-mode looks up
> text properties in the buffer to determine which face to use), it also
> depends on the intricacies of its implementation. E.g. how to reliably
> redraw when such text properties change.

My hunch is that such changes to text-properties would cause
a rerendering of those lines anyway (the decision whether to re-render
doesn't pay attention to which properties have been changed, IIRC), so
it may end up "just working" without any effort in this respect.
Then again, maybe not.


        Stefan

Re: rmsbolt.el [Was: Colorful line numbers]

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Fri, 22 Jul 2022 17:30:46 +0100
> Cc: emacs-devel <emacs-devel@gnu.org>
> 
> Aaanyway, I think I'm happy with the current implementation that _doesn't_ 
> use line numbers at all. I think this discussion has been productive (to some
> degree).  If I ever need this again (or someone else, like Dmitry), I'll be sure
> to ping here again. 

OK.  Let me just say, for the future, that basing this on text
properties is IMO not the best idea.  I think a better idea is to have
some buffer-local (or window-local) variable whose value is an
appropriate data structure that specifies faces for ranges of line
numbers.  Of course that value will need to be updated at suitable
times, which in itself is not a trivial task.

Re: rmsbolt.el [Was: Colorful line numbers]

[João Távora]

Eli Zaretskii <eliz@gnu.org> writes:

>> Aaanyway, I think I'm happy with the current implementation that _doesn't_ 
>> use line numbers at all. I think this discussion has been productive (to some
>> degree).  If I ever need this again (or someone else, like Dmitry), I'll be sure
>> to ping here again. 
>
> OK.  Let me just say, for the future, that basing this on text
> properties is IMO not the best idea.  I think a better idea is to have
> some buffer-local (or window-local) variable whose value is an
> appropriate data structure that specifies faces for ranges of line
> numbers.  Of course that value will need to be updated at suitable
> times, which in itself is not a trivial task.

OK.  Since we're in speculation territory, I don't see a big difference
between that and using text properties, which are also a buffer-local
structure.  Looking up the line we're iterating in that hypothetical
Lisp structure would seem to amount to about the same work as looking
for a particulr text property in a known place of that same line.  But
without the complexity of a new data structure.

But I'd have to see the proposed API and benchmark the two solutions to
make sure.

João

Re: rmsbolt.el [Was: Colorful line numbers]

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Sat, 23 Jul 2022 10:35:07 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Aaanyway, I think I'm happy with the current implementation that _doesn't_ 
> >> use line numbers at all. I think this discussion has been productive (to some
> >> degree).  If I ever need this again (or someone else, like Dmitry), I'll be sure
> >> to ping here again. 
> >
> > OK.  Let me just say, for the future, that basing this on text
> > properties is IMO not the best idea.  I think a better idea is to have
> > some buffer-local (or window-local) variable whose value is an
> > appropriate data structure that specifies faces for ranges of line
> > numbers.  Of course that value will need to be updated at suitable
> > times, which in itself is not a trivial task.
> 
> OK.  Since we're in speculation territory, I don't see a big difference
> between that and using text properties, which are also a buffer-local
> structure.

The big difference from my POV is that text properties slow down
redisplay code in general, whereas buffer-local variables only slow
down (somewhat) the code which examines their values and uses them.

> Looking up the line we're iterating in that hypothetical
> Lisp structure would seem to amount to about the same work as looking
> for a particulr text property in a known place of that same line.

No, referencing a variable is much faster, because it doesn't involve
finding the correct interval for a buffer position.

Re: rmsbolt.el [Was: Colorful line numbers]

[Stefan Monnier]

> The big difference from my POV is that text properties slow down
> redisplay code

If that's indeed a problem, then maybe we should provide "non-display
text-properties".  I.e. text-properties which do not influence the
display, so we can store them in a parallel interval tree (tho
we'd have to check whether the source of the slowdown is the interval
tree itself or the length of the `plist`s stored in its nodes, so maybe
we could have a single tree but with two `plist` fields, one for
display-related properties and the other for non-display-related
properties).


        Stefan

Re: rmsbolt.el [Was: Colorful line numbers]

[Jay Kamat]

João Távora writes:

> Changing the subject completely, I wonder if there's any interest in
> adding rmsbolt.el to GNU Elpa or even to core. It's not just for C/C++,
> it works for a bunch of languages.

I am personally open to putting the package on ELPA (or nongnu ELPA). My main 
concern is that I would prefer to keep upstream where it is to avoid moving 
things around - and ideally I would like to avoid doing extra work for new 
versions (melpa automatically builds new versions every time I push/tag a 
release). I haven't followed elpa very closely but my impression is that you 
might be able to set a remote upstream but you would need to manually keep 
re-syncing to keep things up to date in elpa.

Somewhat related, I think a better candidate for ELPA right now is the 
org-notify package, which was dropped from org-contrib a while ago:

- 
  https://git.sr.ht/~bzg/org-contrib/commit/c6aef31ccfc7c4418c3b51e98f7c3bd8e255f5e6
- https://github.com/melpa/melpa/pull/8116

I'm not the maintainer of this package, but I would like to see this package 
published on ELPA, nongnu ELPA, or Melpa. Depending on what the maintainer 
prefers, I would be happy to put in any work required to get the package back 
on ELPA (either in org-contrib or on its own).

> Apologies if it's old news and everyone's been using this, but
> it's the best package I've seen appear in some time.

Thanks for the kind words :). I wish I had more time to work on it but I've 
been quite busy in recent years unfortunately.

-Jay

Re: rmsbolt.el [Was: Colorful line numbers]

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: João Távora <joaotavora@gmail.com>,
>   emacs-devel@gnu.org
> Date: Sat, 23 Jul 2022 10:43:21 -0400
> 
> > The big difference from my POV is that text properties slow down
> > redisplay code
> 
> If that's indeed a problem, then maybe we should provide "non-display
> text-properties".  I.e. text-properties which do not influence the
> display, so we can store them in a parallel interval tree

That's quite a complication.  For starters, how to make sure the same
property doesn't get stored in both trees?  Next, redisplay aside,
most if not all properties that are important for redisplay are also
important to Lisp, and so Lisp programs will now have to look at two
interval trees.

> (tho
> we'd have to check whether the source of the slowdown is the interval
> tree itself or the length of the `plist`s stored in its nodes, so maybe
> we could have a single tree but with two `plist` fields, one for
> display-related properties and the other for non-display-related
> properties).

The immediate source of slowdown is that the display iterator stops at
every position where the properties change, and runs all of its
"handle stop-point" handlers, each one of which examines the
properties it handles (fontified, face, invisible, and display).

For what we do with the tree and the properties we find, see
compute_stop.

Re: rmsbolt.el [Was: Colorful line numbers]

[Stefan Monnier]

>> > The big difference from my POV is that text properties slow down
>> > redisplay code
>> If that's indeed a problem, then maybe we should provide "non-display
>> text-properties".  I.e. text-properties which do not influence the
>> display, so we can store them in a parallel interval tree
> That's quite a complication.  For starters, how to make sure the same
> property doesn't get stored in both trees?  Next, redisplay aside,
> most if not all properties that are important for redisplay are also
> important to Lisp, and so Lisp programs will now have to look at two
> interval trees.

If we need/want to solve those problems, I don't see any
technical difficulty.  Until we have found it necessary, the question is moot.

>> (tho we'd have to check whether the source of the slowdown is the
>> interval tree itself or the length of the `plist`s stored in its
>> nodes, so maybe we could have a single tree but with two `plist`
>> fields, one for display-related properties and the other for
>> non-display-related properties).
>
> The immediate source of slowdown is that the display iterator stops at
> every position where the properties change, and runs all of its
> "handle stop-point" handlers, each one of which examines the
> properties it handles (fontified, face, invisible, and display).
>
> For what we do with the tree and the properties we find, see
> compute_stop.

So adding a text property for the line numbers's face would (in
the worst case) add 2 "handle stop-points" per line (could be reduced
to one by making those properties cover the whole "line plus newline"
so there's only one stop-point between them).

Could that really cause a measurable slow down?


        Stefan

Re: Colorful line numbers

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: joaotavora@gmail.com,  emacs-devel@gnu.org
> Date: Sat, 23 Jul 2022 12:31:52 -0400
> 
> So adding a text property for the line numbers's face would (in
> the worst case) add 2 "handle stop-points" per line (could be reduced
> to one by making those properties cover the whole "line plus newline"
> so there's only one stop-point between them).
> 
> Could that really cause a measurable slow down?

Depends on what else is in the buffer.  If "one more stop-point per
line" adds significantly to the number of stop-point (which I think it
will in many cases, as code typically has few different faces per
line), it will be measurable, yes.

And using text properties for this kind of information is really not
clean.  It also comes with technical problems, because text properties
cannot overlap.  As I wrote earlier, I'd much prefer some buffer-local
variable.

Re: rmsbolt.el [Was: Colorful line numbers]

[Stefan Monnier]

> I am personally open to putting the package on ELPA (or nongnu ELPA).

I just looked at the code's apparent ownership, and it looks like there
are a few contributors which would need to sign paperwork before we can
add it to GNU ELPA.

Would you willing to help get that paperwork done, or do you prefer we
put it into NonGNU ELPA?

> My main concern is that I would prefer to keep upstream where it is to
> avoid moving things around - and ideally I would like to avoid doing
> extra work for new versions (melpa automatically builds new versions
> every time I push/tag a release). I haven't followed elpa very closely
> but my impression is that you might be able to set a remote upstream
> but you would need to manually keep re-syncing to keep things up to
> date in elpa.

Yes, nowadays most GNU ELPA packages are maintained upstream.
And there's no need to push to elpa.git, instead we have a cron job that
pulls into it (which also means the upstream should *not* force push
even if the last push was "just a few minutes ago so surely noone will
notice") and a new release is made whenever the `Version:` header
is bumped.


        Stefan

Eglot to core [Was: rmsbolt.el [Was: Colorful line numbers]]

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 890 bytes --]

> (which also means the upstream should *not* force push
> even if the last push was "just a few minutes ago so surely noone will
> notice") and a new release is made whenever the `Version:` header

Dang, I shamefully confess I did exactly that some weeks ago to Eglot
(with seconds instead of minutes).  But last I checked Eglot was still in
the push-to-ELPA model, right?  Sorry if I caused any trouble,

Another reason to move Eglot to core I guess :-)

Speaking of that, I'd like to get started with this sometime soon (maybe
August,  around the corner).  I wonder if you know any Git technique that
allows me to keep the history, at least of eglot.el so that is merged with
Emacs's own history.  There was some `git subtree` command  to do this
right?

The other significant piece of work is converting the Eglot README.md
into a half-decent texinfo manual.

João

[-- Attachment #2: Type: text/html, Size: 1171 bytes --]

Re: Eglot to core [Was: rmsbolt.el [Was: Colorful line numbers]]

[Stefan Monnier]

>> (which also means the upstream should *not* force push
>> even if the last push was "just a few minutes ago so surely noone will
>> notice") and a new release is made whenever the `Version:` header
>
> Dang, I shamefully confess I did exactly that some weeks ago to Eglot
> (with seconds instead of minutes).  But last I checked Eglot was still in
> the push-to-ELPA model, right?

Nope.

> Sorry if I caused any trouble,

    % make fetch/eglot
    emacs --no-site-lisp --batch -l admin/elpa-admin.el -f elpaa-batch-fetch-and-show "eglot"
    Fetching updates for eglot...
    Upstream of eglot has DIVERGED!
    
      Local changes:
    c558fd6a24  43703153+ssnno..  Fix #965: Update link for Fortran language server fortls
    e0c08e7f68  47760695+jgart..  Close #961: Add support for jedi-language-server
    
      Upstream changes:
    29690e88e3  joaotavora@gma..  Always default eglot-strict-mode to nil
    c962f6e5f6  joaotavora@gma..  Rework table of contents in README.md again
    a62a388021  joaotavora@gma..  Fix README.md typos and rework section about Workspace configuration
    eed9a65515  joaotavora@gma..  Fix embarrassing paren-matching blunder in eglot.el
    a2d9e18945  joaotavora@gma..  Reply more reasonably to server's workspace/applyEdit
    87e6de3cdf  joaotavora@gma..  Appease byte-compiler warnings about wrong use of quotes
    1db95974a7  joaotavora@gma..  Per #967: eglot-workspace-configuration can be a function
    33c464f658  joaotavora@gma..  Per #131, #314: Be more conservative with the LSP identifier guess
    f62b641b5c  joaotavora@gma..  Per #131, #314: Guess the "LSP identifier at point"
    25f6338741  joaotavora@gma..  Per #131: Tweak some details, fix some bugs
    c64fe76e86  joaotavora@gma..  Per #131: Cosmetic decisions guaranteed to tick off someone somewhere (tm)
    ae7261c1fe  joaotavora@gma..  Per #131: Experiment with grouping in xref-backend-identifier-completion-table
    2f71de72e3  joaotavora@gma..  Fix #131: Make C-u M-. work half decently
    
    %

So, yes, `eglot` in GNU ELPA is now stuck and you'll have to merge
`c558fd6a24` back into your upstream repo before it will start tracking
it again :-(

> Speaking of that, I'd like to get started with this sometime soon (maybe
> August,  around the corner).  I wonder if you know any Git technique that
> allows me to keep the history, at least of eglot.el so that is merged with
> Emacs's own history.  There was some `git subtree` command  to do this
> right?

As a one-time thing it's pretty easy.  You don't really need `git
subtree` for that.  Just do something like:

    git merge --no-commit --allow-unrelated-histories .../eglot
    git mv eglot.el lisp/
    ...
    git commit

Keeping it up if you keep hacking on the upstream is IMO an unsolved
problem (there are various hacks you can use, but they all suck one way
or another).


        Stefan

Re: rmsbolt.el [Was: Colorful line numbers]

[João Távora]

Eli Zaretskii <eliz@gnu.org> writes:

>> Looking up the line we're iterating in that hypothetical
>> Lisp structure would seem to amount to about the same work as looking
>> for a particulr text property in a known place of that same line.
>
> No, referencing a variable is much faster, because it doesn't involve
> finding the correct interval for a buffer position.

I don't know details of this data structure you are proposing but if
it's somehow a set of mappings between line ranges and faces (as you
described it), then you have to somehow traverse it to find, given a
particular line number, the correct mapping and the face to use in
maybe_produce_line_number().  Just as one would have to lookup a text
given for a particular buffer position related to that line.  But I'm
just speculating: I don't know what algorithm you have in mind.

I've understood that having text properties by itself can negatively
impact editing even if they don't contribute anything to the display.

And maybe updating your data structure could be cheaper, since adding
and removing things that aren't newlines doesn't require any update.
Then again, updating a Lisp structure is probably slower than updating a
optimized internal structure (a tree?)  for text properties.

So go ahead and do this structure you propose, if you have time and see
value in it.  Then we'll have something to compare against.

Thanks,
João

Re: rmsbolt.el [Was: Colorful line numbers]

[João Távora]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> So adding a text property for the line numbers's face would (in
> the worst case)

AFAIU the "very worst case": where some application user wants the line
number face to change for every line.  That's not at all the application
I had in mind, and at any rate seems like a "pay for what you use"
thing.  This seems perfectly reasonable to me, just as reasonable as
supporting overlays: if you don't want to suffer overlay-related
slowdown, don't use packages that use them.  They're pretty nifty
rhough.

João

Re: Colorful line numbers

[Dmitry Gutov]

On 23.07.2022 02:32, Stefan Monnier wrote:
>>>> Also seehttps://debbugs.gnu.org/36472.
>>> Have you tried to add such a feature for `(n)linum-mode`?
>> I didn't: that's not what the diff-hl user was asking for.
> 
> But you could add that feature to `diff-hl` for `(n)linum-mode` so
> people can at least try it out.

I don't really use either myself. I hear display-line-numbers-mode has 
improved on some essential performance problems, but I have no practical 
knowledge to substantiate that.

Either way, the person was asking for integration with the latter 
package. So I fear my effort would end up unused even if I took it up.

>> And if we take an alternative approach (display-line-numbers-mode looks up
>> text properties in the buffer to determine which face to use), it also
>> depends on the intricacies of its implementation. E.g. how to reliably
>> redraw when such text properties change.
> 
> My hunch is that such changes to text-properties would cause
> a rerendering of those lines anyway (the decision whether to re-render
> doesn't pay attention to which properties have been changed, IIRC), so
> it may end up "just working" without any effort in this respect.
> Then again, maybe not.

That would be my basic expectation as well, but I dimly recall that 
d-l-n-mode performs some additional optimizations in order not to 
recalculate line numbers.

Re: Eglot to core [Was: rmsbolt.el [Was: Colorful line numbers]]

[João Távora]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> So, yes, `eglot` in GNU ELPA is now stuck and you'll have to merge
> `c558fd6a24` back into your upstream repo before it will start tracking
> it again :-(

OK done.  Let me know when it's running again.

>> Speaking of that, I'd like to get started with this sometime soon (maybe
>> August,  around the corner).  I wonder if you know any Git technique that
>> allows me to keep the history, at least of eglot.el so that is merged with
>> Emacs's own history.  There was some `git subtree` command  to do this
>> right?
>
> As a one-time thing it's pretty easy.  You don't really need `git
> subtree` for that.  Just do something like:
>
>     git merge --no-commit --allow-unrelated-histories .../eglot
>     git mv eglot.el lisp/
>     ...
>     git commit

OK this indeed sounds simple.  lisp/eglot.el sounds fine  I suppose the
tests would go to test/lisp/eglot-tests.el

What about the manual rewritten from the README?  A new section in the
Emacs user manual or a separate manual like Flymake's (the latter has
the unfortunate flaw that it isn't indexed by C-h S).

What about the Eglot NEWS file?  Should I trash it or is there some
place to put it/or integrate it?

> Keeping it up if you keep hacking on the upstream is IMO an unsolved
> problem (there are various hacks you can use, but they all suck one way
> or another).

I plan to hacking in Emacs core, i.e. make commits to emacs.git.  Not in
Eglot's github upstream.  However, I _would_ like the upstream to track
eglot.el from emacs.git (only that file will do), since I suspect some
people still get it from there.  Also often people submit pull requests
to that repo, and if the eglot.el file is up-to-date all I have to do is
extract a patch and commit to emacs.git.

I also wanted to give some grace time to users who file bug reports in
the upstream repo for some time, but that's an easier administrative
issue.

What do you think of this plan, is it feasible?

João

Re: Eglot to core [Was: rmsbolt.el [Was: Colorful line numbers]]

[Stefan Monnier]

>> So, yes, `eglot` in GNU ELPA is now stuck and you'll have to merge
>> `c558fd6a24` back into your upstream repo before it will start tracking
>> it again :-(
> OK done.  Let me know when it's running again.

Thanks, it's fine now.

I'll let others chime in for the details of where to put what.

[ Personally, I think the better option is to leave it in a separate
  repo/branch and integrate it via something like Git submodules.  ]

> What do you think of this plan, is it feasible?

If I were doing it I wouldn't like it (because of having to keep two
sides in sync), but it's probably feasible, yes.


        Stefan

Re: Eglot to core [Was: rmsbolt.el [Was: Colorful line numbers]]

[Po Lu]

João Távora <joaotavora@gmail.com> writes:

> What about the Eglot NEWS file?  Should I trash it or is there some
> place to put it/or integrate it?

How about etc/EGLOT-NEWS?

> I plan to hacking in Emacs core, i.e. make commits to emacs.git.  Not in
> Eglot's github upstream.  However, I _would_ like the upstream to track
> eglot.el from emacs.git (only that file will do), since I suspect some
> people still get it from there.  Also often people submit pull requests
> to that repo, and if the eglot.el file is up-to-date all I have to do is
> extract a patch and commit to emacs.git.
>
> I also wanted to give some grace time to users who file bug reports in
> the upstream repo for some time, but that's an easier administrative
> issue.
>
> What do you think of this plan, is it feasible?
>
> João

I look forward to Eglot being included with Emacs.  But (and I've said
this before) "eglot" is a very bad name that says nothing about its
purpose.  How about renaming it "language-server"?

Thanks.

Re: Eglot to core [Was: rmsbolt.el [Was: Colorful line numbers]]

[Stefan Monnier]

> I look forward to Eglot being included with Emacs.  But (and I've said
> this before) "eglot" is a very bad name that says nothing about its
> purpose.  How about renaming it "language-server"?

I don't like idea conflating the names with the descriptions, because it
inevitably leads to conflicts.  So I see no need to rename it.
But we can definitely add various ways to make it easier to find, such
as adding a `language-server-mode` alias.


        Stefan

Re: Eglot to core [Was: rmsbolt.el [Was: Colorful line numbers]]

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I look forward to Eglot being included with Emacs.  But (and I've said
>> this before) "eglot" is a very bad name that says nothing about its
>> purpose.  How about renaming it "language-server"?
>
> I don't like idea conflating the names with the descriptions, because it
> inevitably leads to conflicts.  

What kind of conflicts?

>                                 So I see no need to rename it.
> But we can definitely add various ways to make it easier to find, such
> as adding a `language-server-mode` alias.

If this is done, it would also be helpful to add alii for commands like
`eglot-code-actions', `eglot-rename', `eglot-format', etc. to avoid
confusion.

Re: Eglot to core [Was: rmsbolt.el [Was: Colorful line numbers]]

[Po Lu]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I don't like idea conflating the names with the descriptions, because it
> inevitably leads to conflicts.

What conflicts?

> So I see no need to rename it.  But we can definitely add various ways
> to make it easier to find, such as adding a `language-server-mode`
> alias.

I think a major improvement would be for searching "Language Server" in
Custom to find variables defined by eglot.

Re: Eglot to core [Was: rmsbolt.el [Was: Colorful line numbers]]

[Bozhidar Batsov]

[-- Attachment #1: Type: text/plain, Size: 823 bytes --]

Also keep in mind there's the super popular lsp-mode package (https://github.com/emacs-lsp/lsp-mode), so most searches about LSP probably end up there. If some renaming/aliasing is done for Eglot it should still be somewhat different from the other package to avoid confusion. Naming is hard... :-)

On Mon, Jul 25, 2022, at 9:23 AM, Po Lu wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> 
> > I don't like idea conflating the names with the descriptions, because it
> > inevitably leads to conflicts.
> 
> What conflicts?
> 
> > So I see no need to rename it.  But we can definitely add various ways
> > to make it easier to find, such as adding a `language-server-mode`
> > alias.
> 
> I think a major improvement would be for searching "Language Server" in
> Custom to find variables defined by eglot.
> 
> 

[-- Attachment #2: Type: text/html, Size: 1374 bytes --]

Re: Eglot to core [Was: rmsbolt.el [Was: Colorful line numbers]]

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1047 bytes --]

On Mon, Jul 25, 2022 at 7:23 AM Po Lu <luangruo@yahoo.com> wrote:

> > So I see no need to rename it.  But we can definitely add various ways
> > to make it easier to find, such as adding a `language-server-mode`
> > alias.
> I think a major improvement would be for searching "Language Server" in
> Custom to find variables defined by eglot.

Makes sense, but keep in mind that in contrast to some other "super
popular"  LSP
packages Eglot provides very few of these.  Only 10 currently.  I intend to
keep this
set minimal, maybe even remove some (but not right away).

One of the points of bringing Eglot into core is that it can be used as
a library by other major modes, who are free to build on top of Eglot's
API.  Thus
language server support becomes somewhat transparent (according to Stefan's
recent definition): users don't need to know -- to some reasonable extent
-- that Eglot
as an LSP client is being used, just as they needn't know that tree-sitter
is being
used to provide, say, a better C++ mode.

João

[-- Attachment #2: Type: text/html, Size: 1241 bytes --]

Re: Eglot to core [Was: rmsbolt.el [Was: Colorful line numbers]]

[Felician Nemeth]

João Távora <joaotavora@gmail.com> writes:

> One of the points of bringing Eglot into core is that it can be used as
> a library by other major modes, who are free to build on top of Eglot's
> API. 

I don't think the current API is good enough for that.  See
https://github.com/joaotavora/eglot/discussions/802#discussioncomment-2171239

More importantly, there are multiple language server implementations for
some languages (python, c) with different quirks or non-standard
extensions to the LSP protocol.  How do you imagine a major-mode would
support these?

Authors of a major-mode even rejected to configure eglot-server-programs
in their mode saying it didn't belong to there.  (Unfortunately, I
cannot find the github discussion for this.)

Re: Eglot to core [Was: rmsbolt.el [Was: Colorful line numbers]]

[João Távora]

Felician Nemeth <felician.nemeth@gmail.com> writes:

> I don't think the current API is good enough for that.  See
> https://github.com/joaotavora/eglot/discussions/802#discussioncomment-2171239

Really depends on what you consider "that" to be.  I know you have an
eglot-x.el package and I haven't had time to look at it.  If you propose
to expand the API of eglot.el with well-grounded use cases, I'm happy to
take it.  Also, once Eglot is in core, I won't be only person deciding.

> More importantly, there are multiple language server implementations for
> some languages (python, c) with different quirks or non-standard
> extensions to the LSP protocol.  How do you imagine a major-mode would
> support these?

They can adds methods to the generic function eglot-handle-request (and
a handful others), set values of Eglot in their major-mode
initialization code, bind keys to `eglot-*` interactive commands, make
compound commands from those commands, etc.

While they _can_, but that doesn't mean they _must_ or even _should_.
In fact, LSP's whole idea, which is surprisingly forgotten so often, is
that server and editor need to know very little about each other to be
able to cooperate.  By and large this idea works very very well.  Eglot
works with tens of servers and hasn't got a line of server specific
code, except for eglot-server-programs invocations.  There are
exceptions, and not everyone has the same luck, but I've been using C++
clangd and C# omnisharp quite a lot lately and I have 0 lines of server
specific config.

> Authors of a major-mode even rejected to configure eglot-server-programs
> in their mode saying it didn't belong to there.  (Unfortunately, I
> cannot find the github discussion for this.)

I can't either, but I do remember it.  One (1) author of a
javascript-ish major mode rejected this, and I'm not sure this person
was seeing the complete picture.  Anyway, I can't foresee the future,
but I don't see many major mode authors objecting to setting font-lock-*
or imenu-* or many other things in their major modes.  Major modes
usually use transversal facilities in Emacs, and I wish for Eglot to
gradually become one such facility.

If you don't work like this you'll have "ghost" major-modes like
`eglot-java.el`, `eglot-python.el`, with their own maintainers, their
own philosophies, and eventually you're back to the beginning
duplicating and rewriting all the code that LSP wants you to avoid
doing.

But it's true: I have seen one author here offer resistance to setting a
single flymake- variable in the major mode, even though Flymake was in
the core.  I think it boils down to the fact that Flymake wasn't even
known or popular with this person or, admittedly, even spectacularly
useful out-of-the-box, at least not when compared to what it does when
Flymake/Eglot/LSP combo.

Nonetheless, I'd say eglot-server-programs and the current status quo
isn't going anywhere for now.  

So there isn't any problem/drama here.  I definitely don't want to be
breaking things in eglot.el.  I was just stating the general direction.

João

Re: Eglot to core [Was: rmsbolt.el [Was: Colorful line numbers]]

[João Távora]

João Távora <joaotavora@gmail.com> writes:

> They can adds methods to the generic function eglot-handle-request (and
> a handful others), set values of Eglot in their major-mode

Doh, this should read "set values of Eglot variables in their major mode"

> initialization code, bind keys to `eglot-*` interactive commands, make
> compound commands from those commands, etc.

Joãoy

Re: Eglot to core [Was: rmsbolt.el [Was: Colorful line numbers]]

[Felician Nemeth]

>> I don't think the current API is good enough for that.  See
>> https://github.com/joaotavora/eglot/discussions/802#discussioncomment-2171239
>
> Really depends on what you consider "that" to be.

To implement non-standard LSP features, which lots of servers propose.

> [...] Also, once Eglot is in core, I won't be only person deciding.

(I welcome Eglot's inclusion in the core and appreciate your work.  I
hope that's clear.)

>> More importantly, there are multiple language server implementations for
>> some languages (python, c) with different quirks or non-standard
>> extensions to the LSP protocol.  How do you imagine a major-mode would
>> support these?
>
> They can adds methods to the generic function eglot-handle-request (and
> a handful others), set values of Eglot in their major-mode
> initialization code, bind keys to `eglot-*` interactive commands, make
> compound commands from those commands, etc.

OK.  This is somewhat similar to the case when a major mode provides an
xref or a flymake backend.

But I think sometimes the reverse approach is more desirable.  Let's
take a not so hypothetical example and say c++-mode wants to provide a
type-hierarchy browser.  It's better to write a generic, language
agnostic hierarchy browser (maybe CEDET even has one), and extend Eglot
with a hierarchy browser backend similarly to its xref backend.  The
hierarchy browser then can have a tree-sitter backend as well.

> While they _can_, but that doesn't mean they _must_ or even _should_.
> In fact, LSP's whole idea, which is surprisingly forgotten so often, is
> that server and editor need to know very little about each other to be
> able to cooperate.  By and large this idea works very very well. 

I agree, but non-standard extensions exist for a reason.

Almost all servers have a VS-Code specific part that contains at least a
list of configuration variables that other clients cannot handle
automatically.  BTW, if Eglot knew about these configuration variables,
could `customize-group` somehow save the settings into a .dir-locals.el
file?

> Eglot works with tens of servers and hasn't got a line of server
> specific code, except for eglot-server-programs invocations.  There
> are exceptions, and not everyone has the same luck, but I've been
> using C++ clangd and C# omnisharp quite a lot lately and I have 0
> lines of server specific config.

Even clangd has a moderately long list of extensions:
https://clangd.llvm.org/extensions.html#type-hierarchy
Sure, clangd is usable without them, but some of the extensions might
make someone's life easier, for example, by providing a type hierarchy
browser.

> [...] So there isn't any problem/drama here.

I don't want to start one either, but it's a good time to think once
more about Eglot's relations with other packages.

Re: Eglot to core [Was: rmsbolt.el [Was: Colorful line numbers]]

[Stefan Monnier]

>>> I look forward to Eglot being included with Emacs.  But (and I've said
>>> this before) "eglot" is a very bad name that says nothing about its
>>> purpose.  How about renaming it "language-server"?
>> I don't like idea conflating the names with the descriptions, because it
>> inevitably leads to conflicts.
> What kind of conflicts?

When several packages offer similar functionality, they fight for the
same name.  Should Gnus or Rmail or MH-E or ... be the one called "Mail"?

> If this is done, it would also be helpful to add alii for commands like
> `eglot-code-actions', `eglot-rename', `eglot-format', etc. to avoid
> confusion.

I don't see much need for that.  Once the users have found the
`language-server-mode`, the usual help system should make it easy for
them to find the rest under their "eglot-" names.

But if/when a such a need arise, of course we can add more aliases.


        Stefan

Re: Eglot to core [Was: rmsbolt.el [Was: Colorful line numbers]]

[Stefan Monnier]

>> So I see no need to rename it.  But we can definitely add various ways
>> to make it easier to find, such as adding a `language-server-mode`
>> alias.
> I think a major improvement would be for searching "Language Server" in
> Custom to find variables defined by eglot.

I don't see any need for the "s" at the end of "variables".
So the one `language-server-mode` variable should be sufficient.


        Stefan

Re: Eglot to core [Was: rmsbolt.el [Was: Colorful line numbers]]

[João Távora]

Felician Nemeth <felician.nemeth@gmail.com> writes:

>>> I don't think the current API is good enough for that.  See
>>> https://github.com/joaotavora/eglot/discussions/802#discussioncomment-2171239
>>
>> Really depends on what you consider "that" to be.
>
> To implement non-standard LSP features, which lots of servers propose.

Yes, that's one of the possibilities.  Another possibility is offering a
different, perhaps richer interface to programming facilities specific
to the major mode which may or may not rely on Eglot's LSP.  For
example, if LSP isn't available, or isn't the best choice for a
particular feature (say completion, or definition-finding) this
interface might delegate to some other tool, to CEDET, etc.

>> [...] Also, once Eglot is in core, I won't be only person deciding.
>
> (I welcome Eglot's inclusion in the core and appreciate your work.  I
> hope that's clear.)

And I hope it's clear that I immensely appreciate your work and your
review efforts in Eglot.  Eglot doesn't have many contributors who
understand the code base well (and bringing it to core is also an effort
to change that).

>>> More importantly, there are multiple language server implementations for
>>> some languages (python, c) with different quirks or non-standard
>>> extensions to the LSP protocol.  How do you imagine a major-mode would
>>> support these?
>>
>> They can adds methods to the generic function eglot-handle-request (and
>> a handful others), set values of Eglot in their major-mode
>> initialization code, bind keys to `eglot-*` interactive commands, make
>> compound commands from those commands, etc.
>
> OK.  This is somewhat similar to the case when a major mode provides an
> xref or a flymake backend.
>
> But I think sometimes the reverse approach is more desirable.  Let's
> take a not so hypothetical example and say c++-mode wants to provide a
> type-hierarchy browser.  It's better to write a generic, language
> agnostic hierarchy browser (maybe CEDET even has one), and extend Eglot
> with a hierarchy browser backend similarly to its xref backend.  The
> hierarchy browser then can have a tree-sitter backend as well.

Yes, I fully agree.

>> While they _can_, but that doesn't mean they _must_ or even _should_.
>> In fact, LSP's whole idea, which is surprisingly forgotten so often, is
>> that server and editor need to know very little about each other to be
>> able to cooperate.  By and large this idea works very very well. 
>
> I agree, but non-standard extensions exist for a reason.

Yup, it's normal.  You see the same in programming extensions that
appear in specific compilers first.  Some of then make it to the
standards, some don't.  Sometimes they make it in largely unchanged and
adapting clients is easy, sometimes not so much.  Standardization is
hard.

> Almost all servers have a VS-Code specific part that contains at least a
> list of configuration variables that other clients cannot handle
> automatically.  BTW, if Eglot knew about these configuration variables,
> could `customize-group` somehow save the settings into a .dir-locals.el
> file?

Very interesting.  Yes, I think this is a great idea to allow the user
to configure server-specific things easily and automatically save it
.dir-locals.el.  People tend to struggle with that, and it's 

The question is how.  Not sure if variables is the way to go.  It
_could_ be (but it could also be some plist/alist).  But let's say it's
variables, i.e. symbols holding values.  Then these symbols belong to
the major mode, not to server-agnostic eglot.el.  Then maybe each such
variable could have a symbol property 'eglot--server-specific-section'
and 'eglot--server-specific-name'.  Then there could indeed be a command
M-x eglot-save-server-specific-variables that creates/updates
.dir-locals.el for the given project.

Maybe custom.el machinery could call that command automatically. But for
which projects?  All of them?  The "current" one?  We don't have
project-specific custom.el

And also when using variables I always think its nice not to force
custom.el the user.  I know others like me prefer to set things from
.emacs.  So it should work without it.

>> Eglot works with tens of servers and hasn't got a line of server
>> specific code, except for eglot-server-programs invocations.  There
>> are exceptions, and not everyone has the same luck, but I've been
>> using C++ clangd and C# omnisharp quite a lot lately and I have 0
>> lines of server specific config.
>
> Even clangd has a moderately long list of extensions:
> https://clangd.llvm.org/extensions.html#type-hierarchy
> Sure, clangd is usable without them, but some of the extensions might
> make someone's life easier, for example, by providing a type hierarchy
> browser.

Writing more C++ lately I do feel this need yes.  I was looking at
tree-widget.el the other day, but didn't make much progress.  I fully
agree to what you we should make some backend-agnostic frontend first
and then plug it into eglot.el or eglot-x.el

>> [...] So there isn't any problem/drama here.
> I don't want to start one either, but it's a good time to think once
> more about Eglot's relations with other packages.

Aye, and that's always the thing with Eglot: it's at the intersection of
many things.

João

Re: Eglot to core [Was: rmsbolt.el [Was: Colorful line numbers]]

[Max Brieiev]

João Távora <joaotavora@gmail.com> writes:

> One of the points of bringing Eglot into core is that it can be used as
> a library by other major modes, who are free to build on top of Eglot's API.  Thus
> language server support becomes somewhat transparent (according to Stefan's
> recent definition): users don't need to know -- to some reasonable extent -- that Eglot
> as an LSP client is being used, just as they needn't know that tree-sitter is being
> used to provide, say, a better C++ mode.

Conceptually, Eglot matches a single major mode to a specific language server.

In some cases this is a limitation, because there are servers that are
supposed to handle the project as a whole.

This is especially the case in web development. The project may contain
various content types: javascript, stylesheets, html, xml, json,
typescript, syntax extensions like jsx, and so on.

This is where it is preferrable to have a single server instance per a
number of major modes. The server may read project configuration file
from the project's root directory, or be passed appropriate
initialization options during server startup.

If major modes were to use Eglot's API in their own ways, wouldn't they
step on each other toes - e.g. each one passing conflicting
initialization options to the server?

Not sure whether Eglot supports this use case, but in the past I wasn't
able to get it working.

Re: Eglot to core [Was: rmsbolt.el [Was: Colorful line numbers]]

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 4220 bytes --]

On Mon, Jul 25, 2022 at 5:08 PM Max Brieiev <max.brieiev@gmail.com> wrote:

> Conceptually, Eglot matches a single major mode to a specific language
server.

That is true.

> In some cases this is a limitation, because there are servers that are
> supposed to handle the project as a whole.

That is not the principal limitation, however.  It could be in a
specific case but it's not a fundamental one.  The limitation is rather
when multiple servers can operate on the same major mode of the same
project, offering different perspectives on your source code. Like one
language server dedicated to spelling, and the other to language things,
one dedicated to a specific linter, etc...

Often the language server itself aggregates these various perspectives
For example the clangd server offers optional support for the clang-tidy
linter, if it finds it.

But this is not always so and I think that some LSP clients do allow
multiple language servers to manage the same file.  Eglot doesn't.  It
requires significant (but not impossible) work on the base model.  I'm
interested in hearing about a "killer use case" for this, i.e. in
hearing about a case where a user has been forced to choose just one of
twogreat language servers for a given major-mode.

On a tangent: the language server jungle is vast and growing every day.
The single greatest difficulty I have in understanding and addressing
bug reports to Eglot is reproducibility.  It always starts with someone
working on language Foo using some special foo-mode.el and some special
foo-server with some special configuration and asking me to debug a
specific problem.  Installing all these things (language toolchain,
foo-mode and foo-server) on my machine is very time-consuming, not to
mention debugging user's .emacs files.  I just don't have the mental
bandwidth.  It would be great if someone could work on some
"server-replayer" application that reads in Eglot event logs collected
from actual servers or synthesized event logs, so that these
installations aren't necessary.  That would really speed up Eglot
development and bug-hunting.

> This is especially the case in web development. The project may contain
> various content types: javascript, stylesheets, html, xml, json,
> typescript, syntax extensions like jsx, and so on.
>
> This is where it is preferrable to have a single server instance per a
> number of major modes. The server may read project configuration file
> from the project's root directory, or be passed appropriate
> initialization options during server startup.
>
> If major modes were to use Eglot's API in their own ways, wouldn't they
> step on each other toes - e.g. each one passing conflicting
> initialization options to the server?

They surely could, but if there isn't some kind of consensus between the
major modes, it's not too difficult to extract a horizontal *minor* mode
that encompasses these differences.  This is a common, effective Emacs
way of solving this problem.  Then have the minor mode set
eglot-server-programs (or better yet, eglot-server-program) with the
correct invocation.

> Not sure whether Eglot supports this use case, but in the past I wasn't
> able to get it working.

It requires more coding, but it isn't extremely difficult.  Have a look
at the SLY Common Lisp IDE and the sly-mode minor mode in particular.
It turns on in lisp-mode source files, and in special SLY buffers whose
major mode isn't lisp-mode and whose content isn't necessarily Common Lisp
source code.  sly-mode sets and manages buffer-local variables that then
work for all intents are purposes as you describe.

The other difficulty you might face is that even doing the above with
Eglot today will probably result in multiple inferior server processes
being spawned (though now with the "correct" invocation).  That may or
may not be a problem and depends on whether the LS has a distributed
architecture.  If it doesn't, then Eglot also offers the TCP transport
model, where one single server is first started and then each inferior
process is actually a networking connection to that one process with a
single address space.

João

[-- Attachment #2: Type: text/html, Size: 4613 bytes --]

Re: Eglot to core [Was: rmsbolt.el [Was: Colorful line numbers]]

[Felician Nemeth]

> Yes, I think this is a great idea to allow the user to configure
> server-specific things easily and automatically save it
> .dir-locals.el.
>
> The question is how.  Not sure if variables is the way to go. 

Rust-analyzer can emit the definition of its configuration variables as
a JSON schema.  There are toml and yaml servers that hopefully
understand this schema description.

So one possibility for Eglot is to help the user to edit a
project/rust.toml or a global ~/.emacs.d/eglot/rust.toml file and
arrange to call eglot-signal-didChangeConfiguration when the user saves
the file.

Re: Eglot to core [Was: rmsbolt.el [Was: Colorful line numbers]]

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1059 bytes --]

On Tue, Jul 26, 2022 at 9:12 AM Felician Nemeth <felician.nemeth@gmail.com>
wrote:

> > Yes, I think this is a great idea to allow the user to configure
> > server-specific things easily and automatically save it
> > .dir-locals.el.
> >
> > The question is how.  Not sure if variables is the way to go.
>
> Rust-analyzer can emit the definition of its configuration variables as
> a JSON schema.  There are toml and yaml servers that hopefully
> understand this schema description.
>
> So one possibility for Eglot is to help the user to edit a
> project/rust.toml or a global ~/.emacs.d/eglot/rust.toml file and
> arrange to call eglot-signal-didChangeConfiguration when the user saves
> the file.
>

I don't disagree with this, though it seems different from your first idea.
When is the .dir-locals.el written in your example, or were you imagining
a use with `dir-locals-set-class`?
Regardless,  if this works, why shouldn't it be rust-mode.el's job to do
exactly
this?  Editing "toml" files is outside the scope of LSP.

João

[-- Attachment #2: Type: text/html, Size: 1467 bytes --]

Re: Eglot to core [Was: rmsbolt.el [Was: Colorful line numbers]]

[Felician Nemeth]

>> > Yes, I think this is a great idea to allow the user to configure
>> > server-specific things easily and automatically save it
>> > .dir-locals.el.
>> >
>> > The question is how.  Not sure if variables is the way to go.
>>
>> Rust-analyzer can emit the definition of its configuration variables as
>> a JSON schema.  There are toml and yaml servers that hopefully
>> understand this schema description.
>>
>> So one possibility for Eglot is to help the user to edit a
>> project/rust.toml or a global ~/.emacs.d/eglot/rust.toml file and
>> arrange to call eglot-signal-didChangeConfiguration when the user saves
>> the file.
>>
>
> I don't disagree with this, though it seems different from your first idea.
> When is the .dir-locals.el written in your example, or were you imagining
> a use with `dir-locals-set-class`?

My first idea was to use `customize` to set project-specific
configuration variables.  This doesn't seem possible or easy.

> Regardless,  if this works, why shouldn't it be rust-mode.el's job to do
> exactly
> this?  Editing "toml" files is outside the scope of LSP.

Yes, but if there was an LSP request that returned the server's
configuration schema, then Eglot could take care it independently of the
major-mode.  Unfortunately, no such LSP request has been proposed at the
moment.  But there was an attempt:
https://github.com/microsoft/language-server-protocol/issues/467

As you said, standardization is hard.  So this approach quite
hypothetical.