bug#65105: Reusing the same string as 'display on consecutive characters evades display

bug#65105: Reusing the same string as 'display on consecutive characters evades display

[JD Smith]

Evaluate:

(let ((s1 "test1")
      (s2 "test2"))
  (insert "\n"
          (propertize " " 'display s1)
          (propertize " " 'display s1)
          (propertize " " 'display s2)
          (propertize " " 'display s1)))


The first space display does not take effect, since the s1 string is used for two consecutive characters.  This has a practical impact for font-lock backends that use the ‘display text-property and would like to minimize string allocation.  

Tested Emacs 27/28/29.

bug#65105: Reusing the same string as 'display on consecutive characters evades display

[Eli Zaretskii]

tags 65105 notabug
thanks

> From: JD Smith <jdtsmith@gmail.com>
> Date: Sat, 5 Aug 2023 14:35:23 -0400
> 
> Evaluate:
> 
> (let ((s1 "test1")
>       (s2 "test2"))
>   (insert "\n"
>           (propertize " " 'display s1)
>           (propertize " " 'display s1)
>           (propertize " " 'display s2)
>           (propertize " " 'display s1)))
> 
> 
> The first space display does not take effect, since the s1 string is used for two consecutive characters.  This has a practical impact for font-lock backends that use the ‘display text-property and would like to minimize string allocation.  

Emacs cannot distinguish between two consecutive characters having
each a text property with the same value, and two characters having
the same property.  If you think about this for a moment, you will
understand why: we use intervals for text properties, so two adjacent
intervals with the identical property values and one interval with
that same value are indistinguishable (and in fact Emacs optimizes
this during GC by making just one interval from these two).

This is not a bug.

bug#65105: Reusing the same string as 'display on consecutive characters evades display

[JD Smith]

[-- Attachment #1: Type: text/plain, Size: 1460 bytes --]


> On Aug 5, 2023, at 3:03 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> tags 65105 notabug
> thanks
> 
>> From: JD Smith <jdtsmith@gmail.com>
>> Date: Sat, 5 Aug 2023 14:35:23 -0400
>> 
>> Evaluate:
>> 
>> (let ((s1 "test1")
>>      (s2 "test2"))
>>  (insert "\n"
>>          (propertize " " 'display s1)
>>          (propertize " " 'display s1)
>>          (propertize " " 'display s2)
>>          (propertize " " 'display s1)))
>> 
>> 
>> The first space display does not take effect, since the s1 string is used for two consecutive characters.  This has a practical impact for font-lock backends that use the ‘display text-property and would like to minimize string allocation.  
> 
> Emacs cannot distinguish between two consecutive characters having
> each a text property with the same value, and two characters having
> the same property.  If you think about this for a moment, you will
> understand why: we use intervals for text properties, so two adjacent
> intervals with the identical property values and one interval with
> that same value are indistinguishable (and in fact Emacs optimizes
> this during GC by making just one interval from these two).
> 
> This is not a bug.

Aha, thanks.  It does make sense from an optimization standpoint to “gang” properties in this manner.  Are you aware of any approach that allow re-using a string for ‘display, but permits consecutive intervals to remain distinct?

[-- Attachment #2: Type: text/html, Size: 9939 bytes --]

bug#65105: Reusing the same string as 'display on consecutive characters evades display

[Dmitry Gutov]

On 05/08/2023 23:49, JD Smith wrote:
> Aha, thanks.  It does make sense from an optimization standpoint to 
> “gang” properties in this manner.  Are you aware of any approach that 
> allow re-using a string for ‘display, but permits consecutive intervals 
> to remain distinct?

Overlays? But they have certain performance problems when their number 
grows.

Or you could try resolving this manually: when the previous character is 
assigned with the same line, do a copy-sequence, and when it's a 
different one, you can use the "singleton" one.

bug#65105: Reusing the same string as 'display on consecutive characters evades display

[JD Smith]

That’s probably the right approach.  Or even simpler, allocate two identical strings and alternate.  Thanks.

> On Aug 5, 2023, at 6:46 PM, Dmitry Gutov <dmitry@gutov.dev> wrote:
> 
> On 05/08/2023 23:49, JD Smith wrote:
>> Aha, thanks.  It does make sense from an optimization standpoint to “gang” properties in this manner.  Are you aware of any approach that allow re-using a string for ‘display, but permits consecutive intervals to remain distinct?
> 
> Overlays? But they have certain performance problems when their number grows.
> 
> Or you could try resolving this manually: when the previous character is assigned with the same line, do a copy-sequence, and when it's a different one, you can use the "singleton" one.

bug#65105: Reusing the same string as 'display on consecutive characters evades display

[Eli Zaretskii]

> From: JD Smith <jdtsmith@gmail.com>
> Date: Sat, 5 Aug 2023 18:49:11 -0400
> Cc: Eli Zaretskii <eliz@gnu.org>,
>  65105@debbugs.gnu.org
> 
> That’s probably the right approach.  Or even simpler, allocate two identical strings and alternate.  Thanks.

And with that, I'm closing this bug.

bug#65105: Reusing the same string as 'display on consecutive characters evades display

[Eli Zaretskii]

> Date: Sun, 6 Aug 2023 01:46:49 +0300
> Cc: 65105@debbugs.gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 05/08/2023 23:49, JD Smith wrote:
> > Aha, thanks.  It does make sense from an optimization standpoint to 
> > “gang” properties in this manner.  Are you aware of any approach that 
> > allow re-using a string for ‘display, but permits consecutive intervals 
> > to remain distinct?
> 
> Overlays? But they have certain performance problems when their number 
> grows.

Not in Emacs 29 and later, right?

bug#65105: Reusing the same string as 'display on consecutive characters evades display

[Dmitry Gutov]

On 06/08/2023 07:54, Eli Zaretskii wrote:
>> Date: Sun, 6 Aug 2023 01:46:49 +0300
>> Cc:65105@debbugs.gnu.org
>> From: Dmitry Gutov<dmitry@gutov.dev>
>>
>> On 05/08/2023 23:49, JD Smith wrote:
>>> Aha, thanks.  It does make sense from an optimization standpoint to
>>> “gang” properties in this manner.  Are you aware of any approach that
>>> allow re-using a string for ‘display, but permits consecutive intervals
>>> to remain distinct?
>> Overlays? But they have certain performance problems when their number
>> grows.
> Not in Emacs 29 and later, right?

Hopefully not -- I haven't benchmarked any particular problem cases 
recently.

But I'm guessing the package JD is working on aims to support previous 
versions anyway.