bug#70784: Abolish string resizing

bug#70784: Abolish string resizing

[Mattias Engdegård]

The Elisp ability to resize strings is high-cost, low-benefit, so we should abolish it.

Strings are performance-critical, and the split allocation (fixed-size metadata + variable-size data) and its indirections are not free. This is one reason why strings are comparatively expensive in Elisp compared to other languages.

With MPS there is even no use for the special-purpose string data defragger in the current GC.

String resizing occurs in exactly one place: `aset` on strings, and then only when the mutation alters the string size, in either of two cases:

* When a unibyte string byte is set to a non-byte value (> 255), which also forces the string to become multibyte.
* When a multibyte string char is set to a char of different length.

String mutation itself is very rare these days, and resizing string mutation exceptionally so.

Best and simplest would be to just turn resizing `aset` on strings into an error now. As an alternative, we could start warning about it now and make it an error in the next release. The (trivial) patches exist.

Note that this is not a proposal for abolishing string mutation in general. For example, `clear-string` would still work on any string, as would arbitrary closed mutations on unibyte strings and on ASCII multibyte strings.

bug#70784: Abolish string resizing

[Eli Zaretskii]

> Cc: Stefan Monnier <monnier@iro.umontreal.ca>
> From: Mattias Engdegård <mattias.engdegard@gmail.com>
> Date: Sun, 5 May 2024 14:33:51 +0200
> 
> The Elisp ability to resize strings is high-cost, low-benefit, so we should abolish it.
> 
> Strings are performance-critical, and the split allocation (fixed-size metadata + variable-size data) and its indirections are not free. This is one reason why strings are comparatively expensive in Elisp compared to other languages.
> 
> With MPS there is even no use for the special-purpose string data defragger in the current GC.
> 
> String resizing occurs in exactly one place: `aset` on strings, and then only when the mutation alters the string size, in either of two cases:
> 
> * When a unibyte string byte is set to a non-byte value (> 255), which also forces the string to become multibyte.
> * When a multibyte string char is set to a char of different length.
> 
> String mutation itself is very rare these days, and resizing string mutation exceptionally so.
> 
> Best and simplest would be to just turn resizing `aset` on strings into an error now. As an alternative, we could start warning about it now and make it an error in the next release. The (trivial) patches exist.

How many places in our own sources call aset on strings?

> 
> Note that this is not a proposal for abolishing string mutation in general. For example, `clear-string` would still work on any string, as would arbitrary closed mutations on unibyte strings and on ASCII multibyte strings.

I'm not yet sure this is a good idea, but one thing I'm sure: it is
not for Emacs 30.  So if we decide to go this way, let's delay
merging the changes until after the emacs-30 branch is cut, which
should happen in a week or two.

Thanks.

bug#70784: Abolish string resizing

[Mattias Engdegård]

5 maj 2024 kl. 16.04 skrev Eli Zaretskii <eliz@gnu.org>:

> How many places in our own sources call aset on strings?

In general, very little. Resizing ones, probably none now.

> I'm not yet sure this is a good idea, but one thing I'm sure: it is
> not for Emacs 30.  So if we decide to go this way, let's delay
> merging the changes until after the emacs-30 branch is cut, which
> should happen in a week or two.

That's fine, but a warning might help in emacs-30 to prepare for making resizing an error in emacs-31.
Here's how that would work: the first 5 times (say), emit a warning like

 Warning: `aset' on a multibyte string with char value #x2022 will become an error in Emacs 31

followed by a truncated stack trace, just enough to help the user (or someone helping out) see what package is responsible.

bug#70784: Abolish string resizing

[Eli Zaretskii]

> From: Mattias Engdegård <mattias.engdegard@gmail.com>
> Date: Sun, 5 May 2024 16:18:16 +0200
> Cc: 70784@debbugs.gnu.org,
>  monnier@iro.umontreal.ca
> 
> 5 maj 2024 kl. 16.04 skrev Eli Zaretskii <eliz@gnu.org>:
> 
> > How many places in our own sources call aset on strings?
> 
> In general, very little. Resizing ones, probably none now.
> 
> > I'm not yet sure this is a good idea, but one thing I'm sure: it is
> > not for Emacs 30.  So if we decide to go this way, let's delay
> > merging the changes until after the emacs-30 branch is cut, which
> > should happen in a week or two.
> 
> That's fine, but a warning might help in emacs-30 to prepare for making resizing an error in emacs-31.
> Here's how that would work: the first 5 times (say), emit a warning like
> 
>  Warning: `aset' on a multibyte string with char value #x2022 will become an error in Emacs 31
> 
> followed by a truncated stack trace, just enough to help the user (or someone helping out) see what package is responsible.

I'm not sure even that is a good idea for Emacs 30.

Let's see what others think.

bug#70784: Abolish string resizing

[Drew Adams via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> I'm not sure even that is a good idea for Emacs 30.
> Let's see what others think.

Next on Matthias's immutable list of "improvements":
doing away with mutable lists.

Poor Lisp.

bug#70784: Abolish string resizing

[Eli Zaretskii]

> From: Drew Adams <drew.adams@oracle.com>
> CC: "70784@debbugs.gnu.org" <70784@debbugs.gnu.org>,
>         "monnier@iro.umontreal.ca" <monnier@iro.umontreal.ca>
> Date: Sun, 5 May 2024 16:55:43 +0000
> 
> > I'm not sure even that is a good idea for Emacs 30.
> > Let's see what others think.
> 
> Next on Matthias's immutable list of "improvements":
> doing away with mutable lists.
> 
> Poor Lisp.

Your opinions might be more important to read and consider if you drop
the sarcasm.

There are no enemies of Lisp here.

bug#70784: Abolish string resizing

[Drew Adams via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> > > I'm not sure even that is a good idea for
> > > Emacs 30.  Let's see what others think.
> >
> > Next on Matthias's immutable list of
> > "improvements": doing away with mutable lists.
> > Poor Lisp.
> 
> Your opinions might be more important to
> read and consider if you drop the sarcasm.
> There are no enemies of Lisp here.

No one (except you) spoke of "enemies" of Lisp.

Without sarcasm: From Mattias in bug 70155 thread:

  Of course if you ask me, I'd prefer it if lists
  were guaranteed to be proper, immutable, with
  the empty list an object distinct from the symbol
  nil and the false boolean value. Maybe next year.

Was _that_ sarcasm, or was it straightforward?  I
took it at face value.  And coincidentally we now
see this string "enhancement" suggestion, along
the same line from the same friend.

My response to this line is "poor Lisp" (or "poor
Elisp") - without sarcasm.  And nothing personal.

OK, yes, Scheme is in the Lisp family, and its nil
is only the empty list, not also false.  (I'd say
"poor Lisp" there too, but not as strongly.)

And Clojure is often considered to be in the Lisp
family, and it uses immutable data more, including
lists.

Elisp is more related to Common Lisp.  And mutable
lists and dotted lists are _fundamental_ for this
part of the family (and for Scheme as well).
Nothing says "Lisp" more than its cons.

Proposing that Elisp lists be only _proper_ (never
dotted) and _immutable_ is a _radical_ departure
from Lisp.

I don't take that proposal as coming from any
"enemy of Lisp".  I don't see any enemies here
(do you?).  I'm not in favor of such a proposal;
that's all.

Likewise for the current proposal, to make strings
immutable (abolish string resizing).  That one
wasn't even on the "Maybe next year" list.  It's
apparently on the this-year list.

Just one opinion.

bug#70784: Abolish string resizing

[Eli Zaretskii]

> From: Drew Adams <drew.adams@oracle.com>
> CC: "mattias.engdegard@gmail.com" <mattias.engdegard@gmail.com>,
>         "70784@debbugs.gnu.org" <70784@debbugs.gnu.org>,
>         "monnier@iro.umontreal.ca"
> 	<monnier@iro.umontreal.ca>
> Date: Sun, 5 May 2024 18:09:38 +0000
> 
> > > > I'm not sure even that is a good idea for
> > > > Emacs 30.  Let's see what others think.
> > >
> > > Next on Matthias's immutable list of
> > > "improvements": doing away with mutable lists.
> > > Poor Lisp.
> > 
> > Your opinions might be more important to
> > read and consider if you drop the sarcasm.
> > There are no enemies of Lisp here.
> 
> No one (except you) spoke of "enemies" of Lisp.

You did ("poor Lisp" etc.).

bug#70784: Abolish string resizing

[Drew Adams via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> > > > > I'm not sure even that is a good idea for
> > > > > Emacs 30.  Let's see what others think.
> > > >
> > > > Next on Matthias's immutable list of
> > > > "improvements": doing away with mutable lists.
> > > > Poor Lisp.
> > >
> > > Your opinions might be more important to
> > > read and consider if you drop the sarcasm.
> > > There are no enemies of Lisp here.
> >
> > No one (except you) spoke of "enemies" of Lisp.
> 
> You did ("poor Lisp" etc.).

No, I didn't - you did.  I said nothing about
any enemies of Lisp, nor did I suggest that
anyone here is an enemy of Lisp.

Please stop with the hyperbole and insinuation.
You appear to be trying to spin my position
into something it's not.

It would be a mistake, IMHO, to do away with
mutable strings or lists in Elisp.  That's my
opinion: that would _impoverish Lisp_.  The
former (make strings immutable) is exactly the
question raised in this thread, IIUC.

It would also be unfortunate to do away with
the use of nil as meaning false, or as the
empty list.  Again, that would _impoverish
Lisp_.  That's my opinion.

Such suggestions don't jibe with Lisp; they
would, in effect mutate Lisp away from its
fundament.  Let's not go down that road.

bug#70784: Abolish string resizing

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Drew Adams <drew.adams@oracle.com> writes:

> No, I didn't - you did.  I said nothing about
> any enemies of Lisp, nor did I suggest that
> anyone here is an enemy of Lisp.
>
> Please stop with the hyperbole and insinuation.
> You appear to be trying to spin my position
> into something it's not.
>
> It would be a mistake, IMHO, to do away with
> mutable strings or lists in Elisp.  That's my
> opinion: that would _impoverish Lisp_.  The
> former (make strings immutable) is exactly the
> question raised in this thread, IIUC.
>
> It would also be unfortunate to do away with
> the use of nil as meaning false, or as the
> empty list.  Again, that would _impoverish
> Lisp_.  That's my opinion.
>
> Such suggestions don't jibe with Lisp; they
> would, in effect mutate Lisp away from its
> fundament.  Let's not go down that road.

I agree, but alas, the dubious practice of removing time-honored
facilities for their supposed disuse, virtually on impulse, and giving
Emacs a new motto inspired by line 6186 of dispnew.c, is much in vogue
with Mattias these days.

One can only hope that calmer heads will prevail.

bug#70784: Abolish string resizing

[Gerd Möllmann]

Po Lu via "Bug reports for GNU Emacs, the Swiss army knife of text
editors" <bug-gnu-emacs@gnu.org> writes:

> Drew Adams <drew.adams@oracle.com> writes:
>
>> No, I didn't - you did.  I said nothing about
>> any enemies of Lisp, nor did I suggest that
>> anyone here is an enemy of Lisp.
>>
>> Please stop with the hyperbole and insinuation.
>> You appear to be trying to spin my position
>> into something it's not.
>>
>> It would be a mistake, IMHO, to do away with
>> mutable strings or lists in Elisp.  That's my
>> opinion: that would _impoverish Lisp_.  The
>> former (make strings immutable) is exactly the
>> question raised in this thread, IIUC.
>>
>> It would also be unfortunate to do away with
>> the use of nil as meaning false, or as the
>> empty list.  Again, that would _impoverish
>> Lisp_.  That's my opinion.
>>
>> Such suggestions don't jibe with Lisp; they
>> would, in effect mutate Lisp away from its
>> fundament.  Let's not go down that road.
>
> I agree, but alas, the dubious practice of removing time-honored
> facilities for their supposed disuse, virtually on impulse, and giving
> Emacs a new motto inspired by line 6186 of dispnew.c, is much in vogue
> with Mattias these days.

I had to look that up: it's bitch_at_user. Yeah, some arrogance is part
of being young, I guess :-).

> One can only hope that calmer heads will prevail.

First of all, I'd like to ask why this feature/these features are not
discussed on emacs-devel but here? I find that strange.

Wrt to this particular feature of aref, I wonder if XEmacs wasn't right
having a character type. That would be an interesting change. The change
in aref or strings doesn't interest me much.

The immutabile list thing and so on. To each his own, live and let live,
and so on, but this direction is not mine. And this all should be on
emacs-devel, but I said that already.

bug#70784: Abolish string resizing

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > First of all, I'd like to ask why this feature/these features are not
  > discussed on emacs-devel but here? I find that strange.

We should always move discussions of feature changes to emacs-devel!

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#70784: Abolish string resizing

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

> Next on Matthias's immutable list of "improvements":
> doing away with mutable lists.

Drew, Eli is right.  Any views we want to state can be expressed
without sarcasm.  Please do it that way.

Another thing to avoid is assuming someone will next propose
some further step.

  > Without sarcasm: From Mattias in bug 70155 thread:

  >   Of course if you ask me, I'd prefer it if lists
  >   were guaranteed to be proper, immutable, with
  >   the empty list an object distinct from the symbol
  >   nil and the false boolean value. Maybe next year.

Those words explicitly highlight the fact that he was _not_ proposing
this.

  > Proposing that Elisp lists be only _proper_ (never
  > dotted) and _immutable_ is a _radical_ departure
  > from Lisp.

That is true.  But he does not proposed we make this change, so please
don't criticize him as if he did.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#70784: Abolish string resizing

[Drew Adams via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>   >   Of course if you ask me, I'd prefer it if lists
>   >   were guaranteed to be proper, immutable, with
>   >   the empty list an object distinct from the symbol
>   >   nil and the false boolean value. Maybe next year.
> 
> Those words explicitly highlight the fact that he was _not_ proposing
> this.

Debatable ("Maybe next year"), but yes, he didn't
explicitly propose those changes.  My reaction was
premature.

>   > Proposing that Elisp lists be only _proper_ (never
>   > dotted) and _immutable_ is a _radical_ departure
>   > from Lisp.
> 
> That is true.  But he does not proposed we make this change, so please
> don't criticize him as if he did.

I didn't criticize him; I criticized such a proposal.
Mea culpa for that.

But I'm glad to see you agree about such a proposal,
should it arise.

bug#70784: Abolish string resizing

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

> I'm not sure even that is a good idea for Emacs 30.
>
> Let's see what others think.

Is there any drawback to adding a warning?  It seems like it would help
Emacs package developers, not all of which track master.

bug#70784: Abolish string resizing

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Sun, 26 May 2024 05:02:01 -0400
> Cc: 70784@debbugs.gnu.org, monnier@iro.umontreal.ca
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I'm not sure even that is a good idea for Emacs 30.
> >
> > Let's see what others think.
> 
> Is there any drawback to adding a warning?

A nuisance, obviously, for something that was valid code for eons.

> It seems like it would help Emacs package developers, not all of
> which track master.

I'd be happier if we waited with this till Emacs 31.  This will let
people more time to see the change and react to it, if they want.

bug#70784: Abolish string resizing

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> I'd be happier if we waited with this till Emacs 31.  This will let
> people more time to see the change and react to it, if they want.

For warnings that can't be caught at compile-time and that can't be
fixed by end-users, maybe we should get into the habit of activating them
only in "dev builds".


        Stefan

bug#70784: Abolish string resizing

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Stefan Kangas <stefankangas@gmail.com>,  mattias.engdegard@gmail.com,
>   70784@debbugs.gnu.org
> Date: Sun, 26 May 2024 14:03:26 -0400
> 
> > I'd be happier if we waited with this till Emacs 31.  This will let
> > people more time to see the change and react to it, if they want.
> 
> For warnings that can't be caught at compile-time and that can't be
> fixed by end-users, maybe we should get into the habit of activating them
> only in "dev builds".

I might agree with this, but I think the intent in this case was to
cause all the Lisp programmers out there to migrate to the brave new
world, and most of those don't do "dev builds".  In fact, my
impression from reading this and the bug list is that most people who
track the master branch build Emacs for production -- with all the
default optimizations and without --enable-checking.  So it could be
that "dev builds" include exactly two people: you and me ;-)

bug#70784: Abolish string resizing

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> I might agree with this, but I think the intent in this case was to
> cause all the Lisp programmers out there to migrate to the brave new
> world, and most of those don't do "dev builds".  In fact, my
> impression from reading this and the bug list is that most people who
> track the master branch build Emacs for production -- with all the
> default optimizations and without --enable-checking.  So it could be
> that "dev builds" include exactly two people: you and me ;-)

Well, yes, I left the definition of "dev build" open.
It could mean a `configure` option, but I was thinking more of something
like "build from Git" or "build from `master`".


        Stefan

bug#70784: Abolish string resizing

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I see no reason to make it impossible to change the size of a string,

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#70784: Abolish string resizing

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> That's fine, but a warning might help in emacs-30 to prepare for making
> resizing an error in emacs-31.
> Here's how that would work: the first 5 times (say), emit a warning like
>
>  Warning: `aset' on a multibyte string with char value #x2022 will become an error in Emacs 31
>
> followed by a truncated stack trace, just enough to help the user (or
> someone helping out) see what package is responsible.

Similar to what I do in `track-changes.el` when we detect a missing
call to `after/before-change-functions`?  I like that.
If the warning is too annoying, we could do like I did there, i.e. only
emit the warning when running in a non-release Emacs (i.e. when
`emacs-version` has 2 dots), and offer a variable to control whether to
emit the warning or not.


        Stefan

bug#70784: Abolish string resizing

[Mattias Engdegård]

[-- Attachment #1: Type: text/plain, Size: 978 bytes --]

5 maj 2024 kl. 20.09 skrev Stefan Monnier <monnier@iro.umontreal.ca>:

> If the warning is too annoying, we could do like I did there, i.e. only
> emit the warning when running in a non-release Emacs (i.e. when
> `emacs-version` has 2 dots), and offer a variable to control whether to
> emit the warning or not.

Thanks, I hadn't seen your work. My approach (attached) is more an instance of outsider art: I didn't know how these things are normally done and just made it up as I went.

The backtrace is included in the message here because it's kind of essential for diagnosing the warning correctly, and it uses the very compact form repurposed from the profiler for brevity and because we probably don't need more to establish the source of the problem. Your idea to stuff stack traces in a variable for later scrutiny is probably more robust.

(Prefer raising actual errors though -- some people will inevitably suppress the warning to avoid fixing the code.)


[-- Attachment #2: warn-aset-string.diff --]
[-- Type: application/octet-stream, Size: 2081 bytes --]

diff --git a/src/data.c b/src/data.c
index ea611ad1abf..a3fba48c7f2 100644
--- a/src/data.c
+++ b/src/data.c
@@ -2570,6 +2570,33 @@ DEFUN ("aref", Faref, Saref, 2, 2, 0,
     }
 }
 
+static void
+warn_aset_string_resize (Lisp_Object string, const char *msg, unsigned ch)
+{
+  /* Only warn the first few times to avoid spamming the user.  */
+  static int remaining = 5;
+  if (remaining <= 0)
+    return;
+  --remaining;
+
+  Lisp_Object bt[16];
+  get_backtrace (bt, ARRAYELTS (bt));
+  Lisp_Object warning
+    = CALLN (Fformat,
+	     build_string ("`aset' on %s string: %s;"
+			   " this will become an error in Emacs 31."
+			   "  Call chain (most recent first):"),
+	     build_string (STRING_MULTIBYTE (string) ? "multibyte" : "unibyte"),
+	     CALLN (Fformat, build_string (msg), make_int (ch)));
+
+  for (ptrdiff_t i = EQ (bt[0], Qaset) ? 1 : 0;
+       i < ARRAYELTS (bt) && !NILP (bt[i]); i++)
+    if (SYMBOLP (bt[i]))
+      warning = CALLN (Fformat, build_string ("%s %s;"), warning, bt[i]);
+  Vdelayed_warnings_list = Fcons (list2 (Qaset, warning),
+				  Vdelayed_warnings_list);
+}
+
 DEFUN ("aset", Faset, Saset, 3, 3, 0,
        doc: /* Store into the element of ARRAY at index IDX the value NEWELT.
 Return NEWELT.  ARRAY may be a vector, a string, a char-table or a
@@ -2623,6 +2650,12 @@ DEFUN ("aset", Faset, Saset, 3, 3, 0,
 	{
 	  idxval_byte = string_char_to_byte (array, idxval);
 	  p1 = SDATA (array) + idxval_byte;
+	  if (c > 127)
+	    warn_aset_string_resize (array,
+				     "replacement char #x%x is non-ASCII", c);
+	  else if (*p1 > 127)
+	    warn_aset_string_resize (array,
+				     "replaced char #x%x is non-ASCII", *p1);
 	  prev_bytes = BYTES_BY_CHAR_HEAD (*p1);
 	}
       else if (SINGLE_BYTE_CHAR_P (c))
@@ -2632,6 +2665,9 @@ DEFUN ("aset", Faset, Saset, 3, 3, 0,
 	}
       else
 	{
+	  warn_aset_string_resize (array,
+				   "replacement char #x%x is not single-byte",
+				   c);
 	  for (ptrdiff_t i = SBYTES (array) - 1; i >= 0; i--)
 	    if (!ASCII_CHAR_P (SREF (array, i)))
 	      args_out_of_range (array, newelt);

bug#70784: Abolish string resizing

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Thanks, I hadn't seen your work. My approach (attached) is more an instance
> of outsider art: I didn't know how these things are normally done and just
> made it up as I went.

Mine was also "invent it as I go along", ...

> The backtrace is included in the message here because it's kind of essential
> for diagnosing the warning correctly, and it uses the very compact form
> repurposed from the profiler for brevity and because we probably don't need
> more to establish the source of the problem.

Very nice.  For track-changes, the backtraces aren't as important,
because we often detect the problem only after the fact so the backtrace
may be unrelated (which is why I also record the recent keys).


        Stefan

bug#70784: Abolish string resizing

[Mattias Engdegård]

5 maj 2024 kl. 23.14 skrev Stefan Monnier <monnier@iro.umontreal.ca>:

> Very nice.  For track-changes, the backtraces aren't as important,
> because we often detect the problem only after the fact so the backtrace
> may be unrelated (which is why I also record the recent keys).

Right, recent keys are probably less important for `aset`. I doubt the proposed warning would trigger very often but if it does, it should given ample information to the user.

bug#70784: Abolish string resizing

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> Very nice.  For track-changes, the backtraces aren't as important,
>> because we often detect the problem only after the fact so the backtrace
>> may be unrelated (which is why I also record the recent keys).
> Right, recent keys are probably less important for `aset`.  I doubt the
> proposed warning would trigger very often but if it does, it should given
> ample information to the user.

Agreed.


        Stefan

bug#70784: Abolish string resizing

[Mattias Engdegård]

To minimise even low-probability trouble, I've looked for standard Elisp functions that might run into the resizing mutation trouble if used in a specific way. There's not a lot, but so far I've done:

- cl-substitute[-if[-not]]
- cl-sort (not that sorting strings is very common)
- subst-char-in-string with INPLACE = nil

I'm going to leave functions that explicitly mutate strings alone since code may depend on them doing that which is completely fine.

The manual would benefit from some light editing, just to make it clear that mutating multibyte strings is not usually the best programming style. I'll post a suggested patch.

Also, I'm leaning toward removing `store-substring` from the manual because while it probably felt like a good addition at the time, circumstances have changed and it has hardly seen any use at all. (We can mark it obsolete or keep it indefinitely but it doesn't really deserve top billing these days.)

bug#70784: Abolish string resizing

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Also, I'm leaning toward removing `store-substring` from the manual because
> while it probably felt like a good addition at the time, circumstances have
> changed and it has hardly seen any use at all. (We can mark it obsolete or
> keep it indefinitely but it doesn't really deserve top billing these days.)

Hmm... wasn't aware of this one.
AFAIK between Emacs, GNU ELPA, and NonGNU ELPA, I can see it used in only
two files:

    elpa/packages/elisp-benchmarks/benchmarks/dhrystone.el
    elpa/packages/mmm-mode/mmm-cmds.el

Deprecation sounds about right.


        Stefan

bug#70784: Abolish string resizing

[Mattias Engdegård]

15 maj 2024 kl. 19.47 skrev Stefan Monnier <monnier@iro.umontreal.ca>:

> AFAIK between Emacs, GNU ELPA, and NonGNU ELPA, I can see it used in only
> two files:
> 
>    elpa/packages/elisp-benchmarks/benchmarks/dhrystone.el
>    elpa/packages/mmm-mode/mmm-cmds.el

Yes, and both are frivolous: in mmm-mode it's for padding a table for display (`format` would be natural), and in dhrystone, where do we even begin...

bug#70784: Abolish string resizing

[Mattias Engdegård]

`store-substring` now removed from the manual (nice to see it getting smaller for a change).
It should probably be marked obsolete as well -- any objections?

Regarding resizing `aset` calls, what about adding the proposed warnings to Emacs 30? They are not expected to be seen much, and it gives users ample time to change some old code that may remain.

In any case, we'll know soon enough (that's one good thing about warnings).

bug#70784: Abolish string resizing

[Eli Zaretskii]

> From: Mattias Engdegård <mattias.engdegard@gmail.com>
> Date: Sat, 25 May 2024 13:24:06 +0200
> Cc: Eli Zaretskii <eliz@gnu.org>,
>  70784@debbugs.gnu.org
> 
> Regarding resizing `aset` calls, what about adding the proposed warnings to Emacs 30? They are not expected to be seen much, and it gives users ample time to change some old code that may remain.

No, it's too late for such changes in Emacs 30.  Please wait until the
emacs-30 release branch is cut.

bug#70784: Abolish string resizing

[Mattias Engdegård]

25 maj 2024 kl. 13.37 skrev Eli Zaretskii <eliz@gnu.org>:

> No, it's too late for such changes in Emacs 30.  Please wait until the
> emacs-30 release branch is cut.

That's perfectly fine. We'll make the changes after the branch has been created then.

bug#70784: Abolish string resizing

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Mattias Engdegård <mattias.engdegard@gmail.com> writes:

> The Elisp ability to resize strings is high-cost, low-benefit, so we
> should abolish it.

What is the improvement to be had by "abolishing" this immemorial
feature?

If it's a notional increase in Lisp evaluation performance, please stop.
Emacs Lisp is not a hot-rod where crucial, fundamental facilities are
dispensable in the face of a performance improvement (which is anything
but a certainly at this early stage) of a few percent or similarly
marginal value on contrived benchmarks testing scenarios unlikely to be
encountered in real Lisp code.

MPS is no justification for degrading the capabilities of the existing
GC, if for no other reason than its being inoperable on systems beyond
the limited selection whose support its developers consider a priority,
and the increase in memory consumption it brings (e.g. it will never
function on Android 13, because the C library's sigaction wrapper is
insufficient to enable MPS and JVM trap handlers to coexist).  What's
more, memory consumption is an aspect that should not be sacrificed for
minor gains in performance, with a program that is designed to be a good
citizen on systems old and new.

Is it only I who am tired of these proposals for complete upheavals
that, somehow, Emacs has fared just fine without, for generations past?
It's precisely this attitude that begins to inspire thoughts of
departure.  Backwards-compatibility is an obligation that cannot be
evaded by means of warnings, which instead serve to annoy and antagonize
users, whose only wish is that Emacs leave them in peace.

bug#70784: Abolish string resizing

[Drew Adams via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Is it only I who am tired of these proposals for complete upheavals
> that, somehow, Emacs has fared just fine without, for generations past?

You're not the only one.

> Backwards-compatibility is an obligation that cannot be
> evaded by means of warnings, which instead serve to annoy
> and antagonize users, whose only wish is that Emacs leave
> them in peace.

Happy to hear that POV.
___

Wrt mutable strings: Consider that Common Lisp, whose
implementations can be very performant, has always had
mutable strings.  I'd even wager that no one ever argued
that its strings shouldn't be mutable.  Inconceivable.

Yes, CL strings are vectors (thus arrays, thus sequences)
of chars, so they're different from Elisp strings.  As
such, you can use sequence functions on them (including
destructive functions such as nreverse).

On the other hand, Elisp strings can have text properties.
(Let's please not hear a proposal to remove that feature
as well, in the name of immutability or performance.)

bug#70784: Abolish string resizing

[Eli Zaretskii]

> Cc: "70784@debbugs.gnu.org" <70784@debbugs.gnu.org>,
>  Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Mon, 6 May 2024 01:56:27 +0000
> From:  Drew Adams via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> Wrt mutable strings: Consider that Common Lisp, whose
> implementations can be very performant, has always had
> mutable strings.  I'd even wager that no one ever argued
> that its strings shouldn't be mutable.  Inconceivable.

Which is why no one is proposing to make string immutable.  This is
not what this proposal is about, and Mattias made a point of saying
that explicitly.

> On the other hand, Elisp strings can have text properties.
> (Let's please not hear a proposal to remove that feature
> as well, in the name of immutability or performance.)

A strawman, if ever there was one.

bug#70784: Abolish string resizing

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

> Which is why no one is proposing to make string immutable.  This is
> not what this proposal is about, and Mattias made a point of saying
> that explicitly.

The point is that CL string implementations are capable of modifying
strings in such a manner as Mattias proposes to "abolish".  I presume
some are implemented as Emacs's are, while others simply use a
representation for multibyte strings where all characters are the same
wide size (which incidentally there's nothing preventing Emacs from
implementing).

bug#70784: Abolish string resizing

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Drew Adams <drew.adams@oracle.com>,  mattias.engdegard@gmail.com,
>   70784@debbugs.gnu.org,  monnier@iro.umontreal.ca
> Date: Mon, 06 May 2024 20:29:05 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Which is why no one is proposing to make string immutable.  This is
> > not what this proposal is about, and Mattias made a point of saying
> > that explicitly.
> 
> The point is that CL string implementations are capable of modifying
> strings in such a manner as Mattias proposes to "abolish".  I presume
> some are implemented as Emacs's are, while others simply use a
> representation for multibyte strings where all characters are the same
> wide size (which incidentally there's nothing preventing Emacs from
> implementing).

That doesn't really matter.  "Other applications" do things that Emacs
doesn't necessarily want to do, or vice versa.  E.g., consider our
support for raw bytes in multibyte strings, which is quite unique.

What _would_ matter is if there are important use cases out there
where replacing one character by another character of a different
multibyte length.  So if you, or anyone else, can come up with such
situations (which are not made up specifically to argue here), please
do, because knowing about such situations would be useful to make this
discussion serious and based on facts.

bug#70784: Abolish string resizing

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

> What _would_ matter is if there are important use cases out there
> where replacing one character by another character of a different
> multibyte length.  So if you, or anyone else, can come up with such
> situations (which are not made up specifically to argue here), please
> do, because knowing about such situations would be useful to make this
> discussion serious and based on facts.

There is certainly plenty of Emacs code that needs to replace characters
within strings whose contents are more or less arbitrary, so that
whether Mattias's proposal will disrupt them will vary in an
unpredictable manner from one string to the next.  I would venture to
suggest that _every_ invocation of aset with multibyte strings will
potentially be disrupted by the proposal in its mildest form, and it
will only be a matter of time before problems surface.  A wildly
variable length of time, for that matter, so it is infeasible to
anticipate their impact ahead of time, at least with the accuracy that
such a change deserves.

Nevertheless, examples are legion and one need search no further than
one's own init files for them:

      (progn
        [...]
	(if allow_unicode
	    (progn
	      (aset string pac-quote-start ?“)
              (aset string (1- pac-quote-close) ?”)
	      )
	  (progn
	    ;; Sometimes [...] prints unicode quotation marks
	    ;; surrounding pac captions so prevent inserting them into
	    ;; source code comments.
	    (aset string pac-quote-start ?\")
	    (aset string (1- pac-quote-close) ?\")
	    )
	  )
        [...]
	)

string is process output from shell-command-to-string, generated by a
Perl script I cannot publish.  Please don't ask for details as to the
script itself, since I am not at liberty to disclose them, and they have
absolutely no bearing on the matter at hand.

bug#70784: Abolish string resizing

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> The Elisp ability to resize strings is high-cost, low-benefit, so we
>> should abolish it.
> What is the improvement to be had by "abolishing" this immemorial
> feature?

Actually, it's not "immemorial", it's just old:

    commit 3c9de1afcde82a99137721436c822059cce79b5b
    Author: Kenichi Handa <handa@gnu.org>
    Date:   Fri Jul 21 06:45:30 2000 +0000

        (Faset): Allow storing any multibyte character in a string.  Convert
        unibyte string to multibyte if necessary.

IOW, since Emacs-21.1.

As for improvements, like a lot of refactoring and maintenance work,
there isn't any immediate benefit.  But it's a "feature" which is *very*
rarely used (thank god: it makes a notionally constant time operation
take time proportional to the size of the string, so if it were used
often we would have heard complaints about the poor performance) and
which imposes pretty significant implementation constraints, so it's
definitely detrimental to long term evolution.


        Stefan

bug#70784: Abolish string resizing

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Actually, it's not "immemorial", it's just old:
>
>     commit 3c9de1afcde82a99137721436c822059cce79b5b
>     Author: Kenichi Handa <handa@gnu.org>
>     Date:   Fri Jul 21 06:45:30 2000 +0000

That's _23_, approaching 24 years, in the past, or 1 generation exactly.
To put this into perspective, a mere 2 years and 7 months after the
introduction of multibyte strings.

>
>         (Faset): Allow storing any multibyte character in a string.  Convert
>         unibyte string to multibyte if necessary.
>
> IOW, since Emacs-21.1.

Only two releases after the distinction between multibyte and unibyte
strings was introduced.  Evidently, this departure from traditional
array behavior was seen as a misdesign at the time, and we should know
better than to second-guess old design decisions, especially those from
a past when computer memory and performance were far scarcer commodities
than today.

> As for improvements, like a lot of refactoring and maintenance work,
> there isn't any immediate benefit.  But it's a "feature" which is *very*
> rarely used (thank god: it makes a notionally constant time operation
> take time proportional to the size of the string, so if it were used
> often we would have heard complaints about the poor performance) and
> which imposes pretty significant implementation constraints, so it's
> definitely detrimental to long term evolution.

Placing quotation marks around the word "feature" does not make it any
less of a feature, and if reports of performance woes be the judge of
whether a feature is sufficiently disused to justify removal, then, why,
virtually all (vastly underreported) sub-optimal code in Emacs would be
eligible for immediate deletion attended by irritating pop-up warnings.

bug#70784: Abolish string resizing

[Eli Zaretskii]

> Cc: Mattias Engdegård <mattias.engdegard@gmail.com>,
>  70784@debbugs.gnu.org
> Date: Sun, 05 May 2024 22:41:54 -0400
> From:  Stefan Monnier via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> >> The Elisp ability to resize strings is high-cost, low-benefit, so we
> >> should abolish it.
> > What is the improvement to be had by "abolishing" this immemorial
> > feature?
> 
> Actually, it's not "immemorial", it's just old:
> 
>     commit 3c9de1afcde82a99137721436c822059cce79b5b
>     Author: Kenichi Handa <handa@gnu.org>
>     Date:   Fri Jul 21 06:45:30 2000 +0000
> 
>         (Faset): Allow storing any multibyte character in a string.  Convert
>         unibyte string to multibyte if necessary.
> 
> IOW, since Emacs-21.1.

That's only half of the story, AFAIR.  The other half is that Emacs
before version 20 allowed aset with any string anywhere, because
characters back then were always unibyte.  We lost that with
introduction of MULE, and the above change by Kenichi restored the
lost functionality (which was evidently deemed important enough back
then).

bug#70784: Abolish string resizing

[Eli Zaretskii]

> Cc: 70784@debbugs.gnu.org, Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Mon, 06 May 2024 08:53:17 +0800
> From:  Po Lu via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> Is it only I who am tired of these proposals for complete upheavals
> that, somehow, Emacs has fared just fine without, for generations past?

If we "get tired" of hearing such proposals, how can we ensure any
significant progress in Emacs, which is a major contributor to its
viability for years to come?

It's okay to discuss such proposals and object to some or all of them,
provided you have good reasons for the objections, but suggestions to
block such proposals up front are counter-productive and will
eventually hamper Emacs's future, IMNSHO.

> It's precisely this attitude that begins to inspire thoughts of
> departure.  Backwards-compatibility is an obligation that cannot be
> evaded by means of warnings, which instead serve to annoy and antagonize
> users, whose only wish is that Emacs leave them in peace.

We are not going to abandon backward-compatibility considerations.
But refusing to discuss significant changes just because they have
compatibility issues is throwing the proverbial baby with the
bathwater.  Refusing changes is of course 110% backward-compatible,
but it has many disadvantages, to say the least.  Instead, we should
see how to keep compatibility, to the extent that we consider it
important, without blocking changes which could potentially help us
adopting new technologies and improving performance.

bug#70784: Abolish string resizing

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

> We are not going to abandon backward-compatibility considerations.
> But refusing to discuss significant changes just because they have
> compatibility issues is throwing the proverbial baby with the
> bathwater.  Refusing changes is of course 110% backward-compatible,
> but it has many disadvantages, to say the least.  Instead, we should
> see how to keep compatibility, to the extent that we consider it
> important, without blocking changes which could potentially help us
> adopting new technologies and improving performance.

These principles are no doubt valid in general, but please consider what
is the feature whose continued existence is being called into question!
`(aset string n foo)' has been possible and countenanced for ages, if
not forever, and, I trust, such behavior of a quarter-century's standing
is among the rudiments of a language that should not change without a
very compelling and immediate reason.  Theoretical or even demonstrable
performance wins, or the hoary old commonplace of facilitating the
adoption of new technologies, both to be found in the release notes of
proprietary programs everywhere, provide no such reason, simply as the
performance of strings has never been a source of user complaint.

Without such a plain justification and a clear strategy for evaluating
whether the results so produced meet expectations, there really is no
detriment in categorically dismissing proposals to alter them, until
such time, if ever, as these conditions are created.  The quantity of
user or package code that would be erratically broken is alone enough to
send shivers down one's spine.

bug#70784: Abolish string resizing

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: mattias.engdegard@gmail.com,  70784@debbugs.gnu.org,
>   monnier@iro.umontreal.ca
> Date: Mon, 06 May 2024 20:23:16 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > We are not going to abandon backward-compatibility considerations.
> > But refusing to discuss significant changes just because they have
> > compatibility issues is throwing the proverbial baby with the
> > bathwater.  Refusing changes is of course 110% backward-compatible,
> > but it has many disadvantages, to say the least.  Instead, we should
> > see how to keep compatibility, to the extent that we consider it
> > important, without blocking changes which could potentially help us
> > adopting new technologies and improving performance.
> 
> These principles are no doubt valid in general, but please consider what
> is the feature whose continued existence is being called into question!
> `(aset string n foo)' has been possible and countenanced for ages

Which is why this is not the feature that was suggested to be
abolished.  Please leave strawmen and red herrings out of this
discussion.  No one in their right mind will agree to removal of
'aset' for strings in general.

> the performance of strings has never been a source of user
> complaint.

You are very wrong.  String performance in Emacs is a known problem,
as evidenced by the fact that we recommend that Lisp programs use
buffers in preference to strings.

I'm not saying that going with this proposal will necessarily make the
problem less severe, just that your over-reaching argument is patently
incorrect.

> Without such a plain justification and a clear strategy for evaluating
> whether the results so produced meet expectations, there really is no
> detriment in categorically dismissing proposals to alter them, until
> such time, if ever, as these conditions are created.

No one said that we will accept this change based on performance
considerations without seeing some performance data.

bug#70784: Abolish string resizing

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Emacs Lisp is not a hot-rod where crucial, fundamental facilities are
  > dispensable in the face of a performance improvement 

That is valid.

There are various reasons why it is useful to be able to relocate the
contents of strings.  Changing the size of a string is just one of
them.

Even if we adopt MPS GC now, that doesn't imply we will never again
change to some other GC.  We have replaced GC in the past.  Maybe in
the future we will once again want to relocate string contents.



-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)