Is it ok to sort a list of overlays destructively?

Is it ok to sort a list of overlays destructively?

[Marcin Borkowski]

Hi all,

I need to sort the list of overlays returned by `overlays-in'.  The docs
for that function do not says explicitly that it creates a new list
every time, although a cursory glance at its source says that it
apparently does.  (Also,

(eq (overlays-in (point-min) (point-max)) (overlays-in (point-min) (point-max)))

is nil also when there are overlays in the buffer, so it seems to be the
case.)  OTOH, if creating a new list every time isn't in the docs, one
shouldn't rely on it, no?

So, is it a good practice to

(sort (overlays-in (point-min) (point-max))
      (lambda (o1 o2)
	(< (overlay-start o1) (overlay-start o2))))

?

TIA,

-- 
Marcin Borkowski
http://mbork.pl

Re: Is it ok to sort a list of overlays destructively?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 909 bytes --]

On Tue, Jul 27, 2021 at 07:22:42AM +0200, Marcin Borkowski wrote:
> Hi all,
> 
> I need to sort the list of overlays returned by `overlays-in'.  The docs
> for that function do not says explicitly that it creates a new list
> every time, although a cursory glance at its source says that it
> apparently does.  (Also,
> 
> (eq (overlays-in (point-min) (point-max)) (overlays-in (point-min) (point-max)))
> 
> is nil also when there are overlays in the buffer, so it seems to be the
> case.)  OTOH, if creating a new list every time isn't in the docs, one
> shouldn't rely on it, no?
> 
> So, is it a good practice to
> 
> (sort (overlays-in (point-min) (point-max))
>       (lambda (o1 o2)
> 	(< (overlay-start o1) (overlay-start o2))))
> 
> ?

Heh. Good question. I wouldn't, unless the docs said explicitly that it
makes a fresh list.

But perhaps I'm too timid :)

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Is it ok to sort a list of overlays destructively?

[Marcin Borkowski]

On 2021-07-27, at 11:40, tomas@tuxteam.de wrote:

> On Tue, Jul 27, 2021 at 07:22:42AM +0200, Marcin Borkowski wrote:
>> Hi all,
>> 
>> I need to sort the list of overlays returned by `overlays-in'.  The docs
>> for that function do not says explicitly that it creates a new list
>> every time, although a cursory glance at its source says that it
>> apparently does.  (Also,
>> 
>> (eq (overlays-in (point-min) (point-max)) (overlays-in (point-min) (point-max)))
>> 
>> is nil also when there are overlays in the buffer, so it seems to be the
>> case.)  OTOH, if creating a new list every time isn't in the docs, one
>> shouldn't rely on it, no?
>> 
>> So, is it a good practice to
>> 
>> (sort (overlays-in (point-min) (point-max))
>>       (lambda (o1 o2)
>> 	(< (overlay-start o1) (overlay-start o2))))
>> 
>> ?
>
> Heh. Good question. I wouldn't, unless the docs said explicitly that it
> makes a fresh list.

Precisely my attitude, but I was curious if it's a good one or only my
OCTD kicking in.

Best,

-- 
Marcin Borkowski
http://mbork.pl

Re: Is it ok to sort a list of overlays destructively?

[Eli Zaretskii]

> From: Marcin Borkowski <mbork@mbork.pl>
> Date: Tue, 27 Jul 2021 07:22:42 +0200
> 
> I need to sort the list of overlays returned by `overlays-in'.  The docs
> for that function do not says explicitly that it creates a new list
> every time, although a cursory glance at its source says that it
> apparently does.

Yes, it's a list that is created specifically for this function, and
every time anew.  (Internally, overlays of a buffer aren't stored as a
single list, so returning anything but a copy would be impossible.)

The doc string kinda hints on that, since it talks about sorting the
list, which would be unthinkable if it were the internal storage.

Re: Is it ok to sort a list of overlays destructively?

[Marcin Borkowski]

On 2021-07-27, at 15:49, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Marcin Borkowski <mbork@mbork.pl>
>> Date: Tue, 27 Jul 2021 07:22:42 +0200
>> 
>> I need to sort the list of overlays returned by `overlays-in'.  The docs
>> for that function do not says explicitly that it creates a new list
>> every time, although a cursory glance at its source says that it
>> apparently does.
>
> Yes, it's a list that is created specifically for this function, and
> every time anew.  (Internally, overlays of a buffer aren't stored as a
> single list, so returning anything but a copy would be impossible.)

That's what I supposed - but I want to be sure I do not give bad advice
(relying on undocumented implementation details) - the reason for this
question was that I have code sorting a list of overlays in my book.

> The doc string kinda hints on that, since it talks about sorting the
> list, which would be unthinkable if it were the internal storage.

--8<---------------cut here---------------start------------->8---
Return a list of the overlays that overlap the region BEG ... END.
Overlap means that at least one character is contained within the overlay
and also contained within the specified region.
Empty overlays are included in the result if they are located at BEG,
between BEG and END, or at END provided END denotes the position at the
end of the buffer.
--8<---------------cut here---------------end--------------->8---

Where does it say anything about sorting?

-- 
Marcin Borkowski
http://mbork.pl

Re: Is it ok to sort a list of overlays destructively?

[Eli Zaretskii]

> From: Marcin Borkowski <mbork@mbork.pl>
> Cc: help-gnu-emacs@gnu.org
> Date: Tue, 27 Jul 2021 22:14:39 +0200
> 
> > The doc string kinda hints on that, since it talks about sorting the
> > list, which would be unthinkable if it were the internal storage.
> 
> --8<---------------cut here---------------start------------->8---
> Return a list of the overlays that overlap the region BEG ... END.
> Overlap means that at least one character is contained within the overlay
> and also contained within the specified region.
> Empty overlays are included in the result if they are located at BEG,
> between BEG and END, or at END provided END denotes the position at the
> end of the buffer.
> --8<---------------cut here---------------end--------------->8---
> 
> Where does it say anything about sorting?

Look at overlays-at.

Re: Is it ok to sort a list of overlays destructively?

[Marcin Borkowski]

On 2021-07-28, at 04:25, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Marcin Borkowski <mbork@mbork.pl>
>> Cc: help-gnu-emacs@gnu.org
>> Date: Tue, 27 Jul 2021 22:14:39 +0200
>> 
>> > The doc string kinda hints on that, since it talks about sorting the
>> > list, which would be unthinkable if it were the internal storage.
>> 
>> --8<---------------cut here---------------start------------->8---
>> Return a list of the overlays that overlap the region BEG ... END.
>> Overlap means that at least one character is contained within the overlay
>> and also contained within the specified region.
>> Empty overlays are included in the result if they are located at BEG,
>> between BEG and END, or at END provided END denotes the position at the
>> end of the buffer.
>> --8<---------------cut here---------------end--------------->8---
>> 
>> Where does it say anything about sorting?
>
> Look at overlays-at.

Ah, thanks.

Would it be a good idea to say explicitly in the docstrings of both
functions that their results may be safely modified destructively?  If
so, I can file a bug report.

TIA,

-- 
Marcin Borkowski
http://mbork.pl

Re: Is it ok to sort a list of overlays destructively?

[Eli Zaretskii]

> From: Marcin Borkowski <mbork@mbork.pl>
> Cc: help-gnu-emacs@gnu.org
> Date: Wed, 28 Jul 2021 08:06:24 +0200
> 
> Would it be a good idea to say explicitly in the docstrings of both
> functions that their results may be safely modified destructively?  If
> so, I can file a bug report.

My personal opinion is that stating this fact in the doc strings of
these APIs alone makes little sense: we have many similar APIs, so
will need to say the same for most or all of them, otherwise users
will become confused.

Basically, you should assume that Emacs protects itself where that is
needed, so if it handed you a list or some other data, and nothing in
the doc string warns you against doing something with that data, you
are free to do that, and if that causes unexpected results, there's
either a documentation bug or a code bug that needs to be fixed.

That said, if you still feel like filing a bug report, please do;
perhaps my is an odd-one-out opinion.

Re: Is it ok to sort a list of overlays destructively?

[John Yates]

On Wed, Jul 28, 2021 at 7:44 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> Basically, you should assume that Emacs protects itself where that is
> needed, so if it handed you a list or some other data, and nothing in
> the doc string warns you against doing something with that data, you
> are free to do that, and if that causes unexpected results, there's
> either a documentation bug or a code bug that needs to be fixed.

I am not sure if such a statement exists in any of the manuals. Should
it not be stated explicitly somewhere?

Re: Is it ok to sort a list of overlays destructively?

[Eli Zaretskii]

> From: John Yates <john@yates-sheets.org>
> Date: Thu, 29 Jul 2021 07:34:07 -0400
> Cc: Help Gnu Emacs mailing list <help-gnu-emacs@gnu.org>
> 
> On Wed, Jul 28, 2021 at 7:44 AM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > Basically, you should assume that Emacs protects itself where that is
> > needed, so if it handed you a list or some other data, and nothing in
> > the doc string warns you against doing something with that data, you
> > are free to do that, and if that causes unexpected results, there's
> > either a documentation bug or a code bug that needs to be fixed.
> 
> I am not sure if such a statement exists in any of the manuals. Should
> it not be stated explicitly somewhere?

I don't think it should be, no.  It's pretty much obvious, IMO.

Re: Is it ok to sort a list of overlays destructively?

[2QdxY4RzWzUUiLuE]

On 2021-07-29 at 15:33:23 +0300,
Eli Zaretskii <eliz@gnu.org> wrote:

> > From: John Yates <john@yates-sheets.org>
> > Date: Thu, 29 Jul 2021 07:34:07 -0400
> > Cc: Help Gnu Emacs mailing list <help-gnu-emacs@gnu.org>
> > 
> > On Wed, Jul 28, 2021 at 7:44 AM Eli Zaretskii <eliz@gnu.org> wrote:
> > >
> > > Basically, you should assume that Emacs protects itself where that is
> > > needed, so if it handed you a list or some other data, and nothing in
> > > the doc string warns you against doing something with that data, you
> > > are free to do that, and if that causes unexpected results, there's
> > > either a documentation bug or a code bug that needs to be fixed.
> > 
> > I am not sure if such a statement exists in any of the manuals. Should
> > it not be stated explicitly somewhere?
> 
> I don't think it should be, no.  It's pretty much obvious, IMO.

I'm not disagreeing, but why is it obvious?  Not all software works that
way (it would be great if it did), whether the authors thought about it
at all and/or intended it to work one way or the other, or not.

Re: Is it ok to sort a list of overlays destructively?

[Eli Zaretskii]

> Date: Thu, 29 Jul 2021 08:11:46 -0700
> From: 2QdxY4RzWzUUiLuE@potatochowder.com
> 
> > > I am not sure if such a statement exists in any of the manuals. Should
> > > it not be stated explicitly somewhere?
> > 
> > I don't think it should be, no.  It's pretty much obvious, IMO.
> 
> I'm not disagreeing, but why is it obvious?

Because anything else is a bug, for the same reason that you
rightfully expect Emacs not to crash when some Lisp program does
something silly.

Re: Is it ok to sort a list of overlays destructively?

[2QdxY4RzWzUUiLuE]

On 2021-07-29 at 18:30:49 +0300,
Eli Zaretskii <eliz@gnu.org> wrote:

> > Date: Thu, 29 Jul 2021 08:11:46 -0700
> > From: 2QdxY4RzWzUUiLuE@potatochowder.com
> > 
> > > > I am not sure if such a statement exists in any of the manuals. Should
> > > > it not be stated explicitly somewhere?
> > > 
> > > I don't think it should be, no.  It's pretty much obvious, IMO.
> > 
> > I'm not disagreeing, but why is it obvious?
> 
> Because anything else is a bug, for the same reason that you
> rightfully expect Emacs not to crash when some Lisp program does
> something silly.

That's very thought provoking.  Thank you.

If I believe (and maybe I'm old!) that "Emacs is the extensible,
customizable, self-documenting, real-time display editor," then I
wouldn't be surprised at all if some Lisp program caused Emacs to crash.
After all, those Lisp programs have [a great deal of] access to Emacs'
internals.

I don't realize the expectation you mentioned unless/until I think of
Emacs as platform of some kind, upon which I build a program.  To that
end, https://www.gnu.org/software/emacs/ does now claim that "[a]t its
core is an interpreter for Emacs Lisp, a dialect of the Lisp programming
language with extensions to support text editing."

I'm definitely closer to "it's obvious" than I was earlier.  :-)

My descriptionb of overlays-in (I don't have the C source code handy,
only what appears when I run describe-function in Emacs 27.2) matches
Marcin's and doesn't describe sorting.

Re: Is it ok to sort a list of overlays destructively?

[Eli Zaretskii]

> Date: Thu, 29 Jul 2021 09:20:21 -0700
> From: 2QdxY4RzWzUUiLuE@potatochowder.com
> 
> > Because anything else is a bug, for the same reason that you
> > rightfully expect Emacs not to crash when some Lisp program does
> > something silly.
> 
> That's very thought provoking.  Thank you.
> 
> If I believe (and maybe I'm old!) that "Emacs is the extensible,
> customizable, self-documenting, real-time display editor," then I
> wouldn't be surprised at all if some Lisp program caused Emacs to crash.
> After all, those Lisp programs have [a great deal of] access to Emacs'
> internals.

Access to the internals is protected by validity checks that signal
Lisp-level errors instead of letting Emacs crash.  Emacs crashing is a
bad problem because you lose your editing session, so Emacs strives to
avoid crashing.

About the only thing I know of that a Lisp program can do to crash
Emacs is to trigger infinite recursion, and even that has protection
in Emacs, albeit not 100% reliable.

Re: Is it ok to sort a list of overlays destructively?

[Marcin Borkowski]

On 2021-07-29, at 14:33, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: John Yates <john@yates-sheets.org>
>> Date: Thu, 29 Jul 2021 07:34:07 -0400
>> Cc: Help Gnu Emacs mailing list <help-gnu-emacs@gnu.org>
>> 
>> On Wed, Jul 28, 2021 at 7:44 AM Eli Zaretskii <eliz@gnu.org> wrote:
>> >
>> > Basically, you should assume that Emacs protects itself where that is
>> > needed, so if it handed you a list or some other data, and nothing in
>> > the doc string warns you against doing something with that data, you
>> > are free to do that, and if that causes unexpected results, there's
>> > either a documentation bug or a code bug that needs to be fixed.
>> 
>> I am not sure if such a statement exists in any of the manuals. Should
>> it not be stated explicitly somewhere?
>
> I don't think it should be, no.  It's pretty much obvious, IMO.

Eli,

thank you very much for all your input here.  After so many years with
Emacs and Elisp, I still learn a lot!

Thanks,

-- 
Marcin Borkowski
http://mbork.pl