bug#35536: 27.0.50; Expose buffer's marker list to Elisp

bug#35536: 27.0.50; Expose buffer's marker list to Elisp

[Basil L. Contovounesios]

[-- Attachment #1: Type: text/plain, Size: 32 bytes --]

Severity: wishlist
Tags: patch


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Add-function-marker-list.patch --]
[-- Type: text/x-diff, Size: 10480 bytes --]

From c3c864a034ceb5c43fb791721a4b5c40f4122228 Mon Sep 17 00:00:00 2001
From: "Basil L. Contovounesios" <contovob@tcd.ie>
Date: Tue, 30 Apr 2019 00:17:02 +0100
Subject: [PATCH] Add function marker-list

Suggested by Martin Rudalics <rudalics@gmx.at> in:
https://debbugs.gnu.org/18#47
* doc/lispref/markers.texi (Marker Information): Rename from
'Information from Markers'.  All references changed.  Document
marker-list.
* etc/NEWS (Changes in Emacs 27.1): Announce marker-list.
* src/data.c (syms_of_data): Define Flss symbol for passing to
Fsort.
* src/marker.c (Fcopy_marker): Fix indentation.
(Fbuffer_has_markers_at): Coerce argument to a fixnum before passing
it to XFIXNUM.
(Fmarker_list): New function for listing markers in a given region.
(syms_of_marker): Define subr Smarker_list.
* test/src/marker-tests.el
(marker-set-window-start-from-other-buffer): Simplify.
(marker-list, marker-list-buffer-change): New tests.
---
 doc/lispref/elisp.texi   |  2 +-
 doc/lispref/markers.texi | 22 +++++++++++-------
 etc/NEWS                 |  5 ++++
 src/data.c               |  1 +
 src/marker.c             | 40 ++++++++++++++++++++++++++------
 test/src/marker-tests.el | 50 ++++++++++++++++++++++++++++++++++------
 6 files changed, 97 insertions(+), 23 deletions(-)

diff --git a/doc/lispref/elisp.texi b/doc/lispref/elisp.texi
index e18759654d..67c3e860aa 100644
--- a/doc/lispref/elisp.texi
+++ b/doc/lispref/elisp.texi
@@ -1168,7 +1168,7 @@ Top
 * Overview of Markers::     The components of a marker, and how it relocates.
 * Predicates on Markers::   Testing whether an object is a marker.
 * Creating Markers::        Making empty markers or markers at certain places.
-* Information from Markers::Finding the marker's buffer or character position.
+* Marker Information::      Finding the marker's buffer or character position.
 * Marker Insertion Types::  Two ways a marker can relocate when you
                               insert where it points.
 * Moving Markers::          Moving the marker to a new buffer or position.
diff --git a/doc/lispref/markers.texi b/doc/lispref/markers.texi
index 27fe1414f0..d18cad2eb8 100644
--- a/doc/lispref/markers.texi
+++ b/doc/lispref/markers.texi
@@ -16,7 +16,7 @@ Markers
 * Overview of Markers::      The components of a marker, and how it relocates.
 * Predicates on Markers::    Testing whether an object is a marker.
 * Creating Markers::         Making empty markers or markers at certain places.
-* Information from Markers:: Finding the marker's buffer or character position.
+* Marker Information::       Finding the marker's buffer or character position.
 * Marker Insertion Types::   Two ways a marker can relocate when you
                                insert where it points.
 * Moving Markers::           Moving the marker to a new buffer or position.
@@ -271,12 +271,12 @@ Creating Markers
 @end group
 @end example
 
-@node Information from Markers
-@section Information from Markers
+@node Marker Information
+@section Marker Information
 @cindex marker information
 
-  This section describes the functions for accessing the components of a
-marker object.
+Several functions return information about markers.  The next two
+functions access components of a marker object.
 
 @defun marker-position marker
 This function returns the position that @var{marker} points to, or
@@ -287,8 +287,6 @@ Information from Markers
 This function returns the buffer that @var{marker} points into, or
 @code{nil} if it points nowhere.
 
-@c FIXME: The 'buffer' argument of 'set-marker' already defaults to
-@c the current buffer, why use '(current-buffer)' explicitly here?
 @example
 @group
 (setq m (make-marker))
@@ -304,7 +302,7 @@ Information from Markers
 @end group
 
 @group
-(set-marker m 3770 (current-buffer))
+(set-marker m 3770)
      @result{} #<marker at 3770 in markers.texi>
 @end group
 @group
@@ -318,6 +316,14 @@ Information from Markers
 @end example
 @end defun
 
+@defun marker-list &optional beg end
+This function returns an ordered list of the markers in the accessible
+range of the current buffer.  If @var{beg} is non-@code{nil}, only
+markers pointing at that position are included.  If @var{end} is also
+non-@code{nil}, all markers in the region @var{beg} through @var{end}
+are included.
+@end defun
+
 @node Marker Insertion Types
 @section Marker Insertion Types
 
diff --git a/etc/NEWS b/etc/NEWS
index 9e3559d27e..dc5923a293 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -340,6 +340,11 @@ longer.
 ** Multicolor fonts such as "Noto Color Emoji" can be displayed on
 Emacs configured with Cairo drawing and linked with cairo >= 1.16.0.
 
++++
+** New function 'marker-list'.
+This function returns a list of markers in the current buffer within a
+given region.
+
 \f
 * Editing Changes in Emacs 27.1
 
diff --git a/src/data.c b/src/data.c
index 476d28eadb..0bf841e928 100644
--- a/src/data.c
+++ b/src/data.c
@@ -3852,6 +3852,7 @@ syms_of_data (void)
   DEFSYM (Qmany, "many");
 
   DEFSYM (Qcdr, "cdr");
+  DEFSYM (Qlss, "<");
 
   error_tail = pure_cons (Qerror, Qnil);
 
diff --git a/src/marker.c b/src/marker.c
index b58051a8c2..d132a43c09 100644
--- a/src/marker.c
+++ b/src/marker.c
@@ -712,7 +712,8 @@ see `marker-insertion-type'.  */)
   register Lisp_Object new;
 
   if (!NILP (marker))
-  CHECK_TYPE (FIXNUMP (marker) || MARKERP (marker), Qinteger_or_marker_p, marker);
+    CHECK_TYPE (FIXNUMP (marker) || MARKERP (marker),
+                Qinteger_or_marker_p, marker);
 
   new = Fmake_marker ();
   Fset_marker (new, marker,
@@ -749,18 +750,42 @@ DEFUN ("buffer-has-markers-at", Fbuffer_has_markers_at, Sbuffer_has_markers_at,
        doc: /* Return t if there are markers pointing at POSITION in the current buffer.  */)
   (Lisp_Object position)
 {
-  register struct Lisp_Marker *tail;
-  register ptrdiff_t charpos;
+  CHECK_FIXNUM_COERCE_MARKER (position);
+  ptrdiff_t charpos = clip_to_bounds (BEG, XFIXNUM (position), Z);
 
-  charpos = clip_to_bounds (BEG, XFIXNUM (position), Z);
-
-  for (tail = BUF_MARKERS (current_buffer); tail; tail = tail->next)
-    if (tail->charpos == charpos)
+  for (struct Lisp_Marker *m = BUF_MARKERS (current_buffer); m; m = m->next)
+    if (m->charpos == charpos)
       return Qt;
 
   return Qnil;
 }
 
+DEFUN ("marker-list", Fmarker_list, Smarker_list, 0, 2, 0,
+       doc: /* Return a list of markers in the accessible range of the buffer.
+If BEG is non-nil, include only markers pointing at that position.
+If END is also non-nil, include all markers in the region BEG
+through END.  */)
+  (Lisp_Object beg, Lisp_Object end)
+{
+  ptrdiff_t b, e;
+  if (NILP (beg))
+    b = BEGV, e = ZV;
+  else
+    {
+      validate_region (&beg, NILP (end) ? &beg : &end);
+      b = XFIXNAT (beg);
+      e = NILP (end) ? b : XFIXNAT (end);
+    }
+
+  Lisp_Object markers = Qnil;
+
+  for (struct Lisp_Marker *m = BUF_MARKERS (current_buffer); m; m = m->next)
+    if (b <= m->charpos && m->charpos <= e)
+      markers = Fcons (make_lisp_ptr (m, Lisp_Vectorlike), markers);
+
+  return Fsort (markers, Qlss);
+}
+
 #ifdef MARKER_DEBUG
 
 /* For debugging -- count the markers in buffer BUF.  */
@@ -807,4 +832,5 @@ syms_of_marker (void)
   defsubr (&Smarker_insertion_type);
   defsubr (&Sset_marker_insertion_type);
   defsubr (&Sbuffer_has_markers_at);
+  defsubr (&Smarker_list);
 }
diff --git a/test/src/marker-tests.el b/test/src/marker-tests.el
index 79e298d8c2..cbafabb556 100644
--- a/test/src/marker-tests.el
+++ b/test/src/marker-tests.el
@@ -29,16 +29,15 @@
 (ert-deftest marker-set-window-start-from-other-buffer ()
   "`set-window-start' from other buffer's marker."
   (let ((text-quoting-style 'curve))
-    (describe-function 'describe-function))
-  (let* ((help (get-buffer "*Help*"))
-         (marker (with-current-buffer help
-                   (copy-marker (point-max)))))
+    (describe-function #'describe-function))
+  (let ((marker (with-current-buffer "*Help*"
+                  (copy-marker (point-max)))))
     (should (set-window-start (selected-window) marker))))
 
 (ert-deftest marker-set-window-point-from-other-buffer ()
   "`set-window-point' from another buffer's marker."
   (let ((text-quoting-style 'curve))
-    (describe-function 'describe-function))
+    (describe-function #'describe-function))
   (let* ((help (get-buffer "*Help*"))
          (marker (with-current-buffer help
                    (copy-marker (point-max)))))
@@ -48,13 +47,50 @@
 (ert-deftest marker-goto-char-from-other-buffer ()
   "`goto-char' from another buffer's marker."
   (let ((text-quoting-style 'curve))
-    (describe-function 'describe-function))
+    (describe-function #'describe-function))
   (let ((marker-1 (make-marker))
         (marker-2 (make-marker)))
-    (describe-function 'describe-function)
+    (describe-function #'describe-function)
     (with-current-buffer "*Help*"
       (set-marker marker-1 (point-max)))
     (set-marker marker-2 marker-1)
     (should (goto-char marker-2))))
 
+(ert-deftest marker-list ()
+  "Test `marker-list' behavior."
+  (with-temp-buffer
+    ;; No markers created yet.
+    (should-not (marker-list))
+    (insert "first\nsecond\n")
+    (forward-line -1)
+    (let ((markers (list (point-min-marker) (point-max-marker))))
+      ;; Check marker arguments.
+      (should (equal (apply #'marker-list markers) markers))
+      (save-restriction
+        (narrow-to-region (point) (line-end-position))
+        ;; Check accessible range of buffer.
+        (should-not (marker-list))
+        ;; Check invalid region.
+        (should-error (apply #'marker-list markers)
+                      :type 'args-out-of-range)))
+    ;; Check single position and that mark is included.
+    (let ((marker (set-marker (mark-marker) (point))))
+      (should (equal (marker-list (point)) (list marker))))
+    ;; Check that unchained markers are not included.
+    (dolist (marker (marker-list))
+      (set-marker marker nil))
+    (should-not (marker-list))))
+
+(ert-deftest marker-list-buffer-change ()
+  "Test `marker-list' behavior across buffer changes."
+  (with-temp-buffer
+    (let ((marker  (point-marker))
+          (markers (marker-list)))
+      (with-temp-buffer
+        (set-marker marker (point))
+        (should (equal (marker-list) (list marker)))
+        (should (equal (mapcar #'marker-buffer markers)
+                       (list (current-buffer)))))
+      (should-not (marker-list)))))
+
 ;;; marker-tests.el ends here.
-- 
2.20.1


[-- Attachment #3: Type: text/plain, Size: 714 bytes --]


The question of listing a buffer's markers has been raised before:

https://debbugs.gnu.org/18#47
https://lists.gnu.org/archive/html/help-gnu-emacs/2016-06/msg00050.html
https://lists.gnu.org/archive/html/emacs-devel/2007-04/msg01391.html

I attach a patch implementing this based on BUF_MARKERS, as per Martin's
suggestion.  Any reasons not to expose such a function?

Thanks,

-- 
Basil

In GNU Emacs 27.0.50 (build 4, x86_64-pc-linux-gnu, X toolkit, Xaw3d scroll bars)
 of 2019-04-30 built on thunk
Repository revision: 910d170771ac74ab76d6dcb2dda3f3167e01b705
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12003000
System Description: Debian GNU/Linux buster/sid

bug#35536: 27.0.50; Expose buffer's marker list to Elisp

[Eli Zaretskii]

> From: "Basil L. Contovounesios" <contovob@tcd.ie>
> Date: Thu, 02 May 2019 16:44:52 +0100
> Cc: Mauro Aranda <maurooaranda@gmail.com>,
> 	Stefan Monnier <monnier@iro.umontreal.ca>
> 
> I attach a patch implementing this based on BUF_MARKERS, as per Martin's
> suggestion.  Any reasons not to expose such a function?

I'm not yet convinced we need something like that, but in any case, is
the order important?  Because the code you propose produces a list in
reverse order.

More generally, I think we should discuss the need for this in more
detail.  Markers are used for several features, and there's internal
stuff like conversion from character to byte positions that depends on
them.  Changing markers could thus easily crash Emacs, especially if
it comes in some in-opportune moment.

It is possible that people actually need higher-level primitives that
manipulate markers internally.  We should first identify the use cases
where this could be needed, and then see how to help solving those use
cases by something like a new marker-related primitive.

bug#35536: 27.0.50; Expose buffer's marker list to Elisp

[Basil L. Contovounesios]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: "Basil L. Contovounesios" <contovob@tcd.ie>
>> Date: Thu, 02 May 2019 16:44:52 +0100
>> Cc: Mauro Aranda <maurooaranda@gmail.com>,
>> 	Stefan Monnier <monnier@iro.umontreal.ca>
>> 
>> I attach a patch implementing this based on BUF_MARKERS, as per Martin's
>> suggestion.  Any reasons not to expose such a function?
>
> I'm not yet convinced we need something like that, but in any case, is
> the order important?  Because the code you propose produces a list in
> reverse order.

The order of the returned list is in increasing buffer position, thanks
to the call to Fsort.  Is that not a reasonable order?

> More generally, I think we should discuss the need for this in more
> detail.  Markers are used for several features, and there's internal
> stuff like conversion from character to byte positions that depends on
> them.  Changing markers could thus easily crash Emacs, especially if
> it comes in some in-opportune moment.

Are you saying that BUF_MARKERS could include markers created by
internal functions which could crash if these markers are changed across
calls to other Lisp functions?

If so, that sounds like a valid concern to a non-expert like me, but it
also sounds like a bug waiting to happen, given that other C code
also traverses and manipulates BUF_MARKERS.

If not, I don't see how manipulating markers returned by marker-list is
any worse than manipulating those created at the Lisp level, with the
usual and documented risks associated with manipulating markers not
owned by the caller.

> It is possible that people actually need higher-level primitives that
> manipulate markers internally.  We should first identify the use cases
> where this could be needed, and then see how to help solving those use
> cases by something like a new marker-related primitive.

I have yet to see a use-case for marker-list which can't be engineered
in a different way (other than as a replacement for the obsolete
buffer-has-markers-at, FWIW).  Perhaps some of the CCed parties might
have examples.

Thanks,

-- 
Basil

bug#35536: 27.0.50; Expose buffer's marker list to Elisp

[Eli Zaretskii]

> From: "Basil L. Contovounesios" <contovob@tcd.ie>
> Cc: <35536@debbugs.gnu.org>,  <maurooaranda@gmail.com>,  <monnier@iro.umontreal.ca>
> Date: Thu, 02 May 2019 17:51:12 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: "Basil L. Contovounesios" <contovob@tcd.ie>
> >> Date: Thu, 02 May 2019 16:44:52 +0100
> >> Cc: Mauro Aranda <maurooaranda@gmail.com>,
> >> 	Stefan Monnier <monnier@iro.umontreal.ca>
> >> 
> >> I attach a patch implementing this based on BUF_MARKERS, as per Martin's
> >> suggestion.  Any reasons not to expose such a function?
> >
> > I'm not yet convinced we need something like that, but in any case, is
> > the order important?  Because the code you propose produces a list in
> > reverse order.
> 
> The order of the returned list is in increasing buffer position, thanks
> to the call to Fsort.  Is that not a reasonable order?

Sorry, missed the sort.  The question whether the order matters still
stands, though.

> > More generally, I think we should discuss the need for this in more
> > detail.  Markers are used for several features, and there's internal
> > stuff like conversion from character to byte positions that depends on
> > them.  Changing markers could thus easily crash Emacs, especially if
> > it comes in some in-opportune moment.
> 
> Are you saying that BUF_MARKERS could include markers created by
> internal functions which could crash if these markers are changed across
> calls to other Lisp functions?

Please don't forget that nowadays we call Lisp from many places in C,
like from redisplay.  We need to be very careful with this because I;m
quite sure the display code doesn't expect markers to change at least
in some of its paths.

> If so, that sounds like a valid concern to a non-expert like me, but it
> also sounds like a bug waiting to happen, given that other C code
> also traverses and manipulates BUF_MARKERS.

Emacs being designed using the MVC pattern, assumes that the buffers
(and thus markers) don't change while they are being displayed.  It
has some probes for when this might happen as result of calling some
hook, and when that is detected, we restart redisplay.  I'm saying
that enlarging the potential for such changes will need careful
auditing of code that didn't expect such changes until now.  It will
also necessarily slow down redisplay.  The question is: is that worth
the hassle?  If what is needed is some higher-level features, then
exposing markers to Lisp will unnecessarily force us to do all that
non-trivial auditing.  So I suggest that we discuss the needs before
coding, to see whether such low-level access to a central data
structure is really needed and justified.

> If not, I don't see how manipulating markers returned by marker-list is
> any worse than manipulating those created at the Lisp level, with the
> usual and documented risks associated with manipulating markers not
> owned by the caller.

Just reading the markers probably won't but do you really believe this
is the last word?  How many hours will it take for someone to ask for
a primitive to set the C-level markers as well, or request the ability
to map a function over all the markers?  If it's really needed, sure,
let's do it.  But is it?  Or are we doing that just because we can?

> I have yet to see a use-case for marker-list which can't be engineered
> in a different way (other than as a replacement for the obsolete
> buffer-has-markers-at, FWIW).

Well, the discussions you cited did express requirements whose
implementation with the existing facilities was either inconvenient or
restricted.  If these problems are still relevant, then why not try
providing some primitives to help them?

IOW, let me turn the table and ask: why would a Lisp program want to
get a list of all the markers in a buffer, especially those not
created from Lisp?

bug#35536: 27.0.50; Expose buffer's marker list to Elisp

[Basil L. Contovounesios]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: "Basil L. Contovounesios" <contovob@tcd.ie>
>> Cc: <35536@debbugs.gnu.org>,  <maurooaranda@gmail.com>,  <monnier@iro.umontreal.ca>
>> Date: Thu, 02 May 2019 17:51:12 +0100
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> From: "Basil L. Contovounesios" <contovob@tcd.ie>
>> >> Date: Thu, 02 May 2019 16:44:52 +0100
>> >> Cc: Mauro Aranda <maurooaranda@gmail.com>,
>> >> 	Stefan Monnier <monnier@iro.umontreal.ca>
>> >> 
>> >> I attach a patch implementing this based on BUF_MARKERS, as per Martin's
>> >> suggestion.  Any reasons not to expose such a function?
>> >
>> > I'm not yet convinced we need something like that, but in any case, is
>> > the order important?  Because the code you propose produces a list in
>> > reverse order.
>> 
>> The order of the returned list is in increasing buffer position, thanks
>> to the call to Fsort.  Is that not a reasonable order?
>
> Sorry, missed the sort.  The question whether the order matters still
> stands, though.

When asked for a list of markers between BEG and END, it makes sense to
me to return a list which ascends from BEG to END.  If it really
matters, we could either return the order of BUF_MARKERS unchanged, or
accept an additional argument which tells the function how to sort.

>> > More generally, I think we should discuss the need for this in more
>> > detail.  Markers are used for several features, and there's internal
>> > stuff like conversion from character to byte positions that depends on
>> > them.  Changing markers could thus easily crash Emacs, especially if
>> > it comes in some in-opportune moment.
>> 
>> Are you saying that BUF_MARKERS could include markers created by
>> internal functions which could crash if these markers are changed across
>> calls to other Lisp functions?
>
> Please don't forget that nowadays we call Lisp from many places in C,
> like from redisplay.  We need to be very careful with this because I;m
> quite sure the display code doesn't expect markers to change at least
> in some of its paths.

Noted.

>> If so, that sounds like a valid concern to a non-expert like me, but it
>> also sounds like a bug waiting to happen, given that other C code
>> also traverses and manipulates BUF_MARKERS.
>
> Emacs being designed using the MVC pattern, assumes that the buffers
> (and thus markers) don't change while they are being displayed.  It
> has some probes for when this might happen as result of calling some
> hook, and when that is detected, we restart redisplay.  I'm saying
> that enlarging the potential for such changes will need careful
> auditing of code that didn't expect such changes until now.  It will
> also necessarily slow down redisplay.  The question is: is that worth
> the hassle?  If what is needed is some higher-level features, then
> exposing markers to Lisp will unnecessarily force us to do all that
> non-trivial auditing.  So I suggest that we discuss the needs before
> coding, to see whether such low-level access to a central data
> structure is really needed and justified.

Thanks for explaining, sounds perfectly reasonable.  I'm not convinced
adding marker-list is worth the trouble.  (FWIW, I'm not eager to
blindly expose marker-list; I just opened a ticket to see what
reservations there are, and to discuss this question I've seen pop up
before.  A bit of code always gives discussions a bit more ground, and
it was an excuse for me to read a bit of the surrounding code.)

>> If not, I don't see how manipulating markers returned by marker-list is
>> any worse than manipulating those created at the Lisp level, with the
>> usual and documented risks associated with manipulating markers not
>> owned by the caller.
>
> Just reading the markers probably won't but do you really believe this
> is the last word?  How many hours will it take for someone to ask for
> a primitive to set the C-level markers as well, or request the ability
> to map a function over all the markers?  If it's really needed, sure,
> let's do it.  But is it?  Or are we doing that just because we can?

So far, the latter.

>> I have yet to see a use-case for marker-list which can't be engineered
>> in a different way (other than as a replacement for the obsolete
>> buffer-has-markers-at, FWIW).
>
> Well, the discussions you cited did express requirements whose
> implementation with the existing facilities was either inconvenient or
> restricted.  If these problems are still relevant, then why not try
> providing some primitives to help them?

A save+restore primitive like the one you suggested in your other
message sounds like it might do the trick without having to expose a
buffer's marker list to Lisp.

> IOW, let me turn the table and ask: why would a Lisp program want to
> get a list of all the markers in a buffer, especially those not
> created from Lisp?

As I say above, I don't have any use-cases which specifically need to
expose a buffer's marker list to Lisp, as opposed to using some other
approach.  The main call for marker-list in bug#18 could probably be
better solved with a different primitive.

Thanks,

-- 
Basil

bug#35536: 27.0.50; Expose buffer's marker list to Elisp

[Drew Adams]

> When asked for a list of markers between BEG and END, it makes sense to
> me to return a list which ascends from BEG to END.
                            ^^^^^^^^^^^^^^^^^^^^^^^
IOW, in buffer-position order.

> If it really matters, we could either return the
> order of BUF_MARKERS unchanged,

Unchanged from what?

> or accept an additional argument which tells the
> function how to sort.

Have not really been following this thread, and
not weighing in on whether such a function is
needed or whether users need access to markers
created by C.

But as for the order of such a list: It's trivial
for users (any Lisp code) to sort by buffer position
or anything else, so why would the default order
be by buffer position?

What's _not_ available to users or Lisp code, I
think, is the order of marker creation or even the
order of last setting.  I'd think that
marker-creation order (either direction) would be
a better default sort order for this, no?

bug#35536: 27.0.50; Expose buffer's marker list to Elisp

[Basil L. Contovounesios]

Drew Adams <drew.adams@oracle.com> writes:

>> When asked for a list of markers between BEG and END, it makes sense to
>> me to return a list which ascends from BEG to END.
>                             ^^^^^^^^^^^^^^^^^^^^^^^
> IOW, in buffer-position order.

Yes.

>> If it really matters, we could either return the
>> order of BUF_MARKERS unchanged,
>
> Unchanged from what?

From the order returned by BUF_MARKERS, i.e. the internal chain of
markers pointing to the current buffer.  This order presumably reflects,
to an extent, the order in which markers were created/chained, but I'm
not sure about this.

>> or accept an additional argument which tells the
>> function how to sort.
>
> Have not really been following this thread, and
> not weighing in on whether such a function is
> needed or whether users need access to markers
> created by C.
>
> But as for the order of such a list: It's trivial
> for users (any Lisp code) to sort by buffer position
> or anything else, so why would the default order
> be by buffer position?

That is the order I would intuitively expect in any enumeration of a
partially ordered set of buffer artifacts in a given region, unless
otherwise stated.

What other order would make sense when talking about markers within a
given region?

> What's _not_ available to users or Lisp code, I
> think, is the order of marker creation or even the
> order of last setting.  I'd think that
> marker-creation order (either direction) would be
> a better default sort order for this, no?

Perhaps when enumerating markers pointing at a single position, yes.
But I think that ordering would make less sense when talking about
markers within a given region.  Assuming something like marker-list is
deemed a useful addition (which is not yet clear), perhaps there should
be two separate functions akin to overlays-in and overlays-at, with
different sorting options and/or default policies.

Thanks,

-- 
Basil

bug#35536: 27.0.50; Expose buffer's marker list to Elisp

[Drew Adams]

I don't want to belabor this, but I'll reply once
about this.

> > But as for the order of such a list: It's trivial
> > for users (any Lisp code) to sort by buffer position
> > or anything else, so why would the default order
> > be by buffer position?
> 
> That is the order I would intuitively expect in any enumeration of a
> partially ordered set of buffer artifacts in a given region, unless
> otherwise stated.
> 
> What other order would make sense when talking about markers within a
> given region?

I think the order that makes sense as the (only?) way
to get the set of markers (including those from C) is
an order that we cannot get otherwise.  Creation order
is one such otherwise-unavailable order.

Anyone can get the buffer-position order at any time.
Or it can be provided as a function, if that's deemed
important.

> > What's _not_ available to users or Lisp code, I
> > think, is the order of marker creation or even the
> > order of last setting.  I'd think that
> > marker-creation order (either direction) would be
> > a better default sort order for this, no?
> 
> Perhaps when enumerating markers pointing at a single position, yes.
> But I think that ordering would make less sense when talking about
> markers within a given region.  Assuming something like marker-list is
> deemed a useful addition (which is not yet clear), perhaps there should
> be two separate functions akin to overlays-in and overlays-at, with
> different sorting options and/or default policies.

It's not about what's most immediately useful for
most imagined uses of the markers in a given region.

It's about _somehow_ getting a set of markers
(wherever, whether within some buffer-position range
or not) in their order of creation (or modification).

Such relative time information is otherwise lost
completely.  The markers themselves contain position
information.  They do not contain time information.

Again, though, I'm not saying anything about whether
we need such a function at all.  I'm just suggesting
that if we provide it it should probably provide info
that is not otherwise obtainable.  Creation time is
one such bit of info.

bug#35536: 27.0.50; Expose buffer's marker list to Elisp

[Stefan Monnier]

>> What's _not_ available to users or Lisp code, I
>> think, is the order of marker creation or even the
>> order of last setting.

There's no reason for the C code to keep track of those things either.
So while the current implementation may happen to have the markers in
"creation order", it would be wrong to provide a primitive that promises
this ordering unless there's a *good* reason why we'd want to force
future Emacsen to also keep track of this info.  So far I don't see any
such reason.


        Stefan

bug#35536: 27.0.50; Expose buffer's marker list to Elisp

[Drew Adams]

> >> What's _not_ available to users or Lisp code, I
> >> think, is the order of marker creation or even the
> >> order of last setting.
> 
> There's no reason for the C code to keep track of those things either.
> So while the current implementation may happen to have the markers in
> "creation order", it would be wrong to provide a primitive that promises
> this ordering unless there's a *good* reason why we'd want to force
> future Emacsen to also keep track of this info.  So far I don't see any
> such reason.

Fair enough.  I was thinking it might be a freebie.

E.g., I was thinking that the list to be returned
would just be created by, say, pushing new markers
onto it.  If that were the case then that order
could be useful, whether or not one could _depend_
on it.

Whether or not one should be able to depend on such
an order is a separate question.

If guaranteeing to provide such a default order is
too restrictive (and I agree that it probably would
be), but if the current implementation did mostly
return the set in such an order, then I'd vote for
not reordering it and just saying nothing about the
return order.

Consider `buffer-list', for example.  The order is
generally useful but we say nothing much about it,
directly.  (But we do show, in (elisp) `Buffer List',
how you can reorder the underlying buffer list.)

We also point out there of saying that `buffer-list'
"is not an internal Emacs data structure, and
modifying it has no effect on the order of buffers."

bug#35536: 27.0.50; Expose buffer's marker list to Elisp

[Stefan Monnier]

> Whether or not one should be able to depend on such
> an order is a separate question.

I'm fine with a `markers-in` which returns markers in the order found in
the C code.  But the docstring should state clearly that you can't rely
on this ordering.

I expect many uses won't care about the ordering anyway, so we don't
need to bother `sort`ing, although the performance cost of `sort`ing will
likely always be negligible.


        Stefan

bug#35536: 27.0.50; Expose buffer's marker list to Elisp

[Drew Adams]

> > Whether or not one should be able to depend on such
> > an order is a separate question.
> 
> I'm fine with a `markers-in` which returns markers in the order found in
> the C code.  But the docstring should state clearly that you can't rely
> on this ordering.
> 
> I expect many uses won't care about the ordering anyway, so we don't
> need to bother `sort`ing, although the performance cost of `sort`ing will
> likely always be negligible.

Agreed on both counts.

bug#35536: 27.0.50; Expose buffer's marker list to Elisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I'm still opposed to making the internal list of markers accessible at all.
Markers can disappear from that list due to GC.

The only reason for having that list is so we can relocate the
markers.  It is not an interface that naturally should exist.

What exactly do people want to do with this list?
Let's look for another way.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#35536: 27.0.50; Expose buffer's marker list to Elisp

[Lars Ingebrigtsen]

Richard Stallman <rms@gnu.org> writes:

> I'm still opposed to making the internal list of markers accessible at all.
> Markers can disappear from that list due to GC.
>
> The only reason for having that list is so we can relocate the
> markers.  It is not an interface that naturally should exist.
>
> What exactly do people want to do with this list?
> Let's look for another way.

I think the "rough consensus" here (i.e., Eli and Richard) is that we
don't really want to expose the marker list.  (This was also discussed
in a different context a couple of weeks ago, and the conclusion was the
same there.)

So I'm closing this bug report.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#35536: 27.0.50; Expose buffer's marker list to Elisp

[Mauro Aranda]

[-- Attachment #1: Type: text/plain, Size: 1723 bytes --]

>>> I have yet to see a use-case for marker-list which can't be engineered
>>> in a different way (other than as a replacement for the obsolete
>>> buffer-has-markers-at, FWIW).
>>
>> Well, the discussions you cited did express requirements whose
>> implementation with the existing facilities was either inconvenient or
>> restricted.  If these problems are still relevant, then why not try
>> providing some primitives to help them?
>
> A save+restore primitive like the one you suggested in your other
> message sounds like it might do the trick without having to expose a
> buffer's marker list to Lisp.

Indeed.  I thought Martin was talking about something like this in his post
in bug#18.  Given a region where text is going to be replaced, save the
positions of markers that would be affected because of the delete+insert,
and then restore them.

>> IOW, let me turn the table and ask: why would a Lisp program want to
>> get a list of all the markers in a buffer, especially those not
>> created from Lisp?
>
> As I say above, I don't have any use-cases which specifically need to
> expose a buffer's marker list to Lisp, as opposed to using some other
> approach.  The main call for marker-list in bug#18 could probably be
> better solved with a different primitive.
>

When I said I didn't find anything at the Lisp level to get the markers,
that didn't fully express my thoughts.  I didn't mean it as a call for a
function to get that information (and certainly, I don't see a use for
getting information about markers created internally).  What I meant was
that I thought about that way of restoring markers, but had no way of
working on it (at least not with my current knowledge of C).

Best regards,
Mauro.

[-- Attachment #2: Type: text/html, Size: 2137 bytes --]

bug#35536: 27.0.50; Expose buffer's marker list to Elisp

[martin rudalics]

 >> A save+restore primitive like the one you suggested in your other
 >> message sounds like it might do the trick without having to expose a
 >> buffer's marker list to Lisp.
 >
 > Indeed.  I thought Martin was talking about something like this in his post
 > in bug#18.  Given a region where text is going to be replaced, save the
 > positions of markers that would be affected because of the delete+insert,
 > and then restore them.

More precisely I would try to save the contexts of the old marker
positions similar to what we do with bookmarks and try to find these
contexts in the replaced region.  The important aspect is obviously
that this step can be skipped for regions left alone by compareseq (or
whatever was used) because in those (hopefully large) regions markers
are preserved automatically.  Some care would be needed for markers
that sit precisely at the bounds of these region and have the "wrong"
insertion type.  And maybe we should optionally give the cursor a
different color if it was not possible to restore it unequivocally.

The approach would be IMHO useful when reverting buffers after code
has been wrongly reindented or commented in or out.  I'm afraid that
it might not be overly useful for auto-reverted dired buffers and
buffers with many fine-grained modifications.

Which markers should be restored this way and whether the
corresponding code should be provided as a primitive or in Elisp is a
decision I leave to knowledgeable people.  I'd be already happy if
'point-marker' were handled that way.

martin

bug#35536: 27.0.50; Expose buffer's marker list to Elisp

[Stefan Monnier]

> I attach a patch implementing this based on BUF_MARKERS, as per Martin's
> suggestion.  Any reasons not to expose such a function?

AFAIK the main reason for such a function is so that you can implement
"replace" functions which preserves markers better than "insert+delete"
does, right?  Arguably `replace-buffer-contents` reduced this need, but
this is just one way to "guess" how to preserve the markers and for
specific replacements there are surely other approaches which would
work better, hence the desire to get access to the marker-list to write
ad-hoc solutions.

Random thoughts:
- I wouldn't expose a `(marker-list)` function but rather `(markers-in
  BEG END)` so you're not bothered by unrelated markers outside of
  the region of interest.
- The main problem I see is that some of the markers in BUF_MARKERS are
  "proper" markers, while others are just the markers that we happen to
  use in the current internal representation of overlays.
  If you can get your hands on those markers, you might end up breaking
  some invariants on which the C code relies (e.g. place the
  overlay-start after the overlay-end, or in a different buffer).
- I think the serious risks (e.g. crashes) are solvable.  E.g. there's
  room for an additional boolean field `lisp_marker` which could be used
  to distinguish those markers which can be safely returned (because
  they're normal Lisp-level markers already accessible from Lisp anyway)
  from the internal ones (such as those from overlays).
- Then we'd probably want to discussion whether markers used within
  `save-excursion` and friends should be marked as `lisp_marker` or not.

This said, as you say later:
> I have yet to see a use-case for marker-list which can't be engineered
> in a different way

So, whether it's worth the trouble: I don't know.


        Stefan

bug#35536: 27.0.50; Expose buffer's marker list to Elisp

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Thu, 02 May 2019 15:59:38 -0400
> Cc: 35536@debbugs.gnu.org, Mauro Aranda <maurooaranda@gmail.com>
> 
> AFAIK the main reason for such a function is so that you can implement
> "replace" functions which preserves markers better than "insert+delete"
> does, right?

If this is the use case, perhaps we could provide a function to save
markers and then restore them, similar to current-window-configuration
and set-window-configuration, but with opaque handles returned to the
caller.

bug#35536: 27.0.50; Expose buffer's marker list to Elisp

[Basil L. Contovounesios]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I attach a patch implementing this based on BUF_MARKERS, as per Martin's
>> suggestion.  Any reasons not to expose such a function?
>
> AFAIK the main reason for such a function is so that you can implement
> "replace" functions which preserves markers better than "insert+delete"
> does, right?

AIUI, yes.

> Random thoughts:
> - I wouldn't expose a `(marker-list)` function but rather `(markers-in
>   BEG END)` so you're not bothered by unrelated markers outside of
>   the region of interest.

The patch in the OP accepts optional BEG and END arguments for the
caller's typing convenience.

> - The main problem I see is that some of the markers in BUF_MARKERS are
>   "proper" markers, while others are just the markers that we happen to
>   use in the current internal representation of overlays.
>   If you can get your hands on those markers, you might end up breaking
>   some invariants on which the C code relies (e.g. place the
>   overlay-start after the overlay-end, or in a different buffer).
> - I think the serious risks (e.g. crashes) are solvable.  E.g. there's
>   room for an additional boolean field `lisp_marker` which could be used
>   to distinguish those markers which can be safely returned (because
>   they're normal Lisp-level markers already accessible from Lisp anyway)
>   from the internal ones (such as those from overlays).
> - Then we'd probably want to discussion whether markers used within
>   `save-excursion` and friends should be marked as `lisp_marker` or not.
>
> This said, as you say later:
>> I have yet to see a use-case for marker-list which can't be engineered
>> in a different way
>
> So, whether it's worth the trouble: I don't know.

Given a sufficiently sufficient save+restore primitive as per Eli's
suggestion, it's not looking worth the trouble.

Thanks,

-- 
Basil

bug#35536: 27.0.50; Expose buffer's marker list to Elisp

[Stefan Monnier]

> Given a sufficiently sufficient save+restore primitive as per Eli's
> suggestion, it's not looking worth the trouble.

FWIW, I think exposing some new opaque "set of markers" plus primitives
to manipulate such objects is likely a lot more work, so I think from
a purely pragmatic point of view, `markers-in` might be a better option
*assuming* we can make it safe-enough (i.e. it can cause weird behaviors
if mis-used but no hard-crashes, and not if used "reasonably", e.g. if
it's not used to change the markers's buffers nor their relative ordering).

In Emacs, we're generally pretty happy to provide tools powerful enough
to shoot yourself in the foot, as long as such mishaps only happen when
you're clearly using the tool in a dangerous way.


        Stefan