point moved despite save-excursion, after deleting/reinserting region

point moved despite save-excursion, after deleting/reinserting region

[Garreau, Alexandre]

Hi,

I spent some time on one or two functions that modified a org source
block, evaled it, and restored it as before the modification: after
narrowing, I saved it using buffer-substring, and restored it using
delete-region then insert what I saved.

However doing that my point ends either at the beginning or the end of
what I inserted, even after save-excursion is over.

How to fix that?  I fill the simple way of restoring changes, without
loosing what evaluation did print outside of narrow (nor exiting this
restriction so that to save the result of evaluation, undo, and insert
it back), is indeed to delete/reinsert everything.  It would be too
complex (though feasible) to store each individual change along with the
position and length of it, to restore it back afterwards, and I’m not
sure it’ll perfectly fix it the same.

Here the two functions (along with original kbd macro implementations,
not working because you can’t nest 0-prefix loops in macros):

#+BEGIN_SRC emacs-lisp
(fset 'src-arg
   [?\C-s ?: ?\C-m ?\C-? ?\C-  ?\C-\M-s ?\\ ?< ?\[ ?_ ?\[ ?: ?a ?l ?p ?h
   ?a ?: ?\] ?\] ?\[ ?_ ?0 ?- ?9 ?\[ ?: ?a ?l ?p ?h ?a ?: ?\] ?\] ?* ?\\
   ?> ?\C-m ?\C-u ?\M-x ?r ?e ?a ?d ?- ?v ?a ?r ?\C-m ?\C-u ?\C-x ?q
   ?\C-m])

(defun src-arg (&optional prefix bound)
  "Search until BOUND for argument to replace for a prompted value.

With 0 prefix argument, repeat until search fails."
  (interactive "p")
  (if (zerop prefix)
      (while (src-arg nil bound))
    (when-let ((search (search-forward ":" bound t)))
      (prog1 search
        (delete-char -1)
        (push-mark)
        (search-forward-regexp "\\<[_[:alpha:]][_0-9[:alpha:]]*\\>" bound)
        (activate-mark)
        (let ((var-value (read-string (format "Enter value for %s: " (match-string 0)))))
          (delete-region (mark) (point))
          (insert (sql-obj-string var-value))
          (pop-mark))))))

(fset 'src-args
   [?\C-x ?n ?b ?\M-> ?\C-  ?\M-< ?\M-w ?\C-n ?\C-u ?0 ?\M-x ?s ?r ?c ?-
   ?a ?r ?g return ?\C-c ?\C-c ?y ?e ?s return ?\M-< ?\C-y ?\C-  ?\M->
   ?\C-w ?\C-a ?\C-x ?n ?w])

(defun src-args (&optional prefix)
  ""
  (interactive "P")
  (save-mark-and-excursion ;; doesn’t work
    (org-narrow-to-block)
    (let* ((end (point-max))
           (orig (buffer-substring (goto-char (point-min)) end)))
      (widen) ;; optional, so that the user still see the rest of
              ;; buffer and isn’t like “OMG WHATS HAPPANING”
      (forward-line)
      (while (src-arg prefix end))
      (let (org-confirm-babel-evaluate)
        (org-ctrl-c-ctrl-c))
      (save-restriction       ;; these may be optional,
        (org-narrow-to-block) ;; see before
        (delete-region (point-min) (point-max))
        (insert orig)))))

#+END_SRC

Re: point moved despite save-excursion, after deleting/reinserting region

[Eli Zaretskii]

> From: "Garreau\, Alexandre" <galex-713@galex-713.eu>
> Date: Wed, 17 Oct 2018 03:43:16 +0200
> 
> I spent some time on one or two functions that modified a org source
> block, evaled it, and restored it as before the modification: after
> narrowing, I saved it using buffer-substring, and restored it using
> delete-region then insert what I saved.
> 
> However doing that my point ends either at the beginning or the end of
> what I inserted, even after save-excursion is over.
> 
> How to fix that?

Please explain what did you want the code to do instead, it's unclear
from your description.

Re: point moved despite save-excursion, after deleting/reinserting region

[Garreau, Alexandre]

On 2018-10-17 at 19:12, Eli Zaretskii wrote:
> Please explain what did you want the code to do instead, it's unclear
> from your description.

I’m not sure of what wasn’t clear so I’m how to reexplain but:

The point was, say, in the middle of the SRC block, and (as I said in
the subject) after the operation, was moved: at the end of the block.

This must be because I deleted a region of text (the whole SRC block)
then reinserted it (or rather, its original content, so that to restore
buffer as before), so the point is after the insertion.

However, I’d have expected save-excursion to put the point again where
it was inside the block.  So either it does save it as a buffer position
but garbled it when deleting region, either it does with a marker and
this very one wasn’t saved within the portion of text I later
reinserted…

Is there something such as `restore-markers'/`save-markers' that would
allow me to restore point after doing this operation?  Or better some
`save-buffer-content' like `save-excursion' but restoring the buffer
afterwards?

Re: point moved despite save-excursion, after deleting/reinserting region

[Eli Zaretskii]

> From: "Garreau\, Alexandre" <galex-713@galex-713.eu>
> Cc: help-gnu-emacs@gnu.org
> Date: Wed, 17 Oct 2018 22:13:28 +0200
> 
> The point was, say, in the middle of the SRC block, and (as I said in
> the subject) after the operation, was moved: at the end of the block.
> 
> This must be because I deleted a region of text (the whole SRC block)
> then reinserted it (or rather, its original content, so that to restore
> buffer as before), so the point is after the insertion.
> 
> However, I’d have expected save-excursion to put the point again where
> it was inside the block.

If that's what you want, why not record the position of point before
the delete/insert operation and restore it afterwards?

> Is there something such as `restore-markers'/`save-markers' that would
> allow me to restore point after doing this operation?  Or better some
> `save-buffer-content' like `save-excursion' but restoring the buffer
> afterwards?

When you delete text, markers pointing into that text cannot follow,
they get relocated to the beginning/end of the deletion/insertion.
What else can Emacs do, given that it doesn't know your future
intentions?

Re: point moved despite save-excursion, after deleting/reinserting region

[Garreau, Alexandre]

On 2018-10-18 at 05:38, Eli Zaretskii wrote:
>> Is there something such as `restore-markers'/`save-markers' that would
>> allow me to restore point after doing this operation?  Or better some
>> `save-buffer-content' like `save-excursion' but restoring the buffer
>> afterwards?
>
> When you delete text, markers pointing into that text cannot follow,
> they get relocated to the beginning/end of the deletion/insertion.
> What else can Emacs do, given that it doesn't know your future
> intentions?

I thought something such as `save-excursion' would do that, or that it
would be saved with the string, or something alike.

> If that's what you want, why not record the position of point before
> the delete/insert operation and restore it afterwards?

So is that the canonical way of doing? that feels like defeating the
purpose of `save-excursion' a little bit (I only wanted to produce a
result for this src-block), and as this is more complex behavior than
just “undoing backward” or anything part of core elisp, I feared
side-effects I wouldn’t be aware of: for instance tho, it means my
function will have the side-effects of moving all the markers (and maybe
other similar things) that were inside that portion of text: it feels
somewhat hardly like “restoring as before”.

Re: point moved despite save-excursion, after deleting/reinserting region

[Yuri Khan]

On Thu, Oct 18, 2018 at 4:51 PM Garreau, Alexandre
<galex-713@galex-713.eu> wrote:

> > When you delete text, markers pointing into that text cannot follow,
> > they get relocated to the beginning/end of the deletion/insertion.
> > What else can Emacs do, given that it doesn't know your future
> > intentions?
>
> I thought something such as `save-excursion' would do that, or that it
> would be saved with the string, or something alike.

Imagine if that was true: that save-excursion could somehow magically
restore point across deletion of a fragment of text and subsequent
re-insertion of the same fragment.

Now comes the next guy (me!) and says, what if I re-insert not the
same fragment but a slight modification of it? Maybe I wrapped it in a
pair of XML tags. Or removed a pair of XML tags. Anyway, plz restore
the point Doing The Right Thing, kthx.

How would the hypothetical implementation of save-excursion do that?

…

You’d probably want to make buffer-substring to return not only the
buffer substring but also a representation of all markers within. You
can’t put them into the string itself because that will confuse the
algorithms that do the modification. You could probably put it into
properties, but then those who use buffer-substring-no-properties
lose. Further, insert would scan the inserted fragment and restore
markers.

The practical answer to my position-preserving modifications is, do
them directly in the buffer. If the modifications do not span the
point, point will be preserved by save-excursion.


Now returning to your modify-evaluate-undo scenario. How important is
it to you that the evaluation happens in the original buffer? Maybe
you could copy the fragment to a temporary buffer, modify and evaluate
it there, have the temporary buffer killed for you? This way, the
original buffer content is unmodified and the point is unmoved.
Moreover, the undo history is untouched.

Re: point moved despite save-excursion, after deleting/reinserting region

[Garreau, Alexandre]

Le 18/10/2018 à 18h32, Yuri Khan a écrit :
> On Thu, Oct 18, 2018 at 4:51 PM Garreau, Alexandre
> <galex-713@galex-713.eu> wrote:
>
>> > When you delete text, markers pointing into that text cannot follow,
>> > they get relocated to the beginning/end of the deletion/insertion.
>> > What else can Emacs do, given that it doesn't know your future
>> > intentions?
>>
>> I thought something such as `save-excursion' would do that, or that it
>> would be saved with the string, or something alike.
>
> Imagine if that was true: that save-excursion could somehow magically
> restore point across deletion of a fragment of text and subsequent
> re-insertion of the same fragment.
>
> Now comes the next guy (me!) and says, what if I re-insert not the
> same fragment but a slight modification of it? Maybe I wrapped it in a
> pair of XML tags. Or removed a pair of XML tags. Anyway, plz restore
> the point Doing The Right Thing, kthx.
>
> How would the hypothetical implementation of save-excursion do that?
>
> …

Yes, if that modification comes from the same string.

> You’d probably want to make buffer-substring to return not only the
> buffer substring but also a representation of all markers within. You
> can’t put them into the string itself because that will confuse the
> algorithms that do the modification. You could probably put it into
> properties, but then those who use buffer-substring-no-properties
> lose. Further, insert would scan the inserted fragment and restore
> markers.

I’d say a function such as buffer-substring-no-properties clearly
indicates you *want* to loose/clear all associated metadata.

Though I’m not sure it may be properly done with text properties: aren’t
these made for storing metadata about *interval* of text, rather than
points? would something such as “#("text" 2 2 plist)” be appropriate?
here it is just taken away at evaluation, and anyway it looks redundant
as well.

> The practical answer to my position-preserving modifications is, do
> them directly in the buffer. If the modifications do not span the
> point, point will be preserved by save-excursion.
>
> Now returning to your modify-evaluate-undo scenario. How important is
> it to you that the evaluation happens in the original buffer? Maybe
> you could copy the fragment to a temporary buffer, modify and evaluate
> it there, have the temporary buffer killed for you? This way, the
> original buffer content is unmodified and the point is unmoved.
> Moreover, the undo history is untouched.

The problem is doing that elsewhere might have other side effects as
well: for instance a source block might use definitions of some other
source block or part of the org-file, or refer to some file, etc. and
anyway copying it in a new buffer, then parsing what its output is, then
putting it in the original buffer, is a lot more complex and error
prone.

The other alternative, of copying the whole widened buffer in a new
buffer, seems overkill, and may, I guess, still cause issues: if there
is a relative path affecting the evaluation, while the
“current-directory” of a such buffer, with no buffer-file, be guaranteed
the same?

Re: point moved despite save-excursion, after deleting/reinserting region

[Eli Zaretskii]

> From: "Garreau\, Alexandre" <galex-713@galex-713.eu>
> Cc: help-gnu-emacs@gnu.org
> Date: Thu, 18 Oct 2018 11:50:40 +0200
> 
> > When you delete text, markers pointing into that text cannot follow,
> > they get relocated to the beginning/end of the deletion/insertion.
> > What else can Emacs do, given that it doesn't know your future
> > intentions?
> 
> I thought something such as `save-excursion' would do that, or that it
> would be saved with the string, or something alike.

What save-excursion does in this case is documented in the ELisp
manual:

     *Warning:* Ordinary insertion of text adjacent to the saved point
  value relocates the saved value, just as it relocates all markers.  More
  precisely, the saved value is a marker with insertion type ‘nil’.  *Note
  Marker Insertion Types::.  Therefore, when the saved point value is
  restored, it normally comes before the inserted text.

> > If that's what you want, why not record the position of point before
> > the delete/insert operation and restore it afterwards?
> 
> So is that the canonical way of doing?

It's the most direct method, AFAIK.

> that feels like defeating the purpose of `save-excursion' a little
> bit

As you can see above, it does not.  save-excursion's main use case is
that you don't edit text around point.  Maybe you should try using
replace-buffer-contents instead (you'll have to copy text to a
temporary buffer and edit it there).

> (I only wanted to produce a result for this src-block), and as this
> is more complex behavior than just “undoing backward” or anything
> part of core elisp, I feared side-effects I wouldn’t be aware of:
> for instance tho, it means my function will have the side-effects of
> moving all the markers (and maybe other similar things) that were
> inside that portion of text: it feels somewhat hardly like
> “restoring as before”.

You should be aware of the fact that point is also a marker, or at
least behaves like one.

Saving (and finding) markers (Was: Re: point moved despite save-excursion, after deleting/reinserting region)

[Garreau, Alexandre]

I’m unaware of what other side-effects might happens because of deleting
and reinserting a region of text, but how, within a region, get all the
markers? so to restore them afterwards.

I guess these must be updated somehow when anything is changed, so there
must be a list of them somewhere, thereas allowing to fetch them
according a given region.

I’d like to try this method as it looks straightforward, I didn’t find
rightaway through describe-function nor manual how to achieve a such
thing, and `clone-buffer' doesn’t seem to work on a file-visiting buffer
(and anyway I’m expecting from org-mode to be complex enough so to
triggers unexpected behaviors and side effects if ever I trigger complex
stuff in another buffer than the original one, or with a different
file-name, etc.).

Re: Saving (and finding) markers (Was: Re: point moved despite save-excursion, after deleting/reinserting region)

[Eli Zaretskii]

> From: "Garreau\, Alexandre" <galex-713@galex-713.eu>
> Cc: help-gnu-emacs@gnu.org
> Date: Thu, 18 Oct 2018 20:08:05 +0200
> 
> I’m unaware of what other side-effects might happens because of deleting
> and reinserting a region of text, but how, within a region, get all the
> markers?

You can't, because it makes no sense to do that.  Markers exist so
that you could _avoid_ doing this stuff by hand as much as possible.
If you are going to move markers by hand, you might as well not use
markers at all, or stop caring for those which happened to be there.

In general, Emacs already does the best it could with markers, when
text around them is edited; you are unlikely to do better.

I pointed you to replace-buffer-contents function, because AFAIU in
your situation it will modify the text while keeping as many markers
(including point) intact as possible.  Did you try using it?

> I guess these must be updated somehow when anything is changed, so there
> must be a list of them somewhere, thereas allowing to fetch them
> according a given region.

We don't expose the list of buffer's markers to Lisp.  There's
buffer-has-marker-at, but it's just a predicate.

> I’d like to try this method as it looks straightforward

I _really_ suggest to try replace-buffer-contents first.  It should do
the job cleanly and efficiently in many cases, and I very much doubt
that you could do better.

Re: Saving (and finding) markers

[Garreau, Alexandre]

On 2018-10-18 at 21:32, Eli Zaretskii wrote:
>> From: "Garreau\, Alexandre" <galex-713@galex-713.eu>
>> Cc: help-gnu-emacs@gnu.org
>> Date: Thu, 18 Oct 2018 20:08:05 +0200
>> 
>> I’m unaware of what other side-effects might happens because of deleting
>> and reinserting a region of text, but how, within a region, get all the
>> markers?
>
> You can't, because it makes no sense to do that.  Markers exist so
> that you could _avoid_ doing this stuff by hand as much as possible.
> If you are going to move markers by hand, you might as well not use
> markers at all, or stop caring for those which happened to be there.
>
> In general, Emacs already does the best it could with markers, when
> text around them is edited; you are unlikely to do better.

So it’s not possible to elisp to twiddle with markers that way?  I find
a such inextensibility sad (even if it’s already great, everything
should be modifiable I believe), but okay.

>> I’d like to try this method as it looks straightforward
>
> I _really_ suggest to try replace-buffer-contents first.  It should do
> the job cleanly and efficiently in many cases, and I very much doubt
> that you could do better.

> I pointed you to replace-buffer-contents function, because AFAIU in
> your situation it will modify the text while keeping as many markers
> (including point) intact as possible.  Did you try using it?

I might then use a separate buffer so to store the original content,
without doing the computation (C-c C-c) on it, and restore it later in
original buffer?

Looking to online manual it seems to be what I want, except it’s
undefined on my GNU Emacs 25.1.1 (i686-pc-linux-gnu, GTK+ Version
3.22.11) of 2017-09-15, modified by Debian.  Must be in next version of
emacs.  Still didn’t compiled it (on this X60 laptop whose thermal paste
change I procrastinated since some weeks, it won’t happen soon).

Re: Saving (and finding) markers

[Eli Zaretskii]

> From: "Garreau\, Alexandre" <galex-713@galex-713.eu>
> Cc: help-gnu-emacs@gnu.org
> Date: Thu, 18 Oct 2018 21:36:14 +0200
> 
> > I pointed you to replace-buffer-contents function, because AFAIU in
> > your situation it will modify the text while keeping as many markers
> > (including point) intact as possible.  Did you try using it?
> 
> I might then use a separate buffer so to store the original content,
> without doing the computation (C-c C-c) on it, and restore it later in
> original buffer?
> 
> Looking to online manual it seems to be what I want, except it’s
> undefined on my GNU Emacs 25.1.1

Indeed, it appeared in Emacs 26.

Re: Saving (and finding) markers

[Stefan Monnier]

> There's buffer-has-marker-at, but it's just a predicate.

It was introduced in 1996, has been marked obsolete since Emacs-24.3 and
I don't think anyone has managed to make use of it between those two ;-)


        Stefan

Re: Saving (and finding) markers

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Thu, 18 Oct 2018 21:15:51 -0400
> 
> > There's buffer-has-marker-at, but it's just a predicate.
> 
> It was introduced in 1996, has been marked obsolete since Emacs-24.3 and
> I don't think anyone has managed to make use of it between those two ;-)

And I can imagine why.