bug#17893: 24.4.50; (error "Marker does not point anywhere")

bug#17893: 24.4.50; (error "Marker does not point anywhere")

[Drew Adams]

No idea what caused this error (I have no code that calls
`clear-transient-map', and here it seems to have been called somehow at
top level (?)), but here is the backtrace, FWIW:

Debugger entered--Lisp error: (error "Marker does not point anywhere")
  pop-mark()
  #[...]()
  #[...]()
  funcall(#[...])
  clear-transient-map()

I was in Info at the time.


In GNU Emacs 24.4.50.1 (i686-pc-mingw32)
 of 2014-06-28 on ODIEONE
Bzr revision: 117431 rgm@gnu.org-20140628015517-eku6hj8mpgcvfnso
Windowing system distributor `Microsoft Corp.', version 6.1.7601
Configured using:
 `configure --prefix=/c/Devel/emacs/snapshot/trunk
 --enable-checking=yes,glyphs 'CFLAGS=-O0 -g3'
 LDFLAGS=-Lc:/Devel/emacs/lib 'CPPFLAGS=-DGC_MCHECK=1
 -Ic:/Devel/emacs/include''

bug#17893: 24.4.50; (error "Marker does not point anywhere")

[Stefan Monnier]

> No idea what caused this error (I have no code that calls
> `clear-transient-map', and here it seems to have been called somehow at
> top level (?)), but here is the backtrace, FWIW:

> Debugger entered--Lisp error: (error "Marker does not point anywhere")
>   pop-mark()
>   #[...]()
>   #[...]()
>   funcall(#[...])
>   clear-transient-map()

That's probably the pop-mark that's in mouse-drag-track.  It has
a `push-mark' before, so naively it seems like the mark should be set.

> I was in Info at the time.

I think we need more data to figure out why the mark isn't set and hence
what to do about it.  A reproducible test case would obviously be best,
but otherwise maybe some indication of what kind of mouse gesture you
did, or what kind of customizations you might have that could interact
with that code and could unset the mark (maybe from
a pre/post-command-hook?).


        Stefan

bug#17893: 24.4.50; (error "Marker does not point anywhere")

[Drew Adams]

> > I was in Info at the time.
> 
> I think we need more data to figure out why the mark isn't set and hence
> what to do about it.  A reproducible test case would obviously be best,
> but otherwise maybe some indication of what kind of mouse gesture you
> did, or what kind of customizations you might have that could interact
> with that code and could unset the mark (maybe from
> a pre/post-command-hook?).

Sorry; If I knew such things I would have mentioned them.  I was not
aware of making any mouse gestures or even using the mouse, IIRC.  But
I might have moved the mouse somehow; dunno.

bug#17893: 24.4.50; (error "Marker does not point anywhere")

[Stefan Monnier]

>> I think we need more data to figure out why the mark isn't set and hence
>> what to do about it.  A reproducible test case would obviously be best,
>> but otherwise maybe some indication of what kind of mouse gesture you
>> did, or what kind of customizations you might have that could interact
>> with that code and could unset the mark (maybe from
>> a pre/post-command-hook?).
> Sorry; If I knew such things I would have mentioned them.  I was not
> aware of making any mouse gestures or even using the mouse, IIRC.  But
> I might have moved the mouse somehow; dunno.

Then maybe it comes from something completely different.
The clear-temporary-map is called when we "exit" a set-temporary-map, so
it can also happen with things like C-u, or C-x z, or users of
`repeat'.  Not sure which ones use `pop-mark', tho.


        Stefan

bug#17893: 24.4.50; (error "Marker does not point anywhere")

[Drew Adams]

> >> I think we need more data to figure out why the mark isn't set and hence
> >> what to do about it.  A reproducible test case would obviously be best,
> >> but otherwise maybe some indication of what kind of mouse gesture you
> >> did, or what kind of customizations you might have that could interact
> >> with that code and could unset the mark (maybe from
> >> a pre/post-command-hook?).
> > Sorry; If I knew such things I would have mentioned them.  I was not
> > aware of making any mouse gestures or even using the mouse, IIRC.  But
> > I might have moved the mouse somehow; dunno.
> 
> Then maybe it comes from something completely different.
> The clear-temporary-map is called when we "exit" a set-temporary-map, so
> it can also happen with things like C-u, or C-x z, or users of
> `repeat'.  Not sure which ones use `pop-mark', tho.

Maybe.  No idea.  FWIW, there is no `set-temporary-map' in my own code.

Doesn't it seem odd that the backtrace would *start* with
`clear-temporary-map'?  That doesn't look much like it was initiated
by a user action, but perhaps some actions have no reflection in a
backtrace.

bug#17893: 24.4.50; (error "Marker does not point anywhere")

[Stefan Monnier]

> Doesn't it seem odd that the backtrace would *start* with
> `clear-temporary-map'?

No, it's run from a pre-command-hook.


        Stefan

bug#17893: 24.4.50; (error "Marker does not point anywhere")

[Drew Adams]

> > Doesn't it seem odd that the backtrace would *start* with
> > `clear-temporary-map'?
> 
> No, it's run from a pre-command-hook.

I see.  Would a feasible enhancement be for an error backtrace to
show which pre-command-hook function was invoked, and indicate that
that function was invoked from `pre-command-hook'?

bug#17893: 24.4.50; (error "Marker does not point anywhere")

[Stefan Monnier]

> I see.  Would a feasible enhancement be for an error backtrace to
> show which pre-command-hook function was invoked,

This is already the case.

> and indicate that that function was invoked from `pre-command-hook'?

That sounds fine, yes.  Patch welcome,


        Stefan

bug#17893: 24.4.50; (error "Marker does not point anywhere")

[Drew Adams]

> > I see.  Would a feasible enhancement be for an error backtrace to
> > show which pre-command-hook function was invoked,
> 
> This is already the case.

I see.  So something added `clear-temporary-map' to `pre-command-hook'
on the fly, I guess (since I don't see it there otherwise).

I guess that must be done in some C code, since I don't find any hits
when I grep for `clear-temporary' in the Emacs Lisp code.  (I don't
have the C code.)

Not very easy for a user to guess (or even discover) what's going on.

> > and indicate that that function was invoked from `pre-command-hook'?
> 
> That sounds fine, yes.

Then consider that enhancement request as part of this bug report.
(Or file an additional report for it, if you prefer that.)

bug#17893: 24.4.50; (error "Marker does not point anywhere")

[Stefan Monnier]

> Not very easy for a user to guess (or even discover) what's going on.

It's added by set-temporary-map (previously known as
set-temporary-overlay-map or something like that).


        Stefan

bug#17893: 24.4.50; (error "Marker does not point anywhere")

[Drew Adams]

> > Not very easy for a user to guess (or even discover) what's going on.
> 
> It's added by set-temporary-map (previously known as
> set-temporary-overlay-map or something like that).

Two errors in this (at least wrt the version I reported, a recent trunk build):

1. It's `clear-transient-map', not `clear-temporary-map'.
2. It's `set-temporary-overlay-map', not `set-temporary-map'.

#1 explains why grepping for `clear-temporary-map' gave no hits.
(Yes, I should have consulted my original report, which mentioned
`clear-transient-map', instead of your reply about `clear-temporary-map'.)

So there is no great difficulty for a user to find this, after all.

bug#17893: 24.4.50; (error "Marker does not point anywhere")

[Stephen Berman]

On Wed, 2 Jul 2014 09:08:42 -0700 (PDT) Drew Adams <drew.adams@oracle.com> wrote:

>> > I see.  Would a feasible enhancement be for an error backtrace to
>> > show which pre-command-hook function was invoked,
>> 
>> This is already the case.
>
> I see.  So something added `clear-temporary-map' to `pre-command-hook'
> on the fly, I guess (since I don't see it there otherwise).
>
> I guess that must be done in some C code, since I don't find any hits
> when I grep for `clear-temporary' in the Emacs Lisp code.  (I don't
> have the C code.)

It's actually called clear-transient-map ...

On Wed, 02 Jul 2014 14:24:25 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote:

>> Not very easy for a user to guess (or even discover) what's going on.
>
> It's added by set-temporary-map

... set-transient-map

Steve Berman

bug#17893: 24.4.50; (error "Marker does not point anywhere")

[Drew Adams]

> That's probably the pop-mark that's in mouse-drag-track.  It has
> a `push-mark' before, so naively it seems like the mark should be set.
> 
> > I was in Info at the time.
> 
> I think we need more data to figure out why the mark isn't set and hence
> what to do about it.  A reproducible test case would obviously be best,
> but otherwise maybe some indication of what kind of mouse gesture you
> did, or what kind of customizations you might have that could interact
> with that code and could unset the mark (maybe from
> a pre/post-command-hook?).

Happened again.  Here's a bit more info.  I did M-n in Info
and got the backtrace:

Debugger entered--Lisp error: (error "Marker does not point anywhere")
  pop-mark()
  #[0 "\303\x11\300\x12\304 \210\305 \207" [t track-mouse auto-hscroll-mode nil deactivate-mark pop-mark] 1 "\n\n(fn)"]()
  #[0 "\301\204\b...
  funcall(#[0 "\301\204\b...
  clear-transient-map()

[Really wish you would fix bug #6991.  Each report of a
backtrace requires manual editing.  Users should be able to
just copy & paste.]

And this is the tail end of C-h l, showing that I really didn't do much after M-n:

<switch-frame> <down-mouse-1>
<mouse-1> <help-echo> <down-mouse-1> <mouse-1> M-n
<help-echo> <down-mouse-2> <mouse-movement> <mouse-2>
<help-echo> <help-echo> <switch-frame> <switch-frame>
<down-mouse-1> <mouse-movement> <mouse-1> C-s <help-echo>
<down-mouse-1> <mouse-movement> <mouse-1> <help-echo>
<help-echo> <help-echo> C-h l

bug#17893: 24.4.50; (error "Marker does not point anywhere")

[Charles A. Roelli]

> Date: Tue, 15 Jul 2014 09:19:34 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> 
> > That's probably the pop-mark that's in mouse-drag-track.  It has
> > a `push-mark' before, so naively it seems like the mark should be set.
> > 
> > > I was in Info at the time.
> > 
> > I think we need more data to figure out why the mark isn't set and hence
> > what to do about it.  A reproducible test case would obviously be best,
> > but otherwise maybe some indication of what kind of mouse gesture you
> > did, or what kind of customizations you might have that could interact
> > with that code and could unset the mark (maybe from
> > a pre/post-command-hook?).
> 
> Happened again.  Here's a bit more info.  I did M-n in Info
> and got the backtrace:
> 
> Debugger entered--Lisp error: (error "Marker does not point anywhere")
>   pop-mark()
>   #[0 "\303\x11\300\x12\304 \210\305 \207" [t track-mouse auto-hscroll-mode nil deactivate-mark pop-mark] 1 "\n\n(fn)"]()
>   #[0 "\301\204\b...
>   funcall(#[0 "\301\204\b...
>   clear-transient-map()
> 
> [Really wish you would fix bug #6991.  Each report of a
> backtrace requires manual editing.  Users should be able to
> just copy & paste.]
> 
> And this is the tail end of C-h l, showing that I really didn't do much after M-n:
> 
> <switch-frame> <down-mouse-1>
> <mouse-1> <help-echo> <down-mouse-1> <mouse-1> M-n
> <help-echo> <down-mouse-2> <mouse-movement> <mouse-2>
> <help-echo> <help-echo> <switch-frame> <switch-frame>
> <down-mouse-1> <mouse-movement> <mouse-1> C-s <help-echo>
> <down-mouse-1> <mouse-movement> <mouse-1> <help-echo>
> <help-echo> <help-echo> C-h l

I recently triggered a similar backtrace after testing some of my own
code that I submitted in Bug#28852, and I suspect the root cause is
similar.  It can be reproduced as follows:

1. Apply the patch at https://debbugs.gnu.org/cgi/bugreport.cgi?bug=28852#8 to emacs-26.
2. src/emacs -q
3. C-x v D RET
4. Click twice with mouse-1, roughly in the middle of the window showing buffer *vc-diff*.
5. M-g c 326 RET
6. C-c C-c RET C-x C-s C-x o g

Now click again in the window showing *vc-diff* and the error is triggered:

  set-transient-map PCH: (error "Marker does not point anywhere")

After M-x toggle-debug-on-error RET and evaluating `pop-mark' manually
with C-M-x, I see this backtrace:

Debugger entered--Lisp error: (error "Marker does not point anywhere")
  +(0 #<marker in no buffer>)
  (set-marker (mark-marker) (+ 0 (car mark-ring)) (current-buffer))
  (progn (setq mark-ring (nconc mark-ring (list (copy-marker (mark-marker))))) (set-marker (mark-marker) (+ 0 (car mark-ring)) (current-buffer)) (move-marker (car mark-ring) nil) (if (null (mark t)) (ding)) (setq mark-ring (cdr mark-ring)))
  (if mark-ring (progn (setq mark-ring (nconc mark-ring (list (copy-marker (mark-marker))))) (set-marker (mark-marker) (+ 0 (car mark-ring)) (current-buffer)) (move-marker (car mark-ring) nil) (if (null (mark t)) (ding)) (setq mark-ring (cdr mark-ring))))
  pop-mark()
  #f(compiled-function () #<bytecode 0x4124aadd>)()
  #f(compiled-function () #<bytecode 0x4124ab01>)()
  clear-transient-map()

The error happens when the marker at the head of the mark ring no
longer points into a buffer.  In this case, we could either delete the
dead marker until we find a live one, and then pop that, or just not
pop any mark at all.

bug#17893: 24.4.50; (error "Marker does not point anywhere")

[Stefan Monnier]

> The error happens when the marker at the head of the mark ring no
> longer points into a buffer.

Hmm... mark-ring is buffer-local, so the marks in there should all point
to current-buffer.  Can you try and figure out why this is not the case?


        Stefan

bug#17893: 24.4.50; (error "Marker does not point anywhere")

[Charles A. Roelli]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Drew Adams <drew.adams@oracle.com>,  17893@debbugs.gnu.org
> Date: Sun, 12 Nov 2017 16:03:09 -0500
> 
> > The error happens when the marker at the head of the mark ring no
> > longer points into a buffer.
> 
> Hmm... mark-ring is buffer-local, so the marks in there should all point
> to current-buffer.  Can you try and figure out why this is not the case?

I was not careful to make sure that the mark-ring contains valid
markers, so it may be an error in my code.  On the other hand, looking
at the definition of `clone-buffer' (which my patch uses), I don't see
anything that would update the markers of the mark-ring in the newly
cloned buffer to point to the new buffer instead of the old one.
Maybe that is also a problem?  And it doesn't help that I find my code
incredibly hard to read one month on...

Drew's problem is probably related to clone-buffer, especially seeing
as he saw the issue right after having hit M-n in an Info mode buffer:

  M-n runs the command clone-buffer (found in Info-mode-map), which is
  an interactive compiled Lisp function in ‘simple.el’.

  It is bound to M-n, <menu-bar> <Info> <Clone Info buffer>.

  (clone-buffer &optional NEWNAME DISPLAY-FLAG)

Ah, turns out that's how you replicate this bug.

1. C-h i
2. Click/drag a few times in the *info* buffer to set some marks
3. M-n C-x o C-x k RET
4. Click in the remaining *info*<2> buffer:

  set-transient-map PCH: (error "Marker does not point anywhere")

bug#17893: 24.4.50; (error "Marker does not point anywhere")

[Stefan Monnier]

> I was not careful to make sure that the mark-ring contains valid
> markers, so it may be an error in my code.  On the other hand, looking
> at the definition of `clone-buffer' (which my patch uses), I don't see
> anything that would update the markers of the mark-ring in the newly
> cloned buffer to point to the new buffer instead of the old one.
> Maybe that is also a problem?

Yup, sounds like we have a bug in clone-buffer, at least.


        Stefan

bug#17893: 24.4.50; (error "Marker does not point anywhere")

[Charles A. Roelli]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Date: Tue, 14 Nov 2017 15:08:08 -0500
> 
> > I was not careful to make sure that the mark-ring contains valid
> > markers, so it may be an error in my code.  On the other hand, looking
> > at the definition of `clone-buffer' (which my patch uses), I don't see
> > anything that would update the markers of the mark-ring in the newly
> > cloned buffer to point to the new buffer instead of the old one.
> > Maybe that is also a problem?
> 
> Yup, sounds like we have a bug in clone-buffer, at least.
> 
> 
>         Stefan

Right-o, here's a fix with respect to the emacs-26 branch:

diff --git a/lisp/simple.el b/lisp/simple.el
index 7d47d0f..64b7bde 100644
--- a/lisp/simple.el
+++ b/lisp/simple.el
@@ -8518,6 +8518,15 @@ clone-buffer
 		(error nil)))
 	    lvars)
 
+      ;; Update markers in the copied mark ring to refer to the new buffer.
+      (setq mark-ring
+            (mapcar (lambda (m)
+                      (let ((new-marker))
+                        (setq new-marker (copy-marker m))
+                        (move-marker new-marker new-marker (current-buffer))
+                        new-marker))
+                    mark-ring))
+
       ;; Run any hooks (typically set up by the major mode
       ;; for cloning to work properly).
       (run-hooks 'clone-buffer-hook))

It fixes the minimal recipe to reproduce this bug:

> 1. C-h i
> 2. Click/drag a few times in the *info* buffer to set some marks
> 3. M-n C-x o C-x k RET
> 4. Click in the remaining *info*<2> buffer:
>
>   set-transient-map PCH: (error "Marker does not point anywhere")

bug#17893: 24.4.50; (error "Marker does not point anywhere")

[Eli Zaretskii]

> Date: Sun, 19 Nov 2017 20:31:06 +0100
> From: charles@aurox.ch (Charles A. Roelli)
> Cc: 17893@debbugs.gnu.org
> 
> > Yup, sounds like we have a bug in clone-buffer, at least.
> > 
> > 
> >         Stefan
> 
> Right-o, here's a fix with respect to the emacs-26 branch:
> 
> diff --git a/lisp/simple.el b/lisp/simple.el
> index 7d47d0f..64b7bde 100644
> --- a/lisp/simple.el
> +++ b/lisp/simple.el
> @@ -8518,6 +8518,15 @@ clone-buffer
>  		(error nil)))
>  	    lvars)
>  
> +      ;; Update markers in the copied mark ring to refer to the new buffer.
> +      (setq mark-ring
> +            (mapcar (lambda (m)
> +                      (let ((new-marker))
> +                        (setq new-marker (copy-marker m))
> +                        (move-marker new-marker new-marker (current-buffer))
> +                        new-marker))
> +                    mark-ring))
> +
>        ;; Run any hooks (typically set up by the major mode
>        ;; for cloning to work properly).
>        (run-hooks 'clone-buffer-hook))
> 
> It fixes the minimal recipe to reproduce this bug:

Bother: this is actually the tip of an iceberg.  Buffer local
variables that get cloned by clone-buffer could hold any number of
markers pointing into the original buffer.  E.g., info.el itself
maintains a per-buffer marker in Info-tag-table-marker; evaluate it
after M-n and see what it tells you.

For mark-ring, I think it's easier to just reset it to nil in the new
clone (it's arguably even more correct, since this is a fresh buffer),
but in general this is a ticking bomb, unless I'm missing something.
If I'm right, the only solution is to walk all the markers that point
to the parent buffer and clone them to point to the cloned buffer
(this has to be done in C).

bug#17893: 24.4.50; (error "Marker does not point anywhere")

[Stefan Monnier]

> Bother: this is actually the tip of an iceberg.  Buffer local
> variables that get cloned by clone-buffer could hold any number of
> markers pointing into the original buffer.

Indeed.

> E.g., info.el itself maintains a per-buffer marker in
> Info-tag-table-marker; evaluate it after M-n and see what it
> tells you.

And for that reason, clone-buffer runs `clone-buffer-hook`, which
Info-mode uses to do:

    (defun Info-clone-buffer ()
      (when (bufferp Info-tag-table-buffer)
        (setq Info-tag-table-buffer
              (with-current-buffer Info-tag-table-buffer (clone-buffer))))
      (let ((m Info-tag-table-marker))
        (when (markerp m)
          (setq Info-tag-table-marker
                (if (and (marker-position m) (bufferp Info-tag-table-buffer))
                    (with-current-buffer Info-tag-table-buffer
                      (copy-marker (marker-position m)))
                  (make-marker))))))

which I believe DTRT.

> For mark-ring, I think it's easier to just reset it to nil in the new
> clone (it's arguably even more correct, since this is a fresh buffer),

I didn't think about it when I saw Charles's patch, but yes resetting it
to nil would make sense.

> but in general this is a ticking bomb, unless I'm missing something.
> If I'm right, the only solution is to walk all the markers that point
> to the parent buffer and clone them to point to the cloned buffer
> (this has to be done in C).

This can't be done in C because after creating those new markers, where
would we store them?  E.g. cloning all the markers that are in
`mark-ring` would just create new markers but it would fail to create
a new list holding those markers, stored in the new buffer-local value
of `mark-ring` (walking the markers doesn't tell us that they're
referred to from `mark-ring`).

Hence clone-buffer-hook, which doesn't solve the problem in itself, but
makes it possible to fix it manually where needed.


        Stefan

bug#17893: 24.4.50; (error "Marker does not point anywhere")

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: charles@aurox.ch (Charles A. Roelli),  17893@debbugs.gnu.org
> Date: Mon, 20 Nov 2017 11:51:45 -0500
> 
> > E.g., info.el itself maintains a per-buffer marker in
> > Info-tag-table-marker; evaluate it after M-n and see what it
> > tells you.
> 
> And for that reason, clone-buffer runs `clone-buffer-hook`, which
> Info-mode uses to do:

If so, it doesn't work well enough:

 emacs -Q
 C-u C-h i elisp.info RET
 M-: Info-tag-table-marker RET
  => #<marker at 3901363 in *info*>
 M-n
 M-: Info-tag-table-marker RET
  => #<marker in no buffer>

> > but in general this is a ticking bomb, unless I'm missing something.
> > If I'm right, the only solution is to walk all the markers that point
> > to the parent buffer and clone them to point to the cloned buffer
> > (this has to be done in C).
> 
> This can't be done in C because after creating those new markers, where
> would we store them?  E.g. cloning all the markers that are in
> `mark-ring` would just create new markers but it would fail to create
> a new list holding those markers, stored in the new buffer-local value
> of `mark-ring` (walking the markers doesn't tell us that they're
> referred to from `mark-ring`).

I guess we will have to walk all the local variables and find markers
in them, like GC does.  Is there any other way?

> Hence clone-buffer-hook, which doesn't solve the problem in itself, but
> makes it possible to fix it manually where needed.

I think this kind of problems is impossible to solve from Lisp, as we
don't expose enough information about markers, and for a good reason.
For starters, that hook assumes that every mode knows exactly what
local variables could need special handling, but that's an illusion.
For example, the user could have set all kinds of local variables
behind the back of Emacs which hold markers.

As another data point, there are 6 users of clone-buffer in Emacs
core, and only one of them bothers to set up a clone-buffer-hook.

bug#17893: 24.4.50; (error "Marker does not point anywhere")

[Stefan Monnier]

>> > E.g., info.el itself maintains a per-buffer marker in
>> > Info-tag-table-marker; evaluate it after M-n and see what it
>> > tells you.
>> And for that reason, clone-buffer runs `clone-buffer-hook`, which
>> Info-mode uses to do:
> If so, it doesn't work well enough:
>  emacs -Q
>  C-u C-h i elisp.info RET
>  M-: Info-tag-table-marker RET
>   => #<marker at 3901363 in *info*>
>  M-n
>  M-: Info-tag-table-marker RET
>   => #<marker in no buffer>

Maybe there's a bug, indeed.  Does the above lead to undesired behavior?
[ It's been a while since I last looked at those things.  ]

> I guess we will have to walk all the local variables and find markers
> in them, like GC does.  Is there any other way?

I don't think it can be done reliably: whether a marker needs to be
cloned or not may depend on what the marker is used for; and some of the
markers that need to be cloned could be stored elsewhere than in
buffer-local vars.

And in order to clone such a marker, we'd need to clone all objects
found on the "spine" between the local var and the marker.  In some
cases such a "clone" is again difficult to do automatically and
reliably, because what needs to be done again will depend on semantic
details to which such a generic code at the C level doesn't have access.

>> Hence clone-buffer-hook, which doesn't solve the problem in itself, but
>> makes it possible to fix it manually where needed.
> I think this kind of problems is impossible to solve from Lisp, as we
> don't expose enough information about markers, and for a good reason.
> For starters, that hook assumes that every mode knows exactly what
> local variables could need special handling, but that's an illusion.

No.  It just assumes that whichever package owns a marker that needs
special treatment should register itself on that hook.  Maybe that's too
much to ask, but it's definitely not "impossible".

> For example, the user could have set all kinds of local variables
> behind the back of Emacs which hold markers.

Then it's his responsibility to add a function to clone-buffer-hook to
handle those (in the unlikely case that she actually cares about
interaction between those vars and clone-buffer, that is).

> As another data point, there are 6 users of clone-buffer in Emacs
> core, and only one of them bothers to set up a clone-buffer-hook.

Do these suffer from problems?  When I introduced clone-buffer
(motivated by Info-mode), I also introduced the Info-clone-buffer hook
function because it was simply indispensable to get it working.
But I remember using it in other cases where such a hook didn't
seem necessary.


        Stefan

bug#17893: 24.4.50; (error "Marker does not point anywhere")

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: charles@aurox.ch,  17893@debbugs.gnu.org
> Date: Mon, 20 Nov 2017 13:59:12 -0500
> 
> >  M-n
> >  M-: Info-tag-table-marker RET
> >   => #<marker in no buffer>
> 
> Maybe there's a bug, indeed.  Does the above lead to undesired behavior?

I didn't dig deep enough to find out.  But it's clear that such a
marker is useless at best.

> No.  It just assumes that whichever package owns a marker that needs
> special treatment should register itself on that hook.  Maybe that's too
> much to ask, but it's definitely not "impossible".

Registering a hook is easy, the problem is doing everything that's
needed in the hook.  Especially given that clone-buffer is not
documented, and therefore all these caveats are nowhere to be learned.

> > As another data point, there are 6 users of clone-buffer in Emacs
> > core, and only one of them bothers to set up a clone-buffer-hook.
> 
> Do these suffer from problems?

I don't know.  But markers are quite common, so I'd be surprised if
they didn't.

bug#17893: 24.4.50; (error "Marker does not point anywhere")

[Andreas Schwab]

On Nov 20 2017, Eli Zaretskii <eliz@gnu.org> wrote:

> If I'm right, the only solution is to walk all the markers that point
> to the parent buffer and clone them to point to the cloned buffer
> (this has to be done in C).

Those cloned markers will be unreferenced, thus useless, with nothing to
be gained.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#17893: 24.4.50; (error "Marker does not point anywhere")

[Charles A. Roelli]

> Bother: this is actually the tip of an iceberg.  Buffer local
> variables that get cloned by clone-buffer could hold any number of
> markers pointing into the original buffer.  E.g., info.el itself
> maintains a per-buffer marker in Info-tag-table-marker; evaluate it
> after M-n and see what it tells you.

> For mark-ring, I think it's easier to just reset it to nil in the new
> clone (it's arguably even more correct, since this is a fresh buffer),
> but in general this is a ticking bomb, unless I'm missing something.
> If I'm right, the only solution is to walk all the markers that point
> to the parent buffer and clone them to point to the cloned buffer
> (this has to be done in C).

By C, do you mean the function `clone-buffer'?

bug#17893: 24.4.50; (error "Marker does not point anywhere")

[Eli Zaretskii]

> Date: Mon, 20 Nov 2017 21:01:58 +0100
> From: charles@aurox.ch (Charles A. Roelli)
> CC: monnier@IRO.UMontreal.CA, 17893@debbugs.gnu.org
> 
> > For mark-ring, I think it's easier to just reset it to nil in the new
> > clone (it's arguably even more correct, since this is a fresh buffer),
> > but in general this is a ticking bomb, unless I'm missing something.
> > If I'm right, the only solution is to walk all the markers that point
> > to the parent buffer and clone them to point to the cloned buffer
> > (this has to be done in C).
> 
> By C, do you mean the function `clone-buffer'?

No, I meant what's in marker.c.

bug#17893: 24.4.50; (error "Marker does not point anywhere")

[Charles A. Roelli]

> Date: Mon, 20 Nov 2017 22:29:55 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> CC: monnier@IRO.UMontreal.CA, 17893@debbugs.gnu.org
> Reply-to: Eli Zaretskii <eliz@gnu.org>
> 
> > Date: Mon, 20 Nov 2017 21:01:58 +0100
> > From: charles@aurox.ch (Charles A. Roelli)
> > CC: monnier@IRO.UMontreal.CA, 17893@debbugs.gnu.org
> > 
> > > For mark-ring, I think it's easier to just reset it to nil in the new
> > > clone (it's arguably even more correct, since this is a fresh buffer),
> > > but in general this is a ticking bomb, unless I'm missing something.
> > > If I'm right, the only solution is to walk all the markers that point
> > > to the parent buffer and clone them to point to the cloned buffer
> > > (this has to be done in C).
> > 
> > By C, do you mean the function `clone-buffer'?
> 
> No, I meant what's in marker.c.

I suppose the main issue is fixed in master now:

> branch: master
> commit 197dd690112e8eef6457b16adbe6e2c1d801c849
> Date: Thu, 23 Nov 2017 13:33:34 -0500 (EST)
> Author: Stefan Monnier <monnier@iro.umontreal.ca>
> Commit: Stefan Monnier <monnier@iro.umontreal.ca>
>
>     * lisp/simple.el (clone-buffer): Adjust `mark-ring'
> ---
>  lisp/simple.el | 7 +++++--
>  1 file changed, 5 insertions(+), 2 deletions(-)
>
> diff --git a/lisp/simple.el b/lisp/simple.el
> index fde6669..41f22b2 100644
> --- a/lisp/simple.el
> +++ b/lisp/simple.el
> @@ -8516,13 +8516,16 @@ after it has been set up properly in other respects."
> 
>        ;; Set up other local variables.
>        (mapc (lambda (v)
> -	      (condition-case ()	;in case var is read-only
> +	      (condition-case ()
>  		  (if (symbolp v)
>  		      (makunbound v)
>  		    (set (make-local-variable (car v)) (cdr v)))
> -		(error nil)))
> +		(setting-constant nil))) ;E.g. for enable-multibyte-characters.
>  	    lvars)
> 
> +      (setq mark-ring (mapcar (lambda (mk) (copy-marker (marker-position mk)))
> +                              mark-ring))
> +
>        ;; Run any hooks (typically set up by the major mode
>        ;; for cloning to work properly).
>        (run-hooks 'clone-buffer-hook))

To solve the problem more generally, maybe clone-buffer could look for
local variables with a non-nil symbol property (called, say,
`clone-buffer-update-function'), the value of which would be a
function that updates the "cloned" variable properly.

So with mark-ring, you would do something like:

(put 'mark-ring 'clone-buffer-update-function
     (lambda (sym old-buf new-buf)
       (with-current-buffer new-buf
	 (set sym (with-current-buffer old-buf
		    (mapcar (lambda (mk) (copy-marker (marker-position mk)))
			    mark-ring))))))

bug#17893: 24.4.50; (error "Marker does not point anywhere")

[Eli Zaretskii]

> Date: Fri, 24 Nov 2017 21:18:36 +0100
> From: charles@aurox.ch (Charles A. Roelli)
> CC: monnier@IRO.UMontreal.CA, 17893@debbugs.gnu.org
> 
> I suppose the main issue is fixed in master now:

That's just one particular case of a more general problem with markers
stored or referenced from local variables of the cloned buffer.

> To solve the problem more generally, maybe clone-buffer could look for
> local variables with a non-nil symbol property (called, say,
> `clone-buffer-update-function'), the value of which would be a
> function that updates the "cloned" variable properly.

This is not different from running a clone-buffer-update-function: it
again lets modes take care of the variables thy know about which need
special handling at clone time.  My problem with that is that I don't
believe this is a complete solution.

bug#17893: 24.4.50; (error "Marker does not point anywhere")

[Charles A. Roelli]

> Date: Fri, 24 Nov 2017 22:39:37 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > Date: Fri, 24 Nov 2017 21:18:36 +0100
> > From: charles@aurox.ch (Charles A. Roelli)
> > CC: monnier@IRO.UMontreal.CA, 17893@debbugs.gnu.org
> > 
> > I suppose the main issue is fixed in master now:
> 
> That's just one particular case of a more general problem with markers
> stored or referenced from local variables of the cloned buffer.
> 
> > To solve the problem more generally, maybe clone-buffer could look for
> > local variables with a non-nil symbol property (called, say,
> > `clone-buffer-update-function'), the value of which would be a
> > function that updates the "cloned" variable properly.
> 
> This is not different from running a clone-buffer-update-function: it
> again lets modes take care of the variables thy know about which need
> special handling at clone time.  My problem with that is that I don't
> believe this is a complete solution.

Can you say what you think is missing?

bug#17893: 24.4.50; (error "Marker does not point anywhere")

[Eli Zaretskii]

> Date: Sat, 25 Nov 2017 15:13:08 +0100
> From: charles@aurox.ch (Charles A. Roelli)
> CC: monnier@IRO.UMontreal.CA, 17893@debbugs.gnu.org
> 
> > > To solve the problem more generally, maybe clone-buffer could look for
> > > local variables with a non-nil symbol property (called, say,
> > > `clone-buffer-update-function'), the value of which would be a
> > > function that updates the "cloned" variable properly.
> > 
> > This is not different from running a clone-buffer-update-function: it
> > again lets modes take care of the variables thy know about which need
> > special handling at clone time.  My problem with that is that I don't
> > believe this is a complete solution.
> 
> Can you say what you think is missing?

What is missing is a way of methodically walking all the markers
reachable from the cloned buffer's local variables, and changing each
marker to point to the cloned buffer instead of the parent buffer.

bug#17893: 24.4.50; (error "Marker does not point anywhere")

[Stefan Monnier]

> What is missing is a way of methodically walking all the markers
> reachable from the cloned buffer's local variables, and changing each
> marker to point to the cloned buffer instead of the parent buffer.

I think this is equivalent to the halting problem.


        Stefan

bug#17893: 24.4.50; (error "Marker does not point anywhere")

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: charles@aurox.ch (Charles A. Roelli), 17893@debbugs.gnu.org
> Date: Sat, 25 Nov 2017 11:48:19 -0500
> 
> > What is missing is a way of methodically walking all the markers
> > reachable from the cloned buffer's local variables, and changing each
> > marker to point to the cloned buffer instead of the parent buffer.
> 
> I think this is equivalent to the halting problem.

Yes, I know.  But that doesn't mean it is, and certainly not that the
problem couldn't be described.

bug#17893: 24.4.50; (error "Marker does not point anywhere")

[Andreas Schwab]

On Nov 25 2017, Eli Zaretskii <eliz@gnu.org> wrote:

> What is missing is a way of methodically walking all the markers
> reachable from the cloned buffer's local variables, and changing each
> marker to point to the cloned buffer instead of the parent buffer.

And how does that help?  It will just move the error from the cloned
buffer to the parent buffer.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#17893: 24.4.50; (error "Marker does not point anywhere")

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: charles@aurox.ch (Charles A. Roelli),  17893@debbugs.gnu.org,  monnier@IRO.UMontreal.CA
> Date: Sat, 25 Nov 2017 19:30:08 +0100
> 
> On Nov 25 2017, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > What is missing is a way of methodically walking all the markers
> > reachable from the cloned buffer's local variables, and changing each
> > marker to point to the cloned buffer instead of the parent buffer.
> 
> And how does that help?  It will just move the error from the cloned
> buffer to the parent buffer.

Sorry, what error are you talking about?

The problem I was thinking of is when buffer-local variables in buffer
A hold markers whose buffer is A; then we clone buffer B from A, and
then we kill buffer A.  Now the markers in the cloned buffer point to
a dead buffer (or actually point nowhere).

bug#17893: 24.4.50; (error "Marker does not point anywhere")

[Andreas Schwab]

On Nov 25 2017, Eli Zaretskii <eliz@gnu.org> wrote:

> then we kill buffer A.

Sorry, I missed that part.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#17893: 24.4.50; (error "Marker does not point anywhere")

[martin rudalics]

 > What is missing is a way of methodically walking all the markers
 > reachable from the cloned buffer's local variables, and changing each
 > marker to point to the cloned buffer instead of the parent buffer.

Have `clone-buffer' record the parent buffer somewhere and when we
encounter a marker referring a dead buffer that is the same as the
recorded parent buffer redirect the marker on the fly to the cloned
buffer.  Not overly clean but what could we lose?

martin

bug#17893: 24.4.50; (error "Marker does not point anywhere")

[Eli Zaretskii]

> Date: Sun, 26 Nov 2017 11:26:29 +0100
> From: martin rudalics <rudalics@gmx.at>
> CC: 17893@debbugs.gnu.org, monnier@IRO.UMontreal.CA
> 
>  > What is missing is a way of methodically walking all the markers
>  > reachable from the cloned buffer's local variables, and changing each
>  > marker to point to the cloned buffer instead of the parent buffer.
> 
> Have `clone-buffer' record the parent buffer somewhere and when we
> encounter a marker referring a dead buffer that is the same as the
> recorded parent buffer redirect the marker on the fly to the cloned
> buffer.  Not overly clean but what could we lose?

When a buffer is deleted, all the markers that point to it get their
buffer wiped out, so I think the above method cannot work, unless you
replace all NULL buffer pointers with the cloned buffer -- which will
probably be too much.

And remember that markers without any buffer do not get adjusted, so
their position will quickly become incorrect or even outside the
cloned buffer's text, and then such markers will become useless.  So
we cannot delay this until the marker is accessed by some Lisp.

bug#17893: 24.4.50; (error "Marker does not point anywhere")

[martin rudalics]

 > When a buffer is deleted, all the markers that point to it get their
 > buffer wiped out, so I think the above method cannot work, unless you
 > replace all NULL buffer pointers with the cloned buffer -- which will
 > probably be too much.
 >
 > And remember that markers without any buffer do not get adjusted, so
 > their position will quickly become incorrect or even outside the
 > cloned buffer's text, and then such markers will become useless.  So
 > we cannot delay this until the marker is accessed by some Lisp.

Note that I've been referring only to your earlier

   The problem I was thinking of is when buffer-local variables in buffer
   A hold markers whose buffer is A; then we clone buffer B from A, and
   then we kill buffer A.  Now the markers in the cloned buffer point to
   a dead buffer (or actually point nowhere).

I'm not sure why this can be a problem because when we delete a base
buffer then the manual says that

   Killing the base buffer effectively kills the indirect buffer in that
   it cannot ever again be the current buffer.

but in fact we kill any indirect buffer before killing its base buffer.
Or I'm misreading the code of `kill-buffer'

martin

bug#17893: 24.4.50; (error "Marker does not point anywhere")

[Andreas Schwab]

On Nov 27 2017, martin rudalics <rudalics@gmx.at> wrote:

> Note that I've been referring only to your earlier
>
>   The problem I was thinking of is when buffer-local variables in buffer
>   A hold markers whose buffer is A; then we clone buffer B from A, and
>   then we kill buffer A.  Now the markers in the cloned buffer point to
>   a dead buffer (or actually point nowhere).
>
> I'm not sure why this can be a problem because when we delete a base
> buffer then the manual says that
>
>   Killing the base buffer effectively kills the indirect buffer in that
>   it cannot ever again be the current buffer.
>
> but in fact we kill any indirect buffer before killing its base buffer.
> Or I'm misreading the code of `kill-buffer'

Indirect buffer != cloned buffer.  An indirect buffer is created by
make-indirect-buffer and shares the buffer text with the parent buffer.
A cloned buffer is created by clone-buffer, and is a new buffer
independent from its origin, with its own buffer text.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

bug#17893: 24.4.50; (error "Marker does not point anywhere")

[martin rudalics]

 > Indirect buffer != cloned buffer.  An indirect buffer is created by
 > make-indirect-buffer and shares the buffer text with the parent buffer.
 > A cloned buffer is created by clone-buffer, and is a new buffer
 > independent from its origin, with its own buffer text.

Thanks!  I confused `clone-buffer' and `clone-indirect-buffer'.

martin

bug#17893: 24.4.50; (error "Marker does not point anywhere")

[Stefan Monnier]

>> > By C, do you mean the function `clone-buffer'?
>> No, I meant what's in marker.c.
> I suppose the main issue is fixed in master now:

Indeed.

> To solve the problem more generally, maybe clone-buffer could look for
> local variables with a non-nil symbol property (called, say,
> `clone-buffer-update-function'), the value of which would be a
> function that updates the "cloned" variable properly.

That's just a different hook, with more or less the same downsides as
the one we already have, so I don't see much benefit.


        Stefan