Tracking down assertion failure

Tracking down assertion failure

[Stefan Monnier]

I regularly get assertion failures in the redisplay that have to do with
an apparent discrepency between charpos and bytepos.

More specifically, the assertion failure is at xdisp.c:6062:

	  xassert (IT_BYTEPOS (*it) == CHAR_TO_BYTE (IT_CHARPOS (*it)));

I've propagated this assertion a bit everywhere and I se it even
triggers around line 11500:

	  /* Skip from tlbufpos to PT and see where it is.  Note that
	     PT may be in invisible text.  If so, we will end at the
	     next visible position.  */
	  init_iterator (&it, w, CHARPOS (tlbufpos), BYTEPOS (tlbufpos),
			 NULL, DEFAULT_FACE_ID);
	  xassert (IT_BYTEPOS (it) == CHAR_TO_BYTE (IT_CHARPOS (it)));

the problem is that as I keep adding such assertions earlier and earlier
in the code I seem to start hitting another problem: I'm not sure at
which point this condition should be true and and at which point it's OK
for it not to be true (because the charpos and bytepos recorded refer
to out-of-data data which will/should simply not be used).

Can someone help me out?  


        Stefan

Re: Tracking down assertion failure

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Wed, 12 Mar 2008 15:18:29 -0400
> 
> 	  /* Skip from tlbufpos to PT and see where it is.  Note that
> 	     PT may be in invisible text.  If so, we will end at the
> 	     next visible position.  */
> 	  init_iterator (&it, w, CHARPOS (tlbufpos), BYTEPOS (tlbufpos),
> 			 NULL, DEFAULT_FACE_ID);
> 	  xassert (IT_BYTEPOS (it) == CHAR_TO_BYTE (IT_CHARPOS (it)));
> 
> the problem is that as I keep adding such assertions earlier and earlier
> in the code I seem to start hitting another problem: I'm not sure at
> which point this condition should be true and and at which point it's OK
> for it not to be true (because the charpos and bytepos recorded refer
> to out-of-data data which will/should simply not be used).
> 
> Can someone help me out?  

I may be forgetting something, but isn't the iterator moving by
characters?  It uses the pair

      IT_BYTEPOS (*it) += it->len;
      IT_CHARPOS (*it) += 1;

to advance, so IT_BYTEPOS and IT_CHARPOS should always be in sync,
right?  Am I missing something?

Re: Tracking down assertion failure

[Stefan Monnier]

>> /* Skip from tlbufpos to PT and see where it is.  Note that
>> PT may be in invisible text.  If so, we will end at the
>> next visible position.  */
>> init_iterator (&it, w, CHARPOS (tlbufpos), BYTEPOS (tlbufpos),
>> NULL, DEFAULT_FACE_ID);
>> xassert (IT_BYTEPOS (it) == CHAR_TO_BYTE (IT_CHARPOS (it)));
>> 
>> the problem is that as I keep adding such assertions earlier and earlier
>> in the code I seem to start hitting another problem: I'm not sure at
>> which point this condition should be true and and at which point it's OK
>> for it not to be true (because the charpos and bytepos recorded refer
>> to out-of-data data which will/should simply not be used).
>> 
>> Can someone help me out?  

> I may be forgetting something, but isn't the iterator moving by
> characters?  It uses the pair

>       IT_BYTEPOS (*it) += it->len;
>       IT_CHARPOS (*it) += 1;

> to advance, so IT_BYTEPOS and IT_CHARPOS should always be in sync,
> right?  Am I missing something?

Yes, they should.  I'm pretty sure that the above xassert should
not fail.  Yet it does (typically the bytepos is 1 larger than the
charpos, even tho the buffer contains only ascii, and typically it
happens when moving around the buffer with the arrow keys, it seems to
have to do with handling of the cursor).

The problem is that when we enter the cursor code, it's not clear to me
which charpos/bytepos pairs are expected to be valid and which ones are
expected to be potentially out-of-date and I don't understand either
enough about how/when we check them being up-to-date.

The bug cannot be reproduced deterministically, but I hit it several
times a day.  After adding xasserts similar to the above further up the
call chain, all I managed to get is that those asserts fail more often,
but I'm not convinced that their failure is really a bug, because I'm
not sure whether the values I check should actually be up-to-date.


        Stefan

Re: Tracking down assertion failure

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Wed, 12 Mar 2008 21:30:48 -0400
> 
> The problem is that when we enter the cursor code, it's not clear to me
> which charpos/bytepos pairs are expected to be valid and which ones are
> expected to be potentially out-of-date and I don't understand either
> enough about how/when we check them being up-to-date.

Can you point me to the place in the sources that you call ``the
cursor code''?  Also, a list of places where the xassert fires would
be useful.

Re: Tracking down assertion failure

[Stefan Monnier]

>> The problem is that when we enter the cursor code, it's not clear to me
>> which charpos/bytepos pairs are expected to be valid and which ones are
>> expected to be potentially out-of-date and I don't understand either
>> enough about how/when we check them being up-to-date.

> Can you point me to the place in the sources that you call ``the
> cursor code''?  Also, a list of places where the xassert fires would
> be useful.

I added assertions into try_cursor_movement that triggered as well.
Can't remember exactly how I got lead to that code.
But I suspect that at least some of those assertions were simply
incorrect (because the data they check is just out of date).

(Almost) everytime that I hit the assertion failure, is when I'm moving
about the buffer with the arrow keys.  Which also points to the cursor
movement code.

Right now, I get assertions failures at (in redisplay_internal):

      /* All text outside that line, including its final newline,
	 must be unchanged */
      && text_outside_line_unchanged_p (w, CHARPOS (tlbufpos),
					CHARPOS (tlendpos)))
    {
      xassert (tlbufpos.bytepos == CHAR_TO_BYTE (tlbufpos.charpos));

      if (CHARPOS (tlbufpos) > BEGV

where the `xassert' is one that I added locally.

I append the end of try_cursor_movement where I'd added 4 xasserts that
tend to trigger too often.  If someone can confirm that those assertions
should never trigger or that it's normal for them to fail sometimes, it
would be helpful.


        Stefan


	  /* Triggers often.  Might not be a bug.
	     xassert (row->start.pos.bytepos == CHAR_TO_BYTE (row->start.pos.charpos)); */
	  if (PT < MATRIX_ROW_START_CHARPOS (row)
	      || PT > MATRIX_ROW_END_CHARPOS (row))
	    {
	      /* if PT is not in the glyph row, give up.  */
	      rc = CURSOR_MOVEMENT_MUST_SCROLL;
	    }
	  else if (rc != CURSOR_MOVEMENT_SUCCESS
		   && MATRIX_ROW_PARTIALLY_VISIBLE_P (w, row)
		   && make_cursor_line_fully_visible_p)
	    {
	      if (PT == MATRIX_ROW_END_CHARPOS (row)
		  && !row->ends_at_zv_p
		  && !MATRIX_ROW_ENDS_IN_MIDDLE_OF_CHAR_P (row))
		rc = CURSOR_MOVEMENT_MUST_SCROLL;
	      else if (row->height > window_box_height (w))
		{
		  /* If we end up in a partially visible line, let's
		     make it fully visible, except when it's taller
		     than the window, in which case we can't do much
		     about it.  */
		  *scroll_step = 1;
		  rc = CURSOR_MOVEMENT_MUST_SCROLL;
		}
	      else
		{
		  set_cursor_from_row (w, row, w->current_matrix, 0, 0, 0, 0);
		  if (!cursor_row_fully_visible_p (w, 0, 1))
		    rc = CURSOR_MOVEMENT_MUST_SCROLL;
		  else
		    rc = CURSOR_MOVEMENT_SUCCESS;
		}
	    }
	  else if (scroll_p)
	    rc = CURSOR_MOVEMENT_MUST_SCROLL;
	  else
	    {
	      /* Triggers often.  Maybe it's not a bug.
		 xassert (row->start.pos.bytepos == CHAR_TO_BYTE (row->start.pos.charpos)); */
	      do
		{
		  /* xassert (row->start.pos.bytepos == CHAR_TO_BYTE (row->start.pos.charpos)); */
		  if (set_cursor_from_row (w, row, w->current_matrix, 0, 0, 0, 0))
		    {
		      rc = CURSOR_MOVEMENT_SUCCESS;
		      break;
		    }
		  ++row;
		  /* xassert (row->start.pos.bytepos == CHAR_TO_BYTE (row->start.pos.charpos)); */
		}
	      while (MATRIX_ROW_BOTTOM_Y (row) < last_y
		     && MATRIX_ROW_START_CHARPOS (row) == PT
		     && cursor_row_p (w, row));
	    }
	}
    }

  return rc;
}

Re: Tracking down assertion failure

[Lennart Borgman (gmail)]

Stefan Monnier wrote:
> (Almost) everytime that I hit the assertion failure, is when I'm moving
> about the buffer with the arrow keys.  Which also points to the cursor
> movement code.

Maybe this could give some clues to the bug I reported here too:

http://lists.gnu.org/archive/html/emacs-devel/2008-01/msg01407.html