Different behaviour while debugging?

Different behaviour while debugging?

[Thorsten Jolitz]

Hi List, 

I have the very strange case that a program - in some special situation
- does not work as expected, but when I instrument the relevant
function(s) for edebug and single-step through them, everythings works
as expected and the output is fine. 

There are no errors involved, so 'toggle-debug-on-error does not help.
Its just that the program should put a buffer temporarily into a given
major-mode and uncomment some comment-lines with 'uncomment-region' before
turning-on Org-mode. 

With other modes it works fine, but in case of html-mode, this step is
simply omitted when running the program as-is, but is successfully
completed when single-stepping with edebug.

How can that be? Has elisp become just *too* fast for correct program
execution and programs work only reliably when slowed down by debugging? 

To be more precise, inserting 'message functions in the relevant
program-function revealed that: 

1. the function is entered
2. the buffer is in the expected (html) mode

but inside the 'cond function in the while loop, the out-commented lines
("comline") are not recognized when run normally

,--------------------------------------
| Entering outorg-convert-to-org....
| html-mode
| Entering while-loop ...
| Entering codeline after codeline ....
| Entering while-loop ... [2 times]
| Entering codeline after codeline ....
| Entering while-loop ...
| Entering codeline after codeline ....
| Entering while-loop ... [2 times]
| Entering codeline after codeline ....
`--------------------------------------

while they are recognized correctly when single-stepping with edebug

,-------------------------------------------
| Edebug: outorg-convert-to-org
| 
| Entering outorg-convert-to-org....
| html-mode
| Entering while-loop ...
| Entering comline after comline or bobp ...
| Entering while-loop ...
| Entering comline after comline or bobp ...
| Entering while-loop ...
| Entering comline after comline or bobp ...
| Entering while-loop ...
| Entering codeline after comline ....
`-------------------------------------------

Strange, isn't it?

-- 
cheers,
Thorsten

RE: Different behaviour while debugging?

[Drew Adams]

> I have the very strange case that a program - in some special 
> situation
> - does not work as expected, but when I instrument the relevant
> function(s) for edebug and single-step through them, everythings works
> as expected and the output is fine. 
> 
> There are no errors involved, so 'toggle-debug-on-error does not help.
> Its just that the program should put a buffer temporarily into a given
> major-mode and uncomment some comment-lines with 'uncomment-region'
> before turning-on Org-mode. 

Yes, it can happen that using the debugger gives you different behavior,
especially when there are changes of selected window or current buffer or frame
focus, or there is interaction with a timer or some events.  It's a case of the
observer affecting the observation, in some sense.

The debugger is pretty useful most of the time, but there are cases where you
need to find another solution.  Typically you can insert calls to `message' in
the code to be debugged, perhaps with accompanying `sleep-for' or `sit-for'
calls.

Before you do that, however, try the normal Emacs debugger, `debug', instead of
`edebug'.  You can use `M-x debug-on-entry' to do that.  Or you can insert calls
`(debug)' in the code itself.  See the manual or `C-h f debug' for more info
about using `debug'.

Re: Different behaviour while debugging?

[Thorsten Jolitz]

"Drew Adams" <drew.adams@oracle.com> writes:

>> I have the very strange case that a program - in some special 
>> situation
>> - does not work as expected, but when I instrument the relevant
>> function(s) for edebug and single-step through them, everythings works
>> as expected and the output is fine. 
>> 
>> There are no errors involved, so 'toggle-debug-on-error does not help.
>> Its just that the program should put a buffer temporarily into a given
>> major-mode and uncomment some comment-lines with 'uncomment-region'
>> before turning-on Org-mode. 
>
> Yes, it can happen that using the debugger gives you different behavior,
> especially when there are changes of selected window or current buffer or frame
> focus, or there is interaction with a timer or some events.  It's a case of the
> observer affecting the observation, in some sense.
>
> The debugger is pretty useful most of the time, but there are cases where you
> need to find another solution.  Typically you can insert calls to `message' in
> the code to be debugged, perhaps with accompanying `sleep-for' or `sit-for'
> calls.

I inserted 'message calls in every condition-case of 'cond, and the
*Messages* buffer showed that the correct condition-cases are simply not
entered when run in normal mode, but are entered when in debugging mode. 

> Before you do that, however, try the normal Emacs debugger, `debug', instead of
> `edebug'.  You can use `M-x debug-on-entry' to do that.  Or you can insert calls
> `(debug)' in the code itself.  See the manual or `C-h f debug' for more info
> about using `debug'.

I tried `M-x debug-on-entry', and just like with edebug, everything
works fine as expected. 


Here the relevant code part with the message calls:
,--------------------------------------------------------------------------------
| [...]
|  (while (not (eobp))
|    (message "Entering while-loop ...")
|    (cond
|     ;; empty line (do nothing)
|     ((looking-at "^[[:space:]]*$"))
|     ;; comment line after comment line or at
|     ;; beginning of buffer
|     ((and
|       (message "Before sleep [comline after comline or bobp] ...")
|       (progn (sleep-for 1) 
|       (message "After sleep [comline after comline or bobp] ..."))
|       (message "Before checking condition [comline after comline or bobp] ...")
|       (save-excursion
|         (eq (comment-on-line-p) (point-at-bol)))
|       (or (bobp) last-line-comment-p))
|      (message "After checking condition [comline after comline or bobp] ...")
|       (message "Entering comline after comline or bobp ...")
|      (uncomment-region (point-at-bol) (point-at-eol))
|      (setq last-line-comment-p t))
| [...]
`--------------------------------------------------------------------------------

Here the different - but only partly correct - output when using sleep
for 1 second:

,--------------------------------------------------------------
| Entering outorg-convert-to-org....
| html-mode
| Entering while-loop ...
| Before sleep [comline after comline or bobp] ...
| After sleep [comline after comline or bobp] ...
| Before checking condition [comline after comline or bobp] ...
| Entering codeline after codeline ....
| Entering while-loop ... [2 times]
| Before sleep [comline after comline or bobp] ...
| After sleep [comline after comline or bobp] ...
| Before checking condition [comline after comline or bobp] ...
| Entering comline after codeline ....
| Entering while-loop ... [2 times]
| Before sleep [comline after comline or bobp] ...
| After sleep [comline after comline or bobp] ...
| Before checking condition [comline after comline or bobp] ...
| Entering codeline after comline ....
`--------------------------------------------------------------

here again the entirely incorrect output after outcommenting  

       ,-------------------------------------------------------------
       | (progn (sleep-for 1) 
       | (message "After sleep [comline after comline or bobp] ..."))
       `-------------------------------------------------------------

in the code example above:

,--------------------------------------------------------------
| Entering outorg-convert-to-org....
| html-mode
| Entering while-loop ...
| Before sleep [comline after comline or bobp] ...
| Before checking condition [comline after comline or bobp] ...
| Entering codeline after codeline ....
| Entering while-loop ... [2 times]
| Before sleep [comline after comline or bobp] ...
| Before checking condition [comline after comline or bobp] ...
| Entering codeline after codeline ....
| Entering while-loop ... [2 times]
| Before sleep [comline after comline or bobp] ...
| Before checking condition [comline after comline or bobp] ...
| Entering codeline after codeline ....
`--------------------------------------------------------------


from this evidence, I would conclude that slowing down the program by 1
second makes it partly behaving correct, but *really* slowing it down by
single-stepping in the debugger (no matter if debug or edebug) makes it
behave correctly. 

Not really what I would expect...

-- 
cheers,
Thorsten

Re: Different behaviour while debugging?

[Stefan Monnier]

> but inside the 'cond function in the while loop, the out-commented lines
> ("comline") are not recognized when run normally

My crystal ball is very cloudy but it seems like it's trying to tell me
that single-stepping via edebug causes the file's content to be
displayed before you run the code, and displaying this content causes it
to be font-locked, which ends up adding some syntax-table properties
which make those comments recognized as such.


        Stefan

RE: Different behaviour while debugging?

[Drew Adams]

> I inserted 'message calls in every condition-case of 'cond, and the
> *Messages* buffer showed that the correct condition-cases are 
> simply not entered when run in normal mode, but are entered when in 
> debugging mode. 
> 
> I tried `M-x debug-on-entry', and just like with edebug, everything
> works fine as expected. 

Yeah, it happens.  Sorry, I don't have a great suggestion to make.  Try somehow
to narrow the problem down to a smaller code segment, perhaps.

I don't see anything in the code you sent that should be affected by timing,
events, etc. (but I didn't examine it closely).  Perhaps you have something else
going on that involves such things?  Can you try to eliminate all extraneous
stuff that might be in your environment, e.g., start from emacs -Q and use just
the minimal code that you want to test?

Or see what happens when you fiddle some more with things that you suspect might
be affecting the behavior.  IOW, poke it a bit here and there to see what
happens. ;-)

Perhaps someone else has a good suggestion.  Stefan's seems like a good place to
start.

Re: Different behaviour while debugging?

[Thorsten Jolitz]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> but inside the 'cond function in the while loop, the out-commented lines
>> ("comline") are not recognized when run normally
>
> My crystal ball is very cloudy but it seems like it's trying to tell me
> that single-stepping via edebug causes the file's content to be
> displayed before you run the code, and displaying this content causes it
> to be font-locked, which ends up adding some syntax-table properties
> which make those comments recognized as such.

that sounds quite reasonable to me, thank you, might 'comment-start',
'comment-end' and 'comment-padding' be among such syntax-table
properties? 

it works as expected with emacs lisp, picolisp and latex, but not in
html-mode (only mode among these that uses 'comment-start' *and*
'comment-end'), so maybe 'comment-end' is involved in the problem.

Is this a mistake of my program not to display the content before
running the code? With which elisp-trickery could I always enforce to
display the content first?

Or is this a problem with Emacs, and my program should not care about
these things? 

-- 
cheers,
Thorsten

Re: Different behaviour while debugging?

[Thorsten Jolitz]

"Drew Adams" <drew.adams@oracle.com> writes:

>> I inserted 'message calls in every condition-case of 'cond, and the
>> *Messages* buffer showed that the correct condition-cases are 
>> simply not entered when run in normal mode, but are entered when in 
>> debugging mode. 
>> 
>> I tried `M-x debug-on-entry', and just like with edebug, everything
>> works fine as expected. 
>
> Yeah, it happens.  Sorry, I don't have a great suggestion to make.  Try somehow
> to narrow the problem down to a smaller code segment, perhaps.
> [...]
> Perhaps someone else has a good suggestion.  Stefan's seems like a good place to
> start.

Thanks a lot for your suggestions, I think they are generally very
useful. In this case maybe the problem is related to some internal Emacs
procedures that I'm simply not aware of (like the order of content
display and code execution) and that don't show up at the surface of my
application code. 

Maybe a call to some kind of 'flush-buffer-content' function (I could
not find one with 'C-h f') at the right moment would solve the problem?

-- 
cheers,
Thorsten

Re: Different behaviour while debugging?

[Stefan Monnier]

> that sounds quite reasonable to me, thank you, might 'comment-start',
> 'comment-end' and 'comment-padding' be among such syntax-table
> properties? 

No, these are variables, not text-properties nor syntax-table properties.

> Is this a mistake of my program not to display the content before
> running the code?

No.

> Or is this a problem with Emacs, and my program should not care about
> these things? 

Hard to tell without first figuring out what's really going on.


        Stefan

Re: Different behaviour while debugging?

[Thorsten Jolitz]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> that sounds quite reasonable to me, thank you, might 'comment-start',
>> 'comment-end' and 'comment-padding' be among such syntax-table
>> properties? 
>
> No, these are variables, not text-properties nor syntax-table properties.
>
>> Is this a mistake of my program not to display the content before
>> running the code?
>
> No.
>
>> Or is this a problem with Emacs, and my program should not care about
>> these things? 
>
> Hard to tell without first figuring out what's really going on.

thats probably quite difficult to figure out, and maybe not worth the
pain, since until now the problems only occur when I run my
outline-minor-mode extensions in 'html-mode', which is quite special
with regards to outline and comments. 

thanks for the hints anyway. 

-- 
cheers,
Thorsten

Re: Different behaviour while debugging?

[Andreas Röhler]

Am 20.02.2013 15:57, schrieb Thorsten Jolitz:
>
> Hi List,
>
> I have the very strange case that a program - in some special situation
> - does not work as expected, but when I instrument the relevant
> function(s) for edebug and single-step through them, everythings works
> as expected and the output is fine.
>
> There are no errors involved, so 'toggle-debug-on-error does not help.
> Its just that the program should put a buffer temporarily into a given
> major-mode and uncomment some comment-lines with 'uncomment-region' before
> turning-on Org-mode.
>
> With other modes it works fine, but in case of html-mode, this step is
> simply omitted when running the program as-is, but is successfully
> completed when single-stepping with edebug.
>
> How can that be? Has elisp become just *too* fast for correct program
> execution and programs work only reliably when slowed down by debugging?
>

IMHO that's possible, for example if some function expects faces, which are only there if
fontification is done

try some (sit-for) then

Re: Different behaviour while debugging?

[Thorsten Jolitz]

Andreas Röhler <andreas.roehler@easy-emacs.de> writes:

>> I have the very strange case that a program - in some special situation
>> - does not work as expected, but when I instrument the relevant
>> function(s) for edebug and single-step through them, everythings works
>> as expected and the output is fine.
>
> IMHO that's possible, for example if some function expects faces, which are only there if
> fontification is done

At the surface of my application program, there are no faces involved,
its about recognizing if there is a comment on a line with
'comment-on-line-p', which calls 'comment-search-forward', which
searches for 'comment-start-skip'. 

Since, when in html-mode, the following condition-case (inside a 'cond
structure) is never entered in normal execution but is correctly entered
when (e)debugging, I would guess the 'comment-on-line-p' makes the
difference. 

         ,---------------------------------------------------
         | ((and
         |   (save-excursion
         |     (not (eq (comment-on-line-p) (point-at-bol))))
         |   last-line-comment-p) ...)
         `---------------------------------------------------

And, actually, if I edebug 'comment-search-forward', things work as
expected, just like when I edebug my own function that contains the
above condition-case. 

> try some (sit-for) then

ok - thanks for the tip. 

-- 
cheers,
Thorsten

Re: Different behaviour while debugging?

[Andreas Röhler]

Am 23.02.2013 10:07, schrieb Thorsten Jolitz:
> Andreas Röhler <andreas.roehler@easy-emacs.de> writes:
>
>>> I have the very strange case that a program - in some special situation
>>> - does not work as expected, but when I instrument the relevant
>>> function(s) for edebug and single-step through them, everythings works
>>> as expected and the output is fine.
>>
>> IMHO that's possible, for example if some function expects faces, which are only there if
>> fontification is done
>
> At the surface of my application program, there are no faces involved,
> its about recognizing if there is a comment on a line with
> 'comment-on-line-p', which calls 'comment-search-forward', which
> searches for 'comment-start-skip'.
>
> Since, when in html-mode, the following condition-case (inside a 'cond
> structure) is never entered in normal execution but is correctly entered
> when (e)debugging, I would guess the 'comment-on-line-p' makes the
> difference.
>
>           ,---------------------------------------------------
>           | ((and
>           |   (save-excursion
>           |     (not (eq (comment-on-line-p) (point-at-bol))))
>           |   last-line-comment-p) ...)
>           `---------------------------------------------------
>
> And, actually, if I edebug 'comment-search-forward', things work as
> expected, just like when I edebug my own function that contains the
> above condition-case.
>
>> try some (sit-for) then
>
> ok - thanks for the tip.
>

Maybe let's have a look at definitions of

comment-on-line-p
point-at-bol