Certain numbers of special forms cause changing behaviour on function calls in --batch

Certain numbers of special forms cause changing behaviour on function calls in --batch

[Wilfred Hughes]

Hi all

I've bumped into an issue where the behaviour of Emacs changes
depending on the number of special forms.

    (defun call-fn-in-let (unused)
      (let (xx)
        (let (yy)
          (split-string "foo" "bar" nil))))

    (with-temp-buffer
      (insert "foofoofoo\nbarbarbar")
      (let ((dd (let (aa)
                  (let (bb)
                    (let (cc)
                      (progn (call-fn-in-let nil)))))))
        (let (ee)
          (save-excursion
            (goto-char (point-min))
            (if t
                (progn
                  (move-end-of-line nil)
                  (message "point is: %s" (point))))))))

In the above code, running it interactively in Emacs with M-x
eval-buffer gives: 'point is: 10'. This is correct.

However, running it in --batch gives:

$ emacs -batch -l ~/projects/pyimport/nasty.el
point is: 20

This is incorrect: for some reason move-end-of-line is behaving
differently here. Removing any of the extra special forms (e.g. the
redundant lets) fixes it.

I can reproduce this on 24.5, but not on HEAD. However,
move-end-of-line has not changed between 24.5 and HEAD.

Is/was this a bug in the elisp interpreter?

Wilfred

Re: Certain numbers of special forms cause changing behaviour on function calls in --batch

[Michael Heerdegen]

Wilfred Hughes <me@wilfred.me.uk> writes:

> However, running it in --batch gives:
>
> $ emacs -batch -l ~/projects/pyimport/nasty.el
> point is: 20
>
> This is incorrect: for some reason move-end-of-line is behaving
> differently here. Removing any of the extra special forms (e.g. the
> redundant lets) fixes it.

Strange indeed.

OTOH, looking at the doc of `move-end-of-line' ("Move point to end of
current line as displayed.") I wouldn't rely on the result in batch mode
at all, since there is no display.

`end-of-line' yields the same result in both situations here.


Michael.

Re: Certain numbers of special forms cause changing behaviour on function calls in --batch

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Date: Tue, 28 Jun 2016 23:12:26 +0200
> Cc: emacs-devel <emacs-devel@gnu.org>
> 
> Wilfred Hughes <me@wilfred.me.uk> writes:
> 
> > However, running it in --batch gives:
> >
> > $ emacs -batch -l ~/projects/pyimport/nasty.el
> > point is: 20
> >
> > This is incorrect: for some reason move-end-of-line is behaving
> > differently here. Removing any of the extra special forms (e.g. the
> > redundant lets) fixes it.
> 
> Strange indeed.

I see no strangeness: I cannot reproduce the problem, neither in Emacs
24.4, nor 24.5, nor the current pretest of Emacs 25.1.

So I'm going to claim that OP's Emacs 24.5 was compiled by a buggy
compiler or something.

Re: Certain numbers of special forms cause changing behaviour on function calls in --batch

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> I see no strangeness: I cannot reproduce the problem, neither in Emacs
> 24.4, nor 24.5, nor the current pretest of Emacs 25.1.
>
> So I'm going to claim that OP's Emacs 24.5 was compiled by a buggy
> compiler or something.

I see the same behavior as the OP.  I use a build of emacs-25, HEAD ->
6cdd8f71 (1 week old), on Debian.


Michael.

Re: Certain numbers of special forms cause changing behaviour on function calls in --batch

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Cc: me@wilfred.me.uk,  emacs-devel@gnu.org
> Date: Wed, 29 Jun 2016 22:39:02 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I see no strangeness: I cannot reproduce the problem, neither in Emacs
> > 24.4, nor 24.5, nor the current pretest of Emacs 25.1.
> >
> > So I'm going to claim that OP's Emacs 24.5 was compiled by a buggy
> > compiler or something.
> 
> I see the same behavior as the OP.  I use a build of emacs-25, HEAD ->
> 6cdd8f71 (1 week old), on Debian.

I need a reproducible recipe to look into this.  So far, it's not
reproducible for me.  If someone can come up with a recipe, there are
good chances we will at least understand the reasons for this.

Re: Certain numbers of special forms cause changing behaviour on function calls in --batch

[Michael Heerdegen]

[-- Attachment #1: Type: text/plain, Size: 353 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

> I need a reproducible recipe to look into this.  So far, it's not
> reproducible for me.  If someone can come up with a recipe, there are
> good chances we will at least understand the reasons for this.

Maybe it's just not reproducible on your system.

I have this file in "/home/micha/Treasure/today/test.el":


[-- Attachment #2: test.el --]
[-- Type: application/emacs-lisp, Size: 483 bytes --]

[-- Attachment #3: Type: text/plain, Size: 165 bytes --]


From the shell:

micha@drachen ~/T/today> emacs -batch -l ./test.el 
point is: 20

When I open the file from Emacs and M-x eval-buffer, Emacs prints 10.


Michael.

Re: Certain numbers of special forms cause changing behaviour on function calls in --batch

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Cc: me@wilfred.me.uk,  emacs-devel@gnu.org
> Date: Thu, 30 Jun 2016 14:34:28 +0200
> 
> Maybe it's just not reproducible on your system.

Maybe.  But that doesn't help me debug the issue, does it?  Perhaps
someone else can, it's not that hard to dig into these things.

> >From the shell:
> 
> micha@drachen ~/T/today> emacs -batch -l ./test.el 
> point is: 20
> 
> When I open the file from Emacs and M-x eval-buffer, Emacs prints 10.

Here, it prints 10 in both cases.

Re: Certain numbers of special forms cause changing behaviour on function calls in --batch

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> Maybe.  But that doesn't help me debug the issue, does it?  Perhaps
> someone else can, it's not that hard to dig into these things.

If you are sure it's a bug (I wasn't), I can give it a quick try (I'm on
vacation).  How would you do proceed?

Michael.

Re: Certain numbers of special forms cause changing behaviour on function calls in --batch

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Cc: me@wilfred.me.uk,  emacs-devel@gnu.org
> Date: Thu, 30 Jun 2016 23:10:58 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Maybe.  But that doesn't help me debug the issue, does it?  Perhaps
> > someone else can, it's not that hard to dig into these things.
> 
> If you are sure it's a bug (I wasn't)

I don't know yet if it is a bug or not.  Debugging can reveal what it
is.

> I can give it a quick try (I'm on vacation).  How would you do
> proceed?

Insert 'message' in strategic places that print values that are
important for the code flow, and see what is different in the batch
session.  In the interactive session, you can step through the code
with Edebug to compare the findings (you will have to manually load
simple.el into a running session).

Just looking at the code, my first suspect would be line-move: it has
a special code for non-interactive sessions, so maybe the issue hides
there somewhere.  Or maybe it's in line-move-1 or the functions it
calls (like vertical-motion).

Thanks.

Re: Certain numbers of special forms cause changing behaviour on function calls in --batch

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> > I can give it a quick try (I'm on vacation).

The first thing that I found out: it makes a difference whether I use
the compiled simple.elc or the source simple.el.  Only with the compiled
simple.elc I get the unexpected result of 20.  But when I load the
uncompiled simple.el into the batch session, I'm not able to reproduce
the issue (I get 10).

Does that give us hints about how I could proceed?


Thanks,

Michael.

Re: Certain numbers of special forms cause changing behaviour on function calls in --batch

[Michael Heerdegen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
> > > I can give it a quick try (I'm on vacation).
>
> The first thing that I found out: it makes a difference whether I use
> the compiled simple.elc or the source simple.el.  Only with the compiled
> simple.elc I get the unexpected result of 20.  But when I load the
> uncompiled simple.el into the batch session, I'm not able to reproduce
> the issue (I get 10).

Hmm, now I changed simple.el like this:

*** /tmp/ediff230416Vp	2016-07-05 23:29:32.557789593 +0200
--- /home/micha/software/emacs/lisp/simple.el	2016-07-05 23:27:12.017360250 +0200
***************
*** 6341,6346 ****
--- 6341,6347 ----
                                        (point) 'invisible)))
  			(backward-char 1)))
  		 (point)))))
+         (message "%s" newpos)
  	(goto-char newpos)
  	(if (and (> (point) newpos)
  		 (eq (preceding-char) ?\n))

and recompiled it.  After that, the issue is not reproducible anymore at
all; neither with the uncompiled nor with compiled "simple" library.

Seems we won't find out anything useful by debugging on the Lisp level.


Michael.

Re: Certain numbers of special forms cause changing behaviour on function calls in --batch

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Cc: me@wilfred.me.uk,  emacs-devel@gnu.org
> Date: Tue, 05 Jul 2016 23:33:29 +0200
> 
> Hmm, now I changed simple.el like this:
> 
> *** /tmp/ediff230416Vp	2016-07-05 23:29:32.557789593 +0200
> --- /home/micha/software/emacs/lisp/simple.el	2016-07-05 23:27:12.017360250 +0200
> ***************
> *** 6341,6346 ****
> --- 6341,6347 ----
>                                         (point) 'invisible)))
>   			(backward-char 1)))
>   		 (point)))))
> +         (message "%s" newpos)
>   	(goto-char newpos)
>   	(if (and (> (point) newpos)
>   		 (eq (preceding-char) ?\n))
> 
> and recompiled it.  After that, the issue is not reproducible anymore at
> all; neither with the uncompiled nor with compiled "simple" library.

Did the experiments with the compiled simple.elc load it into a
running session, or did you re-dump Emacs after byte-compiling
simple.el?

> Seems we won't find out anything useful by debugging on the Lisp level.

What other practical alternatives are there?

Re: Certain numbers of special forms cause changing behaviour on function calls in --batch

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> Did the experiments with the compiled simple.elc load it into a
> running session, or did you re-dump Emacs after byte-compiling
> simple.el?

No, I was lazy and loaded it into the running session.  I can repeat the
test with re-dumping if you think it could change the behavior.

Note: when I (re-)loaded the "original" simple.elc into the batch
session, there was no change (i.e. I still got the "buggy" result of
20).  Only loading the compiled (trivially) modified simple library into
the batch session had an effect (of "fixing" the issue and producing the
"nonbuggy" value of 10).


> What other practical alternatives are there?

I have no idea.


Michael.

Re: Certain numbers of special forms cause changing behaviour on function calls in --batch

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Cc: me@wilfred.me.uk,  emacs-devel@gnu.org
> Date: Wed, 06 Jul 2016 16:47:48 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Did the experiments with the compiled simple.elc load it into a
> > running session, or did you re-dump Emacs after byte-compiling
> > simple.el?
> 
> No, I was lazy and loaded it into the running session.  I can repeat the
> test with re-dumping if you think it could change the behavior.

At this point, I think we have no idea what could change the
behavior.  These differences shouldn't happen, they seem to point to
some macro somewhere.

So more data points might help, yes.

Re: Certain numbers of special forms cause changing behaviour on function calls in --batch

[Yasushi SHOJI]

[-- Attachment #1.1: Type: text/plain, Size: 675 bytes --]

On Thu, Jul 7, 2016 at 12:04 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> So more data points might help, yes.

I can reproduce this bug on my Debian Sid system with
gcc version 5.4.0 20160609 (Debian 5.4.0-6).

git bisect showed me that 7c3d742c357dd6480e813f067435b324dba2b325
is the one to blame.  Farther investigation showed me that the logic of the
commit 7c3d742 isn't the root cause but the additional member in the struct
it is.

7c3d742^ is 7bf54d01159eb09bae3c9cd86f2af0812d9afdf6, which is good.
But if I add void * bad; to struct it, like the attached patch, it turns
into bad one.

So, there might be a bug related to the size of struct it?
-- 
             yashi

[-- Attachment #1.2: Type: text/html, Size: 931 bytes --]

[-- Attachment #2: a.patch --]
[-- Type: text/x-patch, Size: 308 bytes --]

diff --git a/src/dispextern.h b/src/dispextern.h
index fad5bfd..13ca68c 100644
--- a/src/dispextern.h
+++ b/src/dispextern.h
@@ -2533,6 +2533,7 @@ struct it
   /* If what == IT_XWIDGET.  */
   struct xwidget *xwidget;
 #endif
+  void *foo;
 
   /* Values from `slice' property.  */
   struct it_slice slice;

Re: Certain numbers of special forms cause changing behaviour on function calls in --batch

[Yasushi SHOJI]

[-- Attachment #1: Type: text/plain, Size: 1069 bytes --]

On Sun, Jul 10, 2016 at 7:34 PM, Yasushi SHOJI <yasushi.shoji@gmail.com>
wrote:
> On Thu, Jul 7, 2016 at 12:04 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> > So more data points might help, yes.
>
> I can reproduce this bug on my Debian Sid system with
> gcc version 5.4.0 20160609 (Debian 5.4.0-6).

I should have tested with different compilers before posting the
previous comment since Eli predicted that a bad compiler is a problem.

Yes, the results differ depending on the compiler you use.

Here is the results:

 - All test is done with 7bf54d0 + the previous patch applied
 - no option to configure except specifying compiler with CC=

| gcc 6   | good | 6.1.1 20160705 (Debian 6.1.1-9) |
| gcc 5   | bad  | 5.4.0 20160609 (Debian 5.4.0-6) |
| gcc 4.9 | good | 4.9.3 (Debian 4.9.3-14)         |
| gcc 4.8 | good | 4.8.5 (Debian 4.8.5-4)          |

  - good: returns 10
  - bad: returns 20

So the current default compiler on Debian Sid is the one to create a
bad binary with the current source code.

Let me know if you need any other info.
-- 
           yashi

[-- Attachment #2: Type: text/html, Size: 1603 bytes --]

Re: Certain numbers of special forms cause changing behaviour on function calls in --batch

[Noam Postavsky]

On Sun, Jul 10, 2016 at 7:13 AM, Yasushi SHOJI <yasushi.shoji@gmail.com> wrote:
> On Sun, Jul 10, 2016 at 7:34 PM, Yasushi SHOJI <yasushi.shoji@gmail.com>
> wrote:
>> On Thu, Jul 7, 2016 at 12:04 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> > So more data points might help, yes.
>>
>> I can reproduce this bug on my Debian Sid system with
>> gcc version 5.4.0 20160609 (Debian 5.4.0-6).
>
> I should have tested with different compilers before posting the
> previous comment since Eli predicted that a bad compiler is a problem.
>
> Yes, the results differ depending on the compiler you use.

I have gcc 5.3.0 here, and I get 10 (correct) when I compile Emacs
24.5 configured with --enable-checking, but 20 (incorrect) without
that flag. In both cases running temacs with valgrind gives some
errors which look suspicious (the version without --enable-checking
additionally gives a bunch of what I assume are false positives while
starting up). Emacs 25 gives similar errors (I haven't tried Emacs 25
without --enable-check).

(I'm showing here just the results from the --enable-checking
versions, without it's the same except for printing 20 instead of 10)

valgrind emacs-24.5/src/temacs -Q --batch -l nasty-move-end-of-line-batch.el
[...]
==6167== Conditional jump or move depends on uninitialised value(s)
==6167==    at 0x55E74E: CHECK_NUMBER_OR_FLOAT (lisp.h:2627)
==6167==    by 0x5EC762: Fzerop (data.c:2387)
==6167==    by 0x608571: Ffuncall (eval.c:2811)
==6167==    by 0x650F35: exec_byte_code (bytecode.c:916)
==6167==    by 0x608DA1: funcall_lambda (eval.c:2978)
==6167==    by 0x608764: Ffuncall (eval.c:2860)
==6167==    by 0x650F35: exec_byte_code (bytecode.c:916)
==6167==    by 0x608DA1: funcall_lambda (eval.c:2978)
==6167==    by 0x608764: Ffuncall (eval.c:2860)
==6167==    by 0x650F35: exec_byte_code (bytecode.c:916)
==6167==    by 0x608DA1: funcall_lambda (eval.c:2978)
==6167==    by 0x608A99: apply_lambda (eval.c:2919)
==6167==
==6167== Conditional jump or move depends on uninitialised value(s)
==6167==    at 0x5EC76B: Fzerop (data.c:2389)
==6167==    by 0x608571: Ffuncall (eval.c:2811)
==6167==    by 0x650F35: exec_byte_code (bytecode.c:916)
==6167==    by 0x608DA1: funcall_lambda (eval.c:2978)
==6167==    by 0x608764: Ffuncall (eval.c:2860)
==6167==    by 0x650F35: exec_byte_code (bytecode.c:916)
==6167==    by 0x608DA1: funcall_lambda (eval.c:2978)
==6167==    by 0x608764: Ffuncall (eval.c:2860)
==6167==    by 0x650F35: exec_byte_code (bytecode.c:916)
==6167==    by 0x608DA1: funcall_lambda (eval.c:2978)
==6167==    by 0x608A99: apply_lambda (eval.c:2919)
==6167==    by 0x6070E9: eval_sub (eval.c:2226)
==6167==
==6167== Conditional jump or move depends on uninitialised value(s)
==6167==    at 0x5EC7B4: Fzerop (data.c:2396)
==6167==    by 0x608571: Ffuncall (eval.c:2811)
==6167==    by 0x650F35: exec_byte_code (bytecode.c:916)
==6167==    by 0x608DA1: funcall_lambda (eval.c:2978)
==6167==    by 0x608764: Ffuncall (eval.c:2860)
==6167==    by 0x650F35: exec_byte_code (bytecode.c:916)
==6167==    by 0x608DA1: funcall_lambda (eval.c:2978)
==6167==    by 0x608764: Ffuncall (eval.c:2860)
==6167==    by 0x650F35: exec_byte_code (bytecode.c:916)
==6167==    by 0x608DA1: funcall_lambda (eval.c:2978)
==6167==    by 0x608A99: apply_lambda (eval.c:2919)
==6167==    by 0x6070E9: eval_sub (eval.c:2226)
==6167==
point is: 10
==6167==
==6167== HEAP SUMMARY:
[...]



valgrind emacs-25/src/temacs -Q --batch -l nasty-move-end-of-line-batch.el
[...]
==6186== Conditional jump or move depends on uninitialised value(s)
==6186==    at 0x67110E: exec_byte_code (bytecode.c:1367)
==6186==    by 0x625203: funcall_lambda (eval.c:2855)
==6186==    by 0x624B3A: Ffuncall (eval.c:2742)
==6186==    by 0x66F310: exec_byte_code (bytecode.c:880)
==6186==    by 0x625203: funcall_lambda (eval.c:2855)
==6186==    by 0x624B3A: Ffuncall (eval.c:2742)
==6186==    by 0x66F310: exec_byte_code (bytecode.c:880)
==6186==    by 0x625203: funcall_lambda (eval.c:2855)
==6186==    by 0x624EA5: apply_lambda (eval.c:2794)
==6186==    by 0x6234AA: eval_sub (eval.c:2211)
==6186==    by 0x61E3C6: Fprogn (eval.c:426)
==6186==    by 0x622F55: eval_sub (eval.c:2119)
==6186==
==6186== Conditional jump or move depends on uninitialised value(s)
==6186==    at 0x6711D8: exec_byte_code (bytecode.c:1370)
==6186==    by 0x625203: funcall_lambda (eval.c:2855)
==6186==    by 0x624B3A: Ffuncall (eval.c:2742)
==6186==    by 0x66F310: exec_byte_code (bytecode.c:880)
==6186==    by 0x625203: funcall_lambda (eval.c:2855)
==6186==    by 0x624B3A: Ffuncall (eval.c:2742)
==6186==    by 0x66F310: exec_byte_code (bytecode.c:880)
==6186==    by 0x625203: funcall_lambda (eval.c:2855)
==6186==    by 0x624EA5: apply_lambda (eval.c:2794)
==6186==    by 0x6234AA: eval_sub (eval.c:2211)
==6186==    by 0x61E3C6: Fprogn (eval.c:426)
==6186==    by 0x622F55: eval_sub (eval.c:2119)
==6186==
==6186== Conditional jump or move depends on uninitialised value(s)
==6186==    at 0x6712C1: exec_byte_code (bytecode.c:1379)
==6186==    by 0x625203: funcall_lambda (eval.c:2855)
==6186==    by 0x624B3A: Ffuncall (eval.c:2742)
==6186==    by 0x66F310: exec_byte_code (bytecode.c:880)
==6186==    by 0x625203: funcall_lambda (eval.c:2855)
==6186==    by 0x624B3A: Ffuncall (eval.c:2742)
==6186==    by 0x66F310: exec_byte_code (bytecode.c:880)
==6186==    by 0x625203: funcall_lambda (eval.c:2855)
==6186==    by 0x624EA5: apply_lambda (eval.c:2794)
==6186==    by 0x6234AA: eval_sub (eval.c:2211)
==6186==    by 0x61E3C6: Fprogn (eval.c:426)
==6186==    by 0x622F55: eval_sub (eval.c:2119)
==6186==
point is: 10
==6186==
==6186== HEAP SUMMARY:
[...]

Re: Certain numbers of special forms cause changing behaviour on function calls in --batch

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@users.sourceforge.net>
> Date: Sun, 10 Jul 2016 07:33:47 -0400
> Cc: Eli Zaretskii <eliz@gnu.org>, Michael Heerdegen <michael_heerdegen@web.de>, me@wilfred.me.uk, 
> 	Emacs developers <emacs-devel@gnu.org>
> 
> I have gcc 5.3.0 here, and I get 10 (correct) when I compile Emacs
> 24.5 configured with --enable-checking, but 20 (incorrect) without
> that flag.

I've just built 24.5 with GCC 5.3.0 and without --enable-checking, and
I still don't see the problem in the resulting binary.

> In both cases running temacs with valgrind gives some
> errors which look suspicious (the version without --enable-checking
> additionally gives a bunch of what I assume are false positives while
> starting up). Emacs 25 gives similar errors (I haven't tried Emacs 25
> without --enable-check).
> 
> (I'm showing here just the results from the --enable-checking
> versions, without it's the same except for printing 20 instead of 10)
> 
> valgrind emacs-24.5/src/temacs -Q --batch -l nasty-move-end-of-line-batch.el
> [...]
> ==6167== Conditional jump or move depends on uninitialised value(s)
> ==6167==    at 0x55E74E: CHECK_NUMBER_OR_FLOAT (lisp.h:2627)
> ==6167==    by 0x5EC762: Fzerop (data.c:2387)

I don't understand these errors.  Please show the macro-expanded
source of Fzerop with that compiler and configuration options.

Re: Certain numbers of special forms cause changing behaviour on function calls in --batch

[Noam Postavsky]

On Sun, Jul 10, 2016 at 10:53 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Noam Postavsky <npostavs@users.sourceforge.net>
>> Date: Sun, 10 Jul 2016 07:33:47 -0400
>> Cc: Eli Zaretskii <eliz@gnu.org>, Michael Heerdegen <michael_heerdegen@web.de>, me@wilfred.me.uk,
>>       Emacs developers <emacs-devel@gnu.org>
>>
>> I have gcc 5.3.0 here, and I get 10 (correct) when I compile Emacs
>> 24.5 configured with --enable-checking, but 20 (incorrect) without
>> that flag.
>
> I've just built 24.5 with GCC 5.3.0 and without --enable-checking, and
> I still don't see the problem in the resulting binary.

Hmm, that's annoying.

>> valgrind emacs-24.5/src/temacs -Q --batch -l nasty-move-end-of-line-batch.el
>> [...]
>> ==6167== Conditional jump or move depends on uninitialised value(s)
>> ==6167==    at 0x55E74E: CHECK_NUMBER_OR_FLOAT (lisp.h:2627)
>> ==6167==    by 0x5EC762: Fzerop (data.c:2387)
>
> I don't understand these errors.  Please show the macro-expanded
> source of Fzerop with that compiler and configuration options.

I don't think Fzerop is especially relevant, AFAICT, the error just
means Fzerop received an unitialized value as its first argument. The
Emacs-25 error doesn't include Fzerop (because zerop is optimized to
(= <arg> 0) I think).

Lisp_Object Fzerop (Lisp_Object) ; static struct Lisp_Subr
# 2383 "data.c" 3 4
_Alignas
# 2383 "data.c"
(8) Szerop = { { PVEC_SUBR << PSEUDOVECTOR_AREA_BITS }, { .a1 = Fzerop
}, 1, 1, "zerop", 0, 0}; Lisp_Object Fzerop

  (register Lisp_Object number)
{
  CHECK_NUMBER_OR_FLOAT (number); //// <<<<---- this is line 2387

  if ((((enum Lisp_Type) ((number) & ~(
# 2389 "data.c" 3 4
     1
# 2389 "data.c"
     ? - (1 << 3) : (0x7fffffffffffffffL >> (3 - 1))))) == Lisp_Float))
    {
      if (XFLOAT_DATA (number) == 0.0)
 return Qt;
      return Qnil;
    }

  if (!((number) >> INTTYPEBITS))
    return Qt;
  return Qnil;
}

Re: Certain numbers of special forms cause changing behaviour on function calls in --batch

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@users.sourceforge.net>
> Date: Sun, 10 Jul 2016 12:23:17 -0400
> Cc: Yasushi SHOJI <yasushi.shoji@gmail.com>, Michael Heerdegen <michael_heerdegen@web.de>, me@wilfred.me.uk, 
> 	Emacs developers <emacs-devel@gnu.org>
> 
> On Sun, Jul 10, 2016 at 10:53 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> >> From: Noam Postavsky <npostavs@users.sourceforge.net>
> >> Date: Sun, 10 Jul 2016 07:33:47 -0400
> >> Cc: Eli Zaretskii <eliz@gnu.org>, Michael Heerdegen <michael_heerdegen@web.de>, me@wilfred.me.uk,
> >>       Emacs developers <emacs-devel@gnu.org>
> >>
> >> I have gcc 5.3.0 here, and I get 10 (correct) when I compile Emacs
> >> 24.5 configured with --enable-checking, but 20 (incorrect) without
> >> that flag.
> >
> > I've just built 24.5 with GCC 5.3.0 and without --enable-checking, and
> > I still don't see the problem in the resulting binary.
> 
> Hmm, that's annoying.

This whole issue is annoying.

> >> valgrind emacs-24.5/src/temacs -Q --batch -l nasty-move-end-of-line-batch.el
> >> [...]
> >> ==6167== Conditional jump or move depends on uninitialised value(s)
> >> ==6167==    at 0x55E74E: CHECK_NUMBER_OR_FLOAT (lisp.h:2627)
> >> ==6167==    by 0x5EC762: Fzerop (data.c:2387)
> >
> > I don't understand these errors.  Please show the macro-expanded
> > source of Fzerop with that compiler and configuration options.
> 
> I don't think Fzerop is especially relevant, AFAICT, the error just
> means Fzerop received an unitialized value as its first argument.

Sorry, I;m not following: Fzerop is called from Ffuncall, i.e. from
Lisp, so how can the value of Fzerop's argument be uninitialized?
What am I missing?

> Lisp_Object Fzerop (Lisp_Object) ; static struct Lisp_Subr
> # 2383 "data.c" 3 4
> _Alignas
> # 2383 "data.c"
> (8) Szerop = { { PVEC_SUBR << PSEUDOVECTOR_AREA_BITS }, { .a1 = Fzerop
> }, 1, 1, "zerop", 0, 0}; Lisp_Object Fzerop
> 
>   (register Lisp_Object number)
> {
>   CHECK_NUMBER_OR_FLOAT (number); //// <<<<---- this is line 2387
> 
>   if ((((enum Lisp_Type) ((number) & ~(
> # 2389 "data.c" 3 4
>      1
> # 2389 "data.c"
>      ? - (1 << 3) : (0x7fffffffffffffffL >> (3 - 1))))) == Lisp_Float))
>     {
>       if (XFLOAT_DATA (number) == 0.0)
>  return Qt;
>       return Qnil;
>     }
> 
>   if (!((number) >> INTTYPEBITS))
>     return Qt;
>   return Qnil;
> }

That's what I see, thanks.  Very strange, this error from valgrind.
And we are none the wiser.

Re: Certain numbers of special forms cause changing behaviour on function calls in --batch

[Noam Postavsky]

On Sun, Jul 10, 2016 at 12:45 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> I don't think Fzerop is especially relevant, AFAICT, the error just
>> means Fzerop received an unitialized value as its first argument.
>
> Sorry, I;m not following: Fzerop is called from Ffuncall, i.e. from
> Lisp, so how can the value of Fzerop's argument be uninitialized?
> What am I missing?

Possibly there is an error in the bytecode interpreter? Is there an
easy way to trace pushes and pops to the stack?

Re: Certain numbers of special forms cause changing behaviour on function calls in --batch

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@users.sourceforge.net>
> Date: Sun, 10 Jul 2016 13:03:17 -0400
> Cc: Yasushi SHOJI <yasushi.shoji@gmail.com>, Michael Heerdegen <michael_heerdegen@web.de>, me@wilfred.me.uk, 
> 	Emacs developers <emacs-devel@gnu.org>
> 
> On Sun, Jul 10, 2016 at 12:45 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> >> I don't think Fzerop is especially relevant, AFAICT, the error just
> >> means Fzerop received an unitialized value as its first argument.
> >
> > Sorry, I;m not following: Fzerop is called from Ffuncall, i.e. from
> > Lisp, so how can the value of Fzerop's argument be uninitialized?
> > What am I missing?
> 
> Possibly there is an error in the bytecode interpreter?

Could be, but IMO unlikely: such a problem would appear all over the
place, not just in some obscure script.

> Is there an easy way to trace pushes and pops to the stack?

Maybe, but since we already know this is a Heisenbug, I very much
doubt we will be able to catch it that way.  We don't even know if the
call to zerop is related to the problem.

I still think we should try continuing what Michael started, but each
time you make a change in simple.el, re-dump Emacs before trying the
recipe.  (And make very small changes each time, because it looks like
large changes make the bug disappear.)  The idea is to see why does
Emacs move an extra line in batch mode, and I think the answer is
somewhere inside move-end-of-line or functions/subroutines it calls.

Re: Certain numbers of special forms cause changing behaviour on function calls in --batch

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

> That's what I see, thanks.  Very strange, this error from valgrind.
> And we are none the wiser.

--track-origins=yes gives more information.

==25255== Conditional jump or move depends on uninitialised value(s)
==25255==    at 0x5F597E: MARKERP (lisp.h:2614)
==25255==    by 0x5F597E: exec_byte_code (bytecode.c:1367)
==25255==    by 0x5B4582: Ffuncall (eval.c:2754)
==25255==    by 0x5F394C: exec_byte_code (bytecode.c:880)
==25255==    by 0x5B4582: Ffuncall (eval.c:2754)
==25255==    by 0x5F394C: exec_byte_code (bytecode.c:880)
==25255==    by 0x5B336C: apply_lambda (eval.c:2794)
==25255==    by 0x5B36B2: eval_sub (eval.c:2241)
==25255==    by 0x5B3EEC: Fprogn (eval.c:426)
==25255==    by 0x5B3BFC: eval_sub (eval.c:2119)
==25255==    by 0x5B3BFC: eval_sub (eval.c:2119)
==25255==    by 0x5B3EEC: Fprogn (eval.c:426)
==25255==    by 0x5AD694: Fsave_excursion (editfns.c:1014)
==25255==  Uninitialised value was created by a stack allocation
==25255==    at 0x57942A: Fvertical_motion (indent.c:1993)

One of the local variables of Fvertical_motion isn't properly
initialized.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Certain numbers of special forms cause changing behaviour on function calls in --batch

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: Noam Postavsky <npostavs@users.sourceforge.net>,  michael_heerdegen@web.de,  yasushi.shoji@gmail.com,  me@wilfred.me.uk,  emacs-devel@gnu.org
> Date: Sun, 10 Jul 2016 19:10:39 +0200
> 
> --track-origins=yes gives more information.
> 
> ==25255== Conditional jump or move depends on uninitialised value(s)
> ==25255==    at 0x5F597E: MARKERP (lisp.h:2614)
> ==25255==    by 0x5F597E: exec_byte_code (bytecode.c:1367)
> ==25255==    by 0x5B4582: Ffuncall (eval.c:2754)
> ==25255==    by 0x5F394C: exec_byte_code (bytecode.c:880)
> ==25255==    by 0x5B4582: Ffuncall (eval.c:2754)
> ==25255==    by 0x5F394C: exec_byte_code (bytecode.c:880)
> ==25255==    by 0x5B336C: apply_lambda (eval.c:2794)
> ==25255==    by 0x5B36B2: eval_sub (eval.c:2241)
> ==25255==    by 0x5B3EEC: Fprogn (eval.c:426)
> ==25255==    by 0x5B3BFC: eval_sub (eval.c:2119)
> ==25255==    by 0x5B3BFC: eval_sub (eval.c:2119)
> ==25255==    by 0x5B3EEC: Fprogn (eval.c:426)
> ==25255==    by 0x5AD694: Fsave_excursion (editfns.c:1014)
> ==25255==  Uninitialised value was created by a stack allocation
> ==25255==    at 0x57942A: Fvertical_motion (indent.c:1993)
> 
> One of the local variables of Fvertical_motion isn't properly
> initialized.

Thanks.  Does the patch below fix the problem?

diff --git a/src/indent.c b/src/indent.c
index bc59239..ba6612b 100644
--- a/src/indent.c
+++ b/src/indent.c
@@ -2027,6 +2027,7 @@ whether or not it is currently displayed in some window.  */)
       struct position pos;
       pos = *vmotion (PT, PT_BYTE, XINT (lines), w);
       SET_PT_BOTH (pos.bufpos, pos.bytepos);
+      it.vpos = pos.vpos;
     }
   else
     {

Re: Certain numbers of special forms cause changing behaviour on function calls in --batch

[Paul Eggert]

On 07/10/2016 07:37 PM, Eli Zaretskii wrote:
> Does the patch below fix the problem?

Thanks. The patchfixes the problem for me; I reproduced it in an 
artificial environment using GCC 6.

I think this fix should go into emacs-25, as it's small and isolated and 
clearly a fix and the failure symptoms are quite obscure(possibly 
already observed before but never pinned down).

Re: Certain numbers of special forms cause changing behaviour on function calls in --batch

[Eli Zaretskii]

> Cc: michael_heerdegen@web.de, yasushi.shoji@gmail.com, me@wilfred.me.uk,
>  emacs-devel@gnu.org, npostavs@users.sourceforge.net
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Sun, 10 Jul 2016 20:40:52 +0200
> 
> On 07/10/2016 07:37 PM, Eli Zaretskii wrote:
> > Does the patch below fix the problem?
> 
> Thanks. The patchfixes the problem for me; I reproduced it in an 
> artificial environment using GCC 6.

Thanks for testing.

> I think this fix should go into emacs-25, as it's small and isolated and 
> clearly a fix and the failure symptoms are quite obscure(possibly 
> already observed before but never pinned down).

Sure.  Pushed to emacs-25.

Thanks to everybody who helped to dig into this tricky issue.

Re: Certain numbers of special forms cause changing behaviour on function calls in --batch

[Andreas Schwab]

Andreas Schwab <schwab@linux-m68k.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> That's what I see, thanks.  Very strange, this error from valgrind.
>> And we are none the wiser.
>
> --track-origins=yes gives more information.
>
> ==25255== Conditional jump or move depends on uninitialised value(s)
> ==25255==    at 0x5F597E: MARKERP (lisp.h:2614)
> ==25255==    by 0x5F597E: exec_byte_code (bytecode.c:1367)
> ==25255==    by 0x5B4582: Ffuncall (eval.c:2754)
> ==25255==    by 0x5F394C: exec_byte_code (bytecode.c:880)
> ==25255==    by 0x5B4582: Ffuncall (eval.c:2754)
> ==25255==    by 0x5F394C: exec_byte_code (bytecode.c:880)
> ==25255==    by 0x5B336C: apply_lambda (eval.c:2794)
> ==25255==    by 0x5B36B2: eval_sub (eval.c:2241)
> ==25255==    by 0x5B3EEC: Fprogn (eval.c:426)
> ==25255==    by 0x5B3BFC: eval_sub (eval.c:2119)
> ==25255==    by 0x5B3BFC: eval_sub (eval.c:2119)
> ==25255==    by 0x5B3EEC: Fprogn (eval.c:426)
> ==25255==    by 0x5AD694: Fsave_excursion (editfns.c:1014)
> ==25255==  Uninitialised value was created by a stack allocation
> ==25255==    at 0x57942A: Fvertical_motion (indent.c:1993)
>
> One of the local variables of Fvertical_motion isn't properly
> initialized.

This is executing this part from line-move-1:

243:18	constant  vertical-motion
244	constant  1
245	call	  1
246	constant  0
247	eqlsign	  

Thus it is actually it.vpos that is not initialized.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Certain numbers of special forms cause changing behaviour on function calls in --batch

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Date: Sun, 10 Jul 2016 19:37:43 +0200
> Cc: michael_heerdegen@web.de, yasushi.shoji@gmail.com, me@wilfred.me.uk,
> 	Noam Postavsky <npostavs@users.sourceforge.net>, emacs-devel@gnu.org
> 
> > One of the local variables of Fvertical_motion isn't properly
> > initialized.
> 
> This is executing this part from line-move-1:
> 
> 243:18	constant  vertical-motion
> 244	constant  1
> 245	call	  1
> 246	constant  0
> 247	eqlsign	  
> 
> Thus it is actually it.vpos that is not initialized.

Yep, see the proposed patch I sent a few minutes ago.

11.5 years and 3 major versions of GCC to find a bug.  Amazing!

RE: Certain numbers of special forms cause changing behaviour on function calls in --batch

[Drew Adams]

> 11.5 years and 3 major versions of GCC to find a bug.  Amazing!

And (I assume) a good feeling from finding it. ;-)  Congrats!

Re: Certain numbers of special forms cause changing behaviour on function calls in --batch

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 126 bytes --]

On 2016-07-10 19:37, Andreas Schwab wrote:
> Thus it is actually it.vpos that is not initialized.

Brilliant debugging.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Certain numbers of special forms cause changing behaviour on function calls in --batch

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Cc: me@wilfred.me.uk,  emacs-devel@gnu.org
> Date: Tue, 05 Jul 2016 23:21:13 +0200
> 
> The first thing that I found out: it makes a difference whether I use
> the compiled simple.elc or the source simple.el.  Only with the compiled
> simple.elc I get the unexpected result of 20.  But when I load the
> uncompiled simple.el into the batch session, I'm not able to reproduce
> the issue (I get 10).
> 
> Does that give us hints about how I could proceed?

The only idea I have is to try all this in a freshly bootstrapped
build.

Re: Certain numbers of special forms cause changing behaviour on function calls in --batch

[Andreas Röhler]

On 28.06.2016 23:12, Michael Heerdegen wrote:
> Wilfred Hughes <me@wilfred.me.uk> writes:
>
>> However, running it in --batch gives:
>>
>> $ emacs -batch -l ~/projects/pyimport/nasty.el
>> point is: 20
>>
>> This is incorrect: for some reason move-end-of-line is behaving
>> differently here. Removing any of the extra special forms (e.g. the
>> redundant lets) fixes it.
> Strange indeed.
>
> OTOH, looking at the doc of `move-end-of-line' ("Move point to end of
> current line as displayed.") I wouldn't rely on the result in batch mode
> at all, since there is no display.
>
> `end-of-line' yields the same result in both situations here.
>
>
> Michael.
>

A plausible guess. `move-end-of-line' is obviously designed for 
interactive use. It deals with var `line-move-visual' for example. Are 
there reasons not to employ `end-of-line'?

Re: Certain numbers of special forms cause changing behaviour on function calls in --batch

[Eli Zaretskii]

> From: Andreas Röhler <andreas.roehler@online.de>
> Date: Thu, 30 Jun 2016 07:58:57 +0200
> Cc: Wilfred Hughes <me@wilfred.me.uk>
> 
> A plausible guess. `move-end-of-line' is obviously designed for 
> interactive use. It deals with var `line-move-visual' for example.

If that's the reason, setting line-move-visual to nil in the
interactive session will produce the same "buggy" behavior.  Does it?

> Are there reasons not to employ `end-of-line'?

Compare their doc strings, and you will see the reason.  One moves to
the visual end, the other to the logical end.

This bug, whatever it is, will not be solved by guessing.  It will be
solved by a reproducible recipe and debugging.

Re: Certain numbers of special forms cause changing behaviour on function calls in --batch

[Andreas Röhler]

On 30.06.2016 17:13, Eli Zaretskii wrote:
>> From: Andreas Röhler <andreas.roehler@online.de>
>> Date: Thu, 30 Jun 2016 07:58:57 +0200
>> Cc: Wilfred Hughes <me@wilfred.me.uk>
>>
>> A plausible guess. `move-end-of-line' is obviously designed for
>> interactive use. It deals with var `line-move-visual' for example.
> If that's the reason, setting line-move-visual to nil in the
> interactive session will produce the same "buggy" behavior.  Does it?
>
>> Are there reasons not to employ `end-of-line'?
> Compare their doc strings, and you will see the reason.  One moves to
> the visual end, the other to the logical end.
>
> This bug, whatever it is, will not be solved by guessing.  It will be
> solved by a reproducible recipe and debugging.

AFAIU  parts of the display engine obviously work different in batch mode

At

http://debbugs.gnu.org/db/16/16853.html

it reads: "

The point of the
example was to show that pos-visible-in-window-p doesn't work normally
in batch mode. "

Not sure if that must be a bug. It might make sense and being the purpose of batch, not to provide real buffer visibility.

Hence a visual line might not exist for the very same reasons then - at 
least if I had to design a batch mode ;)

Re: Certain numbers of special forms cause changing behaviour on function calls in --batch

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org, me@wilfred.me.uk
> From: Andreas Röhler <andreas.roehler@online.de>
> Date: Thu, 30 Jun 2016 19:20:50 +0200
> 
> AFAIU  parts of the display engine obviously work different in batch mode
> 
> At
> 
> http://debbugs.gnu.org/db/16/16853.html
> 
> it reads: "
> 
> The point of the
> example was to show that pos-visible-in-window-p doesn't work normally
> in batch mode. "
> 
> Not sure if that must be a bug. It might make sense and being the purpose of batch, not to provide real buffer visibility.
> 
> Hence a visual line might not exist for the very same reasons then - at 
> least if I had to design a batch mode ;)

Yes.  Which is why I gave a simple recipe to find out whether this is
the source of the problem.

Re: Certain numbers of special forms cause changing behaviour on function calls in --batch

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> If that's the reason, setting line-move-visual to nil in the
> interactive session will produce the same "buggy" behavior.  Does it?

No, it gives me 10, the "nonbuggy" value.

Michael.

Re: Certain numbers of special forms cause changing behaviour on function calls in --batch

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Date: Thu, 30 Jun 2016 23:22:36 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > If that's the reason, setting line-move-visual to nil in the
> > interactive session will produce the same "buggy" behavior.  Does it?
> 
> No, it gives me 10, the "nonbuggy" value.

Then this theory eats dust.  Which doesn't surprise me, because if it
were the reason, I should have seen the same behavior on my system, as
that behavior is deterministic.