bug#2507: 23.0.91; Stripping emacs.exe on MS-Windows produces an invalid program

bug#2507: 23.0.91; Stripping emacs.exe on MS-Windows produces an invalid program

[Eli Zaretskii]

Please write in English if possible, because the Emacs maintainers
usually do not have translators to read other languages for them.

Your bug report will be posted to the emacs-pretest-bug@gnu.org mailing list.

Please describe exactly what actions triggered the bug
and the precise symptoms of the bug:

  strip emacs.exe
  emacs -Q

results in the OS popping an error dialog saying:

  emacs.exe is not a valid Win32 application

If Emacs crashed, and you have the Emacs process in the gdb debugger,
please include the output from the following gdb commands:
    `bt full' and `xbacktrace'.
If you would like to further debug the crash, please read the file
d:/gnu/emacs/etc/DEBUG for instructions.


In GNU Emacs 23.0.91.1 (i386-mingw-nt5.1.2600)
 of 2009-02-28 on HOME-C4E4A596F7
Windowing system distributor `Microsoft Corp.', version 5.1.2600
configured using `configure --with-gcc (3.4)'

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: ENU
  value of $XMODIFIERS: nil
  locale-coding-system: cp1255
  default-enable-multibyte-characters: t

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  tool-bar-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  global-auto-composition-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
M-x r e p o r t - e m a c s - b u <tab> <return>

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.

bug#2507: 23.0.91; Stripping emacs.exe on MS-Windows produces an invalid program

[Eli Zaretskii]

> Date: Sat, 28 Feb 2009 13:01:07 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 
> 
>   strip emacs.exe
>   emacs -Q
> 
> results in the OS popping an error dialog saying:
> 
>   emacs.exe is not a valid Win32 application

More info: building Emacs like this:

    make USER_LDFLAGS=-s install

produces an already stripped emacs.exe that is 8548352 bytes large.
So this should serve as a stop-gap in case we cannot easily find a
cure for running `strip' on the dumped Emacs.

One further idea to ponder is that the fact we now compile with
DWARF-2 debug info is the reason for the problem with stripping the
dumped Emacs.  Can someone please try rebuilding with "-gstabs"
instead of "-gdwarf-2 -g3", and see if that produces an emacs.exe
which can be safely stripped?

bug#2507: 23.0.91; Stripping emacs.exe on MS-Windows produces an invalid program

[Harald Maier]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Sat, 28 Feb 2009 13:01:07 +0200
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: 
>> 
>>   strip emacs.exe
>>   emacs -Q
>> 
>> results in the OS popping an error dialog saying:
>> 
>>   emacs.exe is not a valid Win32 application
>
> More info: building Emacs like this:
>
>     make USER_LDFLAGS=-s install
>
> produces an already stripped emacs.exe that is 8548352 bytes large.
> So this should serve as a stop-gap in case we cannot easily find a
> cure for running `strip' on the dumped Emacs.
>
> One further idea to ponder is that the fact we now compile with
> DWARF-2 debug info is the reason for the problem with stripping the
> dumped Emacs.  Can someone please try rebuilding with "-gstabs"
> instead of "-gdwarf-2 -g3", and see if that produces an emacs.exe
> which can be safely stripped?

I tried the "-gstabs" debug info, but after stripping it didn't work
too. But the output size of that compilation and linking is far smaller.

Result: 12.55 MB (-gstabs) instead of 31.33 MB (-gdwarf2 -g3).

Harald

bug#2507: 23.0.91; Stripping emacs.exe on MS-Windows produces an invalid program

[Eli Zaretskii]

> From: Harald Maier <harald@maierh.de>
> Cc: 2507@emacsbugs.donarmstrong.com
> Date: Sun, 01 Mar 2009 07:01:27 +0100
> 
> > One further idea to ponder is that the fact we now compile with
> > DWARF-2 debug info is the reason for the problem with stripping the
> > dumped Emacs.  Can someone please try rebuilding with "-gstabs"
> > instead of "-gdwarf-2 -g3", and see if that produces an emacs.exe
> > which can be safely stripped?
> 
> I tried the "-gstabs" debug info, but after stripping it didn't work
> too.

Thanks for trying.

> But the output size of that compilation and linking is far smaller.
> 
> Result: 12.55 MB (-gstabs) instead of 31.33 MB (-gdwarf2 -g3).

This is expected: DWARF-2 produces much more voluminous debug
information (and thus enables several useful debugging features in
GDB).

Btw, the 32MB executable is only that large on disk; when you invoke
it, the debug info is not read at all, so the memory footprint of a
running Emacs is not affected.  The debug info is read only by GDB
when you run Emacs under the debugger.

(I'm not saying that we should ignore this problem, just that it only
matters for disk space and the size of the tarball.)

bug#2507: 23.0.91; Stripping emacs.exe on MS-Windows produces an invalid program

[Juanma Barranquero]

On Sat, Feb 28, 2009 at 13:14, Eli Zaretskii <eliz@gnu.org> wrote:

> a cure for running `strip' on the dumped Emacs.

AFAICS, the problem is not limited to the dumped Emacs. A stripped
temacs.exe fails in the same way.

    Juanma

bug#2507: 23.0.91; Stripping emacs.exe on MS-Windows produces an invalid program

[Eli Zaretskii]

> Date: Mon, 2 Mar 2009 12:55:14 +0100
> From: Juanma Barranquero <lekktu@gmail.com>
> 
> On Sat, Feb 28, 2009 at 13:14, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > a cure for running `strip' on the dumped Emacs.
> 
> AFAICS, the problem is not limited to the dumped Emacs. A stripped
> temacs.exe fails in the same way.

Probably because temacs.exe is a product of dumping temacs.bin.  See
src/makefile.w32-in.  Here's the relevant fragment from a typical
build session:

  "../nt/oo-spd/i386/addsection" "oo-spd/i386/temacs.bin" "oo-spd/i386/temacs.exe" EMHEAP 16
  Dumping from oo-spd/i386/temacs.bin
            to oo-spd/i386/temacs.exe

bug#2507: 23.0.91; Stripping emacs.exe on MS-Windows produces an invalid program

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Please describe exactly what actions triggered the bug
> and the precise symptoms of the bug:
>
>   strip emacs.exe
>   emacs -Q
>
> results in the OS popping an error dialog saying:
>
>   emacs.exe is not a valid Win32 application

Presumably Emacs doesn't strip itself, otherwise it wouldn't work at all
on Windows, which it apparently does.  :-)

So it this still a problem?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

bug#2507: 23.0.91; Stripping emacs.exe on MS-Windows produces an invalid program

[Juanma Barranquero]

On Mon, Jul 11, 2011 at 15:53, Lars Magne Ingebrigtsen <larsi@gnus.org> wrote:

> So it this still a problem?

It still happens, yes.

    Juanma

bug#2507: 23.0.91; Stripping emacs.exe on MS-Windows produces an invalid program

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Cc: 2507@debbugs.gnu.org
> Date: Mon, 11 Jul 2011 15:53:45 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Please describe exactly what actions triggered the bug
> > and the precise symptoms of the bug:
> >
> >   strip emacs.exe
> >   emacs -Q
> >
> > results in the OS popping an error dialog saying:
> >
> >   emacs.exe is not a valid Win32 application
> 
> Presumably Emacs doesn't strip itself, otherwise it wouldn't work at all
> on Windows, which it apparently does.  :-)

The default link command produces an unstripped binary, as on other
supported platforms.

> So it this still a problem?

Yes, of course.  No one did anything about this problem, AFAIK.

bug#2507: 23.0.91; Stripping emacs.exe on MS-Windows produces an invalid program

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Yes, of course.  No one did anything about this problem, AFAIK.

Is it something that should be done anything about, though?  Is
stripping the binary something that you'd want to do?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

bug#2507: 23.0.91; Stripping emacs.exe on MS-Windows produces an invalid program

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Cc: 2507@debbugs.gnu.org
> Date: Mon, 11 Jul 2011 18:09:41 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Yes, of course.  No one did anything about this problem, AFAIK.
> 
> Is it something that should be done anything about, though?

Yes.

> Is stripping the binary something that you'd want to do?

Yes.  For shipping a an official release's binaries, for example.  Or
for taking Emacs on a small mobile device.  The unstripped binary
measures 37MB on Windows, most of which is DWARF2 debug info.

bug#2507: 23.0.91; Stripping emacs.exe on MS-Windows produces an invalid program

[Juanma Barranquero]

On Mon, Jul 11, 2011 at 18:22, Eli Zaretskii <eliz@gnu.org> wrote:

> The unstripped binary
> measures 37MB on Windows

Or more. Mine is at ~42 MiB unstripped, ~10 MiB stripped (but I
doesn't work, of course).

    Juanma

bug#2507: 23.0.91; Stripping emacs.exe on MS-Windows produces an invalid program

[Stefan Monnier]

>> The unstripped binary measures 37MB on Windows
> Or more. Mine is at ~42 MiB unstripped, ~10 MiB stripped (but I
> doesn't work, of course).

So the stripped version can be compressed *much* further without losing
any functionality.  Cool!


        Stefan "feeling silly"

bug#2507: 23.0.91; Stripping emacs.exe on MS-Windows produces an invalid program

[Juanma Barranquero]

On Tue, Jul 12, 2011 at 06:56, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> So the stripped version can be compressed *much* further without losing
> any functionality.  Cool!

Well, yeah. A lot.

>        Stefan "feeling silly"

As we Spaniards use to say: "Ja, ja, me parto".

    Juanma

bug#2507: 23.0.91; Stripping emacs.exe on MS-Windows produces an invalid program

[Jason Rumney]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> The unstripped binary measures 37MB on Windows
>> Or more. Mine is at ~42 MiB unstripped, ~10 MiB stripped (but I
>> doesn't work, of course).
>
> So the stripped version can be compressed *much* further without losing
> any functionality.  Cool!

If you build without debug info in the first place, it does work, at
around the same size. Also if you strip temacs before dumping, it
works. The problem is only in stripping a dumped binary that had
debug info to start with. It seems the strip command removes some info
that Emacs needs to reconstruct the heap from the dumped image.

bug#2507: 23.0.91; Stripping emacs.exe on MS-Windows produces an invalid program

[Eli Zaretskii]

> From: Jason Rumney <jasonr@gnu.org>
> Date: Wed, 13 Jul 2011 22:16:36 +0800
> Cc: Juanma Barranquero <lekktu@gmail.com>,
> 	Lars Magne Ingebrigtsen <larsi@gnus.org>, 2507@debbugs.gnu.org
> 
> If you build without debug info in the first place, it does work, at
> around the same size. Also if you strip temacs before dumping, it
> works. The problem is only in stripping a dumped binary that had
> debug info to start with. It seems the strip command removes some info
> that Emacs needs to reconstruct the heap from the dumped image.

That is correct.

I looked into this some more.  The problem is that we add an extra
section to the Emacs executable (by running addsection), which serves
as the static heap.  Here's the report from "objdump -h":

  temacs.exe:     file format pei-i386

  Sections:
  Idx Name          Size      VMA       LMA       File off  Algn
    0 .text         0032e604  01001000  01001000  00001000  2**4
		    CONTENTS, ALLOC, LOAD, READONLY, CODE, DATA
    1 .data         0020aea0  01330000  01330000  00330000  2**4
		    CONTENTS, ALLOC, LOAD, DATA
    2 .rdata        000dc57c  0153b000  0153b000  0053b000  2**4
		    CONTENTS, ALLOC, LOAD, READONLY, DATA
    3 .bss          0005c410  01618000  01618000  00000000  2**4
		    ALLOC
    4 .idata        000036f4  01675000  01675000  00618000  2**2
		    CONTENTS, ALLOC, LOAD, DATA
    5 .rsrc         0000d5f0  01679000  01679000  0061c000  2**2
		    CONTENTS, ALLOC, LOAD, DATA
    6 .debug_aranges 00000ce0  01687000  01687000  0062a000  2**0
		    CONTENTS, READONLY, DEBUGGING
    7 .debug_pubnames 00011b5a  01688000  01688000  0062b000  2**0
		    CONTENTS, READONLY, DEBUGGING
    8 .debug_info   003992b0  0169a000  0169a000  0063d000  2**0
		    CONTENTS, READONLY, DEBUGGING
    9 .debug_abbrev 0001181b  01a34000  01a34000  009d7000  2**0
		    CONTENTS, READONLY, DEBUGGING
   10 .debug_line   0003f37e  01a46000  01a46000  009e9000  2**0
		    CONTENTS, READONLY, DEBUGGING
   11 .debug_frame  0001b6b8  01a86000  01a86000  00a29000  2**0
		    CONTENTS, READONLY, DEBUGGING
   12 .debug_str    0000d3bb  01aa2000  01aa2000  00a45000  2**0
		    CONTENTS, READONLY, DEBUGGING
   13 .debug_macinfo 01b54308  01ab0000  01ab0000  00a53000  2**0
		    CONTENTS, READONLY, DEBUGGING
   14 EMHEAP        01b00000  03605000  03605000  00000000  2**2
		    ALLOC

The last section, EMHEAP, is the one we add.  Now look what happens
after stripping:

  temacs.exe:     file format pei-i386

  Sections:
  Idx Name          Size      VMA       LMA       File off  Algn
    0 .text         0032e604  01001000  01001000  00000400  2**4
		    CONTENTS, ALLOC, LOAD, READONLY, CODE, DATA
    1 .data         0020aea0  01330000  01330000  0032ec00  2**4
		    CONTENTS, ALLOC, LOAD, DATA
    2 .rdata        000dc57c  0153b000  0153b000  00539c00  2**4
		    CONTENTS, ALLOC, LOAD, READONLY, DATA
    3 .bss          0005c410  01618000  01618000  00000000  2**4
		    ALLOC
    4 .idata        000036f4  01675000  01675000  00616200  2**2
		    CONTENTS, ALLOC, LOAD, DATA
    5 .rsrc         0000d5f0  01679000  01679000  00619a00  2**2
		    CONTENTS, ALLOC, LOAD, DATA
    6 EMHEAP        01b00000  03605000  03605000  00000000  2**2
		    ALLOC

The debug sections are gone, but the VMA and LMA of EMHEAP were left
intact.  By contrast, if we strip temacs.bin _before_ running
addsection, and run addsection on the stripped temacs.bin, we get
this:

  temacs.exe:     file format pei-i386

  Sections:
  Idx Name          Size      VMA       LMA       File off  Algn
    0 .text         0032e604  01001000  01001000  00001000  2**4
		    CONTENTS, ALLOC, LOAD, READONLY, CODE, DATA
    1 .data         0020aea0  01330000  01330000  00330000  2**4
		    CONTENTS, ALLOC, LOAD, DATA
    2 .rdata        000dc57c  0153b000  0153b000  0053b000  2**4
		    CONTENTS, ALLOC, LOAD, READONLY, DATA
    3 .bss          0005c410  01618000  01618000  00000000  2**4
		    ALLOC
    4 .idata        000036f4  01675000  01675000  00618000  2**2
		    CONTENTS, ALLOC, LOAD, DATA
    5 .rsrc         0000d5f0  01679000  01679000  0061c000  2**2
		    CONTENTS, ALLOC, LOAD, DATA
    6 EMHEAP        01b00000  01687000  01687000  00000000  2**2
		    ALLOC

Now EMHEAP's VMA and LMA follow the section before it.

So I think the kind of workaround mentioned in

  http://debbugs.gnu.org/cgi/bugreport.cgi?bug=2507#10

above, or some variant thereof, is the right way of producing a
stripped emacs.exe.  That is, link with -s (or strip temacs.bin after
it is produced), and then run addsection to produce temacs.exe and
finally loadup+dump into emacs.exe.

An alternative is to add code to addsection.c so that it could adjust
the EMHEAP section's VMA and LMA after emacs.exe was stripped.  No,
I'm not volunteering ;-)

P.S. I tried to adjust the VMA/LMA with objcopy, but the result is not
reliable: sometimes works, sometimes crashes.  So more than just
address adjustment is needed.

bug#2507: 23.0.91; Stripping emacs.exe on MS-Windows produces an invalid program

[Eli Zaretskii]

This problem was solved as side effect of the changes to use mmap-like
memory management for buffer text and store the dumped data in a
static array instead of a special section.

Closing.