Size of emacs.pdmp changes between successive installs

Size of emacs.pdmp changes between successive installs

[Ulrich Mueller]

For two successive installs from a clean checkout of the same git
commit, emacs.pdmp ends up having slightly different size:

$ stat -c %s /usr/libexec/emacs/27.0.50/x86_64-pc-linux-gnu/emacs.pdmp
10563744

$ stat -c %s /usr/libexec/emacs/27.0.50/x86_64-pc-linux-gnu/emacs.pdmp
10563728

Why is that so? Shouldn't it be deterministic?

(I've tried option --disable-build-details and disabling parallel make,
but neither of them makes a difference.)

Re: Size of emacs.pdmp changes between successive installs

[Paul Eggert]

On 11/19/19 8:59 AM, Ulrich Mueller wrote:
> Why is that so? Shouldn't it be deterministic?

I would think so, yes. What differs about the two pdump files and why? (I 
realize this might not be an easy question to answer....)

Re: Size of emacs.pdmp changes between successive installs

[Eli Zaretskii]

> From: Ulrich Mueller <ulm@gentoo.org>
> Date: Tue, 19 Nov 2019 17:59:58 +0100
> 
> For two successive installs from a clean checkout of the same git
> commit, emacs.pdmp ends up having slightly different size:
> 
> $ stat -c %s /usr/libexec/emacs/27.0.50/x86_64-pc-linux-gnu/emacs.pdmp
> 10563744
> 
> $ stat -c %s /usr/libexec/emacs/27.0.50/x86_64-pc-linux-gnu/emacs.pdmp
> 10563728
> 
> Why is that so? Shouldn't it be deterministic?

Please describe step by step what you did to produce each of the
installs.  The above is not detailed enough to understand what could
be the cause of the differences.

Re: Size of emacs.pdmp changes between successive installs

[Andreas Schwab]

On Nov 19 2019, Ulrich Mueller wrote:

> For two successive installs from a clean checkout of the same git
> commit, emacs.pdmp ends up having slightly different size:
>
> $ stat -c %s /usr/libexec/emacs/27.0.50/x86_64-pc-linux-gnu/emacs.pdmp
> 10563744
>
> $ stat -c %s /usr/libexec/emacs/27.0.50/x86_64-pc-linux-gnu/emacs.pdmp
> 10563728
>
> Why is that so? Shouldn't it be deterministic?

Try disabling ASLR.  That doesn't make it completely reproducible,
though.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Size of emacs.pdmp changes between successive installs

[Andreas Schwab]

On Nov 19 2019, Eli Zaretskii wrote:

> Please describe step by step what you did to produce each of the
> installs.  The above is not detailed enough to understand what could
> be the cause of the differences.

$ make; mv src/emacs.pdump{,.alt}; make
$ cmp -cl src/emacs.pdmp{,.alt}

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Size of emacs.pdmp changes between successive installs

[Paul Eggert]

On 11/19/19 10:16 AM, Andreas Schwab wrote:
> Try disabling ASLR.

Emacs already tries to disable ASLR when dumping. Perhaps its attempt 
isn't working.

Re: Size of emacs.pdmp changes between successive installs

[Andreas Schwab]

On Nov 19 2019, Paul Eggert wrote:

> Emacs already tries to disable ASLR when dumping.

Only when unexecing.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Size of emacs.pdmp changes between successive installs

[Ulrich Mueller]

>>>>> On Tue, 19 Nov 2019, Andreas Schwab wrote:

> On Nov 19 2019, Ulrich Mueller wrote:
>> For two successive installs from a clean checkout of the same git
>> commit, emacs.pdmp ends up having slightly different size:
>>
>> $ stat -c %s /usr/libexec/emacs/27.0.50/x86_64-pc-linux-gnu/emacs.pdmp
>> 10563744
>>
>> $ stat -c %s /usr/libexec/emacs/27.0.50/x86_64-pc-linux-gnu/emacs.pdmp
>> 10563728
>>
>> Why is that so? Shouldn't it be deterministic?

> Try disabling ASLR.  That doesn't make it completely reproducible,
> though.

Indeed. After doing "echo -n 0 >/proc/sys/kernel/randomize_va_space"
I get at least the same size every time.

This is with Linux 4.14.154 and glibc 2.30.

Re: Size of emacs.pdmp changes between successive installs

[Ulrich Mueller]

>>>>> On Tue, 19 Nov 2019, Andreas Schwab wrote:

> On Nov 19 2019, Paul Eggert wrote:
>> Emacs already tries to disable ASLR when dumping.

> Only when unexecing.

Right, with this trivial patch I'm getting a reproducible size
of emacs.pdmp, too:

--- src/emacs.c~	2019-11-20 00:27:55.016051255 +0100
+++ src/emacs.c	2019-11-20 00:37:56.740331939 +0100
@@ -1055,7 +1055,7 @@
 #endif
 
   argc = maybe_disable_address_randomization (
-    will_dump_with_unexec_p (), argc, argv);
+    will_dump_p (), argc, argv);
 
 #if defined GNU_LINUX && defined HAVE_UNEXEC
   if (!initialized)

Re: Size of emacs.pdmp changes between successive installs

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 227 bytes --]

>> Only when unexecing.

> Right, with this trivial patch I'm getting a reproducible size
> of emacs.pdmp, too:

Thanks to both of you. I installed the attached, which does something 
similar and simplifies the code a bit too.

[-- Attachment #2: 0001-Make-.pdmp-file-more-reproducible.patch --]
[-- Type: text/x-patch, Size: 2873 bytes --]

From 9ac586928bbef16b572bcb34bfcbcb5515c2335a Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Tue, 19 Nov 2019 18:23:01 -0800
Subject: [PATCH] Make .pdmp file more reproducible
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Problem reported by Ulrich Müller
<https://lists.gnu.org/r/emacs-devel/2019-11/msg00757.html>
and diagnosed by Andreas Schwab
<https://lists.gnu.org/r/emacs-devel/2019-11/msg00774.html>.
* src/sysdep.c (maybe_disable_address_randomization):
Disable ASLR if any kind of dumping, instead of merely if
unexec dumping.  Omit first arg for simplicity; all uses changed.
---
 src/emacs.c  | 3 +--
 src/lisp.h   | 4 ++--
 src/sysdep.c | 7 +++++--
 3 files changed, 8 insertions(+), 6 deletions(-)

diff --git a/src/emacs.c b/src/emacs.c
index 21a05d337e..8a6e34deda 100644
--- a/src/emacs.c
+++ b/src/emacs.c
@@ -1054,8 +1054,7 @@ main (int argc, char **argv)
     load_pdump (argc, argv);
 #endif
 
-  argc = maybe_disable_address_randomization (
-    will_dump_with_unexec_p (), argc, argv);
+  argc = maybe_disable_address_randomization (argc, argv);
 
 #if defined GNU_LINUX && defined HAVE_UNEXEC
   if (!initialized)
diff --git a/src/lisp.h b/src/lisp.h
index 1d25add928..e0ae2c4262 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -4576,10 +4576,10 @@ #define DAEMON_RUNNING (w32_daemon_event != INVALID_HANDLE_VALUE)
 
 /* Defined in sysdep.c.  */
 #ifdef HAVE_PERSONALITY_ADDR_NO_RANDOMIZE
-extern int maybe_disable_address_randomization (bool, int, char **);
+extern int maybe_disable_address_randomization (int, char **);
 #else
 INLINE int
-maybe_disable_address_randomization (bool dumping, int argc, char **argv)
+maybe_disable_address_randomization (int argc, char **argv)
 {
   return argc;
 }
diff --git a/src/sysdep.c b/src/sysdep.c
index aa18ee22fd..e34ab2eb58 100644
--- a/src/sysdep.c
+++ b/src/sysdep.c
@@ -158,14 +158,17 @@ #define ULLONG_MAX TYPE_MAXIMUM (unsigned long long int)
 /* Try to disable randomization if the current process needs it and
    does not appear to have it already.  */
 int
-maybe_disable_address_randomization (bool dumping, int argc, char **argv)
+maybe_disable_address_randomization (int argc, char **argv)
 {
   /* Undocumented Emacs option used only by this function.  */
   static char const aslr_disabled_option[] = "--__aslr-disabled";
 
   if (argc < 2 || strcmp (argv[1], aslr_disabled_option) != 0)
     {
-      bool disable_aslr = dumping;
+      /* If dumping via unexec, ASLR must be disabled, as otherwise
+	 data may be scattered and undumpable as a simple executable.
+	 If pdumping, disabling ASLR makes the .pdmp file reproducible.  */
+      bool disable_aslr = will_dump_p ();
 # ifdef __PPC64__
       disable_aslr = true;
 # endif
-- 
2.23.0

Re: Size of emacs.pdmp changes between successive installs

[Andreas Schwab]

On Nov 20 2019, Ulrich Mueller wrote:

> Right, with this trivial patch I'm getting a reproducible size
> of emacs.pdmp, too:

Do you get an identical pdmp, though?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Size of emacs.pdmp changes between successive installs

[Ulrich Mueller]

>>>>> On Wed, 20 Nov 2019, Andreas Schwab wrote:

> On Nov 20 2019, Ulrich Mueller wrote:
>> Right, with this trivial patch I'm getting a reproducible size
>> of emacs.pdmp, too:

> Do you get an identical pdmp, though?

No, cmp shows between 10 and 30 different bytes between successive .pdmp
files. (I have removed src/emacs, src/emacs.pdmp, and src/emacs-27.*
before calling make again, so it is not due to an increasing build
number.)

Re: Size of emacs.pdmp changes between successive installs

[Andreas Schwab]

On Nov 19 2019, Paul Eggert wrote:

> +	 If pdumping, disabling ASLR makes the .pdmp file reproducible.  */

That's not true.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Size of emacs.pdmp changes between successive installs

[Paul Eggert]

On 11/20/19 3:55 AM, Andreas Schwab wrote:
>> +	 If pdumping, disabling ASLR makes the .pdmp file reproducible.  */
> That's not true.

Suggested rewording?

Re: Size of emacs.pdmp changes between successive installs

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 103 bytes --]

On 11/20/19 11:54 AM, Paul Eggert wrote:
> Suggested rewording?

Never mind, I installed the attached.

[-- Attachment #2: 0001-src-sysdep.c-Improve-comment-wording.patch --]
[-- Type: text/x-patch, Size: 861 bytes --]

From 9ca40c4e701e925eaf4b580b20a261737d578706 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Wed, 20 Nov 2019 11:57:17 -0800
Subject: [PATCH] * src/sysdep.c: Improve comment wording.

---
 src/sysdep.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/sysdep.c b/src/sysdep.c
index e34ab2eb58..b436bfe74a 100644
--- a/src/sysdep.c
+++ b/src/sysdep.c
@@ -167,7 +167,7 @@ maybe_disable_address_randomization (int argc, char **argv)
     {
       /* If dumping via unexec, ASLR must be disabled, as otherwise
 	 data may be scattered and undumpable as a simple executable.
-	 If pdumping, disabling ASLR makes the .pdmp file reproducible.  */
+	 If pdumping, disabling ASLR lessens differences in the .pdmp file.  */
       bool disable_aslr = will_dump_p ();
 # ifdef __PPC64__
       disable_aslr = true;
-- 
2.23.0