Re: Mac OS X: Rebuild Require after Security Update 2002-11-21

Re: Mac OS X: Rebuild Require after Security Update 2002-11-21

[Jason Rumney]

Andrew Choi <akochoi_nospam_@shaw.ca> writes:

> The 2002-11-21 Security Update seems to cause Emacs executables built
> before the update to fail.  Rebuilding will make them work again.  I
> realize this is an annoying problem.  But I must admit currently I have
> no idea how to solve it.

Do you use dynamic linking to system libraries on the Mac at all?
This has caused problems between different versions of Windows before.

Re: Mac OS X: Rebuild Require after Security Update 2002-11-21

[Andrew Choi]

Jason Rumney <jasonr@gnu.org> writes:

> Andrew Choi <akochoi_nospam_@shaw.ca> writes:
> 
> > The 2002-11-21 Security Update seems to cause Emacs executables
> > built before the update to fail.  Rebuilding will make them work
> > again.  I realize this is an annoying problem.  But I must admit
> > currently I have no idea how to solve it.
> 
> Do you use dynamic linking to system libraries on the Mac at all?
> This has caused problems between different versions of Windows before.

Thanks.  Yes, that's probably the cause.  Unfortunately dynamically
linked libraries are the only choice for using most of the "system
frameworks" on Mac OS X.  I'm currently trying to determine whether some
variables in the libraries are somehow unexec'ed along with the Emacs
heap.  Of course this will cause a problem if the new libraries change
the variables' locations.  There aren't many tools available to look
into this.  But I'll figure out a way sooner or later.

Re: Mac OS X: Rebuild Require after Security Update 2002-11-21

[Steven Tamm]

Hey Andrew,

Well...  As you have probably figured out, the update changed 
libSystem.B and temacs works perfectly after the update.  So it must be 
in unexec.

The big problem (as you may have figured out) is that the pointers in 
emacs_zone are corrupted.  After the first call to xrealloc (which 
calls unexec_realloc) the program fails pretty much instantly.  The 
culprit is the call to emacs_zone->size().

Here is what emacs_zone looks like before (in the version unexec'd 
before the update):

$3 = {
   reserved1 = 0x0,
   reserved2 = 0x0,
   size = 0x900040c0 <szone_size+160>,
   malloc = 0x90005080 <szone_malloc+160>,
   calloc = 0x90009ac0 <szone_calloc+160>,
   valloc = 0x900159e0 <getattrlist>,
   free = 0x90004380 <szone_free+160>,
   realloc = 0x9000e4a0 <szone_realloc+160>,
   destroy = 0x90064f80 <szone_destroy+160>,
   zone_name = 0x6312c0 "EmacsZone",
   reserved3 = 0x0,
   reserved4 = 0x0,
   introspect = 0xa0001d54,
   reserved5 = 0x0
}

Here is what it looks like after (in the version I redumped after the 
update):

$2 = {
   reserved1 = 0x0,
   reserved2 = 0x0,
   size = 0x90004020 <szone_size>,
   malloc = 0x90004fe0 <szone_malloc>,
   calloc = 0x90009a20 <szone_calloc>,
   valloc = 0x90015940 <szone_valloc>,
   free = 0x900042e0 <szone_free>,
   realloc = 0x9000e400 <szone_realloc>,
   destroy = 0x90064ee0 <szone_destroy>,
   zone_name = 0x6312c0 "EmacsZone",
   reserved3 = 0x0,
   reserved4 = 0x0,
   introspect = 0xa0001d54,
   reserved5 = 0x0
}

szone_size has moved in the system library.  So the pointers are 
corrupted and when unexec_realloc calls emacs_zone->size it goes off 
into never-never land.  The code for malloc_jumpstart in 
scalable_malloc.c has the following code in it:

	/* Set function pointers.  Even the functions that stay the same must 
be
	 * set, since there are no guarantees that they will be mapped to the
	 * same addresses. */
	data->szones[i].basic_zone.size = (void *) szone_size;

This is extremely telling.  My first thought on a fix would be to 
replace the call to emacs_zone->size with a call to szone_size.  
However that is a static function.  All the other malloc zones will 
have the size set correctly, so using any other malloc zone would work. 
  So here is my proposed change.

*** unexmacosx.c.old	Sun Nov 24 12:24:03 2002
--- unexmacosx.c	Sun Nov 24 12:35:41 2002
***************
*** 888,894 ****
   	/* 2002-04-15 T. Ikegami <ikegami@adam.uprr.pr>.  The original
   	   code to get size failed to reallocate read_buffer
   	   (lread.c).  */
! 	int old_size = emacs_zone->size (emacs_zone, old_ptr);
   	int size = new_size > old_size ? old_size : new_size;

   	if (size)
--- 888,894 ----
   	/* 2002-04-15 T. Ikegami <ikegami@adam.uprr.pr>.  The original
   	   code to get size failed to reallocate read_buffer
   	   (lread.c).  */
! 	int old_size = malloc_default_zone()->size (emacs_zone, old_ptr);
   	int size = new_size > old_size ? old_size : new_size;

   	if (size)

I doubt the call to malloc_default_zone will be a performance 
bottleneck.  If it is, during initialization we can call it once and 
assign emacs_zone->size=malloc_default_zone()->size

-Steven

On Saturday, November 23, 2002, at 08:04  AM, Andrew Choi wrote:

> The following message is a courtesy copy of an article
> that has been posted to gnu.emacs.help as well.
>
> The 2002-11-21 Security Update seems to cause Emacs executables built
> before the update to fail.  Rebuilding will make them work again.  I
> realize this is an annoying problem.  But I must admit currently I have
> no idea how to solve it.
>
>
>
> _______________________________________________
> Emacs-devel mailing list
> Emacs-devel@gnu.org
> http://mail.gnu.org/mailman/listinfo/emacs-devel

Re: Mac OS X: Rebuild Require after Security Update 2002-11-21

[Jason Rumney]

Andrew Choi <akochoi@shaw.ca> writes:

> Thanks.  Yes, that's probably the cause.  Unfortunately dynamically
> linked libraries are the only choice for using most of the "system
> frameworks" on Mac OS X.  I'm currently trying to determine whether some
> variables in the libraries are somehow unexec'ed along with the Emacs
> heap.  Of course this will cause a problem if the new libraries change
> the variables' locations.  There aren't many tools available to look
> into this.  But I'll figure out a way sooner or later.

In the case where this caused problems on Windows, it was because the
initialization of the dynamically loaded functions was happening only
when Emacs was dumped.  To work across OS updates, you need to make
sure that the initialization is run every time Emacs runs.

Re: Mac OS X: Rebuild Require after Security Update 2002-11-21

[Andrew Choi]

Steven Tamm <steventamm@mac.com> writes:

> [...]  So here is my proposed change.
> 
> *** unexmacosx.c.old	Sun Nov 24 12:24:03 2002
> --- unexmacosx.c	Sun Nov 24 12:35:41 2002
> ***************
> *** 888,894 ****
>    	/* 2002-04-15 T. Ikegami <ikegami@adam.uprr.pr>.  The original
>    	   code to get size failed to reallocate read_buffer
>    	   (lread.c).  */
> ! 	int old_size = emacs_zone->size (emacs_zone, old_ptr);
>    	int size = new_size > old_size ? old_size : new_size;
> 
>    	if (size)
> --- 888,894 ----
>    	/* 2002-04-15 T. Ikegami <ikegami@adam.uprr.pr>.  The original
>    	   code to get size failed to reallocate read_buffer
>    	   (lread.c).  */
> ! 	int old_size = malloc_default_zone()->size (emacs_zone, old_ptr);
>    	int size = new_size > old_size ? old_size : new_size;
> 
>    	if (size)

Hi Steven,

Hey, that's some great debugging!!  With your change above, I was able
to verify that an executable dumped on 10.2.2 before the security update
can be run on one after the update was applied.  Please install the
change when you can.

Somewhat related, while working in unexmacosx.c, I notice the function
build_region_list.  This function is left over from earlier versions of
the file and is unnecessary since dumping is now based on the emacs_zone
regions.  I'll remove it at some point.

Andrew.

Re: Mac OS X: Rebuild Require after Security Update 2002-11-21

[Steven Tamm]

Yay.  I just checked it in.

	* unexmacosx.c (unexec_realloc): Use malloc_default_zone to
	determine the size of pointers alloced in unexed space instead
	of using possibly invalid emacs_zone pointers.  This fixes the
	binary incompatibility problems caused by updates to libSystem.B.

If anybody is making binary distributions publicly available, you 
should sync the change to unexmacosx.c and rebuild.  And I invite 
anyone to look into adding a pre-flight/post-flight script to the Mac 
OS X installer to "backup" the apple-provided 21.1 binary if installing 
into /usr/bin.  Some people on versiontracker were complaining.

-Steven

> Hi Steven,
>
> Hey, that's some great debugging!!  With your change above, I was able
> to verify that an executable dumped on 10.2.2 before the security 
> update
> can be run on one after the update was applied.  Please install the
> change when you can.
>
> Somewhat related, while working in unexmacosx.c, I notice the function
> build_region_list.  This function is left over from earlier versions of
> the file and is unnecessary since dumping is now based on the 
> emacs_zone
> regions.  I'll remove it at some point.
>
> Andrew.
>
>
>
> _______________________________________________
> Emacs-devel mailing list
> Emacs-devel@gnu.org
> http://mail.gnu.org/mailman/listinfo/emacs-devel

Re: Mac OS X: Rebuild Require after Security Update 2002-11-21

[Thorbjørn Ravn Andersen]

Andrew Choi wrote:

>The 2002-11-21 Security Update seems to cause Emacs executables built
>before the update to fail.  Rebuilding will make them work again.  I
>realize this is an annoying problem.  But I must admit currently I have
>no idea how to solve it.
>
Thank you for this information.

As I have understood it from this list, the problem is with a library 
call that exists solely for Emacs to be able to unexec, which is to 
avoid a lengthy elisp initialization at each startup.

Is this still so complicated that it is almost impossible to do 
"manually" by dumping and restoring the contents of the elisp engine 
similar to what  TeX does?

-- 
  Thorbjørn Ravn Andersen      Scandiatransplant
                               Skejby Sygehus, indgang 3
  +45 89 49 53 01              DK-8200 Århus N
  http://biobase.dk/~tra         

Re: Mac OS X: Rebuild Require after Security Update 2002-11-21

[Andrew Choi]

Thorbjørn Ravn Andersen <tra@biobase.dk> writes:

> [...]  As I have understood it from this list, the problem is with a
> library call that exists solely for Emacs to be able to unexec, which
> is to avoid a lengthy elisp initialization at each startup.
> 
> Is this still so complicated that it is almost impossible to do
> "manually" by dumping and restoring the contents of the elisp engine
> similar to what  TeX does?
> 
> -- 
>   Thorbjørn Ravn Andersen

Steven Tamm has found a solution to this problem.  Please see his
message posted earlier to the mailing list.

Unexec for Mac OS X is implemented in unexmacosx.c.  TeX writes out
format files in a way that is different from how Emacs does unexec.  For
one the former are data files while the latter are executable files.

Re: Mac OS X: Rebuild Require after Security Update 2002-11-21

[Steven Tamm]

So far, it appears that DLL initialization is not the problem.  The 
mach-o format seems to guarantee that library initialization occurs 
upon dynamic linking.  Since the Carbon framework/library isn't called 
during unexec (restricting initialization to UNIXish type things), this 
makes sense.

The problem in this case was that a pointer to an operating system 
function was being stored in the dumped exec without being updated.  
The heartening thing is that the apple/next version of unexec implied 
that these were the only pointers that had to be reinitialized 
post-dumping.  However it does appear that unlike the apple/next 
version, the Carbon version seems to initialize some of the core 
foundation code during unexec (if this was untrue, using 
malloc_freezedry and malloc_jumpstart would work).  This is troublesome 
because none of the Carbon calls are ever made before dumping; just by 
linking with the Framework does this initialization happen.  I put some 
effort into finding out why this happens and it seems to back up the 
first point; the Carbon framework initializes some structures on exec 
regardless of whether it is called during the lifetime of the program.

If anyone familiar with next/darwin/mach-o had any ideas on how to 
prevent the Carbon framework/library from initializing itself, it would 
be greatly appreciated.  That way the unexec code could go back to 
using malloc_freezedry and malloc_jumpstart (and make a lot of these 
issues disappear).

-Steven

P.S.  Andrew, is vfork fixed in Jaguar?  If so, we should probably 
conditionalize its removal

On Sunday, November 24, 2002, at 01:03  PM, Jason Rumney wrote:

> Andrew Choi <akochoi@shaw.ca> writes:
>
>> Thanks.  Yes, that's probably the cause.  Unfortunately dynamically
>> linked libraries are the only choice for using most of the "system
>> frameworks" on Mac OS X.  I'm currently trying to determine whether 
>> some
>> variables in the libraries are somehow unexec'ed along with the Emacs
>> heap.  Of course this will cause a problem if the new libraries change
>> the variables' locations.  There aren't many tools available to look
>> into this.  But I'll figure out a way sooner or later.
>
> In the case where this caused problems on Windows, it was because the
> initialization of the dynamically loaded functions was happening only
> when Emacs was dumped.  To work across OS updates, you need to make
> sure that the initialization is run every time Emacs runs.
>
>
>
> _______________________________________________
> Emacs-devel mailing list
> Emacs-devel@gnu.org
> http://mail.gnu.org/mailman/listinfo/emacs-devel

Re: Mac OS X: Rebuild Require after Security Update 2002-11-21

[Jason Rumney]

Steven Tamm <steventamm@mac.com> writes:

> The problem in this case was that a pointer to an operating system
> function was being stored in the dumped exec without being updated.

Sorry, I wasn't clear. That was what I meant by "initialisation of
dynamically loaded functions".

Re: Mac OS X: Rebuild Require after Security Update 2002-11-21

[Andrew Choi]

Steven Tamm <steventamm@mac.com> writes:

> [...]  P.S.  Andrew, is vfork fixed in Jaguar?  If so, we should
> probably conditionalize its removal

The redirection of vfork to fork is there to solve a problem described
by the following comment in src/s/darwin.h (see also the archive of
emacs-pretest-bug).

/* The following solves the problem that Emacs hangs when evaluating
   (make-comint "test0" "/nodir/nofile" nil "") when /nodir/nofile
   does not exist.  */
#undef HAVE_WORKING_VFORK
#define vfork fork
#define DONT_REOPEN_PTY

I just tried removing these three lines and Emacs (in fact OS X!) still
hangs when evaluating that expression.

Re: Mac OS X: Rebuild Require after Security Update 2002-11-21

[Richard Stallman]

    Is this still so complicated that it is almost impossible to do 
    "manually" by dumping and restoring the contents of the elisp engine 
    similar to what  TeX does?

Starting Emacs initializes lots of C data structures
that are not Lisp data.

Re: Mac OS X: Rebuild Require after Security Update 2002-11-21

[Stefan Monnier]

> Steven Tamm <steventamm@mac.com> writes:
> 
> > [...]  P.S.  Andrew, is vfork fixed in Jaguar?  If so, we should
> > probably conditionalize its removal
> 
> The redirection of vfork to fork is there to solve a problem described
> by the following comment in src/s/darwin.h (see also the archive of
> emacs-pretest-bug).
> 
> /* The following solves the problem that Emacs hangs when evaluating
>    (make-comint "test0" "/nodir/nofile" nil "") when /nodir/nofile
>    does not exist.  */
> #undef HAVE_WORKING_VFORK
> #define vfork fork
> #define DONT_REOPEN_PTY
> 
> I just tried removing these three lines and Emacs (in fact OS X!) still
> hangs when evaluating that expression.

Does it really freeze the kernel or just the UI or just Emacs ?
If it freezes the kernel, post it to bugtraq as a DoS issue and it
should get fixed within a few days.


	Stefan

Re: Mac OS X: Rebuild Require after Security Update 2002-11-21

[Steven Tamm]

It doesn't cause a kernel panic on my box, it just causes the system to  
become totally unresponsive.
The apple version of emacs includes this code around ever call to vfork.

#ifndef PUMA_VFORK_ISSUES_CLEARED_UP
     pid = fork ();
#else
     pid = vfork ();
#endif

Apparently, puma vfork issues are not cleared up, so it appears the  
answer is to use fork.  I started to do some debugging of this issue,  
but the fact that after every run the computer hangs makes things quite  
annoying.  I narrowed down the problem (I think).
This code works:

#include <sys/types.h>
#include <unistd.h>

int main(int argc, char **argv)
{
   pid_t parent = getpid();
   pid_t child;
   printf("Parent pid %x\n", parent);
   child = vfork();
   if (child < 0) {
     perror("vfork");
     _exit(2);
   }
   if (!child) {
     printf("Child pid %x\n", getpid());
     printf("Bad exec return %d\n", execv("/nofile/noplace", argv));
     _exit(1);
   }
   return 0;
}

If you remove the _exit(1), it causes a nasty seg fault (it seems to  
blow away the entire state of the process).
If you remove the _exit(1) and switch to fork() it works.  So I think  
the problem may be with vfork not correctly cleaning up after itself if  
the process goes kablooy.  Andrew, did you report a radar issue  
associated with this?

A couple other people tried to switch bash to use vfork, because in  
later darwin releases the performance of fork seems to have gotten  
worse.  They found similar issues with the system hanging.   
http://lists.apple.com/archives/darwin-kernel/2002/Oct/02/ 
forkexecperformance.txt

-Steven


>> /* The following solves the problem that Emacs hangs when evaluating
>>    (make-comint "test0" "/nodir/nofile" nil "") when /nodir/nofile
>>    does not exist.  */
>> #undef HAVE_WORKING_VFORK
>> #define vfork fork
>> #define DONT_REOPEN_PTY
>>
>> I just tried removing these three lines and Emacs (in fact OS X!)  
>> still
>> hangs when evaluating that expression.
>
> Does it really freeze the kernel or just the UI or just Emacs ?
> If it freezes the kernel, post it to bugtraq as a DoS issue and it
> should get fixed within a few days.
>
>
> 	Stefan
>

Re: Mac OS X: Rebuild Require after Security Update 2002-11-21

[Andrew Choi]

Steven Tamm <steventamm@mac.com> writes:

> [...]  So I think the problem may be with vfork not correctly cleaning
> up after itself if the process goes kablooy.  Andrew, did you report a
> radar issue associated with this?

Hi Steven,

No.  The problem and fix were submitted by Nozomu Ando on
emacs-pretest-bug a while ago.  I thought it was acceptable to just use
fork so I left it at that.  I have not received reports of other
problems related to this one.

Would you like me to write a bug report to the darwin lists?  Seems like
they know about the problem with vfork but perhaps we can remind them.

Andrew.

fork v vfork (was Re: Mac OS X: Rebuild Require after Security Update 2002-11-21)

[Steven Tamm]

I reported a bug with my test case.  It's not that big a deal, I just 
wanted to make sure that something was being done about it.  It's a 
fairly nasty bug if it hangs the system.  It would also make the 
compile a little cleaner.

WRT fork vs. vfork; on darwin the figures I've seen show a difference 
in performance about 100x, and has been getting worse in each release.

What are people's feelings about conditionalizing the three calls to 
vfork (in process.c, callproc.c, and sysdep.c) so that it could become 
fork() on platforms that don't have a working VFORK.  It would probably 
use the HAVE_VFORK conditional.

-Steven

On Tuesday, November 26, 2002, at 09:29  AM, Andrew Choi wrote:

> Steven Tamm <steventamm@mac.com> writes:
>
>> [...]  So I think the problem may be with vfork not correctly cleaning
>> up after itself if the process goes kablooy.  Andrew, did you report a
>> radar issue associated with this?
>
> Hi Steven,
>
> No.  The problem and fix were submitted by Nozomu Ando on
> emacs-pretest-bug a while ago.  I thought it was acceptable to just use
> fork so I left it at that.  I have not received reports of other
> problems related to this one.
>
> Would you like me to write a bug report to the darwin lists?  Seems 
> like
> they know about the problem with vfork but perhaps we can remind them.
>
> Andrew.
>