From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Rich Felker Newsgroups: gmane.emacs.devel Subject: Re: Dumper problems and a possible solutions Date: Wed, 25 Jun 2014 15:57:30 -0400 Message-ID: <20140625195730.GA179@brightrain.aerifal.cx> References: <20140624171955.GS179@brightrain.aerifal.cx> <53AB0EF8.4090608@yandex.ru> <831tucrguf.fsf@gnu.org> <20140625183241.GW179@brightrain.aerifal.cx> <83wqc4q0xl.fsf@gnu.org> <20140625190333.GZ179@brightrain.aerifal.cx> <83vbropzlz.fsf@gnu.org> NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: ger.gmane.org 1403726274 24812 80.91.229.3 (25 Jun 2014 19:57:54 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Wed, 25 Jun 2014 19:57:54 +0000 (UTC) Cc: dmantipov@yandex.ru, emacs-devel@gnu.org To: Eli Zaretskii Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Wed Jun 25 21:57:49 2014 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1WztKG-0005P3-Cz for ged-emacs-devel@m.gmane.org; Wed, 25 Jun 2014 21:57:48 +0200 Original-Received: from localhost ([::1]:40771 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1WztKG-0001Iu-2i for ged-emacs-devel@m.gmane.org; Wed, 25 Jun 2014 15:57:48 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:44140) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1WztK8-0001IL-V6 for emacs-devel@gnu.org; Wed, 25 Jun 2014 15:57:45 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1WztK4-00082m-Ot for emacs-devel@gnu.org; Wed, 25 Jun 2014 15:57:40 -0400 Original-Received: from 216-12-86-13.cv.mvl.ntelos.net ([216.12.86.13]:44433 helo=brightrain.aerifal.cx) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1WztK0-000820-3B; Wed, 25 Jun 2014 15:57:32 -0400 Original-Received: from dalias by brightrain.aerifal.cx with local (Exim 3.15 #2) id 1WztJy-00038Z-00; Wed, 25 Jun 2014 19:57:30 +0000 Content-Disposition: inline In-Reply-To: <83vbropzlz.fsf@gnu.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 216.12.86.13 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.devel:172723 Archived-At: On Wed, Jun 25, 2014 at 10:18:16PM +0300, Eli Zaretskii wrote: > > Date: Wed, 25 Jun 2014 15:03:33 -0400 > > From: Rich Felker > > Cc: dmantipov@yandex.ru, emacs-devel@gnu.org > > > > > > > Is it possible to provide our own implementation of sbrk that > > > > > allocates memory from some large static array? > > > > > > > > That's exactly the hack I described which I'm using right now. But > > > > since I didn't implement a free-like operation and since > > > > load_charset_map_from_file allocates >700k every time it's called, I > > > > had to make the static array 400MB. > > > > > > That's not a problem, because those 700K are free'd before the next > > > one is allocated. And in any case, they are all free'd before we call > > > unexec. Just implement sbrk for negative increment. The Windows port > > > > But load_charset_map_from_file doesn't call an sbrk-like interface; it > > calls (indirectly) xmalloc and xfree. So there's at least some > > nontrivial glue that goes in between. > > Sorry, you are right. What you need is non-toy malloc implementation > that is willing to work with your replacement sbrk. Then support for > negative increments in sbrk isn't required. > > But I still don't understand how you get to 400MB. It's not that we > allocate hundreds of those 700K tables for charsets. Do you have an > explanation for this? Not hundreds at a time, but if the malloc operation is just positive (fake-)sbrk and the free operation is a nop, hundreds of such charset load operations will quickly add up. It might have succeeded with a little less than 400 but that was the first guess I made that didn't run out of memory (guesses were something like 10, 50, 100, 200, 400 IIRC). > > > If supporting deallocation in such an sbrk isn't feasible, how about > > > using gmalloc, as an malloc replacement before dumping? > > > > I suspect it's a lot of work to wire up gmalloc to (1) avoid > > interposing on the malloc/free/etc. names, (2) use the static mini-brk > > buffer, (3) only allocate from the mini-brk buffer before dumping > > (otherwise pass to real malloc), but still check realloc/free calls > > after dumping and handle the case where the old memory was in the > > mini-brk. > > Sorry, I don't see the difficulty. Just make malloc/realloc/free be > pointers that point to gmalloc's implementation before dumping, and to > the libc implementation after it. You may need some #define to rename > malloc to some other symbol, to avoid name clashes. Am I missing > something? Yeah, what happens if, after dumping, the real emacs at runtime ends up calling free() on one of the pre-dump pointers? Can't this happen? This is why I want the whole pre-dump heap in a single C array: so we can range-check addresses and determine if they're pre-dump allocations (and in that case, NOP out free and do malloc+memcpy for realloc). > > > > I would be reasonably happy with this solution (at least it would fix > > > > the problems I'm experiencing), but I don't think it's as elegant as > > > > fixing the portability problem completely by getting rid of the need > > > > to dump executable binary files and instead dumping a C array. > > > > > > But it's conceptually much simpler and reliable. That's "elegant" in > > > my book, when such hairy stuff is concerned. > > > > No, it's less reliable. See my other posts in the thread about what > > happens if you have other libraries linked and they do nontrivial > > things prior to dumping (e.g. from static ctors). > > But in those other posts I thought we agreed that whatever those ctors > do is irrelevant, as the dumped Emacs cannot possibly use what they > allocate, and those ctors will be invoked again in the dumped Emacs. Those ctors are free to inspect global data. For example one might contain (this sort of idiom is necessary if you can't control the relative order of ctors): if (!init) { do_something(); init=1; }. In that case, the dump would save the value of init, and do_something() would fail to happen at runtime. This issue is not something that would be added with our proposed "simple fix"; it's an issue that EXISTS NOW if any such libraries happen to be linked statically (if they're dynamic, the dumper will fail to see/dump the data). > > As an example (I think I mentioned this earlier), if you static > > link, musl libc is remembering the clock_gettime vdso pointer from > > the pre-dump state and attempting to use it later (which is not > > valid because the kernel maps it at a random address). > > You need to fix your libc to detect that it is re-exec'ed, and > re-initialize this stuff. I don't see any easy way to do that that's not a big violation of encapsulation (startup code has no business knowing about clock_gettime internals) and the whole efficiency of static linking (startup code does not pull in any vdso stuff; that only gets pulled in by using a function that needs a vdso lookup). This is some big ugliness for the sake of supporting static-linked emacs. And it ends up being a big case of whack-a-mole. Dumping and re-execing is not a standard usage case and is way out there in the realm of undefined behavior. Emacs is probably the only widely-used (or even known) program that does it. If this issue can't be fixed easily, I'd probably just recommend to musl users not to static link emacs, or to patch in a global ctor that clears AT_SYSINFO_EHDR from the aux vector so that vdso doesn't get used at all. > > > > And it doesn't fix the fact that you can't build a PIE emacs. > > > > > > Why is that important? > > > > Since emacs is processing lots of potentially untrusted data, PIE > > hardening may be beneficial for hardening against vulnerabilities > > where an attacker would otherwise be able to perform arbitrary code > > execution as the user running emacs. I'm not aware of such > > vulnerabilities, but being that I found things that look suspiciously > > like use-after-free while reading the allocator-related code, I > > wouldn't be surprised if they exist. > > Sounds like a secondary requirement to me. We should first make the > dumping work along these lines, and worry about bonuses later. That's fine with me. Rich