From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Rich Felker Newsgroups: gmane.emacs.devel Subject: Re: Dumper problems and a possible solutions Date: Wed, 25 Jun 2014 14:32:41 -0400 Message-ID: <20140625183241.GW179@brightrain.aerifal.cx> References: <20140624171955.GS179@brightrain.aerifal.cx> <53AB0EF8.4090608@yandex.ru> <831tucrguf.fsf@gnu.org> NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: ger.gmane.org 1403721415 28715 80.91.229.3 (25 Jun 2014 18:36:55 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Wed, 25 Jun 2014 18:36:55 +0000 (UTC) Cc: Dmitry Antipov , emacs-devel@gnu.org To: Eli Zaretskii Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Wed Jun 25 20:36:48 2014 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1Wzs3n-000666-QO for ged-emacs-devel@m.gmane.org; Wed, 25 Jun 2014 20:36:43 +0200 Original-Received: from localhost ([::1]:40405 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Wzs3n-0000em-B7 for ged-emacs-devel@m.gmane.org; Wed, 25 Jun 2014 14:36:43 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:49622) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Wzs01-0000F9-0j for emacs-devel@gnu.org; Wed, 25 Jun 2014 14:32:54 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Wzrzu-0002ak-K1 for emacs-devel@gnu.org; Wed, 25 Jun 2014 14:32:48 -0400 Original-Received: from 216-12-86-13.cv.mvl.ntelos.net ([216.12.86.13]:44420 helo=brightrain.aerifal.cx) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Wzrzu-0002ae-DN for emacs-devel@gnu.org; Wed, 25 Jun 2014 14:32:42 -0400 Original-Received: from dalias by brightrain.aerifal.cx with local (Exim 3.15 #2) id 1Wzrzt-0002ob-00; Wed, 25 Jun 2014 18:32:41 +0000 Content-Disposition: inline In-Reply-To: <831tucrguf.fsf@gnu.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 216.12.86.13 X-Mailman-Approved-At: Wed, 25 Jun 2014 14:36:41 -0400 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.devel:172715 Archived-At: On Wed, Jun 25, 2014 at 09:20:40PM +0300, Eli Zaretskii wrote: > > Date: Wed, 25 Jun 2014 22:03:36 +0400 > > From: Dmitry Antipov > > Cc: emacs-devel@gnu.org > > > > What about non-Lisp objects? > > > > It's not too hard to walk through live (reachable) Lisp objects - this > > is exactly what GC mark phase does. It's not too hard to walk through > > all allocated Lisp objects - this is exactly what GC sweep phase does. > > But what about lower-level stuff allocated with malloc at invisible > > from Lisp? Of course, you can do your own serialization for these objects > > as well - but only if you know about their internal structure. > > Is it possible to provide our own implementation of sbrk that > allocates memory from some large static array? That's exactly the hack I described which I'm using right now. But since I didn't implement a free-like operation and since load_charset_map_from_file allocates >700k every time it's called, I had to make the static array 400MB. This obviously isn't acceptable. I think it would work with a "real" mini-malloc implementation using the static array, and a much smaller static array (maybe 8-15 MB) but my attempts to write a quick one have been sloppy and buggy so far. I would be reasonably happy with this solution (at least it would fix the problems I'm experiencing), but I don't think it's as elegant as fixing the portability problem completely by getting rid of the need to dump executable binary files and instead dumping a C array. And it doesn't fix the fact that you can't build a PIE emacs. > Or do modern malloc > implementations avoid calling sbrk when they need more memory? In general avoiding using brk to expand the heap is a bad idea on Linux since alternate methods are considerably slower and can't automatically obtain contiguous virtual space when it's available. However, behaviors may vary. In musl's malloc, we use brk if it's available (note: with PIE, most kernels give you almost no available brk space due to the way the mappings are laid out) for extending the heap that's usef for small allocations, and fallback to constructing a discontiguous heap with mmap if brk fails. However musl always uses mmap for huge allocations (greater than ~100-200k) since it has no freelists for sizes that large and since we want to always return freed memory that large to the system. Also, musl does not provide a working sbrk at all, since synchronizing with an application's use of sbrk would introduce performance costs into all correct applications that don't poke at the brk. > If something like that is possible, we could do what the native and > Cygwin Windows builds already do: record all the objects, Lisp and C, > in that static array, which then gets dumped as part of the data > section. > > > What about stuff allocated by some external library? > > A valid concern, but I don't think we have that problem now. If we > did, the Windows port would not be able to be built, because such > external libraries would call the malloc they were linked against, not > the replacement we provide during "-l loadup dump" phase of the build. Indeed. Rich