From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Rich Felker Newsgroups: gmane.emacs.devel Subject: Re: Dumper problems and a possible solutions Date: Tue, 24 Jun 2014 15:40:27 -0400 Message-ID: <20140624194026.GT179@brightrain.aerifal.cx> References: <20140624171955.GS179@brightrain.aerifal.cx> NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: ger.gmane.org 1403640422 18862 80.91.229.3 (24 Jun 2014 20:07:02 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Tue, 24 Jun 2014 20:07:02 +0000 (UTC) Cc: emacs-devel@gnu.org To: Stefan Monnier Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Tue Jun 24 22:06:55 2014 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1WzWzW-0003Bg-LU for ged-emacs-devel@m.gmane.org; Tue, 24 Jun 2014 22:06:54 +0200 Original-Received: from localhost ([::1]:33632 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1WzWzV-00056q-Up for ged-emacs-devel@m.gmane.org; Tue, 24 Jun 2014 16:06:53 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:57770) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1WzWa5-00012Q-NK for emacs-devel@gnu.org; Tue, 24 Jun 2014 15:40:45 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1WzWZx-0005KL-Sv for emacs-devel@gnu.org; Tue, 24 Jun 2014 15:40:37 -0400 Original-Received: from 216-12-86-13.cv.mvl.ntelos.net ([216.12.86.13]:44411 helo=brightrain.aerifal.cx) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1WzWZx-0005K5-Lq for emacs-devel@gnu.org; Tue, 24 Jun 2014 15:40:29 -0400 Original-Received: from dalias by brightrain.aerifal.cx with local (Exim 3.15 #2) id 1WzWZv-0008DO-00; Tue, 24 Jun 2014 19:40:27 +0000 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 216.12.86.13 X-Mailman-Approved-At: Tue, 24 Jun 2014 16:06:51 -0400 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.devel:172691 Archived-At: On Tue, Jun 24, 2014 at 03:27:39PM -0400, Stefan Monnier wrote: > > To solve ALL of the problems with the dumper (which seems to be a > > recurring theme), I have a proposed design to make it fully portable > > -- even moreso than xemacs "portable dumper" which is still an ugly > > hack. The idea is simple: after loading all of the lisp objects that > > need dumping, walk the lisp heap and output a representation for each > > object as a giant static array in C source format, then compile and > > link this new translation unit with the rest of the emacs .o files to > > produce a final emacs binary. No hacks with binary formats would be > > involved; everything would happen at the C source level. As part of > > the lisp heap dumping, address references to other objects would have > > to be relocated to refer to the object's position in the static array > > rather than the original address at which the object resided when > > created in temacs. That's some non-trivial work, but definitely no > > prohibitive, and as a bonus, the generated address-constant references > > in the static array would transform to load-address-relative > > relocations for the linker, allowing emacs to be built as a > > position-indepdendent executable (PIE) if desired. > > Generating a big static C array against which to link sounds fine and > very portable, indeed. I'm not sure how hard/easy the relocation could > turn out to be. There's the problem of finding *all* the references, > and there's the problem that moving an object means that its "hash" > value changes. Thanks for the feedback. Can you elaborate on how/why the hash changes, and where it's stored that would need to be updated? As far as the relocation, my impression is that it would just need to be able to identify pointers in lisp objects (this is already possible since the GC needs to do it, right?), and rewrite them to (essentially) "static_lisp_heap + offset_of_pointed_to_object" when writing the dump out as a C array. > > If not, or if that's going to be a very long-term project, would a > > cleaned-up version of my current solution be acceptable upstream? > > Making the "dump" portable would be very welcome. Generating a big > static C array sounds OK. So whether the result is acceptable or not > will depend on what's needed to solve the problems linked to relocation. > > Another option is to "dump" the heap into a binary file that we would > later on "mmap". This is the xemacs "portable dumper" approach, and I believe it's inferrior because it depends on being able to map back at the same location. If the region is a page-aligned static buffer in the main executable and you mmap over it with MAP_FIXED, this is safe for the most part, but it's still incompatible with PIE. I think it would be nice to solve this problem in a way that also makes PIE possible. Rich