From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: Daniel Colascione Newsgroups: gmane.emacs.devel Subject: Re: Preview: portable dumper Date: Mon, 28 Nov 2016 15:17:38 -0800 Message-ID: References: <047a67ec-9e29-7e4e-0fb0-24c3e59b5886@dancol.org> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: text/plain X-Trace: blaine.gmane.org 1480375235 23954 195.159.176.226 (28 Nov 2016 23:20:35 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Mon, 28 Nov 2016 23:20:35 +0000 (UTC) User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.0.50 (gnu/linux) Cc: emacs-devel@gnu.org To: Stefan Monnier Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Tue Nov 29 00:20:32 2016 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cBVDq-0005MF-Rs for ged-emacs-devel@m.gmane.org; Tue, 29 Nov 2016 00:20:31 +0100 Original-Received: from localhost ([::1]:33657 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cBVDs-0004AY-P8 for ged-emacs-devel@m.gmane.org; Mon, 28 Nov 2016 18:20:32 -0500 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:59727) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cBVBF-0002wr-CP for emacs-devel@gnu.org; Mon, 28 Nov 2016 18:17:50 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cBVBE-0005f1-D7 for emacs-devel@gnu.org; Mon, 28 Nov 2016 18:17:49 -0500 Original-Received: from dancol.org ([2600:3c01::f03c:91ff:fedf:adf3]:40004) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1cBVBD-0005ej-Vn for emacs-devel@gnu.org; Mon, 28 Nov 2016 18:17:48 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=dancol.org; s=x; h=Content-Type:MIME-Version:Message-ID:Date:References:In-Reply-To:Subject:Cc:To:From; bh=gHwbfc9rBNpBsZGvnhjyWYMcU4TU45U6WFAExIFTAxY=; b=BggJznMiJinoL22K8WJgXf3vSaJsZlBohgc0bqsqjkVPKd+fN/NKE1kN+nT8o/eavHGMKSy+WpuQYvt/MHUyzfjui/32CR9CfLPHFLHjtx3ls+QT5YlcxDeIj/6BPvlx9rl70LKNOmOtQjiZVMX7LLLgi6tcTb/vwXOhWVNxH2F8rX3rN0t6QqdBc/35v2+eAYsnmYUyrY66dfIZZXLOH2ZcFNNA+fEAe08X/84582UA3VriIbBH+tJF/DFjTqb7e0eJY68zkzlHxBqTP3MWj4CXlX56uPwLxSEA+rm5BKB7T5jwU18i9pqqCkmS8+fthSNOICH9VOiFVRHetHe8Wg==; Original-Received: from [2620:0:1008:100b:fd76:a51f:62d3:e4cc] (helo=dancol-glaptop0) by dancol.org with esmtpsa (TLS1.2:DHE_RSA_AES_128_CBC_SHA1:128) (Exim 4.84_2) (envelope-from ) id 1cBVBA-0001V9-T7; Mon, 28 Nov 2016 15:17:44 -0800 In-Reply-To: (Stefan Monnier's message of "Mon, 28 Nov 2016 18:01:00 -0500") X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2600:3c01::f03c:91ff:fedf:adf3 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.org gmane.emacs.devel:209706 Archived-At: On Mon, Nov 28 2016, Stefan Monnier wrote: >> I've been working on a portable dumper for GNU Emacs. The attached patch is >> an early version of this work. > > Cool! > >> An additional optimization follows, although this part isn't implemented >> yet: we can define a "preferred load address" for the dump and write >> relocation information such that if the dump and Emacs end up being where we >> expect them to be, we don't have to perform any relocations at all. > > Oh, so the in-file format is also a valid in-memory format? Great. > AFAIK, that's the crucial difference between a "fast .elc loader" and > your approach. This should be key to get really good performance > (e.g. good enough that it doesn't slow down bootstrap noticeably). Yes. The file is full of valid memory images of Lisp objects --- that's why I'm guessing it might be a bit more verbose than ELC, pre-compression. If you guess correctly at dump time what addresses the Emacs and dump images will have, you can direct mmap it directly and run. With a non-PIC executable, you can do a pretty good job of that and get performance a bit better than unexec, in theory. If you load a dump and find that you've guessed wrong on the base addresses, you need to relocate the image, but that can be done cheaply and gradually, on access. >> would benefit from compression, which will reduce IO loads: LZ4 reduces the >> dump size for me from ~12MB to ~4MB. As in the mmap case, we can throw away > > Hmm... interesting the "dump one big .elc" approach I played with (which > kinda works right now, tho with the known performance issues) leads to > a dumped.elc file of a size a bit below 4MB (which gzip compresses down > to ~1MB). > > I assume your system is using 64bit Lisp_Object, right? So on a 32bit > Lisp_Object your dump size is probably closer to 6-7MB. Right --- I've tested on 64-bit only. I imagine testing 32-bit and wide-int might reveal some latent bugs. > >> functions pdumper.h --- but even so, it's within 100ms or so of an >> unexeced Emacs. > > 0.1s will still lead to a noticeable slow down of bootstrap, but at > least it should be good enough for the usual end-user who doesn't run > Emacs in batch mode. Oh, of course. But that's without any attention paid to optimization. I'm sure it can go much faster. >> It's also possible to dump an already-dumped Emacs, so it should be possible >> for users to have their own dump files. > > [ FWIW in theory my dumped.elc approach could also work on an > already-dumped Emacs, for use by end-users (it basically just prints the > contents of the obarray), tho it would indeed need extra work to weed out > problems printing objects like markers and such. ] Yeah --- then both approaches get into the problem of how to dump arbitrary object graphs, which in practice ends up being "dump the whole heap except for processes, windows, and frames."