From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: Philipp Stephani Newsgroups: gmane.emacs.devel Subject: Re: compiled lisp file format (Re: Skipping unexec via a big .elc file) Date: Sun, 24 Sep 2017 13:57:21 +0000 Message-ID: References: <8A8DA980-13A7-4F8B-9D07-391728C673C9@raeburn.org> <83inmq53xk.fsf@gnu.org> <96D35768-314C-43F5-BD5E-B12187759DCA@raeburn.org> <123104DD-447F-4CDB-B3A0-CED80E3AC8C9@raeburn.org> <20170403165736.GA2851@acm> <2497A2D5-FDB1-47FF-AED3-FD4ABE2FE144@raeburn.org> <83lgrhpalq.fsf@gnu.org> <0D99B4FE-FEEF-4565-87D6-E230A05DEF3C@raeburn.org> <86lgrc4vob.fsf@molnjunk.nocrew.org> <834ly0oew1.fsf@gnu.org> <968E8F50-92F6-43C7-B7E4-EE8378943087@raeburn.org> <83wpawmj4d.fsf@gnu.org> <1e397033-8291-1625-8b78-a1e1c200aea5@gmail.com> <18196f08-408d-8b17-423e-8be54507bb84@gmail.com> <8360hkkcgj.fsf@gnu.org> <26b35c16-33e7-0e08-9cc5-6f9b81e40968@cs.ucla.edu> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="001a113cd06acd02860559efd282" X-Trace: blaine.gmane.org 1506261494 9508 195.159.176.226 (24 Sep 2017 13:58:14 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Sun, 24 Sep 2017 13:58:14 +0000 (UTC) Cc: Paul Eggert , Emacs developers To: Ken Raeburn Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Sun Sep 24 15:58:10 2017 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dw7Q8-00026i-H3 for ged-emacs-devel@m.gmane.org; Sun, 24 Sep 2017 15:58:08 +0200 Original-Received: from localhost ([::1]:38224 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dw7QF-0006L4-Si for ged-emacs-devel@m.gmane.org; Sun, 24 Sep 2017 09:58:15 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:49170) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dw7Pc-0006Kw-8v for emacs-devel@gnu.org; Sun, 24 Sep 2017 09:57:37 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dw7Pa-0000oY-Io for emacs-devel@gnu.org; Sun, 24 Sep 2017 09:57:36 -0400 Original-Received: from mail-oi0-x22e.google.com ([2607:f8b0:4003:c06::22e]:45175) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1dw7Pa-0000kF-B3 for emacs-devel@gnu.org; Sun, 24 Sep 2017 09:57:34 -0400 Original-Received: by mail-oi0-x22e.google.com with SMTP id z73so3828192oia.2 for ; Sun, 24 Sep 2017 06:57:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=K8hESPiyNpo2n2losWuZATSS1vXyi5kgLWMHY3OAeFE=; b=jcJZ3bYbBO9QOLPxr1d6r48Qe/loHitaElKamcbYCoUI2S8xShYi33sSi3dpY3Fr9s zXwiUgfzTqT1S/DKwC39vbpv6d4XoOxvhQ24N666SI+axk6TvcR9LjprJ0jKikA/Xz8U eQpW42FIiptvP+NmW+hZkz/pf8Ny33WiKMWvzJJ9B3BS7zSKABlx/ZsHRnTw4a7w6q9K tS1TmQC1kxniRECC1eZiWCKuXXiyLZAPoiriHEcXMrBY221bB6t7rl7H4rQDGYHiimbH bc7Bjr1GoL4f30kmNW/XMTqxJwcED8Bp+jbhbVnoJhQyIQmzehbvCrM1cy92ppgqfFSk 8olA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=K8hESPiyNpo2n2losWuZATSS1vXyi5kgLWMHY3OAeFE=; b=ii4sIEWVsSHIuQrblANByMvPoOSth9Xj6pnCFYnVeHGAFTbVhVMuF9YuG7hlJ2exN3 eBGRzmod8prLMNlUdjMWrKkHaSxQo10hVVx9TQG55Qf/x9DxzNwibIGBhTjvaBKSgLDm 3YCkuNDVv//JAU/V+hVYsqQ+iiuK+vk3GgSuZvszChrClhEdCF81MWTXMCjP48Z41MJ6 qCHQ1l/imm7yfK1n4FAD0Hwy0baEYRP187CdcJUO79Hw459ZEm+At1fv/W2R1pFxY7+j ydnAoaMVDUIfcq3J9/aU4cmRFK1QTdhFPy4qUOvAlM1r+Ij8RPBCOsiI9aErvrLErmcK Gn6g== X-Gm-Message-State: AHPjjUhZpTdPQz237tiMxrbnyvXZEAA3XZzefXWbzRt23St8zwINq6DH Sr2ayRXqP7XAKzjjqUIcAEYl/PPIQMS4GbK4R7XVFpNG X-Google-Smtp-Source: AOwi7QBZ1DSIoxQVjtzib+5lh6jaY9OufhZhqaxr7x5fY9vbSwjrvCGKjzUxTa0wg45hVbXvtemEWAfYoLyXErri2/0= X-Received: by 10.202.67.194 with SMTP id q185mr1261756oia.52.1506261452258; Sun, 24 Sep 2017 06:57:32 -0700 (PDT) In-Reply-To: X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4003:c06::22e X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.org gmane.emacs.devel:218742 Archived-At: --001a113cd06acd02860559efd282 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Ken Raeburn schrieb am Mo., 3. Juli 2017 um 03:44 Uhr= : > > On Jul 2, 2017, at 11:46, Philipp Stephani wrote: > > Ken Raeburn schrieb am Mo., 29. Mai 2017 um > 11:33 Uhr: > >> >> On May 28, 2017, at 08:43, Philipp Stephani >> wrote: >> >> >> >> Ken Raeburn schrieb am So., 28. Mai 2017 um >> 13:07 Uhr: >> >>> >>> On May 21, 2017, at 04:53, Paul Eggert wrote: >>> >>> > Ken Raeburn wrote: >>> >> The Guile project has taken this idea pretty far; they=E2=80=99re ge= nerating >>> ELF object files with a few special sections for Guile objects, using t= he >>> standard DWARF sections for debug information, etc. While it has a cer= tain >>> appeal (making C modules and Lisp files look much more similar, maybe b= eing >>> able to link Lisp and C together into one executable image, letting GDB >>> understand some of your data), switching to a machine-specific format w= ould >>> be a pretty drastic change, when we can currently share the files acros= s >>> machines. >>> > >>> > Although it does indeed sound like a big change, I don't see why it >>> would prevent us from sharing the files across machines. Emacs can use >>> standard ELF and DWARF format on any platform if Emacs is doing the >>> loading. And there should be some software-engineering benefit in using= the >>> same format that Guile uses. >>> >>> Sorry for the delay in responding. >>> >>> The ELF format has header fields indicating the word size, endianness, >>> machine architecture (though there=E2=80=99s a value for =E2=80=9Cnone= =E2=80=9D), and OS ABI. Some >>> fields vary in size or order depending on whether the 32-bit or 64-bit >>> format is in use. Some other format details (e.g., relocation types, >>> interpretation of certain ranges of values in some fields) are >>> architecture- or OS-dependent; we might not care about many of those >>> details, but relocations are likely needed if we want to play linking g= ames >>> or use DWARF. >>> >>> I think Guile is using whatever the native word size and architecture >>> are. If we do that for Emacs, they=E2=80=99re not portable between pla= tforms. >>> Currently it works for me to put my Lisp files, both source and compile= d, >>> into ~/elisp and use them from different kinds of machines if my home >>> directory is NFS-mounted. >>> >>> We could instead pick fixed values (say, architecture =E2=80=9Cnone=E2= =80=9D, >>> little-endian, 32-bit), but then there=E2=80=99s no guarantee that we c= ould use any >>> of the usual GNU tools on them without a bunch of work, or that we=E2= =80=99d ever >>> be able to use non-GNU tools to treat them as object files. Then again= , we >>> couldn=E2=80=99t expect to do the latter portably anyway, since some of= the >>> platforms don=E2=80=99t even use ELF. >>> >>> >> Is there any significant advantage of using ELF, or could this just use >> one of the standard binary serialization formats (protobuf, flatbuffer, >> ...)? >> >> >> That=E2=80=99s an interesting idea. If one of the popular serialization >> libraries is compatibly licensed, easy to use, and performs well, it may >> be better than rolling our own. >> > > I've tried this out (with flatbuffers), but I haven't seen significant > speed improvements. It might very well be the case that during loading th= e > reader is already fast enough (e.g. for ELC files it doesn't do any > decoding), and it's the evaluator that's too slow. > > > What=E2=80=99s your test case, and how are you measuring the performance? > IIRC I've repeatedly loaded one of the biggest .elc files shipped with Emacs and measured the total loading time. I haven't done any detailed profiling, since I was hoping for a significant speed increase that would justify the work. If people are generally interested in pursuing this further, I'd be happy to put my code into a scratch branch. --001a113cd06acd02860559efd282 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


Ken Ra= eburn <raeburn@raeburn.org>= ; schrieb am Mo., 3. Juli 2017 um 03:44=C2=A0Uhr:

On Ju= l 2, 2017, at 11:46, Philipp Stephani <p.stephani2@gmail.com> wrote:
<= br>
Ken Raeburn <raeburn@raeburn.org> schrieb am Mo., 29. Mai 2017 um 11= :33=C2=A0Uhr:

On May 28, 2017, at 08:43, Philipp Stephani <p.stephani2@gmail.com> wrote:




On May 21, 2017, at 04:53, Paul Eggert <eggert@cs.ucla.edu> wrote:
> Ken Raeburn wrote:
>> The Guile project has taken this idea= pretty far; they=E2=80=99re generating ELF object files with a few special= sections for Guile objects, using the standard DWARF sections for debug in= formation, etc.=C2=A0 While it has a certain appeal (making C modules and L= isp files look much more similar, maybe being able to link Lisp and C toget= her into one executable image, letting GDB understand some of your data), s= witching to a machine-specific format would be a pretty drastic change, whe= n we can currently share the files across machines.
>
> Althoug= h it does indeed sound like a big change, I don't see why it would prev= ent us from sharing the files across machines. Emacs can use standard ELF a= nd DWARF format on any platform if Emacs is doing the loading. And there sh= ould be some software-engineering benefit in using the same format that Gui= le uses.

Sorry for the delay in responding.

The ELF format ha= s header fields indicating the word size, endianness, machine architecture = (though there=E2=80=99s a value for =E2=80=9Cnone=E2=80=9D), and OS ABI.=C2= =A0 Some fields vary in size or order depending on whether the 32-bit or 64= -bit format is in use.=C2=A0 Some other format details (e.g., relocation ty= pes, interpretation of certain ranges of values in some fields) are archite= cture- or OS-dependent; we might not care about many of those details, but = relocations are likely needed if we want to play linking games or use DWARF= .

I think Guile is using whatever the native word size and architect= ure are.=C2=A0 If we do that for Emacs, they=E2=80=99re not portable betwee= n platforms.=C2=A0 Currently it works for me to put my Lisp files, both sou= rce and compiled, into ~/elisp and use them from different kinds of machine= s if my home directory is NFS-mounted.

We could instead pick fixed v= alues (say, architecture =E2=80=9Cnone=E2=80=9D, little-endian, 32-bit), bu= t then there=E2=80=99s no guarantee that we could use any of the usual GNU = tools on them without a bunch of work, or that we=E2=80=99d ever be able to= use non-GNU tools to treat them as object files.=C2=A0 Then again, we coul= dn=E2=80=99t expect to do the latter portably anyway, since some of the pla= tforms don=E2=80=99t even use ELF.


= Is there any significant advantage of using ELF, or could this just use one= of the standard binary serialization formats (protobuf, flatbuffer, ...)?= =C2=A0

That=E2=80=99s an interesting idea.=C2=A0 If one of the popular serializ= ation libraries is compatibly licensed, easy to use, and performs well, it<= /span>=C2=A0may be better than rollin= g our own.

I've trie= d this out (with flatbuffers), but I haven't seen significant speed imp= rovements. It might very well be the case that during loading the reader is= already fast enough (e.g. for ELC files it doesn't do any decoding), a= nd it's the evaluator that's too slow.

What=E2=80= =99s your test case, and how are you measuring the performance?
=

IIRC I've repeatedly loaded one of the= biggest .elc files shipped with Emacs and measured the total loading time.= I haven't done any detailed profiling, since I was hoping for a signif= icant speed increase that would justify the work.
If people are g= enerally interested in pursuing this further, I'd be happy to put my co= de into a scratch branch.
--001a113cd06acd02860559efd282--