From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: Philipp Stephani Newsgroups: gmane.emacs.devel Subject: Re: compiled lisp file format (Re: Skipping unexec via a big .elc file) Date: Sun, 02 Jul 2017 15:46:18 +0000 Message-ID: References: <8A8DA980-13A7-4F8B-9D07-391728C673C9@raeburn.org> <83inmq53xk.fsf@gnu.org> <96D35768-314C-43F5-BD5E-B12187759DCA@raeburn.org> <123104DD-447F-4CDB-B3A0-CED80E3AC8C9@raeburn.org> <20170403165736.GA2851@acm> <2497A2D5-FDB1-47FF-AED3-FD4ABE2FE144@raeburn.org> <83lgrhpalq.fsf@gnu.org> <0D99B4FE-FEEF-4565-87D6-E230A05DEF3C@raeburn.org> <86lgrc4vob.fsf@molnjunk.nocrew.org> <834ly0oew1.fsf@gnu.org> <968E8F50-92F6-43C7-B7E4-EE8378943087@raeburn.org> <83wpawmj4d.fsf@gnu.org> <1e397033-8291-1625-8b78-a1e1c200aea5@gmail.com> <18196f08-408d-8b17-423e-8be54507bb84@gmail.com> <8360hkkcgj.fsf@gnu.org> <26b35c16-33e7-0e08-9cc5-6f9b81e40968@cs.ucla.edu> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="001a113d6350bd3fe20553578d2d" X-Trace: blaine.gmane.org 1499010440 25765 195.159.176.226 (2 Jul 2017 15:47:20 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Sun, 2 Jul 2017 15:47:20 +0000 (UTC) Cc: Paul Eggert , Emacs developers To: Ken Raeburn Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Sun Jul 02 17:47:15 2017 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dRh5Z-00068X-VU for ged-emacs-devel@m.gmane.org; Sun, 02 Jul 2017 17:47:10 +0200 Original-Received: from localhost ([::1]:58413 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dRh5f-0004Qd-EH for ged-emacs-devel@m.gmane.org; Sun, 02 Jul 2017 11:47:15 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:47200) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dRh4x-0004Oy-3O for emacs-devel@gnu.org; Sun, 02 Jul 2017 11:46:32 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dRh4v-0003qY-Qc for emacs-devel@gnu.org; Sun, 02 Jul 2017 11:46:31 -0400 Original-Received: from mail-oi0-x229.google.com ([2607:f8b0:4003:c06::229]:33605) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1dRh4v-0003qM-J2 for emacs-devel@gnu.org; Sun, 02 Jul 2017 11:46:29 -0400 Original-Received: by mail-oi0-x229.google.com with SMTP id p188so66882866oia.0 for ; Sun, 02 Jul 2017 08:46:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=ah+wZmK2jS0QTwBZp6kq3yrAilbCzZOf0resky9vP90=; b=JoNiALiQ1vhYTC4ABfDjc1Vv2Rhxjkc2W6JZGIh5x+uWJWpO9knzT9MqxoIUyMbK8z v0yt6hpu6CzFNsSXIhlZjsUVSMR8ZA5X40lDRzjvuvnpL/5XWgMGGlSukAK9pkfiv1rB UeTE7tdPQBQG/R3if9BdQRbc07hdBmkH+7mQ/MPtvF4qasExVY9GRLOTt6IMztXhAUdk 89inSgYVTC1IkyfGo4zgD1K66VxmJFf80ArBfbxaKPuZVkGwYd72nD2K9XWp00RAphSK N6dmNKU8iVjEW75QoAUfN2EDkmufzW0VepB4UUkdxD6mNnColZh5jAiKOKcn6dEgkurI vCcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ah+wZmK2jS0QTwBZp6kq3yrAilbCzZOf0resky9vP90=; b=hrrrWu1n2Sti3ALkKVIji5GV1+287PCBO6oiEj+6Nv/UNU/O8LMxUkP2QPf+V92b0e JR7f1LaCbIyRXRJvDL3EOoQs/zLk23hMyHcPByPSo2DZ6v3RE1YV11ZBsSIlYeVJXaDV yxypt8J3tFyAP9dYPKlq6jRlSB2pbxU46oms7GiNuW9vhrCTet9PpI5abU6EvPAoOI5E uoS5MPoUkGcR1h5aC+/MdGcERAvtKa0WCKNCzQruLOJHu9VtD4SolMTaxq1zfAMlTck8 AnYPUUYlA1N/5sBMCyRmaSQ1ZcPjMhE/e4U0sqLkPXj7lgfrdO4Z7c+bG0wOyUNJhHKP eL7A== X-Gm-Message-State: AKS2vOzN1PNXCavejiqztX3YdfkVsaKW1B/VEdfrAuGt4OzIM4gMFGzz ngkKbVfRJHVV9l6SDs51bZJiiSHEQA== X-Received: by 10.202.215.213 with SMTP id o204mr13019024oig.23.1499010388801; Sun, 02 Jul 2017 08:46:28 -0700 (PDT) In-Reply-To: X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4003:c06::229 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.org gmane.emacs.devel:216109 Archived-At: --001a113d6350bd3fe20553578d2d Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Ken Raeburn schrieb am Mo., 29. Mai 2017 um 11:33 Uhr= : > > On May 28, 2017, at 08:43, Philipp Stephani wrote= : > > > > Ken Raeburn schrieb am So., 28. Mai 2017 um > 13:07 Uhr: > >> >> On May 21, 2017, at 04:53, Paul Eggert wrote: >> >> > Ken Raeburn wrote: >> >> The Guile project has taken this idea pretty far; they=E2=80=99re gen= erating >> ELF object files with a few special sections for Guile objects, using th= e >> standard DWARF sections for debug information, etc. While it has a cert= ain >> appeal (making C modules and Lisp files look much more similar, maybe be= ing >> able to link Lisp and C together into one executable image, letting GDB >> understand some of your data), switching to a machine-specific format wo= uld >> be a pretty drastic change, when we can currently share the files across >> machines. >> > >> > Although it does indeed sound like a big change, I don't see why it >> would prevent us from sharing the files across machines. Emacs can use >> standard ELF and DWARF format on any platform if Emacs is doing the >> loading. And there should be some software-engineering benefit in using = the >> same format that Guile uses. >> >> Sorry for the delay in responding. >> >> The ELF format has header fields indicating the word size, endianness, >> machine architecture (though there=E2=80=99s a value for =E2=80=9Cnone= =E2=80=9D), and OS ABI. Some >> fields vary in size or order depending on whether the 32-bit or 64-bit >> format is in use. Some other format details (e.g., relocation types, >> interpretation of certain ranges of values in some fields) are >> architecture- or OS-dependent; we might not care about many of those >> details, but relocations are likely needed if we want to play linking ga= mes >> or use DWARF. >> >> I think Guile is using whatever the native word size and architecture >> are. If we do that for Emacs, they=E2=80=99re not portable between plat= forms. >> Currently it works for me to put my Lisp files, both source and compiled= , >> into ~/elisp and use them from different kinds of machines if my home >> directory is NFS-mounted. >> >> We could instead pick fixed values (say, architecture =E2=80=9Cnone=E2= =80=9D, >> little-endian, 32-bit), but then there=E2=80=99s no guarantee that we co= uld use any >> of the usual GNU tools on them without a bunch of work, or that we=E2=80= =99d ever >> be able to use non-GNU tools to treat them as object files. Then again,= we >> couldn=E2=80=99t expect to do the latter portably anyway, since some of = the >> platforms don=E2=80=99t even use ELF. >> >> > Is there any significant advantage of using ELF, or could this just use > one of the standard binary serialization formats (protobuf, flatbuffer, > ...)? > > > That=E2=80=99s an interesting idea. If one of the popular serialization = libraries > is compatibly licensed, easy to use, and performs well, it may be better > than rolling our own. > I've tried this out (with flatbuffers), but I haven't seen significant speed improvements. It might very well be the case that during loading the reader is already fast enough (e.g. for ELC files it doesn't do any decoding), and it's the evaluator that's too slow. --001a113d6350bd3fe20553578d2d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


Ken Ra= eburn <raeburn@raeburn.org>= ; schrieb am Mo., 29. Mai 2017 um 11:33=C2=A0Uhr:

On Ma= y 28, 2017, at 08:43, Philipp Stephani <p.stephani2@gmail.com> wrote:



Ken Raeburn <raeburn@raeburn.org> schrieb am So., 28. Mai 2017 um 13:= 07=C2=A0Uhr:

On May 21, 2017, at 04:53, Paul Eggert <eggert@cs.ucla.edu> wrote:

> Ken Raeburn wrote:
>> The Guile project has taken this idea pretty far; they=E2=80=99re = generating ELF object files with a few special sections for Guile objects, = using the standard DWARF sections for debug information, etc.=C2=A0 While i= t has a certain appeal (making C modules and Lisp files look much more simi= lar, maybe being able to link Lisp and C together into one executable image= , letting GDB understand some of your data), switching to a machine-specifi= c format would be a pretty drastic change, when we can currently share the = files across machines.
>
> Although it does indeed sound like a big change, I don't see why i= t would prevent us from sharing the files across machines. Emacs can use st= andard ELF and DWARF format on any platform if Emacs is doing the loading. = And there should be some software-engineering benefit in using the same for= mat that Guile uses.

Sorry for the delay in responding.

The ELF format has header fields indicating the word size, endianness, mach= ine architecture (though there=E2=80=99s a value for =E2=80=9Cnone=E2=80=9D= ), and OS ABI.=C2=A0 Some fields vary in size or order depending on whether= the 32-bit or 64-bit format is in use.=C2=A0 Some other format details (e.= g., relocation types, interpretation of certain ranges of values in some fi= elds) are architecture- or OS-dependent; we might not care about many of th= ose details, but relocations are likely needed if we want to play linking g= ames or use DWARF.

I think Guile is using whatever the native word size and architecture are.= =C2=A0 If we do that for Emacs, they=E2=80=99re not portable between platfo= rms.=C2=A0 Currently it works for me to put my Lisp files, both source and = compiled, into ~/elisp and use them from different kinds of machines if my = home directory is NFS-mounted.

We could instead pick fixed values (say, architecture =E2=80=9Cnone=E2=80= =9D, little-endian, 32-bit), but then there=E2=80=99s no guarantee that we = could use any of the usual GNU tools on them without a bunch of work, or th= at we=E2=80=99d ever be able to use non-GNU tools to treat them as object f= iles.=C2=A0 Then again, we couldn=E2=80=99t expect to do the latter portabl= y anyway, since some of the platforms don=E2=80=99t even use ELF.


Is there any significant advantage of = using ELF, or could this just use one of the standard binary serialization = formats (protobuf, flatbuffer, ...)?=C2=A0

That=E2=80=99s an int= eresting idea.=C2=A0 If one of the popular serialization libraries is compa= tibly licensed, easy to use, and performs well, it=C2=A0may be better than rolling our own.
<= /div>

I've tried this out (with flatbuf= fers), but I haven't seen significant speed improvements. It might very= well be the case that during loading the reader is already fast enough (e.= g. for ELC files it doesn't do any decoding), and it's the evaluato= r that's too slow.=C2=A0
--001a113d6350bd3fe20553578d2d--