From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: =?utf-8?Q?Ludovic_Court=C3=A8s?= Newsgroups: gmane.lisp.guile.devel Subject: Reducing memory usage of the linker and assembler Date: Mon, 09 Jan 2023 11:46:02 +0100 Message-ID: <87v8lgrno5.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="4107"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) Cc: Andy Wingo To: guile-devel@gnu.org Original-X-From: guile-devel-bounces+guile-devel=m.gmane-mx.org@gnu.org Mon Jan 09 11:47:13 2023 Return-path: Envelope-to: guile-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pEpgO-0000tG-Ly for guile-devel@m.gmane-mx.org; Mon, 09 Jan 2023 11:47:12 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pEpfM-0003NX-LB; Mon, 09 Jan 2023 05:46:08 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pEpfK-0003Mo-9W for guile-devel@gnu.org; Mon, 09 Jan 2023 05:46:06 -0500 Original-Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pEpfJ-00017B-Sk; Mon, 09 Jan 2023 05:46:05 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:Subject:To:From:in-reply-to: references; bh=hsAjYtMgtUbj/IzzNF5bJnZdrh+WRXT6zcQLlkR4gvI=; b=XisX0q1eERrtrp h0ZC/rRpzQbFOLJJwdAoZPKCkX9o482CKHi2aWo9CDkHv0nuU6jz0cm5Kzc6XrYk6CSb1KGmBrNUF jVaAFSCgT1CSkyOmHCG9uDIu98BoJ7QNn0E9zv/Xb+RfP0EYS/2R5V2sEftOzG+lSFH7MWLZ5Ocku 9QbVpy7lgT67sWZvHJk55YsbyXy/qcMvpGfEyAFFkeaeX/sbuEuv/PK9iYk+TqZ8e82oZNRcSaRPC TrBpQ1VUnlNGOoSDgNPGRF26EkFGWX4ve3lvwNOibAiwYUE1TLNODFnaYkOxuYHmC93n5IS9p3318 8or1yLJHwtc0vdicZY5w==; Original-Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pEpfJ-0007i6-8r; Mon, 09 Jan 2023 05:46:05 -0500 X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: =?utf-8?Q?D=C3=A9cadi?= 20 =?utf-8?Q?Niv=C3=B4se?= an 231 de la =?utf-8?Q?R=C3=A9volution=2C?= jour du Van X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu X-BeenThere: guile-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Developers list for Guile, the GNU extensibility library" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-devel-bounces+guile-devel=m.gmane-mx.org@gnu.org Original-Sender: guile-devel-bounces+guile-devel=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.lisp.guile.devel:21551 Archived-At: Hello Guilers! The =E2=80=98wip-linker-assembler-memory-consumption=E2=80=99 branch I rece= ntly pushed aims to reduce the allocation rate and memory consumption of the linker and assembler, as well as the number of bytevector copies. In 3.0.8, the linker and assembler keep roughly two copies of the final ELF file in memory: each contains a =E2=80=98bv=E2=80=99 fi= eld with its wire representation, and then =E2=80=98link-elf=E2=80=99 allocates a byteve= ctor to contain the whole ELF file and copies each bytevector back into that one. When you=E2=80=99re running =E2=80=98guild compile=E2= =80=99, that big bytevector is immediately passed to =E2=80=98put-bytevector=E2=80=99 to wri= te it to disk. The branch replaces the =E2=80=98bv=E2=80=99 field of with = =E2=80=98size=E2=80=99 and =E2=80=98writer=E2=80=99, the latter being a procedure that takes a bytevec= tor and writes to it. At this point, =E2=80=98link-elf=E2=80=99 still allocates one bytevector fo= r each linker object. Eventually we could rewrite those =E2=80=9Cwriter=E2=80=9D procedu= res to use binary I/O primitives instead of expecting a bytevector to write to, and that way we wouldn=E2=80=99t need those temporary bytevectors. It=E2=80=99= s a bit tedious to do though. One big source of memory consumption that remains is the assembler: its =E2=80=98buf=E2=80=99 field contains an in-memory copy of the =E2=80=98.rtl= -text=E2=80=99 section, which is often a significant portion of the ELF file. Addressing that would be more difficult. On the compilation at -O1 of a big file (=E2=80=98gnu/packages/python-xyz.s= cm=E2=80=99 in Guix), I see a 5% reduction of the heap size as measured at the end of the whole compilation process. There are other factors before the linking step that contribute to the peak heap size, which probably explains why the impact is not higher. The current set of commits looks like this: cdf9a0ac4 * DRAFT linker: Do not store entire ELF in memory when writing = to a file. 048651993 * linker: Linker object writer takes a single argument. 2ac4e5a3d * linker, assembler: Avoid intermediate bytevectors. 8c8d4bd37 * DRAFT Add 'bytevector-slice'. 805275882 * linker: Separate effectful part of 'add-elf-objects'. 8194e8a6e * assembler: Separate effectful part of 'link-docstrs'. 0a5768add * assembler: Separate effectful part of 'link-frame-maps'. f6fb95db8 * assembler: Separate effectful part of 'link-procprops'. 099cf8f30 * assembler: Separate effectful part of 'link-dynamic-section'. 43dd2f184 * assembler: Separate effectful part of 'link-symtab'. ad4487091 * assembler: Separate 'process-relocs' from 'patch-relocs!'. There=E2=80=99s no regression. The main issue worth discussing is the last commit, which changes =E2=80=98link-elf=E2=80=99 to optionally return a pro= cedure instead of a bytevector. We also need to discuss =E2=80=98bytevector-slice= =E2=80=99, but that can be done separately. Andy, what do you think of this approach? Cheers, Ludo=E2=80=99.