unofficial mirror of guile-devel@gnu.org 
 help / color / mirror / Atom feed
* Reducing memory usage of the linker and assembler
@ 2023-01-09 10:46 Ludovic Courtès
  2023-01-17 16:54 ` Ludovic Courtès
  0 siblings, 1 reply; 2+ messages in thread
From: Ludovic Courtès @ 2023-01-09 10:46 UTC (permalink / raw)
  To: guile-devel; +Cc: Andy Wingo

Hello Guilers!

The ‘wip-linker-assembler-memory-consumption’ branch I recently pushed
aims to reduce the allocation rate and memory consumption of the linker
and assembler, as well as the number of bytevector copies.

In 3.0.8, the linker and assembler keep roughly two copies of the final
ELF file in memory: each <linker-object> contains a ‘bv’ field with its
wire representation, and then ‘link-elf’ allocates a bytevector to
contain the whole ELF file and copies each <linker-object> bytevector
back into that one.  When you’re running ‘guild compile’, that big
bytevector is immediately passed to ‘put-bytevector’ to write it to
disk.

The branch replaces the ‘bv’ field of <linker-object> with ‘size’ and
‘writer’, the latter being a procedure that takes a bytevector and
writes to it.

At this point, ‘link-elf’ still allocates one bytevector for each linker
object.  Eventually we could rewrite those “writer” procedures to use
binary I/O primitives instead of expecting a bytevector to write to, and
that way we wouldn’t need those temporary bytevectors.  It’s a bit
tedious to do though.

One big source of memory consumption that remains is the assembler: its
‘buf’ field contains an in-memory copy of the ‘.rtl-text’ section, which
is often a significant portion of the ELF file.  Addressing that would
be more difficult.

On the compilation at -O1 of a big file (‘gnu/packages/python-xyz.scm’
in Guix), I see a 5% reduction of the heap size as measured at the end
of the whole compilation process.  There are other factors before the
linking step that contribute to the peak heap size, which probably
explains why the impact is not higher.

The current set of commits looks like this:

  cdf9a0ac4 * DRAFT linker: Do not store entire ELF in memory when writing to a file.
  048651993 * linker: Linker object writer takes a single argument.
  2ac4e5a3d * linker, assembler: Avoid intermediate bytevectors.
  8c8d4bd37 * DRAFT Add 'bytevector-slice'.
  805275882 * linker: Separate effectful part of 'add-elf-objects'.
  8194e8a6e * assembler: Separate effectful part of 'link-docstrs'.
  0a5768add * assembler: Separate effectful part of 'link-frame-maps'.
  f6fb95db8 * assembler: Separate effectful part of 'link-procprops'.
  099cf8f30 * assembler: Separate effectful part of 'link-dynamic-section'.
  43dd2f184 * assembler: Separate effectful part of 'link-symtab'.
  ad4487091 * assembler: Separate 'process-relocs' from 'patch-relocs!'.

There’s no regression.  The main issue worth discussing is the last
commit, which changes ‘link-elf’ to optionally return a procedure
instead of a bytevector.  We also need to discuss ‘bytevector-slice’,
but that can be done separately.

Andy, what do you think of this approach?

Cheers,
Ludo’.



^ permalink raw reply	[flat|nested] 2+ messages in thread

end of thread, other threads:[~2023-01-17 16:54 UTC | newest]

Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-01-09 10:46 Reducing memory usage of the linker and assembler Ludovic Courtès
2023-01-17 16:54 ` Ludovic Courtès

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).