Reducing memory usage of the linker and assembler

Reducing memory usage of the linker and assembler

[Ludovic Courtès]

Hello Guilers!

The ‘wip-linker-assembler-memory-consumption’ branch I recently pushed
aims to reduce the allocation rate and memory consumption of the linker
and assembler, as well as the number of bytevector copies.

In 3.0.8, the linker and assembler keep roughly two copies of the final
ELF file in memory: each <linker-object> contains a ‘bv’ field with its
wire representation, and then ‘link-elf’ allocates a bytevector to
contain the whole ELF file and copies each <linker-object> bytevector
back into that one.  When you’re running ‘guild compile’, that big
bytevector is immediately passed to ‘put-bytevector’ to write it to
disk.

The branch replaces the ‘bv’ field of <linker-object> with ‘size’ and
‘writer’, the latter being a procedure that takes a bytevector and
writes to it.

At this point, ‘link-elf’ still allocates one bytevector for each linker
object.  Eventually we could rewrite those “writer” procedures to use
binary I/O primitives instead of expecting a bytevector to write to, and
that way we wouldn’t need those temporary bytevectors.  It’s a bit
tedious to do though.

One big source of memory consumption that remains is the assembler: its
‘buf’ field contains an in-memory copy of the ‘.rtl-text’ section, which
is often a significant portion of the ELF file.  Addressing that would
be more difficult.

On the compilation at -O1 of a big file (‘gnu/packages/python-xyz.scm’
in Guix), I see a 5% reduction of the heap size as measured at the end
of the whole compilation process.  There are other factors before the
linking step that contribute to the peak heap size, which probably
explains why the impact is not higher.

The current set of commits looks like this:

  cdf9a0ac4 * DRAFT linker: Do not store entire ELF in memory when writing to a file.
  048651993 * linker: Linker object writer takes a single argument.
  2ac4e5a3d * linker, assembler: Avoid intermediate bytevectors.
  8c8d4bd37 * DRAFT Add 'bytevector-slice'.
  805275882 * linker: Separate effectful part of 'add-elf-objects'.
  8194e8a6e * assembler: Separate effectful part of 'link-docstrs'.
  0a5768add * assembler: Separate effectful part of 'link-frame-maps'.
  f6fb95db8 * assembler: Separate effectful part of 'link-procprops'.
  099cf8f30 * assembler: Separate effectful part of 'link-dynamic-section'.
  43dd2f184 * assembler: Separate effectful part of 'link-symtab'.
  ad4487091 * assembler: Separate 'process-relocs' from 'patch-relocs!'.

There’s no regression.  The main issue worth discussing is the last
commit, which changes ‘link-elf’ to optionally return a procedure
instead of a bytevector.  We also need to discuss ‘bytevector-slice’,
but that can be done separately.

Andy, what do you think of this approach?

Cheers,
Ludo’.

Re: Reducing memory usage of the linker and assembler

[Ludovic Courtès]

Hello,

Ludovic Courtès <ludo@gnu.org> skribis:

> The branch replaces the ‘bv’ field of <linker-object> with ‘size’ and
> ‘writer’, the latter being a procedure that takes a bytevector and
> writes to it.
>
> At this point, ‘link-elf’ still allocates one bytevector for each linker
> object.  Eventually we could rewrite those “writer” procedures to use
> binary I/O primitives instead of expecting a bytevector to write to, and
> that way we wouldn’t need those temporary bytevectors.  It’s a bit
> tedious to do though.

I’ve now merged this branch into ‘main’:

  3cd64feb2 * linker: Do not store entire ELF in memory when writing to a file.
  4ab71e1f0 * linker: Linker object writer takes a single argument.
  041f11b35 * linker, assembler: Avoid intermediate bytevectors.
  d0d974360 * linker: Separate effectful part of 'add-elf-objects'.
  d439a3f67 * assembler: Separate effectful part of 'link-docstrs'.
  13e2d5b66 * assembler: Separate effectful part of 'link-frame-maps'.
  dc0c4ccb1 * assembler: Separate effectful part of 'link-procprops'.
  c7f1522c6 * assembler: Separate effectful part of 'link-dynamic-section'.
  fc5eae5d0 * assembler: Separate effectful part of 'link-symtab'.
  15c4c4ceb * assembler: Separate 'process-relocs' from 'patch-relocs!'.

As discussed on IRC, I’d like to tag and upload 3.0.9rc1 hopefully
tomorrow to get some testing (in particular portability testing) and
release roughly a week later.

Thoughts?

Ludo’.