Guile 2.2 .go files are larger

Guile 2.2 .go files are larger

[Ludovic Courtès]

Hello!

The closure of Guix built with 2.0 is 193.8 MiB; when built with 2.2,
it’s 311.8 MiB.  Guix itself goes from 66 to 150 MiB:

--8<---------------cut here---------------start------------->8---
$ du -ms /gnu/store/jh07pwbyf5dbpdd5q0nvgagqkgmh76nh-guix-0.12.0-9.25a4/lib/guile/2.2
101	/gnu/store/jh07pwbyf5dbpdd5q0nvgagqkgmh76nh-guix-0.12.0-9.25a4/lib/guile/2.2
$ du -ms /gnu/store/rnpz1svz4aw75kibb5qb02hhccy2m4y0-guix-0.12.0-7.aabe/lib/guile/2.0
24	/gnu/store/rnpz1svz4aw75kibb5qb02hhccy2m4y0-guix-0.12.0-7.aabe/lib/guile/2.0
--8<---------------cut here---------------end--------------->8---

Would you have any suggestions to shrink the ELF files a bit?

Binutils’ ‘strip’ command bails out on .go files:

  strip: Unable to recognise the format of the input file `/gnu/store/jh07pwbyf5dbpdd5q0nvgagqkgmh76nh-guix-0.12.0-9.25a4/lib/guile/2.2/site-ccache/guix/scripts/refresh.go'

Also we’d rather avoid stripping debug info, if possible, to preserve
debuggability.

Thoughts?

Ludo’.

Re: Guile 2.2 .go files are larger

[Andy Wingo]

On Sat 22 Apr 2017 15:19, ludo@gnu.org (Ludovic Courtès) writes:

> The closure of Guix built with 2.0 is 193.8 MiB; when built with 2.2,
> it’s 311.8 MiB.  Guix itself goes from 66 to 150 MiB:
>
> $ du -ms /gnu/store/jh07pwbyf5dbpdd5q0nvgagqkgmh76nh-guix-0.12.0-9.25a4/lib/guile/2.2
> 101	/gnu/store/jh07pwbyf5dbpdd5q0nvgagqkgmh76nh-guix-0.12.0-9.25a4/lib/guile/2.2
> $ du -ms /gnu/store/rnpz1svz4aw75kibb5qb02hhccy2m4y0-guix-0.12.0-7.aabe/lib/guile/2.0
> 24	/gnu/store/rnpz1svz4aw75kibb5qb02hhccy2m4y0-guix-0.12.0-7.aabe/lib/guile/2.0

Before we begin, some general notes.  My understanding is that the heap
usage corresponding to an individual Guix process will be lower, both
due to allocation of read-only data in read-only, shareable sections of
the .go ELF files, allocation of read-write data in packed sections
known to the GC but not managed by GC, and various optimizations that
can eliminate or simplify some heap allocations (closure optimization
among them).  In short my understanding is that Guile 2.2 (and systems
built on it) should have a smaller run-time footprint than Guile 2.2.

> Would you have any suggestions to shrink the ELF files a bit?

Why?  Have you compared gzipped or lzipped sizes?  I don't want to put
effort in here that's not worth it :)

Part of it is that our page alignment is 64 kB and so average wasteage
per .go is 32kB, and there are a lot of .go files.  These are just zero
bytes though.

If you look at an individual file, you can use readelf to see things, or
that old ELF visualizer I wrote:

  https://wingolog.org/elf-mapper/

Here's a map of gnu/packages/curl.go:

  https://wingolog.org/pub/elf/elf-JWXYrI.png

I think stripping the .debug_* stuff won't get you much of anywhere.

Stripping arities info could reduce size by 10%.  We could figure out a
different way to represent arities that takes less space.

Compiling all the .go files together (requires Guile hacking) could
remove that per-go 64kB alignment overhead.  Alternately we could do
what GNU tools / ld.so do which maps the linear sequence of bytes in the
file to aligned segments in memory.

Honestly though I would punt.  It's not a run-time issue.  I don't think
it's a transport issue.  It's only a disk space issue.  I personally
don't plan to spend time on it but am happy to point out possibilities
to people that will do so.

Andy

Re: Guile 2.2 .go files are larger

[Andy Wingo]

On Mon 24 Apr 2017 10:24, Andy Wingo <wingo@igalia.com> writes:

> On Sat 22 Apr 2017 15:19, ludo@gnu.org (Ludovic Courtès) writes:
>
>> The closure of Guix built with 2.0 is 193.8 MiB; when built with 2.2,
>> it’s 311.8 MiB.  Guix itself goes from 66 to 150 MiB:
>>
>> $ du -ms /gnu/store/jh07pwbyf5dbpdd5q0nvgagqkgmh76nh-guix-0.12.0-9.25a4/lib/guile/2.2
>> 101	/gnu/store/jh07pwbyf5dbpdd5q0nvgagqkgmh76nh-guix-0.12.0-9.25a4/lib/guile/2.2
>> $ du -ms /gnu/store/rnpz1svz4aw75kibb5qb02hhccy2m4y0-guix-0.12.0-7.aabe/lib/guile/2.0
>> 24	/gnu/store/rnpz1svz4aw75kibb5qb02hhccy2m4y0-guix-0.12.0-7.aabe/lib/guile/2.0
>
> Before we begin, some general notes.  My understanding is that the heap
> usage corresponding to an individual Guix process will be lower, both
                                                                 ^
                              lower than Guix-on-Guile-2.0, I mean

> due to allocation of read-only data in read-only, shareable sections of
> the .go ELF files, allocation of read-write data in packed sections
> known to the GC but not managed by GC, and various optimizations that
> can eliminate or simplify some heap allocations (closure optimization
> among them).  In short my understanding is that Guile 2.2 (and systems
> built on it) should have a smaller run-time footprint than Guile 2.2.

Er, than smaller footprint than Guile 2.0.

Sorry for the confusion :)

Re: Guile 2.2 .go files are larger

[Ludovic Courtès]

Hi!

Andy Wingo <wingo@igalia.com> skribis:

> On Sat 22 Apr 2017 15:19, ludo@gnu.org (Ludovic Courtès) writes:
>
>> The closure of Guix built with 2.0 is 193.8 MiB; when built with 2.2,
>> it’s 311.8 MiB.  Guix itself goes from 66 to 150 MiB:
>>
>> $ du -ms /gnu/store/jh07pwbyf5dbpdd5q0nvgagqkgmh76nh-guix-0.12.0-9.25a4/lib/guile/2.2
>> 101	/gnu/store/jh07pwbyf5dbpdd5q0nvgagqkgmh76nh-guix-0.12.0-9.25a4/lib/guile/2.2
>> $ du -ms /gnu/store/rnpz1svz4aw75kibb5qb02hhccy2m4y0-guix-0.12.0-7.aabe/lib/guile/2.0
>> 24	/gnu/store/rnpz1svz4aw75kibb5qb02hhccy2m4y0-guix-0.12.0-7.aabe/lib/guile/2.0
>
> Before we begin, some general notes.  My understanding is that the heap
> usage corresponding to an individual Guix process will be lower, both
> due to allocation of read-only data in read-only, shareable sections of
> the .go ELF files, allocation of read-write data in packed sections
> known to the GC but not managed by GC, and various optimizations that
> can eliminate or simplify some heap allocations (closure optimization
> among them).  In short my understanding is that Guile 2.2 (and systems
> built on it) should have a smaller run-time footprint than Guile 2.2.

Yes of course.

>> Would you have any suggestions to shrink the ELF files a bit?
>
> Why?  Have you compared gzipped or lzipped sizes?  I don't want to put
> effort in here that's not worth it :)

--8<---------------cut here---------------start------------->8---
$ du -ms $(./pre-inst-env guix pack guile@2.0)
35	/gnu/store/25ppdmridc8i1j771s9c498y1sr9xfzb-tarball-pack.tar.gz
$ du -ms $(./pre-inst-env guix pack guile@2.2)
40	/gnu/store/3fkhdpfpjjn8088fs0dxgb6hi38ac46m-tarball-pack.tar.gz
--8<---------------cut here---------------end--------------->8---

(Of course the difference is also due to the additional source code in
2.2.)

> Part of it is that our page alignment is 64 kB and so average wasteage
> per .go is 32kB, and there are a lot of .go files.  These are just zero
> bytes though.

OK.  It might be that file systems can store sparse files reasonably
efficiently.

> If you look at an individual file, you can use readelf to see things, or
> that old ELF visualizer I wrote:
>
>   https://wingolog.org/elf-mapper/
>
> Here's a map of gnu/packages/curl.go:
>
>   https://wingolog.org/pub/elf/elf-JWXYrI.png
>
> I think stripping the .debug_* stuff won't get you much of anywhere.
>
> Stripping arities info could reduce size by 10%.  We could figure out a
> different way to represent arities that takes less space.
>
> Compiling all the .go files together (requires Guile hacking) could
> remove that per-go 64kB alignment overhead.  Alternately we could do
> what GNU tools / ld.so do which maps the linear sequence of bytes in the
> file to aligned segments in memory.
>
> Honestly though I would punt.  It's not a run-time issue.  I don't think
> it's a transport issue.  It's only a disk space issue.  I personally
> don't plan to spend time on it but am happy to point out possibilities
> to people that will do so.

Sure, I understand.  I was just wondering if there was some low-hanging
fruit here.

Thanks for your feedback!

Ludo’.