The size of ‘.go’ files

["Ludovic Courtès" <ludo@gnu.org>]

[-- Attachment #1: Type: text/plain, Size: 3725 bytes --]

Hello Guix!

On IRC there was a discussion about the size of ‘.go’ files.  The
discussion came from this observation:

--8<---------------cut here---------------start------------->8---
$ guix size $(readlink -f /run/current-system) | head -5
store item                                                       total    self
/gnu/store/4d0p06xgaw8lqa9db0d6728kkba8bizj-qemu-5.0.0            1651.6   745.2  18.8%
/gnu/store/abiva5ivq99x30r2s9pa3jj0pv9g16sv-guix-1.1.0-4.bdc801e   468.0   268.8   6.8%
/gnu/store/111zp1qyind7hsnvrm5830jhankmx4ls-linux-libre-5.4.43     243.6   243.6   6.2%
/gnu/store/skxkrhgn9z0fg9hmnbcyfdgzs5w4ryrr-llvm-9.0.1             199.9   128.5   3.2%
--8<---------------cut here---------------end--------------->8---

On disk, those .go files take quite a bit of space (I hear you Btrfs
people, don’t say it! :-)).

The code snippet below sorts the ELF sections of a .go file by size; for
‘python-xyz.go’, I get this:

--8<---------------cut here---------------start------------->8---
$13 = ((".rtl-text" . 3417108)
 (".guile.arities" . 1358536)
 (".data" . 586912)
 (".rodata" . 361599)
 (".symtab" . 117000)
 (".debug_line" . 97342)
 (".debug_info" . 54519)
 (".guile.frame-maps" . 47114)
 ("" . 1344)
 (".guile.arities.strtab" . 681)
 ("" . 232)
 (".shstrtab" . 229)
 (".dynamic" . 112)
 (".debug_str" . 87)
 (".strtab" . 75)
 (".debug_abbrev" . 65)
 (".guile.docstrs.strtab" . 1)
 ("" . 0)
 (".guile.procprops" . 0)
 (".guile.docstrs" . 0)
 (".debug_loc" . 0))
scheme@(guile-user)> (stat:size (stat go))
$14 = 6083445
--8<---------------cut here---------------end--------------->8---

More than half of those 6 MiB is code, and more than 1 MiB is
“.guile.arities” (info "(guile) Object File Format"), which is
surprisingly large; presumably the file only contains thunks (the
‘thunked’ fields of <package>).

Stripping the .debug_* sections (if that works) clearly wouldn’t help.

So I guess we could generate less code (reduce ‘.rtl-text’), perhaps by
tweaking ‘define-record-type*’, but I have little hope there.

We could also investigate where “.guile.arities” could be made denser,
or use fewer thunked fields in <package>.  Currently arity info takes
7x4 = 28 bytes per procedure as documented in (system vm assembler).
With an extra flag we could perhaps save 8 bytes for the simple case
where nopt = 0, nreq is small, and other flags are zero.

But anyway, currently there are (1358536 - 4) / 28 = 48K arity headers
in this file.  However, the file contains 970 packages, so we’re talking
about ~50 procedures per package, even though there are only 5 thunked
fields.  Weird!  Maybe I’m missing something.

But wait, that was with 3.0.2 and -O1.

With 3.0.3-to-be and -O1, python-xyz.go weighs in at 3.4 MiB instead of
5.9 MiB!  Here’s the section size distribution:

--8<---------------cut here---------------start------------->8---
$4 = ((".rtl-text" . 2101168)
 (".data" . 586392)
 (".rodata" . 360703)
 (".guile.arities" . 193106)
 (".symtab" . 117000)
 (".debug_line" . 76685)
 (".debug_info" . 53513)
 ("" . 1280)
 (".guile.arities.strtab" . 517)
 ("" . 232)
 (".shstrtab" . 211)
 (".dynamic" . 96)
 (".debug_str" . 87)
 (".strtab" . 75)
 (".debug_abbrev" . 56)
 (".guile.docstrs.strtab" . 1)
 ("" . 0)
 (".guile.procprops" . 0)
 (".guile.docstrs" . 0)
 (".debug_loc" . 0))
scheme@(guile-user)> (stat:size (stat go))
$5 = 3519323
--8<---------------cut here---------------end--------------->8---

“.rtl-text” is 38% smaller and “.guile.arities” is almost a tenth of
what it was.

Something’s going on here!  Thoughts?

Ludo’.


[-- Attachment #2: the code --]
[-- Type: text/plain, Size: 769 bytes --]

(use-modules (system vm elf)
             (rnrs io ports)
             (ice-9 match))

(define go
  (search-path %load-compiled-path "gnu/packages/python-xyz.go"))

(define elf
  (parse-elf (call-with-input-file go get-bytevector-all)))

(define (elf-section-name-as-string elf section)
  (let ((off (elf-section-offset
              (list-ref (elf-sections elf)
                        (elf-shstrndx elf)))))
    (string-table-ref (elf-bytes elf)
                      (+ off (elf-section-name section)))))

(sort (map (lambda (section)
             (cons (elf-section-name-as-string elf section)
                   (elf-section-size section)))
           (elf-sections elf))
      (match-lambda*
        (((name1 . size1) (name2 . size2))
         (> size1 size2))))