bug#19973: Grafts break debug outputs

bug#19973: Grafts break debug outputs

[Mark H Weaver]

Grafts break debug outputs, or at least that's my guess.
Here's what I see:

mhw@jojen:~$ guix build guile
guix build: warning: ambiguous package specification `guile'
guix build: warning: choosing guile-2.0.11 from gnu/packages/guile.scm:110:2
/gnu/store/3lhr8q28q6f59774di9av7ncy09jd55d-guile-2.0.11
/gnu/store/rgv3fvy6xqp6966rfh8v6fv7m48abcbh-guile-2.0.11-debug
mhw@jojen:~$ guix package -I guile
guile	2.0.11	out	/gnu/store/3lhr8q28q6f59774di9av7ncy09jd55d-guile-2.0.11
guile	2.0.11	debug	/gnu/store/rgv3fvy6xqp6966rfh8v6fv7m48abcbh-guile-2.0.11-debug
mhw@jojen:~$ ls -l .guix-profile/lib/debug/gnu/store/
total 8
lrwxrwxrwx 12 root guixbuild 128 Dec 31  1969 122jv790mv2mlnylbrbzav65vghbw93n-guile-2.0.11 -> /gnu/store/rgv3fvy6xqp6966rfh8v6fv7m48abcbh-guile-2.0.11-debug/lib/debug/gnu/store/122jv790mv2mlnylbrbzav65vghbw93n-guile-2.0.11
lrwxrwxrwx 15 root guixbuild 127 Dec 31  1969 3g20rdmnavpblsmgppyl8jhg67nidhjk-glibc-2.20 -> /gnu/store/hrny2whqg9c3m0klyfpbmmcyiir9yf8m-gcc-toolchain-4.9.2/lib/debug/gnu/store/3g20rdmnavpblsmgppyl8jhg67nidhjk-glibc-2.20

I guess GDB can't find the debugging information because
122jv790mv2mlnylbrbzav65vghbw93n-guile-2.0.11 is the name of the guile
directory *before* grafting.

To work around this, I created my own ~/mydebug directory:

mhw@jojen:~/mydebug/gnu/store$ ln -s /gnu/store/rgv3fvy6xqp6966rfh8v6fv7m48abcbh-guile-2.0.11-debug/lib/debug/gnu/store/122jv790mv2mlnylbrbzav65vghbw93n-guile-2.0.11 3lhr8q28q6f59774di9av7ncy09jd55d-guile-2.0.11

and then pointed GDB at ~/mydebug and then debugging works again.

      Mark

bug#19973: Grafts break debug outputs

[Mark H Weaver]

Mark H Weaver <mhw@netris.org> writes:

> Grafts break debug outputs, or at least that's my guess.
> Here's what I see:
>
> mhw@jojen:~$ guix build guile
> guix build: warning: ambiguous package specification `guile'
> guix build: warning: choosing guile-2.0.11 from gnu/packages/guile.scm:110:2
> /gnu/store/3lhr8q28q6f59774di9av7ncy09jd55d-guile-2.0.11
> /gnu/store/rgv3fvy6xqp6966rfh8v6fv7m48abcbh-guile-2.0.11-debug
> mhw@jojen:~$ guix package -I guile
> guile	2.0.11	out	/gnu/store/3lhr8q28q6f59774di9av7ncy09jd55d-guile-2.0.11
> guile	2.0.11	debug	/gnu/store/rgv3fvy6xqp6966rfh8v6fv7m48abcbh-guile-2.0.11-debug
> mhw@jojen:~$ ls -l .guix-profile/lib/debug/gnu/store/
> total 8
> lrwxrwxrwx 12 root guixbuild 128 Dec 31 1969
> 122jv790mv2mlnylbrbzav65vghbw93n-guile-2.0.11 ->
> /gnu/store/rgv3fvy6xqp6966rfh8v6fv7m48abcbh-guile-2.0.11-debug/lib/debug/gnu/store/122jv790mv2mlnylbrbzav65vghbw93n-guile-2.0.11
> lrwxrwxrwx 15 root guixbuild 127 Dec 31 1969
> 3g20rdmnavpblsmgppyl8jhg67nidhjk-glibc-2.20 ->
> /gnu/store/hrny2whqg9c3m0klyfpbmmcyiir9yf8m-gcc-toolchain-4.9.2/lib/debug/gnu/store/3g20rdmnavpblsmgppyl8jhg67nidhjk-glibc-2.20
>
> I guess GDB can't find the debugging information because
> 122jv790mv2mlnylbrbzav65vghbw93n-guile-2.0.11 is the name of the guile
> directory *before* grafting.
>
> To work around this, I created my own ~/mydebug directory:
>
> mhw@jojen:~/mydebug/gnu/store$ ln -s
> /gnu/store/rgv3fvy6xqp6966rfh8v6fv7m48abcbh-guile-2.0.11-debug/lib/debug/gnu/store/122jv790mv2mlnylbrbzav65vghbw93n-guile-2.0.11
> 3lhr8q28q6f59774di9av7ncy09jd55d-guile-2.0.11
>
> and then pointed GDB at ~/mydebug and then debugging works again.

I've since discovered that because our grafting is not done properly,
for now we need symbolic links for both the pre-grafted and post-grafted
directories.

I installed libgc:debug and then created a proper symlink in ~/mydebug,
but because our guile is still linked against the pre-grafted libgc, we
need a symbolic link with the pre-grafted name.  So here's what I have
now:

mhw@jojen:~/mydebug/gnu/store$ ls -l
total 12
lrwxrwxrwx 1 mhw mhw 128 Mar  1 00:00 3lhr8q28q6f59774di9av7ncy09jd55d-guile-2.0.11 -> /gnu/store/rgv3fvy6xqp6966rfh8v6fv7m48abcbh-guile-2.0.11-debug/lib/debug/gnu/store/122jv790mv2mlnylbrbzav65vghbw93n-guile-2.0.11
lrwxrwxrwx 1 mhw mhw 126 Mar  1 00:18 h6byyxpii73xlbhiq5lxfq3n6qzqwsif-libgc-7.4.2 -> /gnu/store/k0xjv8l196j4nxgbz8dxz6xz7kbysaix-libgc-7.4.2-debug/lib/debug/gnu/store/zshjm1ilbzz0z0zpxr17h13jnmilga7l-libgc-7.4.2
lrwxrwxrwx 1 mhw mhw 126 Mar  1 00:21 zshjm1ilbzz0z0zpxr17h13jnmilga7l-libgc-7.4.2 -> /gnu/store/k0xjv8l196j4nxgbz8dxz6xz7kbysaix-libgc-7.4.2-debug/lib/debug/gnu/store/zshjm1ilbzz0z0zpxr17h13jnmilga7l-libgc-7.4.2

       Mark

bug#19973: Grafts break debug outputs

[Ludovic Courtès]

Mark H Weaver <mhw@netris.org> skribis:

> mhw@jojen:~$ guix build guile
> guix build: warning: ambiguous package specification `guile'
> guix build: warning: choosing guile-2.0.11 from gnu/packages/guile.scm:110:2
> /gnu/store/3lhr8q28q6f59774di9av7ncy09jd55d-guile-2.0.11
> /gnu/store/rgv3fvy6xqp6966rfh8v6fv7m48abcbh-guile-2.0.11-debug
> mhw@jojen:~$ guix package -I guile
> guile	2.0.11	out	/gnu/store/3lhr8q28q6f59774di9av7ncy09jd55d-guile-2.0.11
> guile	2.0.11	debug	/gnu/store/rgv3fvy6xqp6966rfh8v6fv7m48abcbh-guile-2.0.11-debug
> mhw@jojen:~$ ls -l .guix-profile/lib/debug/gnu/store/
> total 8
> lrwxrwxrwx 12 root guixbuild 128 Dec 31  1969 122jv790mv2mlnylbrbzav65vghbw93n-guile-2.0.11 -> /gnu/store/rgv3fvy6xqp6966rfh8v6fv7m48abcbh-guile-2.0.11-debug/lib/debug/gnu/store/122jv790mv2mlnylbrbzav65vghbw93n-guile-2.0.11
> lrwxrwxrwx 15 root guixbuild 127 Dec 31  1969 3g20rdmnavpblsmgppyl8jhg67nidhjk-glibc-2.20 -> /gnu/store/hrny2whqg9c3m0klyfpbmmcyiir9yf8m-gcc-toolchain-4.9.2/lib/debug/gnu/store/3g20rdmnavpblsmgppyl8jhg67nidhjk-glibc-2.20
>
> I guess GDB can't find the debugging information because
> 122jv790mv2mlnylbrbzav65vghbw93n-guile-2.0.11 is the name of the guile
> directory *before* grafting.

I wonder if the problem you described here still exists today.

However, one very likely problem is that .debug files include a CRC of
the binary they correspond to (info "(gdb) Separate Debug Files"), and
that CRC is not updated when we graft things.

We could change our grafting code to specifically address this problem
(using (guix elf) & co.).

Similar issue with build IDs: <https://bugs.gnu.org/25752>.

Ludo’.

bug#19973: Grafts break debug outputs

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> However, one very likely problem is that .debug files include a CRC of
> the binary they correspond to (info "(gdb) Separate Debug Files"), and
> that CRC is not updated when we graft things.

It manifests like this in GDB:

--8<---------------cut here---------------start------------->8---
warning: the debug information found in "/home/ludo/.guix-profile/lib/debug//gnu/store/y8ppqsxiki39n4mqpb4mab6bgwqsnnp7-libgc-7.4.2/lib/libgc.so.debug" does not match "/gnu/store/y8ppqsxiki39n4mqpb4mab6bgwqsnnp7-libgc-7.4.2/lib/libgc.so.1" (CRC mismatch).
--8<---------------cut here---------------end--------------->8---

Ludo’.

bug#19973: Grafts break debug outputs

[Ludovic Courtès]

Hello!

ludo@gnu.org (Ludovic Courtès) skribis:

> However, one very likely problem is that .debug files include a CRC of
> the binary they correspond to (info "(gdb) Separate Debug Files"), and
> that CRC is not updated when we graft things.
>
> We could change our grafting code to specifically address this problem
> (using (guix elf) & co.).

The patches below address this by adding code to update the CRC in the
‘.gnu_debuglink’ section upon grafting.  It uses a simple hook
mechanism, a simplified version of what Timothy proposed.

The ‘.gnu_debuglink’ hook does nothing for packages that lack a “debug”
output.  For packages that have a “debug” output, only ELF files from
bin/, lib/, etc. are considered.  Overall the run-time cost should be
low.

I tested it with guile-ssh on current master (commit
bab4dc58f7278e481c7eb8c6f954d6f1751deb23).  When running:

  guix build guile-ssh

you see messages like this:

  updating '.gnu_debuglink' CRC in '/gnu/store/523lgdnmr4x0q7cnn8wr7f2zzcnlvrpp-guile-ssh-0.11.3/lib/guile/2.2/site-ccache/ssh/tunnel.go'
  updating '.gnu_debuglink' CRC in '/gnu/store/523lgdnmr4x0q7cnn8wr7f2zzcnlvrpp-guile-ssh-0.11.3/lib/guile/2.2/site-ccache/ssh/version.go'
  updating '.gnu_debuglink' CRC in '/gnu/store/523lgdnmr4x0q7cnn8wr7f2zzcnlvrpp-guile-ssh-0.11.3/lib/libguile-ssh.so.11.0.0'

The thing about .go files isn’t very useful (the ‘strip’ phase should
ignore them in the first place probably).  However I’ve confirmed that
gdb is happy with the grafted libguile-ssh.so: if you add breakpoints
with “rb guile_ssh” and then call one of the guile-ssh procedures from
the REPL, you see that you get proper debugging info and all (don’t
forget to “set debug-file-directory” upfront.)

That’s it!  It’s a relief to fix a 3-year old bug.  :-)

Next we can fix build IDs similarly (see <https://bugs.gnu.org/25752>),
and maybe the Racket CRC issue that Timothy and Chris looked at
recently, and maybe the Java manifest issue as well (is it still
relevant?).

Thanks,
Ludo’.

bug#19973: [PATCH 1/2] grafts: Add high-level 'graft' procedure on the build side.

[Ludovic Courtès]

* guix/build/graft.scm (graft): New procedure.
* guix/grafts.scm (graft-derivation/shallow)[build]: Use it instead of
inline code.
---
 guix/build/graft.scm | 21 +++++++++++++++++++--
 guix/grafts.scm      | 13 ++-----------
 2 files changed, 21 insertions(+), 13 deletions(-)

diff --git a/guix/build/graft.scm b/guix/build/graft.scm
index e567bff4f..8d79e8a50 100644
--- a/guix/build/graft.scm
+++ b/guix/build/graft.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2014, 2015, 2016 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2014, 2015, 2016, 2018 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2016 Mark H Weaver <mhw@netris.org>
 ;;;
 ;;; This file is part of GNU Guix.
@@ -27,7 +27,8 @@
   #:use-module (srfi srfi-1)   ; list library
   #:use-module (srfi srfi-26)  ; cut and cute
   #:export (replace-store-references
-            rewrite-directory))
+            rewrite-directory
+            graft))
 
 ;;; Commentary:
 ;;;
@@ -321,4 +322,20 @@ file name pairs."
                               #:directories? #t))
   (rename-matching-files output mapping))
 
+(define* (graft old-outputs new-outputs mapping
+                #:key (log-port (current-output-port)))
+  "Apply the grafts described by MAPPING on OLD-OUTPUTS, leading to
+NEW-OUTPUTS.  MAPPING must be a list of file name pairs; OLD-OUTPUTS and
+NEW-OUTPUTS are lists of output name/file name pairs."
+  (for-each (lambda (input output)
+              (format log-port "grafting '~a' -> '~a'...~%" input output)
+              (force-output)
+              (rewrite-directory input output mapping))
+            (match old-outputs
+              (((names . files) ...)
+               files))
+            (match new-outputs
+              (((names . files) ...)
+               files))))
+
 ;;; graft.scm ends here
diff --git a/guix/grafts.scm b/guix/grafts.scm
index d6b0e93e8..4b10b3efd 100644
--- a/guix/grafts.scm
+++ b/guix/grafts.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2014, 2015, 2016, 2017 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2014, 2015, 2016, 2017, 2018 Ludovic Courtès <ludo@gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -117,16 +117,7 @@ are not recursively applied to dependencies of DRV."
                                        (cons (assoc-ref old-outputs name)
                                              file)))
                                     %outputs))))
-         (for-each (lambda (input output)
-                     (format #t "grafting '~a' -> '~a'...~%" input output)
-                     (force-output)
-                     (rewrite-directory input output mapping))
-                   (match old-outputs
-                     (((names . files) ...)
-                      files))
-                   (match %outputs
-                     (((names . files) ...)
-                      files))))))
+         (graft old-outputs %outputs mapping))))
 
   (define add-label
     (cut cons "x" <>))
-- 
2.18.0

bug#19973: [PATCH 2/2] grafts: Add (guix build debug-link) and use it.

[Ludovic Courtès]

Fixes <https://bugs.gnu.org/19973>.
Reported by Mark H Weaver <mhw@netris.org>.

* guix/build/debug-link.scm: New file.
* guix/build/graft.scm (%graft-hooks): New variable.
(graft): Add #:hooks and honor it.
* guix/grafts.scm (graft-derivation/shallow): Add (guix build
debug-link) and (guix elf) to #:modules.
* tests/debug-link.scm: New file.
* Makefile.am (MODULES): Add guix/build/debug-link.scm.
(SCM_TESTS): Add tests/debug-link.scm.
---
 Makefile.am               |   2 +
 guix/build/debug-link.scm | 210 ++++++++++++++++++++++++++++++++++++++
 guix/build/graft.scm      |  14 ++-
 guix/grafts.scm           |   4 +-
 tests/debug-link.scm      | 154 ++++++++++++++++++++++++++++
 5 files changed, 381 insertions(+), 3 deletions(-)
 create mode 100644 guix/build/debug-link.scm
 create mode 100644 tests/debug-link.scm

diff --git a/Makefile.am b/Makefile.am
index bdf1c63ce..324674a60 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -171,6 +171,7 @@ MODULES =					\
   guix/build/svn.scm				\
   guix/build/syscalls.scm                       \
   guix/build/gremlin.scm			\
+  guix/build/debug-link.scm			\
   guix/build/emacs-utils.scm			\
   guix/build/java-utils.scm			\
   guix/build/lisp-utils.scm			\
@@ -363,6 +364,7 @@ SCM_TESTS =					\
   tests/syscalls.scm				\
   tests/inferior.scm				\
   tests/gremlin.scm				\
+  tests/debug-link.scm				\
   tests/bournish.scm				\
   tests/lint.scm				\
   tests/publish.scm				\
diff --git a/guix/build/debug-link.scm b/guix/build/debug-link.scm
new file mode 100644
index 000000000..9167737fb
--- /dev/null
+++ b/guix/build/debug-link.scm
@@ -0,0 +1,210 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2018 Ludovic Courtès <ludo@gnu.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix build debug-link)
+  #:use-module (guix elf)
+  #:use-module ((guix build utils)
+                #:select (find-files elf-file? make-file-writable))
+  #:use-module (rnrs bytevectors)
+  #:use-module (rnrs io ports)
+  #:use-module (srfi srfi-1)
+  #:use-module (system foreign)
+  #:use-module (ice-9 match)
+  #:export (debuglink-crc32
+            elf-debuglink
+            set-debuglink-crc
+
+            graft-debug-links))
+
+;;; Commentary:
+;;;
+;;; This module provides tools to deal with '.gnu_debuglink' sections in ELF
+;;; files.  These sections are created by 'objcopy --add-gnu-debuglink' to
+;;; create separate debug files (info "(gdb) Separate Debug Files").
+;;;
+;;; The main facility of this module is 'graft-debug-links', which allows us
+;;; to update the CRC that appears in '.gnu_debuglink' sections when grafting,
+;;; such that separate debug files remain usable after grafting.  Failing to
+;;; do that, GDB would complain about CRC mismatch---see
+;;; <https://bugs.gnu.org/19973>.
+;;;
+;;; Code:
+
+(define %crc32-table
+  ;; CRC table taken from "(gdb) Separate Debug Files".
+  ;; TODO: Wouldn't it be nice to generate it "from source" with a macro?
+  #(#x00000000 #x77073096 #xee0e612c #x990951ba #x076dc419
+    #x706af48f #xe963a535 #x9e6495a3 #x0edb8832 #x79dcb8a4
+    #xe0d5e91e #x97d2d988 #x09b64c2b #x7eb17cbd #xe7b82d07
+    #x90bf1d91 #x1db71064 #x6ab020f2 #xf3b97148 #x84be41de
+    #x1adad47d #x6ddde4eb #xf4d4b551 #x83d385c7 #x136c9856
+    #x646ba8c0 #xfd62f97a #x8a65c9ec #x14015c4f #x63066cd9
+    #xfa0f3d63 #x8d080df5 #x3b6e20c8 #x4c69105e #xd56041e4
+    #xa2677172 #x3c03e4d1 #x4b04d447 #xd20d85fd #xa50ab56b
+    #x35b5a8fa #x42b2986c #xdbbbc9d6 #xacbcf940 #x32d86ce3
+    #x45df5c75 #xdcd60dcf #xabd13d59 #x26d930ac #x51de003a
+    #xc8d75180 #xbfd06116 #x21b4f4b5 #x56b3c423 #xcfba9599
+    #xb8bda50f #x2802b89e #x5f058808 #xc60cd9b2 #xb10be924
+    #x2f6f7c87 #x58684c11 #xc1611dab #xb6662d3d #x76dc4190
+    #x01db7106 #x98d220bc #xefd5102a #x71b18589 #x06b6b51f
+    #x9fbfe4a5 #xe8b8d433 #x7807c9a2 #x0f00f934 #x9609a88e
+    #xe10e9818 #x7f6a0dbb #x086d3d2d #x91646c97 #xe6635c01
+    #x6b6b51f4 #x1c6c6162 #x856530d8 #xf262004e #x6c0695ed
+    #x1b01a57b #x8208f4c1 #xf50fc457 #x65b0d9c6 #x12b7e950
+    #x8bbeb8ea #xfcb9887c #x62dd1ddf #x15da2d49 #x8cd37cf3
+    #xfbd44c65 #x4db26158 #x3ab551ce #xa3bc0074 #xd4bb30e2
+    #x4adfa541 #x3dd895d7 #xa4d1c46d #xd3d6f4fb #x4369e96a
+    #x346ed9fc #xad678846 #xda60b8d0 #x44042d73 #x33031de5
+    #xaa0a4c5f #xdd0d7cc9 #x5005713c #x270241aa #xbe0b1010
+    #xc90c2086 #x5768b525 #x206f85b3 #xb966d409 #xce61e49f
+    #x5edef90e #x29d9c998 #xb0d09822 #xc7d7a8b4 #x59b33d17
+    #x2eb40d81 #xb7bd5c3b #xc0ba6cad #xedb88320 #x9abfb3b6
+    #x03b6e20c #x74b1d29a #xead54739 #x9dd277af #x04db2615
+    #x73dc1683 #xe3630b12 #x94643b84 #x0d6d6a3e #x7a6a5aa8
+    #xe40ecf0b #x9309ff9d #x0a00ae27 #x7d079eb1 #xf00f9344
+    #x8708a3d2 #x1e01f268 #x6906c2fe #xf762575d #x806567cb
+    #x196c3671 #x6e6b06e7 #xfed41b76 #x89d32be0 #x10da7a5a
+    #x67dd4acc #xf9b9df6f #x8ebeeff9 #x17b7be43 #x60b08ed5
+    #xd6d6a3e8 #xa1d1937e #x38d8c2c4 #x4fdff252 #xd1bb67f1
+    #xa6bc5767 #x3fb506dd #x48b2364b #xd80d2bda #xaf0a1b4c
+    #x36034af6 #x41047a60 #xdf60efc3 #xa867df55 #x316e8eef
+    #x4669be79 #xcb61b38c #xbc66831a #x256fd2a0 #x5268e236
+    #xcc0c7795 #xbb0b4703 #x220216b9 #x5505262f #xc5ba3bbe
+    #xb2bd0b28 #x2bb45a92 #x5cb36a04 #xc2d7ffa7 #xb5d0cf31
+    #x2cd99e8b #x5bdeae1d #x9b64c2b0 #xec63f226 #x756aa39c
+    #x026d930a #x9c0906a9 #xeb0e363f #x72076785 #x05005713
+    #x95bf4a82 #xe2b87a14 #x7bb12bae #x0cb61b38 #x92d28e9b
+    #xe5d5be0d #x7cdcefb7 #x0bdbdf21 #x86d3d2d4 #xf1d4e242
+    #x68ddb3f8 #x1fda836e #x81be16cd #xf6b9265b #x6fb077e1
+    #x18b74777 #x88085ae6 #xff0f6a70 #x66063bca #x11010b5c
+    #x8f659eff #xf862ae69 #x616bffd3 #x166ccf45 #xa00ae278
+    #xd70dd2ee #x4e048354 #x3903b3c2 #xa7672661 #xd06016f7
+    #x4969474d #x3e6e77db #xaed16a4a #xd9d65adc #x40df0b66
+    #x37d83bf0 #xa9bcae53 #xdebb9ec5 #x47b2cf7f #x30b5ffe9
+    #xbdbdf21c #xcabac28a #x53b39330 #x24b4a3a6 #xbad03605
+    #xcdd70693 #x54de5729 #x23d967bf #xb3667a2e #xc4614ab8
+    #x5d681b02 #x2a6f2b94 #xb40bbe37 #xc30c8ea1 #x5a05df1b
+    #x2d02ef8d))
+
+(define (debuglink-crc32 port)
+  "Compute the 32-bit CRC used in in '.gnu_debuglink' over the data read from
+PORT and return it."                      ;(info "(gdb) Separate Debug Files")
+  (let loop ((crc #xffffffff))
+    (let ((byte (get-u8 port)))
+      (if (eof-object? byte)
+          (logand (lognot crc) #xffffffff)
+          (let* ((index (logand (logxor crc byte) #xff))
+                 (lhs   (vector-ref %crc32-table index)))
+            (loop (logxor lhs (ash crc -8))))))))
+
+(define (section-contents elf section)     ;XXX: copied from linux-modules.scm
+  "Return the contents of SECTION in ELF as a bytevector."
+  (let* ((contents (make-bytevector (elf-section-size section))))
+    (bytevector-copy! (elf-bytes elf) (elf-section-offset section)
+                      contents 0
+                      (elf-section-size section))
+    contents))
+
+(define null-terminated-bytevector->string
+  (compose pointer->string bytevector->pointer))
+
+(define (elf-debuglink elf)
+  "Return two values: the '.gnu_debuglink' file name of ELF and its CRC.
+Return #f for both if ELF lacks a '.gnu_debuglink' section."
+  (let ((section (elf-section-by-name elf ".gnu_debuglink")))
+    (if section
+        (let ((size       (elf-section-size section))
+              (bv         (section-contents elf section))
+              (endianness (elf-byte-order elf)))
+          (values (null-terminated-bytevector->string bv)
+                  (bytevector-u32-ref bv (- size 4) endianness)))
+        (values #f #f))))
+
+(define (elf-debuglink-crc-offset elf)
+  "Return the offset of the '.gnu_debuglink' 32-bit CRC, or #f if ELF lacks a
+'.gnu_debuglink' section."
+  (let ((section (elf-section-by-name elf ".gnu_debuglink")))
+    (and section
+         (+ (elf-section-offset section)
+            (elf-section-size section)
+            -4))))
+
+(define (set-debuglink-crc file debug-file)
+  "Compute the CRC of DEBUG-FILE and set it as the '.gnu_debuglink' CRC in
+FILE."
+  (let* ((elf    (parse-elf (call-with-input-file file get-bytevector-all)))
+         (offset (elf-debuglink-crc-offset elf)))
+    (and offset
+         (let* ((crc (call-with-input-file debug-file debuglink-crc32))
+                (bv  (make-bytevector 4)))
+           (bytevector-u32-set! bv 0 crc (elf-byte-order elf))
+           (let ((port (open file O_RDWR)))
+             (set-port-position! port offset)
+             (put-bytevector port bv)
+             (close-port port))))))
+
+\f
+;;;
+;;; Updating debuglink CRC.
+;;;
+
+(define (find-elf-files outputs)
+  "Return the list of ELF files found in OUTPUTS, a list of top-level store
+directories."
+  (define directories
+    (append-map (lambda (output)
+                  (list (string-append output "/bin")
+                        (string-append output "/sbin")
+                        (string-append output "/lib")
+                        (string-append output "/libexec")))
+                outputs))
+
+  (append-map (lambda (directory)
+                (filter elf-file?
+                        (with-error-to-port (%make-void-port "w")
+                          (lambda ()
+                            (find-files directory)))))
+              directories))
+
+(define* (graft-debug-links old-outputs new-outputs mapping
+                            #:key (log-port (current-error-port)))
+  "Update the '.gnu_debuglink' CRCs found in ELF files of NEW-OUTPUTS,
+provided NEW-OUTPUTS contains a \"debug\" output, such that those CRCs match
+those of the corresponding '.debug' files found in the \"debug\" output.
+
+This procedure is meant to be used as a \"grafting hook\" by (guix build
+graft)."
+  (match (assoc-ref new-outputs "debug")
+    (#f #t)                                       ;nothing to do
+    (debug-directory
+     (let ((files (find-elf-files (filter-map (match-lambda
+                                                (("debug" . _)
+                                                 #f)
+                                                ((name . directory)
+                                                 directory))
+                                              new-outputs))))
+       (for-each (lambda (file)
+                   (let ((debug (string-append debug-directory
+                                               "/lib/debug" file ".debug")))
+                     (when (file-exists? debug)
+                       (format log-port
+                               "updating '.gnu_debuglink' CRC in '~a'~%"
+                               file)
+                       (make-file-writable file)
+                       (set-debuglink-crc file debug))))
+                 files)))))
diff --git a/guix/build/graft.scm b/guix/build/graft.scm
index 8d79e8a50..c119ee71d 100644
--- a/guix/build/graft.scm
+++ b/guix/build/graft.scm
@@ -19,6 +19,7 @@
 
 (define-module (guix build graft)
   #:use-module (guix build utils)
+  #:use-module (guix build debug-link)
   #:use-module (rnrs bytevectors)
   #:use-module (ice-9 vlist)
   #:use-module (ice-9 match)
@@ -322,8 +323,13 @@ file name pairs."
                               #:directories? #t))
   (rename-matching-files output mapping))
 
+(define %graft-hooks
+  ;; Default list of hooks run after grafting.
+  (list graft-debug-links))
+
 (define* (graft old-outputs new-outputs mapping
-                #:key (log-port (current-output-port)))
+                #:key (log-port (current-output-port))
+                (hooks %graft-hooks))
   "Apply the grafts described by MAPPING on OLD-OUTPUTS, leading to
 NEW-OUTPUTS.  MAPPING must be a list of file name pairs; OLD-OUTPUTS and
 NEW-OUTPUTS are lists of output name/file name pairs."
@@ -336,6 +342,10 @@ NEW-OUTPUTS are lists of output name/file name pairs."
                files))
             (match new-outputs
               (((names . files) ...)
-               files))))
+               files)))
+  (for-each (lambda (hook)
+              (hook old-outputs new-outputs mapping
+                    #:log-port log-port))
+            hooks))
 
 ;;; graft.scm ends here
diff --git a/guix/grafts.scm b/guix/grafts.scm
index 4b10b3efd..f303e925f 100644
--- a/guix/grafts.scm
+++ b/guix/grafts.scm
@@ -130,7 +130,9 @@ are not recursively applied to dependencies of DRV."
                                      #:system system
                                      #:guile-for-build guile
                                      #:modules '((guix build graft)
-                                                 (guix build utils))
+                                                 (guix build utils)
+                                                 (guix build debug-link)
+                                                 (guix elf))
                                      #:inputs `(,@(map (lambda (out)
                                                          `("x" ,drv ,out))
                                                        outputs)
diff --git a/tests/debug-link.scm b/tests/debug-link.scm
new file mode 100644
index 000000000..a57f360b2
--- /dev/null
+++ b/tests/debug-link.scm
@@ -0,0 +1,154 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2018 Ludovic Courtès <ludo@gnu.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (test-debug-link)
+  #:use-module (guix elf)
+  #:use-module (guix build utils)
+  #:use-module (guix build debug-link)
+  #:use-module (guix gexp)
+  #:use-module (guix store)
+  #:use-module (guix tests)
+  #:use-module (guix monads)
+  #:use-module (guix derivations)
+  #:use-module (gnu packages bootstrap)
+  #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-26)
+  #:use-module (srfi srfi-64)
+  #:use-module (rnrs io ports)
+  #:use-module (ice-9 match))
+
+(define %guile-executable
+  (match (false-if-exception (readlink "/proc/self/exe"))
+    ((? string? program)
+     (and (file-exists? program) (elf-file? program)
+          program))
+    (_
+     #f)))
+
+(define read-elf
+  (compose parse-elf get-bytevector-all))
+
+(define %store
+  (open-connection-for-tests))
+
+(define-syntax-rule (test-assertm name exp)
+  (test-assert name
+    (run-with-store %store exp
+                    #:guile-for-build (%guile-for-build))))
+
+\f
+(test-begin "debug-link")
+
+(unless %guile-executable (test-skip 1))
+(test-assert "elf-debuglink"
+  (let ((elf (call-with-input-file %guile-executable read-elf)))
+    (match (call-with-values (lambda () (elf-debuglink elf)) list)
+      ((#f #f)                                    ;no '.gnu_debuglink' section
+       (pk 'no-debuglink #t))
+      (((? string? file) (? integer? crc))
+       (string-suffix? ".debug" file)))))
+
+(test-assertm "elf-debuglink"
+  ;; Check whether we can compute the CRC just like objcopy, and whether we
+  ;; can retrieve it.
+  (let* ((code (plain-file "test.c" "int main () { return 42; }"))
+         (exp  (with-imported-modules '((guix build utils)
+                                        (guix build debug-link)
+                                        (guix elf))
+                 #~(begin
+                     (use-modules (guix build utils)
+                                  (guix build debug-link)
+                                  (guix elf)
+                                  (rnrs io ports))
+
+                     (define read-elf
+                       (compose parse-elf get-bytevector-all))
+
+                     (setenv "PATH" (string-join '(#$%bootstrap-gcc
+                                                   #$%bootstrap-binutils)
+                                                 "/bin:" 'suffix))
+                     (invoke "gcc" "-O0" "-g" #$code "-o" "exe")
+                     (copy-file "exe" "exe.debug")
+                     (invoke "strip" "--only-keep-debug" "exe.debug")
+                     (invoke "strip" "--strip-debug" "exe")
+                     (invoke "objcopy" "--add-gnu-debuglink=exe.debug"
+                             "exe")
+                     (call-with-values (lambda ()
+                                         (elf-debuglink
+                                          (call-with-input-file "exe"
+                                            read-elf)))
+                       (lambda (file crc)
+                         (call-with-output-file #$output
+                           (lambda (port)
+                             (let ((expected (call-with-input-file "exe.debug"
+                                               debuglink-crc32)))
+                               (write (list file (= crc expected))
+                                      port))))))))))
+    (mlet* %store-monad ((drv (gexp->derivation "debuglink" exp))
+                         (x   (built-derivations (list drv))))
+      (call-with-input-file (derivation->output-path drv)
+        (lambda (port)
+          (return (match (read port)
+                    (("exe.debug" #t) #t)
+                    (x                (pk 'fail x #f)))))))))
+
+(test-assertm "set-debuglink-crc"
+  ;; Check whether 'set-debuglink-crc' successfully updates the CRC.
+  (let* ((code  (plain-file "test.c" "int main () { return 42; }"))
+         (debug (plain-file "exe.debug" "a"))
+         (exp   (with-imported-modules '((guix build utils)
+                                         (guix build debug-link)
+                                         (guix elf))
+                  #~(begin
+                      (use-modules (guix build utils)
+                                   (guix build debug-link)
+                                   (guix elf)
+                                   (rnrs io ports))
+
+                      (define read-elf
+                        (compose parse-elf get-bytevector-all))
+
+                      (setenv "PATH" (string-join '(#$%bootstrap-gcc
+                                                    #$%bootstrap-binutils)
+                                                  "/bin:" 'suffix))
+                      (invoke "gcc" "-O0" "-g" #$code "-o" "exe")
+                      (copy-file "exe" "exe.debug")
+                      (invoke "strip" "--only-keep-debug" "exe.debug")
+                      (invoke "strip" "--strip-debug" "exe")
+                      (invoke "objcopy" "--add-gnu-debuglink=exe.debug"
+                              "exe")
+                      (set-debuglink-crc "exe" #$debug)
+                      (call-with-values (lambda ()
+                                          (elf-debuglink
+                                           (call-with-input-file "exe"
+                                             read-elf)))
+                        (lambda (file crc)
+                          (call-with-output-file #$output
+                            (lambda (port)
+                              (write (list file crc) port)))))))))
+    (mlet* %store-monad ((drv (gexp->derivation "debuglink" exp))
+                         (x   (built-derivations (list drv))))
+      (call-with-input-file (derivation->output-path drv)
+        (lambda (port)
+          (return (match (read port)
+                    (("exe.debug" crc)
+                     (= crc (debuglink-crc32 (open-input-string "a"))))
+                    (x
+                     (pk 'fail x #f)))))))))
+
+(test-end "debug-link")
-- 
2.18.0

bug#19973: Grafts break debug outputs

[Timothy Sample]

Hi Ludo,

ludo@gnu.org (Ludovic Courtès) writes:

> Hello!
>
> ludo@gnu.org (Ludovic Courtès) skribis:
>
>> However, one very likely problem is that .debug files include a CRC of
>> the binary they correspond to (info "(gdb) Separate Debug Files"), and
>> that CRC is not updated when we graft things.
>>
>> We could change our grafting code to specifically address this problem
>> (using (guix elf) & co.).
>
> The patches below address this by adding code to update the CRC in the
> ‘.gnu_debuglink’ section upon grafting.  It uses a simple hook
> mechanism, a simplified version of what Timothy proposed.
>
> The ‘.gnu_debuglink’ hook does nothing for packages that lack a “debug”
> output.  For packages that have a “debug” output, only ELF files from
> bin/, lib/, etc. are considered.  Overall the run-time cost should be
> low.
>
> I tested it with guile-ssh on current master (commit
> bab4dc58f7278e481c7eb8c6f954d6f1751deb23).  When running:
>
>   guix build guile-ssh
>
> you see messages like this:
>
>   updating '.gnu_debuglink' CRC in
> '/gnu/store/523lgdnmr4x0q7cnn8wr7f2zzcnlvrpp-guile-ssh-0.11.3/lib/guile/2.2/site-ccache/ssh/tunnel.go'
>   updating '.gnu_debuglink' CRC in
> '/gnu/store/523lgdnmr4x0q7cnn8wr7f2zzcnlvrpp-guile-ssh-0.11.3/lib/guile/2.2/site-ccache/ssh/version.go'
>   updating '.gnu_debuglink' CRC in
> '/gnu/store/523lgdnmr4x0q7cnn8wr7f2zzcnlvrpp-guile-ssh-0.11.3/lib/libguile-ssh.so.11.0.0'
>
> The thing about .go files isn’t very useful (the ‘strip’ phase should
> ignore them in the first place probably).  However I’ve confirmed that
> gdb is happy with the grafted libguile-ssh.so: if you add breakpoints
> with “rb guile_ssh” and then call one of the guile-ssh procedures from
> the REPL, you see that you get proper debugging info and all (don’t
> forget to “set debug-file-directory” upfront.)
>
> That’s it!  It’s a relief to fix a 3-year old bug.  :-)
>
> Next we can fix build IDs similarly (see <https://bugs.gnu.org/25752>),
> and maybe the Racket CRC issue that Timothy and Chris looked at
> recently, and maybe the Java manifest issue as well (is it still
> relevant?).

The only concern I have is the level at which the hooks operate.  In my
draft patch¹ I had the hooks running both on the client side and the
build side.  This made it possible to get a bit more information about
the derivation being grafted.  If everything happens at the build level
based on outputs, we will only be able to look at the structures and
names of the outputs.

That being said, this is probably okay.  The Racket hook will just have
to check for “share/racket” to determine if it runs (and fail safely if
anything is amiss).

> Thanks,
> Ludo’.


-- Tim

¹ https://lists.gnu.org/archive/html/guix-devel/2018-08/msg00113.html

bug#19973: Grafts break debug outputs

[Ludovic Courtès]

Hello Timothy,

Timothy Sample <samplet@ngyro.com> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:

[...]

>> Next we can fix build IDs similarly (see <https://bugs.gnu.org/25752>),
>> and maybe the Racket CRC issue that Timothy and Chris looked at
>> recently, and maybe the Java manifest issue as well (is it still
>> relevant?).
>
> The only concern I have is the level at which the hooks operate.  In my
> draft patch¹ I had the hooks running both on the client side and the
> build side.  This made it possible to get a bit more information about
> the derivation being grafted.  If everything happens at the build level
> based on outputs, we will only be able to look at the structures and
> names of the outputs.

Yes, I agree that your proposal had the appeal of being possibly more
extensible that what I posted here.  However, as I wrote there, there
are hooks that we’ll always want to run, independently of the input
packages, such as the .gnu_debuglink and build-ID hooks; also, it costs
nothing to have them unconditionally, we only pay for their
functionality when candidate files exist.

> That being said, this is probably okay.  The Racket hook will just have
> to check for “share/racket” to determine if it runs (and fail safely if
> anything is amiss).

Yes, that should work well enough.

Thanks for your feedback!

Ludo’.

bug#19973: Grafts break debug outputs

[Ludovic Courtès]

Hello,

ludo@gnu.org (Ludovic Courtès) skribis:

> The patches below address this by adding code to update the CRC in the
> ‘.gnu_debuglink’ section upon grafting.  It uses a simple hook
> mechanism, a simplified version of what Timothy proposed.
>
> The ‘.gnu_debuglink’ hook does nothing for packages that lack a “debug”
> output.  For packages that have a “debug” output, only ELF files from
> bin/, lib/, etc. are considered.  Overall the run-time cost should be
> low.

I went ahead and pushed these two patches as
93c333895a4e2dc9baabec8ade60d9d2ac0a91e2 and
e4297aa8b95cefa32e2595ce58886fc03b0561f7.

I modified tests/debug-link.scm compared to what I posted such that it
would skip tests that require %bootstrap-gcc when networking is lacking.

Thanks for your feedback,
Ludo’.