Using GDB to debug Guix-installed software (e.g. virsh)

Using GDB to debug Guix-installed software (e.g. virsh)

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 10830 bytes --]

Hi,

I sometimes want to debug Guix-installed software using GDB.
Unfortunately, I've only been successful with trivial programs like GNU
Hello.  All of my attempts to debug actual problems have failed because
I can't seem to get GDB to behave.  It's a bit frustrating to bang my
head on stuff like this by myself, so I'm hoping somebody with more
experience can offer some advice.

Let's start with what might be a real bug.  I've noticed that Guix's
virsh command (from the libvirt package) emits a suspicious error when
you try to list devices:

--8<---------------cut here---------------start------------->8---
$ virsh nodedev-list
error: Failed to count node devices
error: this function is not supported by the connection driver: virNodeNumOfDevices
--8<---------------cut here---------------end--------------->8---

Apparently, because this function is "not supported", it is also not
possible to use virt-manager to assign PCI devices to a libvirt domain.
That's what I was trying to do when I stumbled across this issue.

Anyway, this virsh problem occurs even when I invoke the command as
root, so it probably isn't a permissions issue.  I searched the Internet
for errors like this, but I didn't find anything helpful.  Every guide
I've read so far seems to suggest that this invocation should just work.
But it doesn't.  Why?

If you want to try reproducing this issue on GuixSD, make sure you have
a libvirt-service-type service and a virtlog-service-type service in
your operating system configuration declaration:

--8<---------------cut here---------------start------------->8---
(service libvirt-service-type
         (libvirt-configuration
          (unix-sock-group "libvirt")))
(service virtlog-service-type)
--8<---------------cut here---------------end--------------->8---

For good measure, make sure your user is in the "libvirt" group, too:

--8<---------------cut here---------------start------------->8---
(user-account
  (name "marusich")
  (comment "Chris Marusich")
  (group "users")
  (supplementary-groups '("wheel"
                          "netdev"
                          "video"
                          "libvirt"))
  (home-directory "/home/marusich"))
--8<---------------cut here---------------end--------------->8---

Reconfigure and restart if necessary.  Then run virsh:

--8<---------------cut here---------------start------------->8---
$ virsh nodedev-list
error: Failed to count node devices
error: this function is not supported by the connection driver: virNodeNumOfDevices
--8<---------------cut here---------------end--------------->8---

At this point, there are two possibilities: either everything is fine,
and this error is expected, or something is wrong.  If somebody knows
that this is expected, I'd love to hear about it.  However, let's
operate on the assumption that something is wrong.  How might we debug
it?

One way to debug it is to use GDB to investigate precisely why this
failure occurred.  There are probably other ways to debug the issue, but
I want to focus on using GDB because this email is more about the
problems I've had with GDB than the virsh issue.

To begin, I create a directory where I'll do my debugging:

--8<---------------cut here---------------start------------->8---
$ mkdir ~/debug
$ cd ~/debug
--8<---------------cut here---------------end--------------->8---

Let's get the virsh source so we can get GDB to tell us where we are in
the code as we debug it:

--8<---------------cut here---------------start------------->8---
$ tar -xf $(guix build -S libvirt)
--8<---------------cut here---------------end--------------->8---

For me, this unpacks the source to:

    /home/marusich/debug/libvirt-4.3.0

Note that the function virNodeNumOfDevices is defined in

    /home/marusich/debug/libvirt-4.3.0/libvirt-4.3.0/src/libvirt-nodedev.c

and called on line 254 of

    /home/marusich/debug/libvirt-4.3.0/tools/virsh-nodedev.c

in the virshNodeDeviceListCollect function.

I'd like to debug the code for virNodeNumOfDevices using GDB to see
what's going on.  To do this, I'm going to need the debug symbols, but
the libvirt package doesn't have a debug output.  Let's define a version
of it that does.  I put the following package definition into the file
/home/marusich/debug/my-libvirt.scm:

--8<---------------cut here---------------start------------->8---
(define-module (my-libvirt)
  #:use-module (guix packages)
  #:use-module (gnu packages virtualization))

(define-public my-libvirt
  (package
   (inherit libvirt)
   (name "my-libvirt")
   (outputs '("out" "debug"))))
--8<---------------cut here---------------end--------------->8---

Let's build it and install both outputs into a new profile:

--8<---------------cut here---------------start------------->8---
$ GUIX_PACKAGE_PATH=/home/marusich/debug guix package -p /home/marusich/debug/profile -i my-libvirt my-libvirt:debug
--8<---------------cut here---------------end--------------->8---

Let's make sure the new virsh still reports the same error:

--8<---------------cut here---------------start------------->8---
$ /home/marusich/debug/profile/bin/virsh nodedev-list
error: Failed to count node devices
error: this function is not supported by the connection driver: virNodeNumOfDevices
--8<---------------cut here---------------end--------------->8---

Great!  Let's debug it with GDB.  First, make sure your ~/.gdbinit
doesn't exist, otherwise your results might be different from mine.
Then let's start GDB:

--8<---------------cut here---------------start------------->8---
$ gdb
--8<---------------cut here---------------end--------------->8---

Tell it where the debug files live:

--8<---------------cut here---------------start------------->8---
(gdb) set debug-file-directory /home/marusich/debug/profile/lib/debug
--8<---------------cut here---------------end--------------->8---

Tell it where the source lives:

--8<---------------cut here---------------start------------->8---
(gdb) directory /home/marusich/debug/libvirt-4.3.0/src
Source directories searched: /home/marusich/debug/libvirt-4.3.0/src:$cdir:$cwd
(gdb) directory /home/marusich/debug/libvirt-4.3.0/tools
Source directories searched: /home/marusich/debug/libvirt-4.3.0/tools:/home/marusich/debug/libvirt-4.3.0/src:$cdir:$cwd
--8<---------------cut here---------------end--------------->8---

Tell it to use the file and read the symbols:
--8<---------------cut here---------------start------------->8---
(gdb) file /home/marusich/debug/profile/bin/virsh
Reading symbols from /home/marusich/debug/profile/bin/virsh...Reading symbols from /home/marusich/debug/profile/lib/debug//gnu/store/mx3rmbpg6lhl0yxl9djbx49nfps9lwqi-my-libvirt-4.3.0/bin/virsh.debug...done.
done.
--8<---------------cut here---------------end--------------->8---

Set the program's arguments:

--8<---------------cut here---------------start------------->8---
(gdb) set args nodedev-list
--8<---------------cut here---------------end--------------->8---

Set a breakpoint on the function virNodeNumOfDevices:

--8<---------------cut here---------------start------------->8---
(gdb) break virNodeNumOfDevices
Breakpoint 1 at 0x28610
--8<---------------cut here---------------end--------------->8---

Uh oh.  This is our first sign of a problem: The breakpoint is
associated with some sort of memory address, rather than a location in a
file.  Anyway, let's run the program:

--8<---------------cut here---------------start------------->8---
(gdb) run
Starting program: /gnu/store/mx3rmbpg6lhl0yxl9djbx49nfps9lwqi-my-libvirt-4.3.0/bin/virsh nodedev-list
warning: the debug information found in "/home/marusich/debug/profile/lib/debug//gnu/store/mx3rmbpg6lhl0yxl9djbx49nfps9lwqi-my-libvirt-4.3.0/lib/libvirt.so.0.4003.0.debug" does not match "/gnu/store/mx3rmbpg6lhl0yxl9djbx49nfps9lwqi-my-libvirt-4.3.0/lib/libvirt.so.0" (CRC mismatch).

warning: the debug information found in "/home/marusich/debug/profile/lib/debug//gnu/store/mx3rmbpg6lhl0yxl9djbx49nfps9lwqi-my-libvirt-4.3.0/lib/libvirt.so.0.4003.0.debug" does not match "/gnu/store/mx3rmbpg6lhl0yxl9djbx49nfps9lwqi-my-libvirt-4.3.0/lib/libvirt.so.0" (CRC mismatch).

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/gnu/store/l4lr0f5cjd0nbsaaf8b5dmcw1a1yypr3-glibc-2.27/lib/libthread_db.so.1".
[New Thread 0x7ffff2219700 (LWP 16097)]

Thread 1 "virsh" hit Breakpoint 1, 0x00007ffff768cdc0 in virNodeNumOfDevices ()
   from /gnu/store/mx3rmbpg6lhl0yxl9djbx49nfps9lwqi-my-libvirt-4.3.0/lib/libvirt.so.0
--8<---------------cut here---------------end--------------->8---

We hit the breakpoint - great!  However, it seems GDB did not load the
debug information for libvirt because of a CRC mismatch.  Indeed, the
backtrace seems to suggest that GDB knows about some of the source
files, but not all of them:

--8<---------------cut here---------------start------------->8---
(gdb) bt
#0  0x00007ffff768cdc0 in virNodeNumOfDevicesw ()
   from /gnu/store/mx3rmbpg6lhl0yxl9djbx49nfps9lwqi-my-libvirt-4.3.0/lib/libvirt.so.0
#1  0x00005555555a816e in virshNodeDeviceListCollect (flags=0, 
    ncapnames=<optimized out>, capnames=0x0, ctl=0x7fffffffb460)
    at virsh-nodedev.c:254
#2  cmdNodeListDevices (ctl=0x7fffffffb460, cmd=<optimized out>)
    at virsh-nodedev.c:472
#3  0x00005555555b8911 in vshCommandRun (ctl=0x7fffffffb460, 
    cmd=0x55555583d850) at vsh.c:1318
#4  0x000055555557ea65 in main (argc=2, argv=0x7fffffffb7f8) at virsh.c:932
--8<---------------cut here---------------end--------------->8---

I wanted to see what was happening in the virNodeNumOfDevices function,
which came from libvirt.so.0.  Unfortunately, that's the library with
the CRC mismatch.  This means I'm totally blocked from investigating any
further using GDB.  I could set step-mode to "on" to step through the
machine code without debug symbols, but as they say: "That is an
exercise left to the reader."

I have seen this CRC mismatch problem twice now when trying to debug
issues with Guix-installed software.  The other time was while
attempting to debug a segfault in vinagre:

    https://debbugs.gnu.org/cgi/bugreport.cgi?bug=30591

What is wrong?  Am I using GDB wrong?  Is there a bug in the part of the
gnu-build-system that creates the debug files which might be causing the
CRC mismatch?  I'm aware of the fact that the gnu-build-system takes
advantage of the .gnu-debuglink stuff ((gdb) Separate Debug Files), but
to be honest I haven't done a lot of GDB debugging, so part of me
wonders if this is just a case of "user error".  If so, please help me
understand what I'm doing wrong.

Thank you,

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Using GDB to debug Guix-installed software (e.g. virsh)

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 11677 bytes --]

Chris Marusich <cmmarusich@gmail.com> ezt írta (időpont: 2018. júl. 17., K,
8:36):

> Hi,
>
> I sometimes want to debug Guix-installed software using GDB.
> Unfortunately, I've only been successful with trivial programs like GNU
> Hello.  All of my attempts to debug actual problems have failed because
> I can't seem to get GDB to behave.  It's a bit frustrating to bang my
> head on stuff like this by myself, so I'm hoping somebody with more
> experience can offer some advice.
>
> Let's start with what might be a real bug.  I've noticed that Guix's
> virsh command (from the libvirt package) emits a suspicious error when
> you try to list devices:
>
> --8<---------------cut here---------------start------------->8---
> $ virsh nodedev-list
> error: Failed to count node devices
> error: this function is not supported by the connection driver:
> virNodeNumOfDevices
> --8<---------------cut here---------------end--------------->8---
>
> Apparently, because this function is "not supported", it is also not
> possible to use virt-manager to assign PCI devices to a libvirt domain.
> That's what I was trying to do when I stumbled across this issue.
>
> Anyway, this virsh problem occurs even when I invoke the command as
> root, so it probably isn't a permissions issue.  I searched the Internet
> for errors like this, but I didn't find anything helpful.  Every guide
> I've read so far seems to suggest that this invocation should just work.
> But it doesn't.  Why?
>
> If you want to try reproducing this issue on GuixSD, make sure you have
> a libvirt-service-type service and a virtlog-service-type service in
> your operating system configuration declaration:
>
> --8<---------------cut here---------------start------------->8---
> (service libvirt-service-type
>          (libvirt-configuration
>           (unix-sock-group "libvirt")))
> (service virtlog-service-type)
> --8<---------------cut here---------------end--------------->8---
>
> For good measure, make sure your user is in the "libvirt" group, too:
>
> --8<---------------cut here---------------start------------->8---
> (user-account
>   (name "marusich")
>   (comment "Chris Marusich")
>   (group "users")
>   (supplementary-groups '("wheel"
>                           "netdev"
>                           "video"
>                           "libvirt"))
>   (home-directory "/home/marusich"))
> --8<---------------cut here---------------end--------------->8---
>
> Reconfigure and restart if necessary.  Then run virsh:
>
> --8<---------------cut here---------------start------------->8---
> $ virsh nodedev-list
> error: Failed to count node devices
> error: this function is not supported by the connection driver:
> virNodeNumOfDevices
> --8<---------------cut here---------------end--------------->8---
>
> At this point, there are two possibilities: either everything is fine,
> and this error is expected, or something is wrong.  If somebody knows
> that this is expected, I'd love to hear about it.  However, let's
> operate on the assumption that something is wrong.  How might we debug
> it?
>
> One way to debug it is to use GDB to investigate precisely why this
> failure occurred.  There are probably other ways to debug the issue, but
> I want to focus on using GDB because this email is more about the
> problems I've had with GDB than the virsh issue.
>
> To begin, I create a directory where I'll do my debugging:
>
> --8<---------------cut here---------------start------------->8---
> $ mkdir ~/debug
> $ cd ~/debug
> --8<---------------cut here---------------end--------------->8---
>
> Let's get the virsh source so we can get GDB to tell us where we are in
> the code as we debug it:
>
> --8<---------------cut here---------------start------------->8---
> $ tar -xf $(guix build -S libvirt)
> --8<---------------cut here---------------end--------------->8---
>
> For me, this unpacks the source to:
>
>     /home/marusich/debug/libvirt-4.3.0
>
> Note that the function virNodeNumOfDevices is defined in
>
>     /home/marusich/debug/libvirt-4.3.0/libvirt-4.3.0/src/libvirt-nodedev.c
>
> and called on line 254 of
>
>     /home/marusich/debug/libvirt-4.3.0/tools/virsh-nodedev.c
>
> in the virshNodeDeviceListCollect function.
>
> I'd like to debug the code for virNodeNumOfDevices using GDB to see
> what's going on.  To do this, I'm going to need the debug symbols, but
> the libvirt package doesn't have a debug output.  Let's define a version
> of it that does.  I put the following package definition into the file
> /home/marusich/debug/my-libvirt.scm:
>
> --8<---------------cut here---------------start------------->8---
> (define-module (my-libvirt)
>   #:use-module (guix packages)
>   #:use-module (gnu packages virtualization))
>
> (define-public my-libvirt
>   (package
>    (inherit libvirt)
>    (name "my-libvirt")
>    (outputs '("out" "debug"))))
> --8<---------------cut here---------------end--------------->8---
>
> Let's build it and install both outputs into a new profile:
>
> --8<---------------cut here---------------start------------->8---
> $ GUIX_PACKAGE_PATH=/home/marusich/debug guix package -p
> /home/marusich/debug/profile -i my-libvirt my-libvirt:debug
> --8<---------------cut here---------------end--------------->8---
>
> Let's make sure the new virsh still reports the same error:
>
> --8<---------------cut here---------------start------------->8---
> $ /home/marusich/debug/profile/bin/virsh nodedev-list
> error: Failed to count node devices
> error: this function is not supported by the connection driver:
> virNodeNumOfDevices
> --8<---------------cut here---------------end--------------->8---
>
> Great!  Let's debug it with GDB.  First, make sure your ~/.gdbinit
> doesn't exist, otherwise your results might be different from mine.
> Then let's start GDB:
>
> --8<---------------cut here---------------start------------->8---
> $ gdb
> --8<---------------cut here---------------end--------------->8---
>
> Tell it where the debug files live:
>
> --8<---------------cut here---------------start------------->8---
> (gdb) set debug-file-directory /home/marusich/debug/profile/lib/debug
> --8<---------------cut here---------------end--------------->8---
>
> Tell it where the source lives:
>
> --8<---------------cut here---------------start------------->8---
> (gdb) directory /home/marusich/debug/libvirt-4.3.0/src
> Source directories searched:
> /home/marusich/debug/libvirt-4.3.0/src:$cdir:$cwd
> (gdb) directory /home/marusich/debug/libvirt-4.3.0/tools
> Source directories searched:
> /home/marusich/debug/libvirt-4.3.0/tools:/home/marusich/debug/libvirt-4.3.0/src:$cdir:$cwd
> --8<---------------cut here---------------end--------------->8---
>
> Tell it to use the file and read the symbols:
> --8<---------------cut here---------------start------------->8---
> (gdb) file /home/marusich/debug/profile/bin/virsh
> Reading symbols from /home/marusich/debug/profile/bin/virsh...Reading
> symbols from
> /home/marusich/debug/profile/lib/debug//gnu/store/mx3rmbpg6lhl0yxl9djbx49nfps9lwqi-my-libvirt-4.3.0/bin/virsh.debug...done.
> done.
> --8<---------------cut here---------------end--------------->8---
>
> Set the program's arguments:
>
> --8<---------------cut here---------------start------------->8---
> (gdb) set args nodedev-list
> --8<---------------cut here---------------end--------------->8---
>
> Set a breakpoint on the function virNodeNumOfDevices:
>
> --8<---------------cut here---------------start------------->8---
> (gdb) break virNodeNumOfDevices
> Breakpoint 1 at 0x28610
> --8<---------------cut here---------------end--------------->8---
>
> Uh oh.  This is our first sign of a problem: The breakpoint is
> associated with some sort of memory address, rather than a location in a
> file.  Anyway, let's run the program:
>
> --8<---------------cut here---------------start------------->8---
> (gdb) run
> Starting program:
> /gnu/store/mx3rmbpg6lhl0yxl9djbx49nfps9lwqi-my-libvirt-4.3.0/bin/virsh
> nodedev-list
> warning: the debug information found in
> "/home/marusich/debug/profile/lib/debug//gnu/store/mx3rmbpg6lhl0yxl9djbx49nfps9lwqi-my-libvirt-4.3.0/lib/libvirt.so.0.4003.0.debug"
> does not match
> "/gnu/store/mx3rmbpg6lhl0yxl9djbx49nfps9lwqi-my-libvirt-4.3.0/lib/libvirt.so.0"
> (CRC mismatch).
>
> warning: the debug information found in
> "/home/marusich/debug/profile/lib/debug//gnu/store/mx3rmbpg6lhl0yxl9djbx49nfps9lwqi-my-libvirt-4.3.0/lib/libvirt.so.0.4003.0.debug"
> does not match
> "/gnu/store/mx3rmbpg6lhl0yxl9djbx49nfps9lwqi-my-libvirt-4.3.0/lib/libvirt.so.0"
> (CRC mismatch).
>
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library
> "/gnu/store/l4lr0f5cjd0nbsaaf8b5dmcw1a1yypr3-glibc-2.27/lib/libthread_db.so.1".
> [New Thread 0x7ffff2219700 (LWP 16097)]
>
> Thread 1 "virsh" hit Breakpoint 1, 0x00007ffff768cdc0 in
> virNodeNumOfDevices ()
>    from
> /gnu/store/mx3rmbpg6lhl0yxl9djbx49nfps9lwqi-my-libvirt-4.3.0/lib/libvirt.so.0
> --8<---------------cut here---------------end--------------->8---
>
> We hit the breakpoint - great!  However, it seems GDB did not load the
> debug information for libvirt because of a CRC mismatch.  Indeed, the
> backtrace seems to suggest that GDB knows about some of the source
> files, but not all of them:
>
> --8<---------------cut here---------------start------------->8---
> (gdb) bt
> #0  0x00007ffff768cdc0 in virNodeNumOfDevicesw ()
>    from
> /gnu/store/mx3rmbpg6lhl0yxl9djbx49nfps9lwqi-my-libvirt-4.3.0/lib/libvirt.so.0
> #1  0x00005555555a816e in virshNodeDeviceListCollect (flags=0,
>     ncapnames=<optimized out>, capnames=0x0, ctl=0x7fffffffb460)
>     at virsh-nodedev.c:254
> #2  cmdNodeListDevices (ctl=0x7fffffffb460, cmd=<optimized out>)
>     at virsh-nodedev.c:472
> #3  0x00005555555b8911 in vshCommandRun (ctl=0x7fffffffb460,
>     cmd=0x55555583d850) at vsh.c:1318
> #4  0x000055555557ea65 in main (argc=2, argv=0x7fffffffb7f8) at virsh.c:932
> --8<---------------cut here---------------end--------------->8---
>
> I wanted to see what was happening in the virNodeNumOfDevices function,
> which came from libvirt.so.0.  Unfortunately, that's the library with
> the CRC mismatch.  This means I'm totally blocked from investigating any
> further using GDB.  I could set step-mode to "on" to step through the
> machine code without debug symbols, but as they say: "That is an
> exercise left to the reader."
>
> I have seen this CRC mismatch problem twice now when trying to debug
> issues with Guix-installed software.  The other time was while
> attempting to debug a segfault in vinagre:
>
>     https://debbugs.gnu.org/cgi/bugreport.cgi?bug=30591
>
> What is wrong?  Am I using GDB wrong?  Is there a bug in the part of the
> gnu-build-system that creates the debug files which might be causing the
> CRC mismatch?  I'm aware of the fact that the gnu-build-system takes
> advantage of the .gnu-debuglink stuff ((gdb) Separate Debug Files), but
> to be honest I haven't done a lot of GDB debugging, so part of me
> wonders if this is just a case of "user error".  If so, please help me
> understand what I'm doing wrong.
>
>
Actually this is about the same thing I've found out. I'm also suffering
CRC mismatches. The workaround I used was defining a package where I didn't
strip the debugging symbols in the first place. I don't know what this is
about either, but it is annoying.


> Thank you,
>
> --
> Chris
>

[-- Attachment #2: Type: text/html, Size: 13344 bytes --]

Re: Using GDB to debug Guix-installed software (e.g. virsh)

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 12169 bytes --]

Gábor Boskovits <boskovits@gmail.com> ezt írta (időpont: 2018. júl. 17., K,
17:18):

> Chris Marusich <cmmarusich@gmail.com> ezt írta (időpont: 2018. júl. 17.,
> K, 8:36):
>
>> Hi,
>>
>> I sometimes want to debug Guix-installed software using GDB.
>> Unfortunately, I've only been successful with trivial programs like GNU
>> Hello.  All of my attempts to debug actual problems have failed because
>> I can't seem to get GDB to behave.  It's a bit frustrating to bang my
>> head on stuff like this by myself, so I'm hoping somebody with more
>> experience can offer some advice.
>>
>> Let's start with what might be a real bug.  I've noticed that Guix's
>> virsh command (from the libvirt package) emits a suspicious error when
>> you try to list devices:
>>
>> --8<---------------cut here---------------start------------->8---
>> $ virsh nodedev-list
>> error: Failed to count node devices
>> error: this function is not supported by the connection driver:
>> virNodeNumOfDevices
>> --8<---------------cut here---------------end--------------->8---
>>
>> Apparently, because this function is "not supported", it is also not
>> possible to use virt-manager to assign PCI devices to a libvirt domain.
>> That's what I was trying to do when I stumbled across this issue.
>>
>> Anyway, this virsh problem occurs even when I invoke the command as
>> root, so it probably isn't a permissions issue.  I searched the Internet
>> for errors like this, but I didn't find anything helpful.  Every guide
>> I've read so far seems to suggest that this invocation should just work.
>> But it doesn't.  Why?
>>
>> If you want to try reproducing this issue on GuixSD, make sure you have
>> a libvirt-service-type service and a virtlog-service-type service in
>> your operating system configuration declaration:
>>
>> --8<---------------cut here---------------start------------->8---
>> (service libvirt-service-type
>>          (libvirt-configuration
>>           (unix-sock-group "libvirt")))
>> (service virtlog-service-type)
>> --8<---------------cut here---------------end--------------->8---
>>
>> For good measure, make sure your user is in the "libvirt" group, too:
>>
>> --8<---------------cut here---------------start------------->8---
>> (user-account
>>   (name "marusich")
>>   (comment "Chris Marusich")
>>   (group "users")
>>   (supplementary-groups '("wheel"
>>                           "netdev"
>>                           "video"
>>                           "libvirt"))
>>   (home-directory "/home/marusich"))
>> --8<---------------cut here---------------end--------------->8---
>>
>> Reconfigure and restart if necessary.  Then run virsh:
>>
>> --8<---------------cut here---------------start------------->8---
>> $ virsh nodedev-list
>> error: Failed to count node devices
>> error: this function is not supported by the connection driver:
>> virNodeNumOfDevices
>> --8<---------------cut here---------------end--------------->8---
>>
>> At this point, there are two possibilities: either everything is fine,
>> and this error is expected, or something is wrong.  If somebody knows
>> that this is expected, I'd love to hear about it.  However, let's
>> operate on the assumption that something is wrong.  How might we debug
>> it?
>>
>> One way to debug it is to use GDB to investigate precisely why this
>> failure occurred.  There are probably other ways to debug the issue, but
>> I want to focus on using GDB because this email is more about the
>> problems I've had with GDB than the virsh issue.
>>
>> To begin, I create a directory where I'll do my debugging:
>>
>> --8<---------------cut here---------------start------------->8---
>> $ mkdir ~/debug
>> $ cd ~/debug
>> --8<---------------cut here---------------end--------------->8---
>>
>> Let's get the virsh source so we can get GDB to tell us where we are in
>> the code as we debug it:
>>
>> --8<---------------cut here---------------start------------->8---
>> $ tar -xf $(guix build -S libvirt)
>> --8<---------------cut here---------------end--------------->8---
>>
>> For me, this unpacks the source to:
>>
>>     /home/marusich/debug/libvirt-4.3.0
>>
>> Note that the function virNodeNumOfDevices is defined in
>>
>>     /home/marusich/debug/libvirt-4.3.0/libvirt-4.3.0/src/libvirt-nodedev.c
>>
>> and called on line 254 of
>>
>>     /home/marusich/debug/libvirt-4.3.0/tools/virsh-nodedev.c
>>
>> in the virshNodeDeviceListCollect function.
>>
>> I'd like to debug the code for virNodeNumOfDevices using GDB to see
>> what's going on.  To do this, I'm going to need the debug symbols, but
>> the libvirt package doesn't have a debug output.  Let's define a version
>> of it that does.  I put the following package definition into the file
>> /home/marusich/debug/my-libvirt.scm:
>>
>> --8<---------------cut here---------------start------------->8---
>> (define-module (my-libvirt)
>>   #:use-module (guix packages)
>>   #:use-module (gnu packages virtualization))
>>
>> (define-public my-libvirt
>>   (package
>>    (inherit libvirt)
>>    (name "my-libvirt")
>>    (outputs '("out" "debug"))))
>> --8<---------------cut here---------------end--------------->8---
>>
>> Let's build it and install both outputs into a new profile:
>>
>> --8<---------------cut here---------------start------------->8---
>> $ GUIX_PACKAGE_PATH=/home/marusich/debug guix package -p
>> /home/marusich/debug/profile -i my-libvirt my-libvirt:debug
>> --8<---------------cut here---------------end--------------->8---
>>
>> Let's make sure the new virsh still reports the same error:
>>
>> --8<---------------cut here---------------start------------->8---
>> $ /home/marusich/debug/profile/bin/virsh nodedev-list
>> error: Failed to count node devices
>> error: this function is not supported by the connection driver:
>> virNodeNumOfDevices
>> --8<---------------cut here---------------end--------------->8---
>>
>> Great!  Let's debug it with GDB.  First, make sure your ~/.gdbinit
>> doesn't exist, otherwise your results might be different from mine.
>> Then let's start GDB:
>>
>> --8<---------------cut here---------------start------------->8---
>> $ gdb
>> --8<---------------cut here---------------end--------------->8---
>>
>> Tell it where the debug files live:
>>
>> --8<---------------cut here---------------start------------->8---
>> (gdb) set debug-file-directory /home/marusich/debug/profile/lib/debug
>> --8<---------------cut here---------------end--------------->8---
>>
>> Tell it where the source lives:
>>
>> --8<---------------cut here---------------start------------->8---
>> (gdb) directory /home/marusich/debug/libvirt-4.3.0/src
>> Source directories searched:
>> /home/marusich/debug/libvirt-4.3.0/src:$cdir:$cwd
>> (gdb) directory /home/marusich/debug/libvirt-4.3.0/tools
>> Source directories searched:
>> /home/marusich/debug/libvirt-4.3.0/tools:/home/marusich/debug/libvirt-4.3.0/src:$cdir:$cwd
>> --8<---------------cut here---------------end--------------->8---
>>
>> Tell it to use the file and read the symbols:
>> --8<---------------cut here---------------start------------->8---
>> (gdb) file /home/marusich/debug/profile/bin/virsh
>> Reading symbols from /home/marusich/debug/profile/bin/virsh...Reading
>> symbols from
>> /home/marusich/debug/profile/lib/debug//gnu/store/mx3rmbpg6lhl0yxl9djbx49nfps9lwqi-my-libvirt-4.3.0/bin/virsh.debug...done.
>> done.
>> --8<---------------cut here---------------end--------------->8---
>>
>> Set the program's arguments:
>>
>> --8<---------------cut here---------------start------------->8---
>> (gdb) set args nodedev-list
>> --8<---------------cut here---------------end--------------->8---
>>
>> Set a breakpoint on the function virNodeNumOfDevices:
>>
>> --8<---------------cut here---------------start------------->8---
>> (gdb) break virNodeNumOfDevices
>> Breakpoint 1 at 0x28610
>> --8<---------------cut here---------------end--------------->8---
>>
>> Uh oh.  This is our first sign of a problem: The breakpoint is
>> associated with some sort of memory address, rather than a location in a
>> file.  Anyway, let's run the program:
>>
>> --8<---------------cut here---------------start------------->8---
>> (gdb) run
>> Starting program:
>> /gnu/store/mx3rmbpg6lhl0yxl9djbx49nfps9lwqi-my-libvirt-4.3.0/bin/virsh
>> nodedev-list
>> warning: the debug information found in
>> "/home/marusich/debug/profile/lib/debug//gnu/store/mx3rmbpg6lhl0yxl9djbx49nfps9lwqi-my-libvirt-4.3.0/lib/libvirt.so.0.4003.0.debug"
>> does not match
>> "/gnu/store/mx3rmbpg6lhl0yxl9djbx49nfps9lwqi-my-libvirt-4.3.0/lib/libvirt.so.0"
>> (CRC mismatch).
>>
>> warning: the debug information found in
>> "/home/marusich/debug/profile/lib/debug//gnu/store/mx3rmbpg6lhl0yxl9djbx49nfps9lwqi-my-libvirt-4.3.0/lib/libvirt.so.0.4003.0.debug"
>> does not match
>> "/gnu/store/mx3rmbpg6lhl0yxl9djbx49nfps9lwqi-my-libvirt-4.3.0/lib/libvirt.so.0"
>> (CRC mismatch).
>>
>> [Thread debugging using libthread_db enabled]
>> Using host libthread_db library
>> "/gnu/store/l4lr0f5cjd0nbsaaf8b5dmcw1a1yypr3-glibc-2.27/lib/libthread_db.so.1".
>> [New Thread 0x7ffff2219700 (LWP 16097)]
>>
>> Thread 1 "virsh" hit Breakpoint 1, 0x00007ffff768cdc0 in
>> virNodeNumOfDevices ()
>>    from
>> /gnu/store/mx3rmbpg6lhl0yxl9djbx49nfps9lwqi-my-libvirt-4.3.0/lib/libvirt.so.0
>> --8<---------------cut here---------------end--------------->8---
>>
>> We hit the breakpoint - great!  However, it seems GDB did not load the
>> debug information for libvirt because of a CRC mismatch.  Indeed, the
>> backtrace seems to suggest that GDB knows about some of the source
>> files, but not all of them:
>>
>> --8<---------------cut here---------------start------------->8---
>> (gdb) bt
>> #0  0x00007ffff768cdc0 in virNodeNumOfDevicesw ()
>>    from
>> /gnu/store/mx3rmbpg6lhl0yxl9djbx49nfps9lwqi-my-libvirt-4.3.0/lib/libvirt.so.0
>> #1  0x00005555555a816e in virshNodeDeviceListCollect (flags=0,
>>     ncapnames=<optimized out>, capnames=0x0, ctl=0x7fffffffb460)
>>     at virsh-nodedev.c:254
>> #2  cmdNodeListDevices (ctl=0x7fffffffb460, cmd=<optimized out>)
>>     at virsh-nodedev.c:472
>> #3  0x00005555555b8911 in vshCommandRun (ctl=0x7fffffffb460,
>>     cmd=0x55555583d850) at vsh.c:1318
>> #4  0x000055555557ea65 in main (argc=2, argv=0x7fffffffb7f8) at
>> virsh.c:932
>> --8<---------------cut here---------------end--------------->8---
>>
>> I wanted to see what was happening in the virNodeNumOfDevices function,
>> which came from libvirt.so.0.  Unfortunately, that's the library with
>> the CRC mismatch.  This means I'm totally blocked from investigating any
>> further using GDB.  I could set step-mode to "on" to step through the
>> machine code without debug symbols, but as they say: "That is an
>> exercise left to the reader."
>>
>> I have seen this CRC mismatch problem twice now when trying to debug
>> issues with Guix-installed software.  The other time was while
>> attempting to debug a segfault in vinagre:
>>
>>     https://debbugs.gnu.org/cgi/bugreport.cgi?bug=30591
>>
>> What is wrong?  Am I using GDB wrong?  Is there a bug in the part of the
>> gnu-build-system that creates the debug files which might be causing the
>> CRC mismatch?  I'm aware of the fact that the gnu-build-system takes
>> advantage of the .gnu-debuglink stuff ((gdb) Separate Debug Files), but
>> to be honest I haven't done a lot of GDB debugging, so part of me
>> wonders if this is just a case of "user error".  If so, please help me
>> understand what I'm doing wrong.
>>
>>
> Actually this is about the same thing I've found out. I'm also suffering
> CRC mismatches. The workaround I used was defining a package where I didn't
> strip the debugging symbols in the first place. I don't know what this is
> about either, but it is annoying.
>

Hello Chris,

I was thinking about this yesterday, is it possible that this is related to
grafting? I


>
>
>> Thank you,
>>
>> --
>> Chris
>>
>

[-- Attachment #2: Type: text/html, Size: 14030 bytes --]

Re: Using GDB to debug Guix-installed software (e.g. virsh)

[Ludovic Courtès]

Hi Chris,

Chris Marusich <cmmarusich@gmail.com> skribis:

> (gdb) run
> Starting program: /gnu/store/mx3rmbpg6lhl0yxl9djbx49nfps9lwqi-my-libvirt-4.3.0/bin/virsh nodedev-list
> warning: the debug information found in "/home/marusich/debug/profile/lib/debug//gnu/store/mx3rmbpg6lhl0yxl9djbx49nfps9lwqi-my-libvirt-4.3.0/lib/libvirt.so.0.4003.0.debug" does not match "/gnu/store/mx3rmbpg6lhl0yxl9djbx49nfps9lwqi-my-libvirt-4.3.0/lib/libvirt.so.0" (CRC mismatch).
>
> warning: the debug information found in "/home/marusich/debug/profile/lib/debug//gnu/store/mx3rmbpg6lhl0yxl9djbx49nfps9lwqi-my-libvirt-4.3.0/lib/libvirt.so.0.4003.0.debug" does not match "/gnu/store/mx3rmbpg6lhl0yxl9djbx49nfps9lwqi-my-libvirt-4.3.0/lib/libvirt.so.0" (CRC mismatch).

This is a bug: like Gábor guessed, it has to do with grafting:

  https://bugs.gnu.org/19973

GDB documents how this CRC is computed so we could implement that method
and use that in (guix build grafts).  Would be nice!

Ludo’.