Performance issues with /gnu/store in virtual machines

Performance issues with /gnu/store in virtual machines

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 3414 bytes --]

Hey,

So I was trying out using VMs created through Guix for more things,
particularly as they can be run as part of derivations, e.g. the system
tests for Guix.

However, I'm seeing anything that does I/O performing very poorly
compared with the host system. See the system call profiles from strace
below, for the following command:

  strace -c find /gnu/store/*-guile* -name guile | wc -l

Is anyone else having difficulties with this, or have any experience
with performance tuning for the software in use here?

Thanks,

Chris



On the host system, I see the following performance:

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 76.23    0.001286           0     31978           fcntl
 10.20    0.000172           0      6974           getdents
  7.53    0.000127           0     18673           newfstatat
  3.32    0.000056           0     14208           close
  1.60    0.000027           0      3872           fstat
  1.13    0.000019           0      3853           openat
  0.00    0.000000           0        20           read
  0.00    0.000000           0         4           write
  0.00    0.000000           0        69        50 open
  0.00    0.000000           0         6         6 stat
  0.00    0.000000           0        21           mmap
  0.00    0.000000           0         8           mprotect
  0.00    0.000000           0        33           brk
  0.00    0.000000           0         2         1 ioctl
  0.00    0.000000           0         1         1 access
  0.00    0.000000           0         1           execve
  0.00    0.000000           0         1           uname
  0.00    0.000000           0         1           fchdir
  0.00    0.000000           0         1           arch_prctl
------ ----------- ----------- --------- --------- ----------------
100.00    0.001687                 79726        58 total

Within the VM, I see the following performance:

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 59.83    0.851601          46     18673           newfstatat
 13.56    0.193014          50      3853           openat
 10.88    0.154935          22      6974           getdents
  7.95    0.113196           8     14208           close
  4.00    0.056956          15      3872           fstat
  3.58    0.050908           2     31978           fcntl
  0.14    0.002051          66        31        12 open
  0.02    0.000232          12        20           read
  0.01    0.000199           7        30           brk
  0.01    0.000107          18         6         6 stat
  0.01    0.000076           4        21           mmap
  0.00    0.000045           6         8           mprotect
  0.00    0.000030           8         4           write
  0.00    0.000018          18         1         1 access
  0.00    0.000006           3         2         1 ioctl
  0.00    0.000004           4         1           execve
  0.00    0.000003           3         1           uname
  0.00    0.000003           3         1           fchdir
  0.00    0.000001           1         1           arch_prctl
------ ----------- ----------- --------- --------- ----------------
100.00    1.423385                 79685        20 total



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: Performance issues with /gnu/store in virtual machines

[Ricardo Wurmus]

Hi Chris,

> So I was trying out using VMs created through Guix for more things,
> particularly as they can be run as part of derivations, e.g. the system
> tests for Guix.
>
> However, I'm seeing anything that does I/O performing very poorly
> compared with the host system.

(I’m not using VMs as KVM doesn’t work on my laptop.)

Have you tried specifying “virtio” drivers for the virtual disks?  I
don’t think they are the default and I’m hopeful that with virtio
drivers I/O performance will be comparable.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: Performance issues with /gnu/store in virtual machines

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1548 bytes --]

On Sun, 20 Aug 2017 11:02:16 +0200
Ricardo Wurmus <rekado@elephly.net> wrote:

> Hi Chris,
> 
> > So I was trying out using VMs created through Guix for more things,
> > particularly as they can be run as part of derivations, e.g. the
> > system tests for Guix.
> >
> > However, I'm seeing anything that does I/O performing very poorly
> > compared with the host system.  
> 
> (I’m not using VMs as KVM doesn’t work on my laptop.)
> 
> Have you tried specifying “virtio” drivers for the virtual disks?  I
> don’t think they are the default and I’m hopeful that with virtio
> drivers I/O performance will be comparable.

I think I may already be using virtio, at least that is what is
specified as the trans option for the mount in the VM? I think this is
the relevant bit from the vm start script:

  -virtfs local,path="/gnu/store",security_model=none,mount_tag="TAG_gnu_store"

This is how the mount shows up in the vm:

  TAG_gnu_store on /gnu/store type 9p (ro,relatime,sync,dirsync,trans=virtio)


One thing I have just tried, is adding the cache=loose option in (by
directly editing the mapping->file-system procedure in (gnu system
vm)). For my really quick test [1], this did increase the initial
runtime from ~8 to ~11 seconds, but subsequent runs were down to ~5
seconds from the original ~8.

5 seconds though, is still 16 times longer than the ~0.3 seconds this
command takes to run on the host, so it's still slow by comparison.

1: time find /gnu/store/*-guile* -name guile | wc -l

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: Performance issues with /gnu/store in virtual machines

[Ludovic Courtès]

Hi Chris,

Christopher Baines <mail@cbaines.net> skribis:

>   1.13    0.000019           0      3853           openat

[...]

>  13.56    0.193014          50      3853           openat

So openat(2) is 4 orders of magnitude slower in the VM, which is a
problem.

I guess a large part of that is due to the fact that VM created with
‘guix system vm’ use unionfs as their root file system, which is a FUSE
thing (see the “volatile” options in (gnu system vm), (gnu system
linux-initrd), and (gnu build linux-boot).)  I’m not sure how we could
avoid it.

Can you confirm that the same ‘find’ command is faster in a full-blown
“guix system disk-image” VM?

Thanks,
Ludo’.

Re: Performance issues with /gnu/store in virtual machines

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1283 bytes --]

On Tue, 22 Aug 2017 12:11:43 +0200
ludo@gnu.org (Ludovic Courtès) wrote:

> Hi Chris,
> 
> Christopher Baines <mail@cbaines.net> skribis:
> 
> >   1.13    0.000019           0      3853           openat  
> 
> [...]
> 
> >  13.56    0.193014          50      3853           openat  
> 
> So openat(2) is 4 orders of magnitude slower in the VM, which is a
> problem.
> 
> I guess a large part of that is due to the fact that VM created with
> ‘guix system vm’ use unionfs as their root file system, which is a
> FUSE thing (see the “volatile” options in (gnu system vm), (gnu system
> linux-initrd), and (gnu build linux-boot).)  I’m not sure how we could
> avoid it.

I'm unsure if the unionfs will be involved though, as find is searching
in the /gnu/store directory, which is shared with the host:

  TAG_gnu_store on /gnu/store type 9p (ro,relatime,sync,dirsync,trans=virtio)

This might be misunderstanding how mounts work, but this could be
either layered on top, or below the unionfs, and I'm guessing its on
top, which in my mind would mean that the unionfs isn't involved?

> Can you confirm that the same ‘find’ command is faster in a full-blown
> “guix system disk-image” VM?

Yep, I see 8 usecs/call (compared to 50).

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: Performance issues with /gnu/store in virtual machines

[宋文武]

ludo@gnu.org (Ludovic Courtès) writes:

> Hi Chris,
>
> Christopher Baines <mail@cbaines.net> skribis:
>
>>   1.13    0.000019           0      3853           openat
>
> [...]
>
>>  13.56    0.193014          50      3853           openat
>
> So openat(2) is 4 orders of magnitude slower in the VM, which is a
> problem.
>
> I guess a large part of that is due to the fact that VM created with
> ‘guix system vm’ use unionfs as their root file system, which is a FUSE
> thing (see the “volatile” options in (gnu system vm), (gnu system
> linux-initrd), and (gnu build linux-boot).)  I’m not sure how we could
> avoid it.

We might switch to the linux kernel's bultin overlayfs, which should be
faster.

https://wiki.archlinux.org/index.php/Overlay_filesystem

Re: Performance issues with /gnu/store in virtual machines

[Ludovic Courtès]

iyzsong@member.fsf.org (宋文武) skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> Hi Chris,
>>
>> Christopher Baines <mail@cbaines.net> skribis:
>>
>>>   1.13    0.000019           0      3853           openat
>>
>> [...]
>>
>>>  13.56    0.193014          50      3853           openat
>>
>> So openat(2) is 4 orders of magnitude slower in the VM, which is a
>> problem.
>>
>> I guess a large part of that is due to the fact that VM created with
>> ‘guix system vm’ use unionfs as their root file system, which is a FUSE
>> thing (see the “volatile” options in (gnu system vm), (gnu system
>> linux-initrd), and (gnu build linux-boot).)  I’m not sure how we could
>> avoid it.
>
> We might switch to the linux kernel's bultin overlayfs, which should be
> faster.
>
> https://wiki.archlinux.org/index.php/Overlay_filesystem

Good point, we should probably do that.

Ludo’.

Re: Performance issues with /gnu/store in virtual machines

[Ludovic Courtès]

Christopher Baines <mail@cbaines.net> skribis:

> On Tue, 22 Aug 2017 12:11:43 +0200
> ludo@gnu.org (Ludovic Courtès) wrote:
>
>> Hi Chris,
>> 
>> Christopher Baines <mail@cbaines.net> skribis:
>> 
>> >   1.13    0.000019           0      3853           openat  
>> 
>> [...]
>> 
>> >  13.56    0.193014          50      3853           openat  
>> 
>> So openat(2) is 4 orders of magnitude slower in the VM, which is a
>> problem.
>> 
>> I guess a large part of that is due to the fact that VM created with
>> ‘guix system vm’ use unionfs as their root file system, which is a
>> FUSE thing (see the “volatile” options in (gnu system vm), (gnu system
>> linux-initrd), and (gnu build linux-boot).)  I’m not sure how we could
>> avoid it.
>
> I'm unsure if the unionfs will be involved though, as find is searching
> in the /gnu/store directory, which is shared with the host:
>
>   TAG_gnu_store on /gnu/store type 9p (ro,relatime,sync,dirsync,trans=virtio)
>
> This might be misunderstanding how mounts work, but this could be
> either layered on top, or below the unionfs, and I'm guessing its on
> top, which in my mind would mean that the unionfs isn't involved?

Oh you’re probably right.  It might be that 9p over virtio is this slow.
We should look at what others are doing, I guess it’s a fairly common
use case.

Ludo’.