* Combining Guix, direnv and Emacs for environment customisation
[not found] <20170527105641.9426-1-mail@cbaines.net>
@ 2017-05-27 11:31 ` Christopher Baines
2017-05-30 15:03 ` Ludovic Courtès
2017-08-29 19:02 ` Combining Guix, direnv and Emacs for environment customisation Thompson, David
0 siblings, 2 replies; 37+ messages in thread
From: Christopher Baines @ 2017-05-27 11:31 UTC (permalink / raw)
To: guix-devel; +Cc: 27097
[-- Attachment #1: Type: text/plain, Size: 2776 bytes --]
Hey,
In summary, using direnv provides a convinient way to manage different
environments created from Guix. There is now support for using direnv
directly from Emacs.
So for a while now I've been using direnv and Guix, but thanks to
recent improvements with the Emacs integration, its now become even
more useful.
direnv [1] is an environment switcher for shells, for example, you want
to have a specific environment variable set when working on a particular
project, you drop a .envrc file in to the relevant directory and
providing direnv is hooked in to your shell, it will get loaded and
unloaded as you move in and out of that directory.
1: https://direnv.net/
While direnv is useful for simple environment variables, guix
environment can output environment variables with the --shell-paths
option. Using guix environment in a .envrc file would look something
like:
eval "$(guix environment --ad-hoc guile --search-paths)"
There is a use_guix helper function in the direnv stdlib [2] that helps
with this, so you can just do:
use guix --ad-hoc guile
2: https://github.com/direnv/direnv/blob/master/stdlib.sh#L574-L586
I've been using direnv and Guix for a while now, but to use both with
Emacs, I've been starting a shell outside of Emacs, navigating to the
relevant directory so that direnv sets up the environment, and then
starting Emacs from the shell such that running programs (e.g. linters,
ruby, ...) from Emacs works within the environment setup through Guix
and direnv.
I've recently become aware of emacs-direnv [3], which provides access
to the functionality of direnv from Emacs. When the global minor mode
is active, this means that moving around between buffers in Emacs can
completely change the environment within Emacs. This had made my
workflow simpler, as I now just open Emacs, and navigate to the
relevant directory, and direnv just works behind the scenes.
3: https://github.com/wbolster/emacs-direnv
One issue with this is that running guix environment from direnv will
slow down switching buffers. To make it a bit more useable, I found
some bash code that caches the results of running commands, and wrapped
that around guix environment when invoked from direnv. This helps speed
things up, but I don't think its useful in the long term.
For this particular use case, it would help if guix environment was
faster, perhaps by doing caching internally? On my system, running guix
environment --ad-hoc guile --search-paths repeatedly takes ~2 seconds,
I haven't looked at what the breakdown of this is yet.
I'd be interested in hearing if anyone does something similar for using
Guix, or if anyone does something different, but to the same effect?
Thanks,
Chris
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 963 bytes --]
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Combining Guix, direnv and Emacs for environment customisation
2017-05-27 11:31 ` Combining Guix, direnv and Emacs for environment customisation Christopher Baines
@ 2017-05-30 15:03 ` Ludovic Courtès
2017-06-01 13:17 ` Roel Janssen
2017-08-29 19:02 ` Combining Guix, direnv and Emacs for environment customisation Thompson, David
1 sibling, 1 reply; 37+ messages in thread
From: Ludovic Courtès @ 2017-05-30 15:03 UTC (permalink / raw)
To: Christopher Baines; +Cc: guix-devel, 27097
Hi Christopher!
Christopher Baines <mail@cbaines.net> skribis:
> direnv [1] is an environment switcher for shells, for example, you want
> to have a specific environment variable set when working on a particular
> project, you drop a .envrc file in to the relevant directory and
> providing direnv is hooked in to your shell, it will get loaded and
> unloaded as you move in and out of that directory.
>
> 1: https://direnv.net/
>
> While direnv is useful for simple environment variables, guix
> environment can output environment variables with the --shell-paths
> option. Using guix environment in a .envrc file would look something
> like:
>
> eval "$(guix environment --ad-hoc guile --search-paths)"
>
> There is a use_guix helper function in the direnv stdlib [2] that helps
> with this, so you can just do:
>
> use guix --ad-hoc guile
This is pretty cool!
However, using ‘guix environment --search-paths’ is kinda unsafe: the
items mentioned in its output are not protected from GC. This is why
‘guix environment’ normally spawns a shell or some other process while
keep its guix-daemon session open.
> I've recently become aware of emacs-direnv [3], which provides access
> to the functionality of direnv from Emacs. When the global minor mode
> is active, this means that moving around between buffers in Emacs can
> completely change the environment within Emacs. This had made my
> workflow simpler, as I now just open Emacs, and navigate to the
> relevant directory, and direnv just works behind the scenes.
>
> 3: https://github.com/wbolster/emacs-direnv
I think it’d be great Emacs-Guix could do something similar, i.e.,
associate a ‘guix environment’ to a buffer. :-)
> One issue with this is that running guix environment from direnv will
> slow down switching buffers. To make it a bit more useable, I found
> some bash code that caches the results of running commands, and wrapped
> that around guix environment when invoked from direnv. This helps speed
> things up, but I don't think its useful in the long term.
>
> For this particular use case, it would help if guix environment was
> faster, perhaps by doing caching internally? On my system, running guix
> environment --ad-hoc guile --search-paths repeatedly takes ~2 seconds,
> I haven't looked at what the breakdown of this is yet.
I agree that we could do a lot more things with a faster ‘guix
environment’. My guess is that it won’t be easy to go optimize, and
very hard to go below 1 second. We should profile that and see what can
be done.
Cheers,
Ludo’.
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Combining Guix, direnv and Emacs for environment customisation
2017-05-30 15:03 ` Ludovic Courtès
@ 2017-06-01 13:17 ` Roel Janssen
2017-06-03 13:59 ` Ludovic Courtès
0 siblings, 1 reply; 37+ messages in thread
From: Roel Janssen @ 2017-06-01 13:17 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: guix-devel, 27097
Ludovic Courtès writes:
> I agree that we could do a lot more things with a faster ‘guix
> environment’. My guess is that it won’t be easy to go optimize, and
> very hard to go below 1 second. We should profile that and see what can
> be done.
FWIW, on our NFS-mounted /gnu, the 'guix environment' command takes at
least 20 seconds, but for any reasonably big environment it takes more
than one minute. The biggest bottleneck here is the disk latency.
Could it be that 'guix environment' accesses many files? If we could
reduce that, it would speed things up (at least for us ;)).
Kind regards,
Roel Janssen
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Combining Guix, direnv and Emacs for environment customisation
2017-06-01 13:17 ` Roel Janssen
@ 2017-06-03 13:59 ` Ludovic Courtès
2017-06-03 21:08 ` Roel Janssen
0 siblings, 1 reply; 37+ messages in thread
From: Ludovic Courtès @ 2017-06-03 13:59 UTC (permalink / raw)
To: Roel Janssen; +Cc: guix-devel, 27097
Roel Janssen <roel@gnu.org> skribis:
> Ludovic Courtès writes:
>
>> I agree that we could do a lot more things with a faster ‘guix
>> environment’. My guess is that it won’t be easy to go optimize, and
>> very hard to go below 1 second. We should profile that and see what can
>> be done.
>
> FWIW, on our NFS-mounted /gnu, the 'guix environment' command takes at
> least 20 seconds, but for any reasonably big environment it takes more
> than one minute. The biggest bottleneck here is the disk latency.
> Could it be that 'guix environment' accesses many files? If we could
> reduce that, it would speed things up (at least for us ;)).
Interesting. Does guix-daemon access /gnu over NFS too?
Could you send the output of (first run the same command without
‘strace’):
strace -c guix environment --ad-hoc coreutils --pure -- true
On my laptop I get:
--8<---------------cut here---------------start------------->8---
$ strace -c guix environment --ad-hoc coreutils --pure -- true
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
37.84 0.017268 1 14036 10965 stat
15.13 0.006904 1 9990 read
14.02 0.006398 1 10000 write
8.75 0.003991 3991 1 wait4
6.06 0.002764 1 5218 4 lstat
4.73 0.002157 1 1930 136 open
2.58 0.001176 1 1794 close
2.08 0.000949 1 1717 3 lseek
2.04 0.000932 1 689 mmap
1.89 0.000861 1 645 mprotect
1.71 0.000781 4 189 32 futex
[…]
--8<---------------cut here---------------end--------------->8---
We stat a lot mostly to access all the Guix modules. Are they on NFS
too?
Thanks,
Ludo’.
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Combining Guix, direnv and Emacs for environment customisation
2017-06-03 13:59 ` Ludovic Courtès
@ 2017-06-03 21:08 ` Roel Janssen
2017-06-04 21:15 ` Ludovic Courtès
0 siblings, 1 reply; 37+ messages in thread
From: Roel Janssen @ 2017-06-03 21:08 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: guix-devel, 27097
Ludovic Courtès writes:
> Roel Janssen <roel@gnu.org> skribis:
>
>> Ludovic Courtès writes:
>>
>>> I agree that we could do a lot more things with a faster ‘guix
>>> environment’. My guess is that it won’t be easy to go optimize, and
>>> very hard to go below 1 second. We should profile that and see what can
>>> be done.
>>
>> FWIW, on our NFS-mounted /gnu, the 'guix environment' command takes at
>> least 20 seconds, but for any reasonably big environment it takes more
>> than one minute. The biggest bottleneck here is the disk latency.
>> Could it be that 'guix environment' accesses many files? If we could
>> reduce that, it would speed things up (at least for us ;)).
>
> Interesting. Does guix-daemon access /gnu over NFS too?
Yes.
> Could you send the output of (first run the same command without
> ‘strace’):
>
> strace -c guix environment --ad-hoc coreutils --pure -- true
>
> On my laptop I get:
>
> --8<---------------cut here---------------start------------->8---
> $ strace -c guix environment --ad-hoc coreutils --pure -- true
> % time seconds usecs/call calls errors syscall
> ------ ----------- ----------- --------- --------- ----------------
> 37.84 0.017268 1 14036 10965 stat
> 15.13 0.006904 1 9990 read
> 14.02 0.006398 1 10000 write
> 8.75 0.003991 3991 1 wait4
> 6.06 0.002764 1 5218 4 lstat
> 4.73 0.002157 1 1930 136 open
> 2.58 0.001176 1 1794 close
> 2.08 0.000949 1 1717 3 lseek
> 2.04 0.000932 1 689 mmap
> 1.89 0.000861 1 645 mprotect
> 1.71 0.000781 4 189 32 futex
>
> […]
> --8<---------------cut here---------------end--------------->8---
>
> We stat a lot mostly to access all the Guix modules. Are they on NFS
> too?
Yes.
> Thanks,
> Ludo’.
Here's my output:
[rjanssen2@hpcguix ~]$ strace -c guix environment --ad-hoc coreutils --pure -- true
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
41.79 0.038363 22 1782 188 open
14.71 0.013500 3 4769 1747 stat
13.19 0.012106 2 8033 read
6.06 0.005561 1854 3 1 wait4
5.83 0.005356 8 633 147 futex
4.15 0.003814 0 10020 write
3.25 0.002987 157 19 clone
3.23 0.002968 1 5086 2 lstat
2.86 0.002627 25 105 sendfile
1.04 0.000954 954 1 connect
0.97 0.000889 889 1 socket
0.63 0.000582 0 1596 close
0.50 0.000460 1 685 mmap
0.45 0.000414 0 1025 fstat
0.42 0.000386 0 1520 3 lseek
0.38 0.000350 1 644 mprotect
0.28 0.000255 12 22 getdents
0.11 0.000103 1 186 brk
0.06 0.000051 26 2 madvise
0.02 0.000021 4 5 munmap
0.02 0.000021 0 65 clock_gettime
0.01 0.000009 5 2 execve
0.01 0.000006 0 50 rt_sigprocmask
0.01 0.000005 0 13 1 fcntl
0.01 0.000005 1 5 prlimit64
0.00 0.000002 0 25 rt_sigaction
0.00 0.000002 1 2 arch_prctl
0.00 0.000001 0 10 3 ioctl
0.00 0.000001 0 3 3 access
0.00 0.000001 0 3 dup2
0.00 0.000000 0 1 rt_sigreturn
0.00 0.000000 0 1 pipe
0.00 0.000000 0 2 getpid
0.00 0.000000 0 2 uname
0.00 0.000000 0 1 getcwd
0.00 0.000000 0 2 readlink
0.00 0.000000 0 1 sysinfo
0.00 0.000000 0 1 getuid
0.00 0.000000 0 1 getgid
0.00 0.000000 0 1 geteuid
0.00 0.000000 0 1 getegid
0.00 0.000000 0 1 getppid
0.00 0.000000 0 1 getpgrp
0.00 0.000000 0 2 sched_getaffinity
0.00 0.000000 0 1 set_tid_address
0.00 0.000000 0 1 faccessat
0.00 0.000000 0 1 set_robust_list
0.00 0.000000 0 3 pipe2
------ ----------- ----------- --------- --------- ----------------
100.00 0.091800 36339 2095 total
So the main difference is that calls to 'open' take a lot longer.
There's little we can do, other than come up with a good caching
mechanism.
Kind regards,
Roel Janssen
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Combining Guix, direnv and Emacs for environment customisation
2017-06-03 21:08 ` Roel Janssen
@ 2017-06-04 21:15 ` Ludovic Courtès
2017-06-05 22:37 ` Roel Janssen
0 siblings, 1 reply; 37+ messages in thread
From: Ludovic Courtès @ 2017-06-04 21:15 UTC (permalink / raw)
To: Roel Janssen; +Cc: guix-devel, 27097
Hi Roel,
Roel Janssen <roel@gnu.org> skribis:
> Ludovic Courtès writes:
[...]
>>> FWIW, on our NFS-mounted /gnu, the 'guix environment' command takes at
>>> least 20 seconds, but for any reasonably big environment it takes more
>>> than one minute. The biggest bottleneck here is the disk latency.
>>> Could it be that 'guix environment' accesses many files? If we could
>>> reduce that, it would speed things up (at least for us ;)).
>>
>> Interesting. Does guix-daemon access /gnu over NFS too?
>
> Yes.
That is going to make builds verrrry slow. I would suggest having
guix-daemon operate on the local /gnu (that’s what Ricardo did, IIRC).
>> We stat a lot mostly to access all the Guix modules. Are they on NFS
>> too?
>
> Yes.
OK.
> Here's my output:
>
> [rjanssen2@hpcguix ~]$ strace -c guix environment --ad-hoc coreutils --pure -- true
> % time seconds usecs/call calls errors syscall
> ------ ----------- ----------- --------- --------- ----------------
> 41.79 0.038363 22 1782 188 open
> 14.71 0.013500 3 4769 1747 stat
> 13.19 0.012106 2 8033 read
> 6.06 0.005561 1854 3 1 wait4
> 5.83 0.005356 8 633 147 futex
> 4.15 0.003814 0 10020 write
> 3.25 0.002987 157 19 clone
[...]
> ------ ----------- ----------- --------- --------- ----------------
> 100.00 0.091800 36339 2095 total
>
>
> So the main difference is that calls to 'open' take a lot longer.
> There's little we can do, other than come up with a good caching
> mechanism.
We should make sure these 1782 calls are all warranted (e.g., make sure
we don’t open the same file twice; this could be easily diagnosed by
looking at the output of ‘strace’.)
But then, as you say, it’s a process that opens lots of files.
We could cache package lookups, and maybe that’d help a little bit.
Currently we use ‘fold-packages’, which loads every single package
module, and then we look for a package named “coreutils” among them;
with a cache, we could directly load (gnu packages base) and its
dependencies.
Ludo’.
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Combining Guix, direnv and Emacs for environment customisation
2017-06-04 21:15 ` Ludovic Courtès
@ 2017-06-05 22:37 ` Roel Janssen
2017-06-07 10:35 ` Performance on NFS Ludovic Courtès
2017-06-07 11:01 ` Combining Guix, direnv and Emacs for environment customisation Ricardo Wurmus
0 siblings, 2 replies; 37+ messages in thread
From: Roel Janssen @ 2017-06-05 22:37 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: guix-devel, 27097
Ludovic Courtès writes:
> Hi Roel,
>
> Roel Janssen <roel@gnu.org> skribis:
>
>> Ludovic Courtès writes:
>
> [...]
>
>>>> FWIW, on our NFS-mounted /gnu, the 'guix environment' command takes at
>>>> least 20 seconds, but for any reasonably big environment it takes more
>>>> than one minute. The biggest bottleneck here is the disk latency.
>>>> Could it be that 'guix environment' accesses many files? If we could
>>>> reduce that, it would speed things up (at least for us ;)).
>>>
>>> Interesting. Does guix-daemon access /gnu over NFS too?
>>
>> Yes.
>
> That is going to make builds verrrry slow. I would suggest having
> guix-daemon operate on the local /gnu (that’s what Ricardo did, IIRC).
That definitely speeds things up about 50%. So, running the environment
command from a node that is connected to the node that runs
guix-daemon, will take about 30 seconds instead of one minute.
This could be made faster by letting 'guix-daemon' do more of the work,
and let the client-side 'guix' only do the minimum.
(I know.. this remote-guix-daemon stuff voids the warranty.. ;))
The NFS overhead is pretty large. Maybe we can better tune it, and if
so, document how to tune it for GNU Guix. I already talked to our
storage expert, and tuning boils down to using fast disks, a
large-enough NFS cache, and low-latency network equipment.
The reason we have /gnu NFS-mounted on the build node is that we can
then very easily replace the node when it would have a hardware failure,
without even losing the ability to run programs that were already
installed.
>
>>> We stat a lot mostly to access all the Guix modules. Are they on NFS
>>> too?
>>
>> Yes.
>
> OK.
>
>> Here's my output:
>>
>> [rjanssen2@hpcguix ~]$ strace -c guix environment --ad-hoc coreutils --pure -- true
>> % time seconds usecs/call calls errors syscall
>> ------ ----------- ----------- --------- --------- ----------------
>> 41.79 0.038363 22 1782 188 open
>> 14.71 0.013500 3 4769 1747 stat
>> 13.19 0.012106 2 8033 read
>> 6.06 0.005561 1854 3 1 wait4
>> 5.83 0.005356 8 633 147 futex
>> 4.15 0.003814 0 10020 write
>> 3.25 0.002987 157 19 clone
>
> [...]
>
>> ------ ----------- ----------- --------- --------- ----------------
>> 100.00 0.091800 36339 2095 total
>>
>>
>> So the main difference is that calls to 'open' take a lot longer.
>> There's little we can do, other than come up with a good caching
>> mechanism.
>
> We should make sure these 1782 calls are all warranted (e.g., make sure
> we don’t open the same file twice; this could be easily diagnosed by
> looking at the output of ‘strace’.)
>
> But then, as you say, it’s a process that opens lots of files.
>
> We could cache package lookups, and maybe that’d help a little bit.
> Currently we use ‘fold-packages’, which loads every single package
> module, and then we look for a package named “coreutils” among them;
> with a cache, we could directly load (gnu packages base) and its
> dependencies.
>
> Ludo’.
Running 'guix package -A' only takes a couple of seconds. So I don't
think that loading the Scheme modules is really a big issue.
Kind regards,
Roel Janssen
^ permalink raw reply [flat|nested] 37+ messages in thread
* Performance on NFS
2017-06-05 22:37 ` Roel Janssen
@ 2017-06-07 10:35 ` Ludovic Courtès
2017-06-07 13:06 ` Roel Janssen
2017-06-07 11:01 ` Combining Guix, direnv and Emacs for environment customisation Ricardo Wurmus
1 sibling, 1 reply; 37+ messages in thread
From: Ludovic Courtès @ 2017-06-07 10:35 UTC (permalink / raw)
To: Roel Janssen; +Cc: guix-devel, 27097
Hi Roel,
Roel Janssen <roel@gnu.org> skribis:
>> That is going to make builds verrrry slow. I would suggest having
>> guix-daemon operate on the local /gnu (that’s what Ricardo did, IIRC).
>
> That definitely speeds things up about 50%. So, running the environment
> command from a node that is connected to the node that runs
> guix-daemon, will take about 30 seconds instead of one minute.
>
> This could be made faster by letting 'guix-daemon' do more of the work,
> and let the client-side 'guix' only do the minimum.
Right, but it’s the client that computes the derivation, which involves
quite a bit of I/O and computation.
> (I know.. this remote-guix-daemon stuff voids the warranty.. ;))
>
> The NFS overhead is pretty large. Maybe we can better tune it, and if
> so, document how to tune it for GNU Guix. I already talked to our
> storage expert, and tuning boils down to using fast disks, a
> large-enough NFS cache, and low-latency network equipment.
>
> The reason we have /gnu NFS-mounted on the build node is that we can
> then very easily replace the node when it would have a hardware failure,
> without even losing the ability to run programs that were already
> installed.
I guess you could also use a RAID device for /gnu/store. Or you could
have a local /gnu/store and replicate it elsewhere, maybe
opportunistically via ‘guix publish’.
>> We should make sure these 1782 calls are all warranted (e.g., make sure
>> we don’t open the same file twice; this could be easily diagnosed by
>> looking at the output of ‘strace’.)
>>
>> But then, as you say, it’s a process that opens lots of files.
>>
>> We could cache package lookups, and maybe that’d help a little bit.
>> Currently we use ‘fold-packages’, which loads every single package
>> module, and then we look for a package named “coreutils” among them;
>> with a cache, we could directly load (gnu packages base) and its
>> dependencies.
>>
>> Ludo’.
>
> Running 'guix package -A' only takes a couple of seconds. So I don't
> think that loading the Scheme modules is really a big issue.
How does:
time guix environment --ad-hoc coreutils --pure -- true
compare to:
time guix environment --ad-hoc -e '(@ (gnu packages base) coreutils)' --pure -- true
? That would give us an estimate of how much the cache I describe would
help.
Thanks,
Ludo’.
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Combining Guix, direnv and Emacs for environment customisation
2017-06-05 22:37 ` Roel Janssen
2017-06-07 10:35 ` Performance on NFS Ludovic Courtès
@ 2017-06-07 11:01 ` Ricardo Wurmus
2017-06-07 12:25 ` Performance on NFS Ludovic Courtès
1 sibling, 1 reply; 37+ messages in thread
From: Ricardo Wurmus @ 2017-06-07 11:01 UTC (permalink / raw)
To: Roel Janssen; +Cc: guix-devel, 27097
Roel Janssen <roel@gnu.org> writes:
> The NFS overhead is pretty large. Maybe we can better tune it, and if
> so, document how to tune it for GNU Guix. I already talked to our
> storage expert, and tuning boils down to using fast disks, a
> large-enough NFS cache, and low-latency network equipment.
We may get a faster NFS share soon (with some of the slow storage
settings disabled), but I don’t hold my breath when it comes to NFS
performance.
> The reason we have /gnu NFS-mounted on the build node is that we can
> then very easily replace the node when it would have a hardware failure,
> without even losing the ability to run programs that were already
> installed.
That’s the same reason I have for keeping /gnu on NFS. I have been
experimenting with lsync (writing to local disks and then pushing
changes asynchronously from local-gnu to remote-gnu), but I wasn’t
convinced it would be reliable.
I’m not sure if this would help, though, when the nodes keep mounting
/gnu over NFS.
Is there a way to put /gnu on a Samba share instead? I’d like to give
that a try, but I’m not sure about what would happen to ownership and
permissions (I don’t worry about hardlinks because I disabled
deduplication).
--
Ricardo
GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC
https://elephly.net
^ permalink raw reply [flat|nested] 37+ messages in thread
* Performance on NFS
2017-06-07 11:01 ` Combining Guix, direnv and Emacs for environment customisation Ricardo Wurmus
@ 2017-06-07 12:25 ` Ludovic Courtès
2017-06-07 12:59 ` Ricardo Wurmus
0 siblings, 1 reply; 37+ messages in thread
From: Ludovic Courtès @ 2017-06-07 12:25 UTC (permalink / raw)
To: Ricardo Wurmus; +Cc: guix-devel, 27097
Ricardo Wurmus <rekado@elephly.net> skribis:
> Roel Janssen <roel@gnu.org> writes:
[...]
>> The reason we have /gnu NFS-mounted on the build node is that we can
>> then very easily replace the node when it would have a hardware failure,
>> without even losing the ability to run programs that were already
>> installed.
>
> That’s the same reason I have for keeping /gnu on NFS. I have been
> experimenting with lsync (writing to local disks and then pushing
> changes asynchronously from local-gnu to remote-gnu), but I wasn’t
> convinced it would be reliable.
>
> I’m not sure if this would help, though, when the nodes keep mounting
> /gnu over NFS.
The nodes don’t do as much I/O the store as the daemon, so I think it
makes sense to use a local store for the node that runs guix-daemon. In
general, Guix aside, building software on NFS is unavoidably slow.
> Is there a way to put /gnu on a Samba share instead? I’d like to give
> that a try, but I’m not sure about what would happen to ownership and
> permissions (I don’t worry about hardlinks because I disabled
> deduplication).
I’m not sure if SMB is faster than NFS, is it? 9p (which Linux
supports) might work well.
Thanks,
Ludo’.
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance on NFS
2017-06-07 12:25 ` Performance on NFS Ludovic Courtès
@ 2017-06-07 12:59 ` Ricardo Wurmus
0 siblings, 0 replies; 37+ messages in thread
From: Ricardo Wurmus @ 2017-06-07 12:59 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: guix-devel, 27097
Ludovic Courtès <ludo@gnu.org> writes:
> I’m not sure if SMB is faster than NFS, is it? 9p (which Linux
> supports) might work well.
Sadly, our “storage appliances” only offer NFS or SMB (and maybe SCP).
I’d gladly try *anything* to get away from our slow NFS.
--
Ricardo
GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC
https://elephly.net
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance on NFS
2017-06-07 10:35 ` Performance on NFS Ludovic Courtès
@ 2017-06-07 13:06 ` Roel Janssen
2017-06-09 13:46 ` Ludovic Courtès
0 siblings, 1 reply; 37+ messages in thread
From: Roel Janssen @ 2017-06-07 13:06 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: guix-devel, 27097
Ludovic Courtès writes:
> How does:
>
> time guix environment --ad-hoc coreutils --pure -- true
>
> compare to:
>
> time guix environment --ad-hoc -e '(@ (gnu packages base) coreutils)' --pure -- true
>
> ? That would give us an estimate of how much the cache I describe would
> help.
>
> Thanks,
> Ludo’.
You should know that we have 'submit' nodes that use the guixr wrapper
script to connect to the guix-daemon that runs on the 'hpcguix' node.
Both have a /gnu mounted by a storage subsystem.
I couldn't run the second command on a 'submit' node. But I could run
it in the 'hpcguix' node.
The first command:
------------------
[roel@hpc-submit1 ~]$ time guixr environment --ad-hoc coreutils --pure -- true
real 0m38.415s
user 0m6.075s
sys 0m0.611s
[roel@hpcguix ~]$ time guix environment --ad-hoc coreutils --pure -- true
real 0m27.054s
user 0m4.254s
sys 0m0.383s
The second command:
-------------------
[roel@hpcguix ~]$ time guix environment --ad-hoc -e '(@ (gnu packages base) coreutils)' --pure --no-substitutes --no-grafts -- true
The following derivations will be built:
/gnu/store/9wczighnyz1bz43j4wawf09z180g3ywv-profile.drv
/gnu/store/ffsyhajbdcp1lcq6x65czghya1iydly8-info-dir.drv
/gnu/store/5gyl3l23ps6f8dgay4awybwq7n9j9pzk-fonts-dir.drv
/gnu/store/l2mwj2q4vnq2v5raxz64ra7jyphd2jyd-manual-database.drv
Creating manual page database for 1 packages... done in 5.524 s
real 1m6.812s
user 0m2.969s
sys 0m0.325s
[roel@hpcguix ~]$ time guix environment --ad-hoc -e '(@ (gnu packages base) coreutils)' --pure --no-substitutes --no-grafts -- true
real 0m23.357s
user 0m2.802s
sys 0m0.340s
I suspect that the difference between the two commands is that one only
looks for one module, while the other looks in all modules. Looking at
the second run, I suppose the difference is quite small.
Kind regards,
Roel Janssen
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance on NFS
2017-06-07 13:06 ` Roel Janssen
@ 2017-06-09 13:46 ` Ludovic Courtès
2017-06-12 8:45 ` Roel Janssen
0 siblings, 1 reply; 37+ messages in thread
From: Ludovic Courtès @ 2017-06-09 13:46 UTC (permalink / raw)
To: Roel Janssen; +Cc: guix-devel, 27097
[-- Attachment #1: Type: text/plain, Size: 9697 bytes --]
Hi Roel,
Roel Janssen <roel@gnu.org> skribis:
> You should know that we have 'submit' nodes that use the guixr wrapper
> script to connect to the guix-daemon that runs on the 'hpcguix' node.
>
> Both have a /gnu mounted by a storage subsystem.
>
> I couldn't run the second command on a 'submit' node. But I could run
> it in the 'hpcguix' node.
OK.
Side note: I think you can replace your ‘guixr’ wrapper by just doing:
export GUIX_DAEMON_SOCKET=guix://hpcguix:1234
See <https://www.gnu.org/software/guix/manual/html_node/The-Store.html>.
> The first command:
> ------------------
>
> [roel@hpc-submit1 ~]$ time guixr environment --ad-hoc coreutils --pure -- true
>
> real 0m38.415s
> user 0m6.075s
> sys 0m0.611s
>
> [roel@hpcguix ~]$ time guix environment --ad-hoc coreutils --pure -- true
>
> real 0m27.054s
> user 0m4.254s
> sys 0m0.383s
>
>
> The second command:
> -------------------
>
> [roel@hpcguix ~]$ time guix environment --ad-hoc -e '(@ (gnu packages base) coreutils)' --pure --no-substitutes --no-grafts -- true
> The following derivations will be built:
> /gnu/store/9wczighnyz1bz43j4wawf09z180g3ywv-profile.drv
> /gnu/store/ffsyhajbdcp1lcq6x65czghya1iydly8-info-dir.drv
> /gnu/store/5gyl3l23ps6f8dgay4awybwq7n9j9pzk-fonts-dir.drv
> /gnu/store/l2mwj2q4vnq2v5raxz64ra7jyphd2jyd-manual-database.drv
> Creating manual page database for 1 packages... done in 5.524 s
>
> real 1m6.812s
> user 0m2.969s
> sys 0m0.325s
> [roel@hpcguix ~]$ time guix environment --ad-hoc -e '(@ (gnu packages base) coreutils)' --pure --no-substitutes --no-grafts -- true
>
> real 0m23.357s
> user 0m2.802s
> sys 0m0.340s
>
>
> I suspect that the difference between the two commands is that one only
> looks for one module, while the other looks in all modules. Looking at
> the second run, I suppose the difference is quite small.
Yeah, -e doesn’t seem to be much faster (there are still a lot of
modules to load anyway.)
At any rate, let’s see what we can do; 23 seconds is not okay.
I did a quick experiment:
--8<---------------cut here---------------start------------->8---
$ strace -o ,,s -s 123 guix environment --ad-hoc -e '(@ (gnu packages base) coreutils)' --pure --no-substitutes --no-grafts -- true
$ grep ^open ,,s |wc -l
1095
$ grep '^open.*ENOENT' ,,s |wc -l
136
$ grep -E '^(open|stat|lstat).*patches/gcc-arm-bug' ,,s |wc -l
27
$ grep -E '^(open|stat|lstat).*guix/build/utils' ,,s |wc -l
2190
--8<---------------cut here---------------end--------------->8---
After the patch below, I get:
--8<---------------cut here---------------start------------->8---
$ grep -E '^(open|stat|lstat).*guix/build/utils' ,,s2 |wc -l
14
$ grep -E '^(open|stat|lstat).*patches/gcc-arm-bug' ,,s2 |wc -l
4
--8<---------------cut here---------------end--------------->8---
Here’s the big picture before and after:
--8<---------------cut here---------------start------------->8---
$ strace -c guix environment --ad-hoc -e '(@ (gnu packages base) coreutils)' --pure --no-substitutes --no-grafts -- true
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
32.55 0.009781 1 10463 9158 stat
15.55 0.004673 1 8780 write
11.26 0.003385 3385 1 wait4
7.94 0.002387 20 122 12 futex
6.38 0.001917 0 5052 4 lstat
5.70 0.001713 2 1095 136 open
5.54 0.001664 1 2919 read
3.02 0.000909 2 525 mmap
2.96 0.000889 148 6 clone
2.50 0.000751 2 481 mprotect
2.00 0.000600 1 959 close
1.56 0.000469 1 898 3 lseek
1.10 0.000330 3 100 sendfile
0.88 0.000264 0 541 fstat
0.42 0.000127 1 175 brk
0.15 0.000044 2 22 rt_sigaction
0.09 0.000026 5 5 munmap
0.06 0.000019 1 18 rt_sigprocmask
0.06 0.000019 2 10 3 ioctl
0.03 0.000010 3 3 3 access
0.03 0.000010 2 6 1 fcntl
0.03 0.000009 0 23 clock_gettime
0.03 0.000008 2 5 prlimit64
0.02 0.000006 6 1 statfs
0.02 0.000005 2 3 pipe2
0.01 0.000004 2 2 getpid
0.01 0.000004 4 1 sysinfo
0.01 0.000003 3 1 dup2
0.01 0.000003 2 2 arch_prctl
0.01 0.000003 3 1 set_tid_address
0.01 0.000002 1 2 execve
0.01 0.000002 1 2 uname
0.01 0.000002 2 1 getuid
0.01 0.000002 2 1 getgid
0.01 0.000002 2 1 geteuid
0.01 0.000002 2 1 getegid
0.01 0.000002 2 1 getppid
0.01 0.000002 2 1 getpgrp
0.01 0.000002 2 1 set_robust_list
0.00 0.000001 0 16 readlink
0.00 0.000000 0 3 madvise
0.00 0.000000 0 1 socket
0.00 0.000000 0 1 connect
0.00 0.000000 0 1 getcwd
0.00 0.000000 0 2 sched_getaffinity
0.00 0.000000 0 1 getrandom
------ ----------- ----------- --------- --------- ----------------
100.00 0.030051 32256 9320 total
$ ./pre-inst-env strace -c guix environment --ad-hoc -e '(@ (gnu packages base) coreutils)' --pure --no-substitutes --no-grafts -- true
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
35.19 0.004311 0 9423 8117 stat
25.25 0.003094 0 8780 write
9.99 0.001224 0 2895 read
7.88 0.000965 1 1062 122 open
6.39 0.000783 7 117 10 futex
3.80 0.000466 1 502 mmap
3.34 0.000409 1 474 mprotect
2.91 0.000357 0 940 close
2.14 0.000262 0 900 3 lseek
1.90 0.000233 0 518 fstat
0.59 0.000072 1 100 sendfile
0.26 0.000032 0 123 lstat
0.20 0.000025 0 78 brk
0.04 0.000005 0 23 clock_gettime
0.02 0.000003 1 5 munmap
0.02 0.000003 1 6 clone
0.02 0.000002 0 5 rt_sigaction
0.02 0.000002 0 15 rt_sigprocmask
0.01 0.000001 1 1 wait4
0.01 0.000001 1 1 arch_prctl
0.01 0.000001 1 2 sched_getaffinity
0.01 0.000001 0 3 prlimit64
0.00 0.000000 0 7 1 ioctl
0.00 0.000000 0 2 2 access
0.00 0.000000 0 1 madvise
0.00 0.000000 0 1 socket
0.00 0.000000 0 1 connect
0.00 0.000000 0 1 execve
0.00 0.000000 0 1 uname
0.00 0.000000 0 3 fcntl
0.00 0.000000 0 1 getcwd
0.00 0.000000 0 1 statfs
0.00 0.000000 0 1 set_tid_address
0.00 0.000000 0 1 set_robust_list
0.00 0.000000 0 3 pipe2
0.00 0.000000 0 1 getrandom
------ ----------- ----------- --------- --------- ----------------
100.00 0.012252 25998 8255 total
--8<---------------cut here---------------end--------------->8---
So that’s shaving ~20% of the syscalls (notice the number of ‘lstat’
calls), which may make a difference on NFS.
Could you give it a try and report back?
This patch optimizes the ‘add-to-store’ on the client side. Until now
the caching strategy in ‘add-to-store’ was very conservative: at each
call it would ‘lstat’ the given file and use the stat as a key in
caching. Thus, if the file changed, we would notice and reopen it.
The obvious downside is that we’d keep doing ‘lstat’ for a case that’s
unlikely. With this patch, we simply use the file name as the key in
the ‘add-to-store’ cache.
Though as you wrote before, ‘open’ is much more expensive that ‘stat’ on
NFS, so that’s where we should focus.
Thanks,
Ludo’.
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 958 bytes --]
diff --git a/guix/store.scm b/guix/store.scm
index c94dfea95..8acde18d0 100644
--- a/guix/store.scm
+++ b/guix/store.scm
@@ -830,10 +830,9 @@ bits are kept. HASH-ALGO must be a string such as \"sha256\".
When RECURSIVE? is true, call (SELECT? FILE STAT) for each directory entry,
where FILE is the entry's absolute file name and STAT is the result of
'lstat'; exclude entries for which SELECT? does not return true."
- (let* ((st (false-if-exception (lstat file-name)))
- (args `(,st ,basename ,recursive? ,hash-algo ,select?))
+ (let* ((args `(,file-name ,basename ,recursive? ,hash-algo ,select?))
(cache (nix-server-add-to-store-cache server)))
- (or (and st (hash-ref cache args))
+ (or (hash-ref cache args)
(let ((path (add-to-store server basename recursive?
hash-algo file-name
#:select? select?)))
^ permalink raw reply related [flat|nested] 37+ messages in thread
* Re: Performance on NFS
2017-06-09 13:46 ` Ludovic Courtès
@ 2017-06-12 8:45 ` Roel Janssen
2017-06-12 15:58 ` Ludovic Courtès
0 siblings, 1 reply; 37+ messages in thread
From: Roel Janssen @ 2017-06-12 8:45 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: guix-devel, 27097
Ludovic Courtès writes:
> Hi Roel,
>
> Roel Janssen <roel@gnu.org> skribis:
>
>> You should know that we have 'submit' nodes that use the guixr wrapper
>> script to connect to the guix-daemon that runs on the 'hpcguix' node.
>>
>> Both have a /gnu mounted by a storage subsystem.
>>
>> I couldn't run the second command on a 'submit' node. But I could run
>> it in the 'hpcguix' node.
>
> OK.
>
> Side note: I think you can replace your ‘guixr’ wrapper by just doing:
>
> export GUIX_DAEMON_SOCKET=guix://hpcguix:1234
>
> See <https://www.gnu.org/software/guix/manual/html_node/The-Store.html>.
>
>> The first command:
>> ------------------
>>
>> [roel@hpc-submit1 ~]$ time guixr environment --ad-hoc coreutils --pure -- true
>>
>> real 0m38.415s
>> user 0m6.075s
>> sys 0m0.611s
>>
>> [roel@hpcguix ~]$ time guix environment --ad-hoc coreutils --pure -- true
>>
>> real 0m27.054s
>> user 0m4.254s
>> sys 0m0.383s
>>
>>
>> The second command:
>> -------------------
>>
>> [roel@hpcguix ~]$ time guix environment --ad-hoc -e '(@ (gnu packages base) coreutils)' --pure --no-substitutes --no-grafts -- true
>> The following derivations will be built:
>> /gnu/store/9wczighnyz1bz43j4wawf09z180g3ywv-profile.drv
>> /gnu/store/ffsyhajbdcp1lcq6x65czghya1iydly8-info-dir.drv
>> /gnu/store/5gyl3l23ps6f8dgay4awybwq7n9j9pzk-fonts-dir.drv
>> /gnu/store/l2mwj2q4vnq2v5raxz64ra7jyphd2jyd-manual-database.drv
>> Creating manual page database for 1 packages... done in 5.524 s
>>
>> real 1m6.812s
>> user 0m2.969s
>> sys 0m0.325s
>> [roel@hpcguix ~]$ time guix environment --ad-hoc -e '(@ (gnu packages base) coreutils)' --pure --no-substitutes --no-grafts -- true
>>
>> real 0m23.357s
>> user 0m2.802s
>> sys 0m0.340s
>>
>>
>> I suspect that the difference between the two commands is that one only
>> looks for one module, while the other looks in all modules. Looking at
>> the second run, I suppose the difference is quite small.
>
> Yeah, -e doesn’t seem to be much faster (there are still a lot of
> modules to load anyway.)
>
> At any rate, let’s see what we can do; 23 seconds is not okay.
>
> I did a quick experiment:
>
> --8<---------------cut here---------------start------------->8---
> $ strace -o ,,s -s 123 guix environment --ad-hoc -e '(@ (gnu packages base) coreutils)' --pure --no-substitutes --no-grafts -- true
> $ grep ^open ,,s |wc -l
> 1095
> $ grep '^open.*ENOENT' ,,s |wc -l
> 136
> $ grep -E '^(open|stat|lstat).*patches/gcc-arm-bug' ,,s |wc -l
> 27
> $ grep -E '^(open|stat|lstat).*guix/build/utils' ,,s |wc -l
> 2190
> --8<---------------cut here---------------end--------------->8---
>
> After the patch below, I get:
>
> --8<---------------cut here---------------start------------->8---
> $ grep -E '^(open|stat|lstat).*guix/build/utils' ,,s2 |wc -l
> 14
> $ grep -E '^(open|stat|lstat).*patches/gcc-arm-bug' ,,s2 |wc -l
> 4
> --8<---------------cut here---------------end--------------->8---
>
> Here’s the big picture before and after:
>
> --8<---------------cut here---------------start------------->8---
> $ strace -c guix environment --ad-hoc -e '(@ (gnu packages base) coreutils)' --pure --no-substitutes --no-grafts -- true
> % time seconds usecs/call calls errors syscall
> ------ ----------- ----------- --------- --------- ----------------
> 32.55 0.009781 1 10463 9158 stat
> 15.55 0.004673 1 8780 write
> 11.26 0.003385 3385 1 wait4
> 7.94 0.002387 20 122 12 futex
> 6.38 0.001917 0 5052 4 lstat
> 5.70 0.001713 2 1095 136 open
> 5.54 0.001664 1 2919 read
> 3.02 0.000909 2 525 mmap
> 2.96 0.000889 148 6 clone
> 2.50 0.000751 2 481 mprotect
> 2.00 0.000600 1 959 close
> 1.56 0.000469 1 898 3 lseek
> 1.10 0.000330 3 100 sendfile
> 0.88 0.000264 0 541 fstat
> 0.42 0.000127 1 175 brk
> 0.15 0.000044 2 22 rt_sigaction
> 0.09 0.000026 5 5 munmap
> 0.06 0.000019 1 18 rt_sigprocmask
> 0.06 0.000019 2 10 3 ioctl
> 0.03 0.000010 3 3 3 access
> 0.03 0.000010 2 6 1 fcntl
> 0.03 0.000009 0 23 clock_gettime
> 0.03 0.000008 2 5 prlimit64
> 0.02 0.000006 6 1 statfs
> 0.02 0.000005 2 3 pipe2
> 0.01 0.000004 2 2 getpid
> 0.01 0.000004 4 1 sysinfo
> 0.01 0.000003 3 1 dup2
> 0.01 0.000003 2 2 arch_prctl
> 0.01 0.000003 3 1 set_tid_address
> 0.01 0.000002 1 2 execve
> 0.01 0.000002 1 2 uname
> 0.01 0.000002 2 1 getuid
> 0.01 0.000002 2 1 getgid
> 0.01 0.000002 2 1 geteuid
> 0.01 0.000002 2 1 getegid
> 0.01 0.000002 2 1 getppid
> 0.01 0.000002 2 1 getpgrp
> 0.01 0.000002 2 1 set_robust_list
> 0.00 0.000001 0 16 readlink
> 0.00 0.000000 0 3 madvise
> 0.00 0.000000 0 1 socket
> 0.00 0.000000 0 1 connect
> 0.00 0.000000 0 1 getcwd
> 0.00 0.000000 0 2 sched_getaffinity
> 0.00 0.000000 0 1 getrandom
> ------ ----------- ----------- --------- --------- ----------------
> 100.00 0.030051 32256 9320 total
> $ ./pre-inst-env strace -c guix environment --ad-hoc -e '(@ (gnu packages base) coreutils)' --pure --no-substitutes --no-grafts -- true
> % time seconds usecs/call calls errors syscall
> ------ ----------- ----------- --------- --------- ----------------
> 35.19 0.004311 0 9423 8117 stat
> 25.25 0.003094 0 8780 write
> 9.99 0.001224 0 2895 read
> 7.88 0.000965 1 1062 122 open
> 6.39 0.000783 7 117 10 futex
> 3.80 0.000466 1 502 mmap
> 3.34 0.000409 1 474 mprotect
> 2.91 0.000357 0 940 close
> 2.14 0.000262 0 900 3 lseek
> 1.90 0.000233 0 518 fstat
> 0.59 0.000072 1 100 sendfile
> 0.26 0.000032 0 123 lstat
> 0.20 0.000025 0 78 brk
> 0.04 0.000005 0 23 clock_gettime
> 0.02 0.000003 1 5 munmap
> 0.02 0.000003 1 6 clone
> 0.02 0.000002 0 5 rt_sigaction
> 0.02 0.000002 0 15 rt_sigprocmask
> 0.01 0.000001 1 1 wait4
> 0.01 0.000001 1 1 arch_prctl
> 0.01 0.000001 1 2 sched_getaffinity
> 0.01 0.000001 0 3 prlimit64
> 0.00 0.000000 0 7 1 ioctl
> 0.00 0.000000 0 2 2 access
> 0.00 0.000000 0 1 madvise
> 0.00 0.000000 0 1 socket
> 0.00 0.000000 0 1 connect
> 0.00 0.000000 0 1 execve
> 0.00 0.000000 0 1 uname
> 0.00 0.000000 0 3 fcntl
> 0.00 0.000000 0 1 getcwd
> 0.00 0.000000 0 1 statfs
> 0.00 0.000000 0 1 set_tid_address
> 0.00 0.000000 0 1 set_robust_list
> 0.00 0.000000 0 3 pipe2
> 0.00 0.000000 0 1 getrandom
> ------ ----------- ----------- --------- --------- ----------------
> 100.00 0.012252 25998 8255 total
> --8<---------------cut here---------------end--------------->8---
>
> So that’s shaving ~20% of the syscalls (notice the number of ‘lstat’
> calls), which may make a difference on NFS.
>
> Could you give it a try and report back?
>
> This patch optimizes the ‘add-to-store’ on the client side. Until now
> the caching strategy in ‘add-to-store’ was very conservative: at each
> call it would ‘lstat’ the given file and use the stat as a key in
> caching. Thus, if the file changed, we would notice and reopen it.
>
> The obvious downside is that we’d keep doing ‘lstat’ for a case that’s
> unlikely. With this patch, we simply use the file name as the key in
> the ‘add-to-store’ cache.
>
> Though as you wrote before, ‘open’ is much more expensive that ‘stat’ on
> NFS, so that’s where we should focus.
>
> Thanks,
> Ludo’.
>
> diff --git a/guix/store.scm b/guix/store.scm
> index c94dfea95..8acde18d0 100644
> --- a/guix/store.scm
> +++ b/guix/store.scm
> @@ -830,10 +830,9 @@ bits are kept. HASH-ALGO must be a string such as \"sha256\".
> When RECURSIVE? is true, call (SELECT? FILE STAT) for each directory entry,
> where FILE is the entry's absolute file name and STAT is the result of
> 'lstat'; exclude entries for which SELECT? does not return true."
> - (let* ((st (false-if-exception (lstat file-name)))
> - (args `(,st ,basename ,recursive? ,hash-algo ,select?))
> + (let* ((args `(,file-name ,basename ,recursive? ,hash-algo ,select?))
> (cache (nix-server-add-to-store-cache server)))
> - (or (and st (hash-ref cache args))
> + (or (hash-ref cache args)
> (let ((path (add-to-store server basename recursive?
> hash-algo file-name
> #:select? select?)))
Here are the timings. The patched version is used with ./pre-inst-env,
and the unpatched version is not invoked using ./pre-inst-env.
With patch:
-----------------------------------------------------------------------------------------------------
[roel@hpc-submit1 guix]$ time ./pre-inst-env guix environment --pure coreutils --no-substitutes --no-grafts -- true
The following derivations will be built:
/gnu/store/zfyjc6zmhjj19vx4h6z22kjbz75izm2d-profile.drv
/gnu/store/p9nxlcyhvkiyfxllbb81m8jjnv1hfkhq-ca-certificate-bundle.drv
/gnu/store/any2b7k23b16xq4pbym21dx0sg6ybmyi-info-dir.drv
/gnu/store/55mmjkg47n58gql1yys053gl3vqfxlml-fonts-dir.drv
/gnu/store/s5vinzd4rc9yffigamb5k76r4ipqim6z-manual-database.drv
Creating manual page database for 19 packages... done in 26.220 s
warning: collision encountered: /gnu/store/ri56wnmzkgzrajdyl5ydc55lrwy1164k-ld-wrapper-0/bin/ld /gnu/store/zq65kpvwwxgc3qqbf9apic8gyss2l0zq-binutils-2.27/bin/ld
warning: arbitrarily choosing /gnu/store/ri56wnmzkgzrajdyl5ydc55lrwy1164k-ld-wrapper-0/bin/ld
real 6m14.571s
user 0m3.702s
sys 0m0.361s
[roel@hpc-submit1 guix]$ time ./pre-inst-env guix environment --pure coreutils --no-substitutes --no-grafts -- true
real 0m43.446s
user 0m3.393s
sys 0m0.452s
[roel@hpc-submit1 guix]$ time ./pre-inst-env guix environment --pure coreutils --no-substitutes --no-grafts -- true
real 0m22.405s
user 0m3.532s
sys 0m0.336s
[roel@hpc-submit1 guix]$ time ./pre-inst-env guix environment --pure coreutils --no-substitutes --no-grafts -- true
real 0m22.361s
user 0m3.356s
sys 0m0.326s
[roel@hpc-submit1 guix]$ time ./pre-inst-env guix environment --pure coreutils --no-substitutes --no-grafts -- true
real 0m17.993s
user 0m3.422s
sys 0m0.394s
Without patch:
-----------------------------------------------------------------------------------------------------
[roel@hpc-submit1 guix]$ time guix environment --pure coreutils --no-substitutes --no-grafts -- true
real 0m41.746s
user 0m3.471s
sys 0m0.552s
[roel@hpc-submit1 guix]$ time guix environment --pure coreutils --no-substitutes --no-grafts -- true
real 0m26.406s
user 0m3.451s
sys 0m0.523s
[roel@hpc-submit1 guix]$ time guix environment --pure coreutils --no-substitutes --no-grafts -- true
real 0m37.703s
user 0m3.445s
sys 0m0.579s
[roel@hpc-submit1 guix]$ time guix environment --pure coreutils --no-substitutes --no-grafts -- true
real 0m27.084s
user 0m3.538s
sys 0m0.428s
On hpcguix with patch:
-----------------------------------------------------------------------------------------------------
[roel@hpcguix guix]$ time ./pre-inst-env guix environment --pure coreutils --no-substitutes --no-grafts -- true
real 0m42.971s
user 0m3.335s
sys 0m0.386s
[roel@hpcguix guix]$ time ./pre-inst-env guix environment --pure coreutils --no-substitutes --no-grafts -- true
real 0m19.428s
user 0m3.239s
sys 0m0.367s
[roel@hpcguix guix]$ time ./pre-inst-env guix environment --pure coreutils --no-substitutes --no-grafts -- true
real 0m27.086s
user 0m3.073s
sys 0m0.348s
[roel@hpcguix guix]$ time ./pre-inst-env guix environment --pure coreutils --no-substitutes --no-grafts -- true
real 0m33.214s
user 0m3.068s
sys 0m0.296s
[roel@hpcguix guix]$ time ./pre-inst-env guix environment --pure coreutils --no-substitutes --no-grafts -- true
real 0m22.686s
user 0m3.010s
sys 0m0.305s
On hpcguix without patch:
-----------------------------------------------------------------------------------------------------
[roel@hpcguix guix]$ time guix environment --pure coreutils --no-substitutes --no-grafts -- true
real 0m28.502s
user 0m3.201s
sys 0m0.447s
[roel@hpcguix guix]$ time guix environment --pure coreutils --no-substitutes --no-grafts -- true
real 0m41.693s
user 0m3.061s
sys 0m0.301s
[roel@hpcguix guix]$ time guix environment --pure coreutils --no-substitutes --no-grafts -- true
real 0m26.178s
user 0m3.034s
sys 0m0.358s
From these timings, I don't think it has a big impact. This makes me
wonder, can't we replace the disk-intensive stuff with a database?
If we only have to read the files on disk once, after which we extracted
the information (the hashes?) needed to compute which links have to be
created to make an environment, then actually creating the environment
can be as fast as only creating those links.
Maybe this is too vague, because I don't know what Guix needs to read
exactly here.
Kind regards,
Roel Janssen
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance on NFS
2017-06-12 8:45 ` Roel Janssen
@ 2017-06-12 15:58 ` Ludovic Courtès
2017-06-16 15:23 ` Ludovic Courtès
0 siblings, 1 reply; 37+ messages in thread
From: Ludovic Courtès @ 2017-06-12 15:58 UTC (permalink / raw)
To: Roel Janssen; +Cc: guix-devel, 27097
Hi!
Roel Janssen <roel@gnu.org> skribis:
> From these timings, I don't think it has a big impact.
Wait, wait. If we take the best timings of each series of runs, we get:
hpc-submit1: 26.4s -> 18.0s (-30%)
hpc-guix: 26.2s -> 22.7s (-13%)
This is arguably insufficient but still non-negligible. I’ve committed
it as b46712159c15f72fc28b71d17d5a7c74fcb64ed0.
With commit 015f17e8b9eff97f656852180ac51c75438d7f9d, the number of
open(2) calls for that same command drops from 991 to 795 (including 122
errors). I suspect we can’t reduce it further:
--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env strace -o ,,s guix build coreutils -n
$ grep '^open.*' < ,,s |wc -l
795
$ grep '^open.*\.go"' < ,,s |wc -l
563
$ grep '^open.*\.patch"' < ,,s |wc -l
29
$ grep '^open.*\.scm"' < ,,s |wc -l
6
--8<---------------cut here---------------end--------------->8---
Could you check how this affects performance on your NFS system?
There’s possibly another low-hanging fruit, which is to disable file
name canonicalization (via ‘%file-port-name-canonicalization’.) It
special care though, so I’ll try that later.
> This makes me wonder, can't we replace the disk-intensive stuff with a
> database? If we only have to read the files on disk once, after which
> we extracted the information (the hashes?) needed to compute which
> links have to be created to make an environment, then actually
> creating the environment can be as fast as only creating those links.
Essentially we need to compute derivations as a function of local files
(sent to the daemon with ‘add-to-store’) and other derivations. We
cannot avoid that.
In the case of a remote server, communications with the daemon play an
important role too. Have you tried setting ‘GUIX_DAEMON_SOCKET’ as
suggested before instead of using the “socat hack”? I think this should
be faster (see commit 950d51c9d9a5107c5dac279da7d2e431134b5f43.)
HTH,
Ludo’.
^ permalink raw reply [flat|nested] 37+ messages in thread
* Performance on NFS
2017-06-12 15:58 ` Ludovic Courtès
@ 2017-06-16 15:23 ` Ludovic Courtès
2017-06-17 7:36 ` Roel Janssen
0 siblings, 1 reply; 37+ messages in thread
From: Ludovic Courtès @ 2017-06-16 15:23 UTC (permalink / raw)
To: Roel Janssen; +Cc: guix-devel
Hello!
I just pushed another improvement:
d27cc3bfaafe6b5b0831e88afb1c46311d382a0b significantly reduces the
number of ‘stat’ calls when discovering .scm files. I’d be curious to
see what impact it has on NFS.
Commit cbee955901b3e252ebdeb8066a2196055149198a also reduces the number
of ‘stat’ calls.
After that I have:
--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env strace -c guix environment --ad-hoc coreutils -- true
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
27.03 0.006916 1 13585 read
24.64 0.006306 1 9701 8029 stat
21.25 0.005438 1 10698 write
11.05 0.002828 2828 1 wait4
3.60 0.000922 1 1598 122 open
2.88 0.000738 4 203 28 futex
1.79 0.000458 1 670 mmap
1.75 0.000448 1 642 mprotect
1.64 0.000419 0 1476 close
1.36 0.000347 0 1411 3 lseek
0.74 0.000190 2 107 sendfile
0.70 0.000180 4 51 getdents
0.62 0.000159 0 886 fstat
0.54 0.000137 1 196 lstat
0.22 0.000057 1 99 brk
0.09 0.000024 4 6 clone
0.02 0.000005 1 5 munmap
0.02 0.000005 0 19 rt_sigprocmask
0.02 0.000004 0 27 clock_gettime
0.01 0.000003 3 1 getrandom
0.01 0.000002 2 1 connect
0.01 0.000002 1 3 prlimit64
0.00 0.000001 1 2 2 access
0.00 0.000001 1 1 socket
0.00 0.000000 0 5 rt_sigaction
0.00 0.000000 0 7 1 ioctl
0.00 0.000000 0 1 madvise
0.00 0.000000 0 1 execve
0.00 0.000000 0 1 uname
0.00 0.000000 0 3 fcntl
0.00 0.000000 0 1 getcwd
0.00 0.000000 0 1 statfs
0.00 0.000000 0 1 arch_prctl
0.00 0.000000 0 2 sched_getaffinity
0.00 0.000000 0 1 set_tid_address
0.00 0.000000 0 1 set_robust_list
0.00 0.000000 0 3 pipe2
------ ----------- ----------- --------- --------- ----------------
100.00 0.025590 41417 8185 total
$ git describe
v0.13.0-804-gf4453df9a
--8<---------------cut here---------------end--------------->8---
The number of ‘stat’ calls is a function of the number of entries in
‘GUILE_LOAD_PATH’ and ‘GUILE_LOAD_COMPILED_PATH’. I can’t think of any
easy way to reduce it further.
The many ‘read’ and ‘write’ are due to the unbuffered port used for
RPCs, and definitely not great when talking to a remote store. I’ve
tried to add some buffering but that turned out to be trickier than I
had hoped for.
To be continued!
Ludo’.
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance on NFS
2017-06-16 15:23 ` Ludovic Courtès
@ 2017-06-17 7:36 ` Roel Janssen
2017-06-17 22:40 ` Ludovic Courtès
0 siblings, 1 reply; 37+ messages in thread
From: Roel Janssen @ 2017-06-17 7:36 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: guix-devel
Hello!
Ludovic Courtès writes:
> Hello!
>
> I just pushed another improvement:
> d27cc3bfaafe6b5b0831e88afb1c46311d382a0b significantly reduces the
> number of ‘stat’ calls when discovering .scm files. I’d be curious to
> see what impact it has on NFS.
>
> Commit cbee955901b3e252ebdeb8066a2196055149198a also reduces the number
> of ‘stat’ calls.
>
> After that I have:
>
> --8<---------------cut here---------------start------------->8---
> $ ./pre-inst-env strace -c guix environment --ad-hoc coreutils -- true
> % time seconds usecs/call calls errors syscall
> ------ ----------- ----------- --------- --------- ----------------
> 27.03 0.006916 1 13585 read
> 24.64 0.006306 1 9701 8029 stat
> 21.25 0.005438 1 10698 write
> 11.05 0.002828 2828 1 wait4
> 3.60 0.000922 1 1598 122 open
> 2.88 0.000738 4 203 28 futex
> 1.79 0.000458 1 670 mmap
> 1.75 0.000448 1 642 mprotect
> 1.64 0.000419 0 1476 close
> 1.36 0.000347 0 1411 3 lseek
> 0.74 0.000190 2 107 sendfile
> 0.70 0.000180 4 51 getdents
> 0.62 0.000159 0 886 fstat
> 0.54 0.000137 1 196 lstat
> 0.22 0.000057 1 99 brk
> 0.09 0.000024 4 6 clone
> 0.02 0.000005 1 5 munmap
> 0.02 0.000005 0 19 rt_sigprocmask
> 0.02 0.000004 0 27 clock_gettime
> 0.01 0.000003 3 1 getrandom
> 0.01 0.000002 2 1 connect
> 0.01 0.000002 1 3 prlimit64
> 0.00 0.000001 1 2 2 access
> 0.00 0.000001 1 1 socket
> 0.00 0.000000 0 5 rt_sigaction
> 0.00 0.000000 0 7 1 ioctl
> 0.00 0.000000 0 1 madvise
> 0.00 0.000000 0 1 execve
> 0.00 0.000000 0 1 uname
> 0.00 0.000000 0 3 fcntl
> 0.00 0.000000 0 1 getcwd
> 0.00 0.000000 0 1 statfs
> 0.00 0.000000 0 1 arch_prctl
> 0.00 0.000000 0 2 sched_getaffinity
> 0.00 0.000000 0 1 set_tid_address
> 0.00 0.000000 0 1 set_robust_list
> 0.00 0.000000 0 3 pipe2
> ------ ----------- ----------- --------- --------- ----------------
> 100.00 0.025590 41417 8185 total
> $ git describe
> v0.13.0-804-gf4453df9a
> --8<---------------cut here---------------end--------------->8---
>
> The number of ‘stat’ calls is a function of the number of entries in
> ‘GUILE_LOAD_PATH’ and ‘GUILE_LOAD_COMPILED_PATH’. I can’t think of any
> easy way to reduce it further.
>
> The many ‘read’ and ‘write’ are due to the unbuffered port used for
> RPCs, and definitely not great when talking to a remote store. I’ve
> tried to add some buffering but that turned out to be trickier than I
> had hoped for.
>
> To be continued!
>
> Ludo’.
I applied the patch, and here are the results:
[roel@hpcguix guix]$ time guixr environment --ad-hoc coreutils --pure -- true
The following derivations will be built:
/gnu/store/0hz8g844432b5h9zbqr9cpsjy0brg15h-profile.drv
/gnu/store/wkksb7bbx3jr0p6p5cj4kkphbwday0yd-info-dir.drv
/gnu/store/cd2mwx9qprdy23p7j3pik2zs14nifn36-manual-database.drv
Creating manual page database for 1 packages... done in 1.816 s
real 1m14.686s
user 0m5.761s
sys 0m0.498s
[roel@hpcguix guix]$ time guixr environment --ad-hoc coreutils --pure -- true
real 0m34.100s
user 0m5.599s
sys 0m0.414s
[roel@hpcguix guix]$ time guixr environment --ad-hoc coreutils --pure -- true
real 0m33.821s
user 0m5.140s
sys 0m0.432s
[roel@hpcguix guix]$ time guixr environment --ad-hoc coreutils --pure --no-substitutes --no-grafts -- true
The following derivations will be built:
/gnu/store/rvh0imjdimwm90nzr0fmr5gmp97lyiix-profile.drv
/gnu/store/5hm3v4afjf9gix92ixqzv9bwc11a608s-fonts-dir.drv
real 0m37.200s
user 0m3.408s
sys 0m0.284s
[roel@hpcguix guix]$ time guixr environment --ad-hoc coreutils --pure --no-substitutes --no-grafts -- true
real 0m19.415s
user 0m3.466s
sys 0m0.306s
[roel@hpcguix guix]$ time guixr environment --ad-hoc coreutils --pure --no-substitutes --no-grafts -- true
real 0m18.850s
user 0m3.536s
sys 0m0.346s
[roel@hpcguix guix]$ time guixr environment --ad-hoc coreutils --pure --no-grafts -- true
real 0m16.003s
user 0m3.246s
sys 0m0.301s
[roel@hpcguix guix]$ time guixr environment --ad-hoc coreutils --pure --no-grafts -- true
real 0m18.205s
user 0m3.470s
sys 0m0.314s
[roel@hpcguix guix]$ time guixr environment --ad-hoc coreutils --pure --no-substitutes -- true
real 0m33.731s
user 0m5.111s
sys 0m0.428s
[roel@hpcguix guix]$ time guixr environment --ad-hoc coreutils --pure --no-substitutes -- true
real 0m30.993s
user 0m5.049s
sys 0m0.458s
Why is grafting so slow, even if it doesn't have to graft anything?
So, because grafting is disk-intensive rather than CPU-intensive, it might
be a good idea to be able to globally disable grafting. (it would
reduce the after-we-build-it-time considerably for our cluster.)
Kind regards,
Roel Janssen
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance on NFS
2017-06-17 7:36 ` Roel Janssen
@ 2017-06-17 22:40 ` Ludovic Courtès
2017-06-17 23:15 ` Roel Janssen
2017-06-18 8:43 ` Ricardo Wurmus
0 siblings, 2 replies; 37+ messages in thread
From: Ludovic Courtès @ 2017-06-17 22:40 UTC (permalink / raw)
To: Roel Janssen; +Cc: guix-devel
Hi!
Roel Janssen <roel@gnu.org> skribis:
> I applied the patch, and here are the results:
>
> [roel@hpcguix guix]$ time guixr environment --ad-hoc coreutils --pure -- true
> The following derivations will be built:
> /gnu/store/0hz8g844432b5h9zbqr9cpsjy0brg15h-profile.drv
> /gnu/store/wkksb7bbx3jr0p6p5cj4kkphbwday0yd-info-dir.drv
> /gnu/store/cd2mwx9qprdy23p7j3pik2zs14nifn36-manual-database.drv
> Creating manual page database for 1 packages... done in 1.816 s
>
> real 1m14.686s
> user 0m5.761s
> sys 0m0.498s
> [roel@hpcguix guix]$ time guixr environment --ad-hoc coreutils --pure -- true
>
> real 0m34.100s
> user 0m5.599s
> sys 0m0.414s
> [roel@hpcguix guix]$ time guixr environment --ad-hoc coreutils --pure -- true
>
> real 0m33.821s
> user 0m5.140s
> sys 0m0.432s
You’re telling me it’s just as bad as before, right?
> [roel@hpcguix guix]$ time guixr environment --ad-hoc coreutils --pure --no-substitutes --no-grafts -- true
> The following derivations will be built:
> /gnu/store/rvh0imjdimwm90nzr0fmr5gmp97lyiix-profile.drv
> /gnu/store/5hm3v4afjf9gix92ixqzv9bwc11a608s-fonts-dir.drv
>
> real 0m37.200s
> user 0m3.408s
> sys 0m0.284s
> [roel@hpcguix guix]$ time guixr environment --ad-hoc coreutils --pure --no-substitutes --no-grafts -- true
>
> real 0m19.415s
> user 0m3.466s
> sys 0m0.306s
> [roel@hpcguix guix]$ time guixr environment --ad-hoc coreutils --pure --no-substitutes --no-grafts -- true
>
> real 0m18.850s
> user 0m3.536s
> sys 0m0.346s
> [roel@hpcguix guix]$ time guixr environment --ad-hoc coreutils --pure --no-grafts -- true
>
> real 0m16.003s
> user 0m3.246s
> sys 0m0.301s
> [roel@hpcguix guix]$ time guixr environment --ad-hoc coreutils --pure --no-grafts -- true
>
> real 0m18.205s
> user 0m3.470s
> sys 0m0.314s
> [roel@hpcguix guix]$ time guixr environment --ad-hoc coreutils --pure --no-substitutes -- true
>
> real 0m33.731s
> user 0m5.111s
> sys 0m0.428s
> [roel@hpcguix guix]$ time guixr environment --ad-hoc coreutils --pure --no-substitutes -- true
>
> real 0m30.993s
> user 0m5.049s
> sys 0m0.458s
>
> Why is grafting so slow, even if it doesn't have to graft anything?
Grafting leads to a bunch of additional RPCs:
--8<---------------cut here---------------start------------->8---
$ GUIX_PROFILING=rpc ./pre-inst-env guix build coreutils
/gnu/store/mskh7zisxa313anqv68c5lr4hajldjc5-coreutils-8.27-debug
/gnu/store/xbvwxf4k5njnb3hn93xwqlppjkiz4hdv-coreutils-8.27
Remote procedure call summary: 379 RPCs
build-things ... 1
built-in-builders ... 1
valid-path? ... 5
query-substitutable-path-infos ... 8
query-references ... 22
query-valid-derivers ... 48
add-text-to-store ... 294
$ GUIX_PROFILING=rpc ./pre-inst-env guix build coreutils --no-grafts
/gnu/store/mskh7zisxa313anqv68c5lr4hajldjc5-coreutils-8.27-debug
/gnu/store/xbvwxf4k5njnb3hn93xwqlppjkiz4hdv-coreutils-8.27
Remote procedure call summary: 294 RPCs
built-in-builders ... 1
query-substitutable-path-infos ... 1
build-things ... 1
valid-path? ... 5
add-text-to-store ... 286
--8<---------------cut here---------------end--------------->8---
So the problem is probably not NFS in this case but rather RPC
performance.
However, I can’t help with this until you drop ‘guixr’ and use
GUIX_DAEMON_SOCKET=guix:// instead. Hint hint. ;-)
Thanks for your feedback,
Ludo’.
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance on NFS
2017-06-17 22:40 ` Ludovic Courtès
@ 2017-06-17 23:15 ` Roel Janssen
2017-06-18 8:43 ` Ricardo Wurmus
1 sibling, 0 replies; 37+ messages in thread
From: Roel Janssen @ 2017-06-17 23:15 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: guix-devel
Ludovic Courtès writes:
> Hi!
>
> Roel Janssen <roel@gnu.org> skribis:
>
>> I applied the patch, and here are the results:
>>
>> [roel@hpcguix guix]$ time guixr environment --ad-hoc coreutils --pure -- true
>> The following derivations will be built:
>> /gnu/store/0hz8g844432b5h9zbqr9cpsjy0brg15h-profile.drv
>> /gnu/store/wkksb7bbx3jr0p6p5cj4kkphbwday0yd-info-dir.drv
>> /gnu/store/cd2mwx9qprdy23p7j3pik2zs14nifn36-manual-database.drv
>> Creating manual page database for 1 packages... done in 1.816 s
>>
>> real 1m14.686s
>> user 0m5.761s
>> sys 0m0.498s
>> [roel@hpcguix guix]$ time guixr environment --ad-hoc coreutils --pure -- true
>>
>> real 0m34.100s
>> user 0m5.599s
>> sys 0m0.414s
>> [roel@hpcguix guix]$ time guixr environment --ad-hoc coreutils --pure -- true
>>
>> real 0m33.821s
>> user 0m5.140s
>> sys 0m0.432s
>
> You’re telling me it’s just as bad as before, right?
Sorry for the somewhat hasted response.
Well, before it was more variable what the speed was. Now it seems to
be pretty stable around ~30 to ~35 seconds with grafting, and ~15 to ~20
seconds without grafting.
I really appreciate the effort for optimizing. And I feel it is
improving.
>
>> [roel@hpcguix guix]$ time guixr environment --ad-hoc coreutils --pure --no-substitutes --no-grafts -- true
>> The following derivations will be built:
>> /gnu/store/rvh0imjdimwm90nzr0fmr5gmp97lyiix-profile.drv
>> /gnu/store/5hm3v4afjf9gix92ixqzv9bwc11a608s-fonts-dir.drv
>>
>> real 0m37.200s
>> user 0m3.408s
>> sys 0m0.284s
>> [roel@hpcguix guix]$ time guixr environment --ad-hoc coreutils --pure --no-substitutes --no-grafts -- true
>>
>> real 0m19.415s
>> user 0m3.466s
>> sys 0m0.306s
>> [roel@hpcguix guix]$ time guixr environment --ad-hoc coreutils --pure --no-substitutes --no-grafts -- true
>>
>> real 0m18.850s
>> user 0m3.536s
>> sys 0m0.346s
>> [roel@hpcguix guix]$ time guixr environment --ad-hoc coreutils --pure --no-grafts -- true
>>
>> real 0m16.003s
>> user 0m3.246s
>> sys 0m0.301s
>> [roel@hpcguix guix]$ time guixr environment --ad-hoc coreutils --pure --no-grafts -- true
>>
>> real 0m18.205s
>> user 0m3.470s
>> sys 0m0.314s
>> [roel@hpcguix guix]$ time guixr environment --ad-hoc coreutils --pure --no-substitutes -- true
>>
>> real 0m33.731s
>> user 0m5.111s
>> sys 0m0.428s
>> [roel@hpcguix guix]$ time guixr environment --ad-hoc coreutils --pure --no-substitutes -- true
>>
>> real 0m30.993s
>> user 0m5.049s
>> sys 0m0.458s
>>
>> Why is grafting so slow, even if it doesn't have to graft anything?
>
> Grafting leads to a bunch of additional RPCs:
>
> --8<---------------cut here---------------start------------->8---
> $ GUIX_PROFILING=rpc ./pre-inst-env guix build coreutils
> /gnu/store/mskh7zisxa313anqv68c5lr4hajldjc5-coreutils-8.27-debug
> /gnu/store/xbvwxf4k5njnb3hn93xwqlppjkiz4hdv-coreutils-8.27
> Remote procedure call summary: 379 RPCs
> build-things ... 1
> built-in-builders ... 1
> valid-path? ... 5
> query-substitutable-path-infos ... 8
> query-references ... 22
> query-valid-derivers ... 48
> add-text-to-store ... 294
> $ GUIX_PROFILING=rpc ./pre-inst-env guix build coreutils --no-grafts
> /gnu/store/mskh7zisxa313anqv68c5lr4hajldjc5-coreutils-8.27-debug
> /gnu/store/xbvwxf4k5njnb3hn93xwqlppjkiz4hdv-coreutils-8.27
> Remote procedure call summary: 294 RPCs
> built-in-builders ... 1
> query-substitutable-path-infos ... 1
> build-things ... 1
> valid-path? ... 5
> add-text-to-store ... 286
> --8<---------------cut here---------------end--------------->8---
>
> So the problem is probably not NFS in this case but rather RPC
> performance.
>
> However, I can’t help with this until you drop ‘guixr’ and use
> GUIX_DAEMON_SOCKET=guix:// instead. Hint hint. ;-)
This is what guixr is already doing, see:
https://github.com/UMCUGenetics/guix-additions/blob/master/umcu/packages/guix.scm#L95
So I went a little bit further and did this:
[roel@hpcguix ~]$ export GUIX_DAEMON_SOCKET="/gnu/daemon-socket/socket"
[roel@hpcguix ~]$ export NIX_STATE_DIR=/gnu
This means that if I run "guix" on the same machine as where guix-daemon
is running, and communicate over the UNIX socket, I should not
experience a performance problem, other than a little bit of NFS
latency..
Here are the timings:
[roel@hpcguix ~]$ time guix environment --ad-hoc coreutils --pure --no-grafts -- true
real 0m16.293s
user 0m2.550s
sys 0m0.274s
[roel@hpcguix ~]$ time guix environment --ad-hoc coreutils --pure --no-grafts -- true
real 0m15.746s
user 0m2.411s
sys 0m0.260s
[roel@hpcguix ~]$ time guix environment --ad-hoc coreutils --pure -- true
real 0m32.821s
user 0m4.342s
sys 0m0.375s
[roel@hpcguix ~]$ time guix environment --ad-hoc coreutils --pure -- true
real 0m31.140s
user 0m4.361s
sys 0m0.312s
Looks pretty much the same. Now, the only thing that can be trouble,
other than NFS's overhead, is that the daemon socket is on the NFS mount
as well. I don't think that matters because the socket is not an actual
file from the kernel's perspective, so there shouldn't be any NFS
involvement there. Nevertheless, I will try mounting the
/gnu/daemon-socket directory on a tmpfs (in memory) for maximum speed.
There's "nfsstat" that can display some NFS numbers. Here's what the
client information looks like (run on hpcguix):
[roel@hpcguix ~]$ nfsstat -c
Client rpc stats:
calls retrans authrefrsh
21090135 0 21097679
Client nfs v3:
null getattr setattr lookup access readlink
0 0% 17580080 83% 500076 2% 842458 3% 930443 4% 1142 0%
read write create mkdir symlink mknod
153231 0% 313668 1% 232301 1% 60846 0% 10604 0% 80 0%
remove rmdir rename link readdir readdirplus
181436 0% 53549 0% 74060 0% 75984 0% 278 0% 76912 0%
fsstat fsinfo pathconf commit
1402 0% 6 0% 3 0% 2481 0%
The high number of getattr is probably due to the attribute cache. I
also tried disabling the attribute cache with the "noac" mount option.
This lead to timings of (stably) around 2 minutes (with grafts enabled).
What worries me from these numbers is the authrefrsh, which are almost
equal to the number of calls. What I think it means is that for almost
every call, it also checks whether the client is still authorized to get
data from the NFS server, effectively doubling the network packets,
doubling the network overhead. What I can try here is disable
authentication. I am not sure whether that is a good solution, but at
least we can get some performance impact numbers here.
I will report back with my findings here.
>
> Thanks for your feedback,
> Ludo’.
Thanks a lot for your efforts in speeding things up!
Kind regards,
Roel Janssen
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Performance on NFS
2017-06-17 22:40 ` Ludovic Courtès
2017-06-17 23:15 ` Roel Janssen
@ 2017-06-18 8:43 ` Ricardo Wurmus
2017-06-19 8:01 ` RPC performance Ludovic Courtès
1 sibling, 1 reply; 37+ messages in thread
From: Ricardo Wurmus @ 2017-06-18 8:43 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: guix-devel
Ludovic Courtès <ludovic.courtes@inria.fr> writes:
[…]
> So the problem is probably not NFS in this case but rather RPC
> performance.
>
> However, I can’t help with this until you drop ‘guixr’ and use
> GUIX_DAEMON_SOCKET=guix:// instead. Hint hint. ;-)
FWIW: I’m using GUIX_DAEMON_SOCKET=guix:// since a few days. It allows
for a simpler “guixr”, which now only prints extra info and refuses to
run on the login node. It’s really nice!
I haven’t noticed any difference in performance, but I certainly could
test patches.
Performance differs a lot for different packages. “guix build bwa” (~ 7
seconds) is a lot faster than “guix build python2-numpy” (> 40 seconds),
for example.
--
Ricardo
GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC
https://elephly.net
^ permalink raw reply [flat|nested] 37+ messages in thread
* RPC performance
2017-06-18 8:43 ` Ricardo Wurmus
@ 2017-06-19 8:01 ` Ludovic Courtès
2017-06-19 8:15 ` Ludovic Courtès
2017-06-19 21:25 ` Ludovic Courtès
0 siblings, 2 replies; 37+ messages in thread
From: Ludovic Courtès @ 2017-06-19 8:01 UTC (permalink / raw)
To: Ricardo Wurmus; +Cc: guix-devel
Hello,
Ricardo Wurmus <rekado@elephly.net> skribis:
> Performance differs a lot for different packages. “guix build bwa” (~ 7
> seconds) is a lot faster than “guix build python2-numpy” (> 40 seconds),
> for example.
My guess is that this is mostly due to RPC performance:
--8<---------------cut here---------------start------------->8---
$ GUIX_PROFILING=rpc time ./pre-inst-env guix build bwa -n
Remote procedure call summary: 266 RPCs
built-in-builders ... 1
query-substitutable-path-infos ... 1
valid-path? ... 3
add-text-to-store ... 261
0.88user 0.04system 0:00.83elapsed 110%CPU (0avgtext+0avgdata 497792maxresident)k
0inputs+0outputs (0major+11771minor)pagefaults 0swaps
$ GUIX_PROFILING=rpc time ./pre-inst-env guix build python2-numpy -n
Remote procedure call summary: 668 RPCs
built-in-builders ... 1
query-substitutable-path-infos ... 1
valid-path? ... 3
add-text-to-store ... 663
1.22user 0.06system 0:01.33elapsed 95%CPU (0avgtext+0avgdata 508288maxresident)k
1432inputs+0outputs (0major+10752minor)pagefaults 0swaps
--8<---------------cut here---------------end--------------->8---
There are several ways we can improve it, and over time we should try to
implement all of them:
1. Buffer writes to the server socket (currently it’s terrible if you
look at the ‘write’ calls in ‘strace’).
2. Make fewer ‘add-text-to-store’ calls. This can be done by, for
instance, always using the same ‘-guile-builder’ script for
everything that uses ‘gnu-build-system’, and parameterize the
script through environment variables or similar.
3. Replace N ‘add-text-to-store’ calls with a single ‘import-paths’
call. It’s not possible yet because ‘import-paths’ requires a
signature, which is pointless for content-address items.
4. Alternatively, change the protocol to allow for RPC pipelining
(send N requests, then read N responses). Currently this isn’t
possible because all the RPCs call ‘process-stderr’ (which makes
one or more round trips) because reading their response.
I’ll look into #1 soon and we can investigate #2 on the next
‘core-updates’ cycle.
Thanks,
Ludo’.
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: RPC performance
2017-06-19 8:01 ` RPC performance Ludovic Courtès
@ 2017-06-19 8:15 ` Ludovic Courtès
2017-06-19 14:25 ` Ricardo Wurmus
2017-06-22 14:03 ` Andy Wingo
2017-06-19 21:25 ` Ludovic Courtès
1 sibling, 2 replies; 37+ messages in thread
From: Ludovic Courtès @ 2017-06-19 8:15 UTC (permalink / raw)
To: Ricardo Wurmus; +Cc: guix-devel
[-- Attachment #1: Type: text/plain, Size: 649 bytes --]
ludovic.courtes@inria.fr (Ludovic Courtès) skribis:
> There are several ways we can improve it, and over time we should try to
> implement all of them:
>
> 1. Buffer writes to the server socket (currently it’s terrible if you
> look at the ‘write’ calls in ‘strace’).
Could you test the effect of the patch below? It reduces the number of
‘write’ calls from 9.5K to 2.0K on “guix build python2-numpy”.
(It’s not acceptable as-is because it allocates a new port and
associated buffer at each RPC, meaning that on my laptop the cost in
user time slightly outweighs the cost in system time.)
Ludo’.
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 3457 bytes --]
diff --git a/guix/store.scm b/guix/store.scm
index 2acab6b1a..47fa3447f 100644
--- a/guix/store.scm
+++ b/guix/store.scm
@@ -321,13 +321,15 @@
;; remote-store.cc
(define-record-type <nix-server>
- (%make-nix-server socket major minor
+ (%make-nix-server socket major minor buffer
ats-cache atts-cache)
nix-server?
(socket nix-server-socket)
(major nix-server-major-version)
(minor nix-server-minor-version)
+ (buffer nix-server-output-buffer)
+
;; Caches. We keep them per-connection, because store paths build
;; during the session are temporary GC roots kept for the duration of
;; the session.
@@ -499,6 +501,7 @@ for this connection will be pinned. Return a server object."
(let ((conn (%make-nix-server port
(protocol-major v)
(protocol-minor v)
+ (make-bytevector 8192)
(make-hash-table 100)
(make-hash-table 100))))
(let loop ((done? (process-stderr conn)))
@@ -718,6 +721,41 @@ encoding conversion errors."
(let loop ((done? (process-stderr server)))
(or done? (process-stderr server)))))
+(define (buffering-output-port port buffer)
+ ;; Note: In Guile 2.2.2, custom binary output ports already have their own
+ ;; 4K internal buffer.
+ (define size
+ (bytevector-length buffer))
+
+ (define total 0)
+
+ (define (flush)
+ (put-bytevector port buffer 0 total)
+ (set! total 0))
+
+ (define (write bv offset count)
+ (if (zero? count) ;end of file
+ (flush)
+ (let loop ((offset offset)
+ (count count)
+ (written 0))
+ (cond ((= total size)
+ (flush)
+ (loop offset count written))
+ ((zero? count)
+ written)
+ (else
+ (let ((to-copy (min count (- size total))))
+ (bytevector-copy! bv offset buffer total to-copy)
+ (set! total (+ total to-copy))
+ (loop (+ offset to-copy) (- count to-copy)
+ (+ written to-copy))))))))
+
+ (let ((port (make-custom-binary-output-port "buffering-output-port"
+ write #f #f flush)))
+ (setvbuf port _IONBF)
+ port))
+
(define %rpc-calls
;; Mapping from RPC names (symbols) to invocation counts.
(make-hash-table))
@@ -755,11 +793,15 @@ encoding conversion errors."
((_ (name (type arg) ...) docstring return ...)
(lambda (server arg ...)
docstring
- (let ((s (nix-server-socket server)))
+ (let* ((s (nix-server-socket server))
+ (buffered (buffering-output-port
+ s (nix-server-output-buffer server))))
(record-operation 'name)
- (write-int (operation-id name) s)
- (write-arg type arg s)
+ (write-int (operation-id name) buffered)
+ (write-arg type arg buffered)
...
+ (close-port buffered)
+
;; Loop until the server is done sending error output.
(let loop ((done? (process-stderr server)))
(or done? (loop (process-stderr server))))
^ permalink raw reply related [flat|nested] 37+ messages in thread
* Re: RPC performance
2017-06-19 8:15 ` Ludovic Courtès
@ 2017-06-19 14:25 ` Ricardo Wurmus
2017-06-22 14:03 ` Andy Wingo
1 sibling, 0 replies; 37+ messages in thread
From: Ricardo Wurmus @ 2017-06-19 14:25 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: guix-devel
Ludovic Courtès <ludovic.courtes@inria.fr> writes:
> ludovic.courtes@inria.fr (Ludovic Courtès) skribis:
>
>> There are several ways we can improve it, and over time we should try to
>> implement all of them:
>>
>> 1. Buffer writes to the server socket (currently it’s terrible if you
>> look at the ‘write’ calls in ‘strace’).
>
> Could you test the effect of the patch below? It reduces the number of
> ‘write’ calls from 9.5K to 2.0K on “guix build python2-numpy”.
I have tried it but my setup is way too variable to allow me to compare
performance changes. For “guix build bwa” when “bwa” has already been
built I get waiting times between 6 and 42 seconds. Any improvement
would disappear in this terribly wide range.
I will soon have access to a new NFS share on a separate server and with
disabled read timestamps, which I hope will be more reliable.
--
Ricardo
GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC
https://elephly.net
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: RPC performance
2017-06-19 8:01 ` RPC performance Ludovic Courtès
2017-06-19 8:15 ` Ludovic Courtès
@ 2017-06-19 21:25 ` Ludovic Courtès
2017-06-22 8:04 ` Ricardo Wurmus
1 sibling, 1 reply; 37+ messages in thread
From: Ludovic Courtès @ 2017-06-19 21:25 UTC (permalink / raw)
To: Ricardo Wurmus; +Cc: guix-devel
ludovic.courtes@inria.fr (Ludovic Courtès) skribis:
> There are several ways we can improve it, and over time we should try to
> implement all of them:
>
> 1. Buffer writes to the server socket (currently it’s terrible if you
> look at the ‘write’ calls in ‘strace’).
Done in commit e037e9dbec1ab5a8cfaf65d73aa3afb2eeb98d71. It doesn’t
significantly improve performance, probably because the other problems
are more acute, but it’s good to have.
Ludo’.
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: RPC performance
2017-06-19 21:25 ` Ludovic Courtès
@ 2017-06-22 8:04 ` Ricardo Wurmus
0 siblings, 0 replies; 37+ messages in thread
From: Ricardo Wurmus @ 2017-06-22 8:04 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: guix-devel
Ludovic Courtès <ludo@gnu.org> writes:
> ludovic.courtes@inria.fr (Ludovic Courtès) skribis:
>
>> There are several ways we can improve it, and over time we should try to
>> implement all of them:
>>
>> 1. Buffer writes to the server socket (currently it’s terrible if you
>> look at the ‘write’ calls in ‘strace’).
>
> Done in commit e037e9dbec1ab5a8cfaf65d73aa3afb2eeb98d71. It doesn’t
> significantly improve performance, probably because the other problems
> are more acute, but it’s good to have.
Thanks! I’m looking forward to getting the new NFS server ready and
benefiting from these improvemets.
--
Ricardo
GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC
https://elephly.net
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: RPC performance
2017-06-19 8:15 ` Ludovic Courtès
2017-06-19 14:25 ` Ricardo Wurmus
@ 2017-06-22 14:03 ` Andy Wingo
2017-06-22 16:05 ` Ludovic Courtès
1 sibling, 1 reply; 37+ messages in thread
From: Andy Wingo @ 2017-06-22 14:03 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: guix-devel
On Mon 19 Jun 2017 10:15, ludovic.courtes@inria.fr (Ludovic Courtès) writes:
> +(define (buffering-output-port port buffer)
> + ;; Note: In Guile 2.2.2, custom binary output ports already have their own
> + ;; 4K internal buffer.
> + (define size
> + (bytevector-length buffer))
> +
> + (define total 0)
> +
> + (define (flush)
> + (put-bytevector port buffer 0 total)
> + (set! total 0))
> +
> + (define (write bv offset count)
> + (if (zero? count) ;end of file
> + (flush)
> + (let loop ((offset offset)
> + (count count)
> + (written 0))
> + (cond ((= total size)
> + (flush)
> + (loop offset count written))
> + ((zero? count)
> + written)
> + (else
> + (let ((to-copy (min count (- size total))))
> + (bytevector-copy! bv offset buffer total to-copy)
> + (set! total (+ total to-copy))
> + (loop (+ offset to-copy) (- count to-copy)
> + (+ written to-copy))))))))
> +
> + (let ((port (make-custom-binary-output-port "buffering-output-port"
> + write #f #f flush)))
> + (setvbuf port _IONBF)
> + port))
> +
Why not just set to _IOFBF and let Guile 2.2's buffering handle it?
Andy
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: RPC performance
2017-06-22 14:03 ` Andy Wingo
@ 2017-06-22 16:05 ` Ludovic Courtès
2017-06-23 9:09 ` Andy Wingo
0 siblings, 1 reply; 37+ messages in thread
From: Ludovic Courtès @ 2017-06-22 16:05 UTC (permalink / raw)
To: Andy Wingo; +Cc: guix-devel
Heya,
Andy Wingo <wingo@igalia.com> skribis:
> On Mon 19 Jun 2017 10:15, ludovic.courtes@inria.fr (Ludovic Courtès) writes:
>
>> +(define (buffering-output-port port buffer)
>> + ;; Note: In Guile 2.2.2, custom binary output ports already have their own
>> + ;; 4K internal buffer.
>> + (define size
>> + (bytevector-length buffer))
>> +
>> + (define total 0)
>> +
>> + (define (flush)
>> + (put-bytevector port buffer 0 total)
>> + (set! total 0))
>> +
>> + (define (write bv offset count)
>> + (if (zero? count) ;end of file
>> + (flush)
>> + (let loop ((offset offset)
>> + (count count)
>> + (written 0))
>> + (cond ((= total size)
>> + (flush)
>> + (loop offset count written))
>> + ((zero? count)
>> + written)
>> + (else
>> + (let ((to-copy (min count (- size total))))
>> + (bytevector-copy! bv offset buffer total to-copy)
>> + (set! total (+ total to-copy))
>> + (loop (+ offset to-copy) (- count to-copy)
>> + (+ written to-copy))))))))
>> +
>> + (let ((port (make-custom-binary-output-port "buffering-output-port"
>> + write #f #f flush)))
>> + (setvbuf port _IONBF)
>> + port))
>> +
>
> Why not just set to _IOFBF and let Guile 2.2's buffering handle it?
Because we want controlled buffering when writing (we need to flush
pending output when we’re done writing the RPC request), and no
buffering at all when reading.
In C/C++ the way to do that is to have unbuffered streams and to do
application-level buffering by allocating output buffers of the right
size.
Thoughts?
Ludo’.
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: RPC performance
2017-06-22 16:05 ` Ludovic Courtès
@ 2017-06-23 9:09 ` Andy Wingo
2017-06-23 9:24 ` Ludovic Courtès
0 siblings, 1 reply; 37+ messages in thread
From: Andy Wingo @ 2017-06-23 9:09 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: guix-devel
On Thu 22 Jun 2017 18:05, ludovic.courtes@inria.fr (Ludovic Courtès) writes:
> Andy Wingo <wingo@igalia.com> skribis:
>
>> Why not just set to _IOFBF and let Guile 2.2's buffering handle it?
>
> Because we want controlled buffering when writing (we need to flush
> pending output when we’re done writing the RPC request), and no
> buffering at all when reading.
For controlling output buffering, there is the setvbuf buffer size, and
"force-output". In Guile 2.2 the CBOP's "write" function is really a
"flush" function -- it only gets called when the internal buffer is
filled, or when flush-output is called, or (for line-buffered ports)
when a newline is written.
Why do you not want buffering when reading? Do you need to hand off
this FD to some other process?
Andy
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: RPC performance
2017-06-23 9:09 ` Andy Wingo
@ 2017-06-23 9:24 ` Ludovic Courtès
2017-06-23 9:46 ` Andy Wingo
0 siblings, 1 reply; 37+ messages in thread
From: Ludovic Courtès @ 2017-06-23 9:24 UTC (permalink / raw)
To: Andy Wingo; +Cc: guix-devel
Hi Andy,
Andy Wingo <wingo@igalia.com> skribis:
> On Thu 22 Jun 2017 18:05, ludovic.courtes@inria.fr (Ludovic Courtès) writes:
>
>> Andy Wingo <wingo@igalia.com> skribis:
>>
>>> Why not just set to _IOFBF and let Guile 2.2's buffering handle it?
>>
>> Because we want controlled buffering when writing (we need to flush
>> pending output when we’re done writing the RPC request), and no
>> buffering at all when reading.
>
> For controlling output buffering, there is the setvbuf buffer size, and
> "force-output". In Guile 2.2 the CBOP's "write" function is really a
> "flush" function -- it only gets called when the internal buffer is
> filled, or when flush-output is called, or (for line-buffered ports)
> when a newline is written.
>
> Why do you not want buffering when reading? Do you need to hand off
> this FD to some other process?
With the current protocol, often we’re just reading a handful of bytes.
Full buffering would mean that Guile would block on an 8K read or so
that will never be fulfilled.
Ludo’.
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: RPC performance
2017-06-23 9:24 ` Ludovic Courtès
@ 2017-06-23 9:46 ` Andy Wingo
2017-06-26 11:54 ` Ludovic Courtès
0 siblings, 1 reply; 37+ messages in thread
From: Andy Wingo @ 2017-06-23 9:46 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: guix-devel
Hi!
On Fri 23 Jun 2017 11:24, ludovic.courtes@inria.fr (Ludovic Courtès) writes:
> With the current protocol, often we’re just reading a handful of bytes.
> Full buffering would mean that Guile would block on an 8K read or so
> that will never be fulfilled.
That's not how it works :) The "read" function of a port should only
block if no byte can be read. If 1K bytes are available for an 8K
buffer, then the read function should return after filling only 1K
bytes; looping to fill at least 8K is some other code's responsibility.
In particular, "read" functions should not use get-bytevector-n, as
get-bytevector-n is defined to block until N bytes are available.
Instead they should use get-bytevector-some. See:
https://git.savannah.gnu.org/cgit/guile.git/commit/?h=stable-2.2&id=0c102b56e98da39b5a3213bdc567a31ad8ef3e73
Andy
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: RPC performance
2017-06-23 9:46 ` Andy Wingo
@ 2017-06-26 11:54 ` Ludovic Courtès
2017-06-26 14:19 ` Andy Wingo
0 siblings, 1 reply; 37+ messages in thread
From: Ludovic Courtès @ 2017-06-26 11:54 UTC (permalink / raw)
To: Andy Wingo; +Cc: guix-devel
Andy Wingo <wingo@igalia.com> skribis:
> Hi!
>
> On Fri 23 Jun 2017 11:24, ludovic.courtes@inria.fr (Ludovic Courtès) writes:
>
>> With the current protocol, often we’re just reading a handful of bytes.
>> Full buffering would mean that Guile would block on an 8K read or so
>> that will never be fulfilled.
>
> That's not how it works :) The "read" function of a port should only
> block if no byte can be read. If 1K bytes are available for an 8K
> buffer, then the read function should return after filling only 1K
> bytes; looping to fill at least 8K is some other code's responsibility.
I must be missing something. With full buffering, when my code does:
(get-bytevector-n port 8)
I see read(2) hanging on an 8K read:
--8<---------------cut here---------------start------------->8---
#0 0x00007fb0b36baaed in read () at ../sysdeps/unix/syscall-template.S:84
#1 0x00007fb0b3b91c47 in fport_read (port=<optimized out>, dst=<optimized out>, start=<optimized out>,
count=8192) at fports.c:604
#2 0x00007fb0b3bbed77 in scm_i_read_bytes (port=port@entry=0x194f700, dst=0x195c000, start=start@entry=0,
count=8192) at ports.c:1544
#3 0x00007fb0b3bc25fe in scm_fill_input (port=port@entry=0x194f700, minimum_size=1, minimum_size@entry=0,
cur_out=cur_out@entry=0x7ffd7eee5f30, avail_out=avail_out@entry=0x7ffd7eee5f38) at ports.c:2677
#4 0x00007fb0b3bc3384 in scm_c_read_bytes (port=port@entry=0x194f700, dst=dst@entry=0x1952510,
start=start@entry=0, count=count@entry=8) at ports.c:1610
#5 0x00007fb0b3bc9838 in scm_get_bytevector_n (port=0x194f700, count=<optimized out>) at r6rs-ports.c:421
#6 0x00007fb0b3bfdc4d in vm_regular_engine (thread=0xe, vp=0x143df30, registers=0x2000, resume=-1284789523)
at vm-engine.c:784
--8<---------------cut here---------------end--------------->8---
(That’s not Guile-specific.) I agree that read(2) could return less
than 8K and not block, but it doesn’t have to.
I’ll see if I can investigate more later.
Thanks for your input,
Ludo’.
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: RPC performance
2017-06-26 11:54 ` Ludovic Courtès
@ 2017-06-26 14:19 ` Andy Wingo
0 siblings, 0 replies; 37+ messages in thread
From: Andy Wingo @ 2017-06-26 14:19 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: guix-devel
On Mon 26 Jun 2017 13:54, ludovic.courtes@inria.fr (Ludovic Courtès) writes:
> Andy Wingo <wingo@igalia.com> skribis:
>
>> Hi!
>>
>> On Fri 23 Jun 2017 11:24, ludovic.courtes@inria.fr (Ludovic Courtès) writes:
>>
>>> With the current protocol, often we’re just reading a handful of bytes.
>>> Full buffering would mean that Guile would block on an 8K read or so
>>> that will never be fulfilled.
>>
>> That's not how it works :) The "read" function of a port should only
>> block if no byte can be read. If 1K bytes are available for an 8K
>> buffer, then the read function should return after filling only 1K
>> bytes; looping to fill at least 8K is some other code's responsibility.
>
> I must be missing something. With full buffering, when my code does:
>
> (get-bytevector-n port 8)
Note that get-bytevector-n will block until there are 8 bytes.
> I see read(2) hanging on an 8K read:
>
> #0 0x00007fb0b36baaed in read () at ../sysdeps/unix/syscall-template.S:84
> #1 0x00007fb0b3b91c47 in fport_read (port=<optimized out>, dst=<optimized out>, start=<optimized out>,
> count=8192) at fports.c:604
> #2 0x00007fb0b3bbed77 in scm_i_read_bytes (port=port@entry=0x194f700, dst=0x195c000, start=start@entry=0,
> count=8192) at ports.c:1544
> #3 0x00007fb0b3bc25fe in scm_fill_input (port=port@entry=0x194f700, minimum_size=1, minimum_size@entry=0,
> cur_out=cur_out@entry=0x7ffd7eee5f30, avail_out=avail_out@entry=0x7ffd7eee5f38) at ports.c:2677
Here this indicates that the buffer is empty, and that it's blocking on
receiving *any* bytes at all.
> (That’s not Guile-specific.) I agree that read(2) could return less
> than 8K and not block, but it doesn’t have to.
I think this is incorrect. Read returns when it has any bytes at all.
From read(2):
It is not an error if this number is smaller than the number of
bytes requested; this may happen for example because fewer bytes
are actually available right now (maybe because we were close to
end-of-file, or because we are reading from a pipe, or from a
terminal), or because read() was interrupted by a signal. See
also NOTES.
In short the reason read is blocking for you is that there are no bytes
available -- if there were 8 bytes and only 8 bytes, the read(2) would
return directly.
If you have blocking problems related to 8K buffers, it's likely related
to using get-bytevector-n inside CBIP read() functions.
Andy
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Combining Guix, direnv and Emacs for environment customisation
2017-05-27 11:31 ` Combining Guix, direnv and Emacs for environment customisation Christopher Baines
2017-05-30 15:03 ` Ludovic Courtès
@ 2017-08-29 19:02 ` Thompson, David
2017-08-29 22:30 ` Maxim Cournoyer
` (2 more replies)
1 sibling, 3 replies; 37+ messages in thread
From: Thompson, David @ 2017-08-29 19:02 UTC (permalink / raw)
To: Christopher Baines; +Cc: guix-devel, 27097
Hello Christopher,
Reviving this old thread, if you don't mind. :)
On Sat, May 27, 2017 at 7:31 AM, Christopher Baines <mail@cbaines.net> wrote:
>
> In summary, using direnv provides a convinient way to manage different
> environments created from Guix. There is now support for using direnv
> directly from Emacs.
This is great. I need to try it out sometime.
> One issue with this is that running guix environment from direnv will
> slow down switching buffers. To make it a bit more useable, I found
> some bash code that caches the results of running commands, and wrapped
> that around guix environment when invoked from direnv. This helps speed
> things up, but I don't think its useful in the long term.
>
> For this particular use case, it would help if guix environment was
> faster, perhaps by doing caching internally? On my system, running guix
> environment --ad-hoc guile --search-paths repeatedly takes ~2 seconds,
> I haven't looked at what the breakdown of this is yet.
>
> I'd be interested in hearing if anyone does something similar for using
> Guix, or if anyone does something different, but to the same effect?
There is a feature missing in 'guix environment': saving the resulting
profile for easy access later. I often want to build an environment
once and not update it for awhile, but with the current state of 'guix
environment' it's not possible. This leads me to pulling my hair out
when I do an upgrade of Guix and have to rebuild all of my development
environments. What was supposed to be a quick hack turns into a Guix
maintenance session.
Let's imagine that the first time `guix environment` is invoked it
creates a symlink in $PWD to the generated profile named
.guix-environment. Future invocations of 'guix environment' would
short-circuit all the daemon communication, package module loading,
etc. and just apply the environment variables in the already built
profile. Once we have that built, emacs-direnv integration is as
simple as sourcing .guix-environment/etc/profile, or running 'guix
environment' if the profile is not yet built.
WDYT?
(If time permits, I'd like to start contributing to Guix again by
overhauling 'guix environment' to be more convenient for day-to-day
development)
- Dave
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Combining Guix, direnv and Emacs for environment customisation
2017-08-29 19:02 ` Combining Guix, direnv and Emacs for environment customisation Thompson, David
@ 2017-08-29 22:30 ` Maxim Cournoyer
2017-08-30 5:52 ` Carlo Zancanaro
2017-08-30 10:16 ` Ludovic Courtès
2 siblings, 0 replies; 37+ messages in thread
From: Maxim Cournoyer @ 2017-08-29 22:30 UTC (permalink / raw)
To: Thompson, David; +Cc: guix-devel, 27097
Hi!
"Thompson, David" <dthompson2@worcester.edu> writes:
> There is a feature missing in 'guix environment': saving the resulting
> profile for easy access later. I often want to build an environment
> once and not update it for awhile, but with the current state of 'guix
> environment' it's not possible. This leads me to pulling my hair out
> when I do an upgrade of Guix and have to rebuild all of my development
> environments. What was supposed to be a quick hack turns into a Guix
> maintenance session.
Have you tried creating extra persistent profiles? This
would seem to suite your use case better (although a 'save-env' feature
for environments would be neat!).
The doc is sparsed about the use of profiles, but Ricardo published some
useful write-up here: https://elephly.net/posts/latest.html where you'll
find some commands making use of it.
HTH,
Maxim
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Combining Guix, direnv and Emacs for environment customisation
2017-08-29 19:02 ` Combining Guix, direnv and Emacs for environment customisation Thompson, David
2017-08-29 22:30 ` Maxim Cournoyer
@ 2017-08-30 5:52 ` Carlo Zancanaro
2017-08-30 10:09 ` Ludovic Courtès
2017-08-30 10:16 ` Ludovic Courtès
2 siblings, 1 reply; 37+ messages in thread
From: Carlo Zancanaro @ 2017-08-30 5:52 UTC (permalink / raw)
To: David Thompson; +Cc: guix-devel
[-- Attachment #1: Type: text/plain, Size: 916 bytes --]
On Tue, Aug 29 2017, David wrote:
> There is a feature missing in 'guix environment': saving the resulting
> profile for easy access later. I often want to build an environment
> once and not update it for awhile, but with the current state of 'guix
> environment' it's not possible. This leads me to pulling my hair out
> when I do an upgrade of Guix and have to rebuild all of my development
> environments. What was supposed to be a quick hack turns into a Guix
> maintenance session.
I also got frustrated by this when trying to keep my own checkout of
guix. Then I found out you can do this:
guix environment guix --root=environment
This puts a link to the environment under `environment` in the local
directory. Then you can run `source environment/etc/profile; make` to
rebuild guix. This also protects the environment from garbage
collection, which was the actual problem I was trying to solve.
Carlo
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Combining Guix, direnv and Emacs for environment customisation
2017-08-30 5:52 ` Carlo Zancanaro
@ 2017-08-30 10:09 ` Ludovic Courtès
0 siblings, 0 replies; 37+ messages in thread
From: Ludovic Courtès @ 2017-08-30 10:09 UTC (permalink / raw)
To: Carlo Zancanaro; +Cc: guix-devel
Carlo Zancanaro <carlo@zancanaro.id.au> skribis:
> On Tue, Aug 29 2017, David wrote:
>> There is a feature missing in 'guix environment': saving the resulting
>> profile for easy access later. I often want to build an environment
>> once and not update it for awhile, but with the current state of 'guix
>> environment' it's not possible. This leads me to pulling my hair out
>> when I do an upgrade of Guix and have to rebuild all of my development
>> environments. What was supposed to be a quick hack turns into a Guix
>> maintenance session.
>
> I also got frustrated by this when trying to keep my own checkout of
> guix. Then I found out you can do this:
>
> guix environment guix --root=environment
>
> This puts a link to the environment under `environment` in the local
> directory. Then you can run `source environment/etc/profile; make` to
> rebuild guix. This also protects the environment from garbage
> collection, which was the actual problem I was trying to solve.
It surely helps, but I think it would be even better if ‘guix
environment’ would directly include a way to reload the environment in
its UI, like Dave was suggesting.
Maybe it’s just a matter of adding a --reload option that does what you
wrote above for you?
Ludo’.
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Combining Guix, direnv and Emacs for environment customisation
2017-08-29 19:02 ` Combining Guix, direnv and Emacs for environment customisation Thompson, David
2017-08-29 22:30 ` Maxim Cournoyer
2017-08-30 5:52 ` Carlo Zancanaro
@ 2017-08-30 10:16 ` Ludovic Courtès
2 siblings, 0 replies; 37+ messages in thread
From: Ludovic Courtès @ 2017-08-30 10:16 UTC (permalink / raw)
To: Thompson, David; +Cc: guix-devel, 27097
Hi!
"Thompson, David" <dthompson2@worcester.edu> skribis:
> There is a feature missing in 'guix environment': saving the resulting
> profile for easy access later. I often want to build an environment
> once and not update it for awhile, but with the current state of 'guix
> environment' it's not possible. This leads me to pulling my hair out
> when I do an upgrade of Guix and have to rebuild all of my development
> environments. What was supposed to be a quick hack turns into a Guix
> maintenance session.
>
> Let's imagine that the first time `guix environment` is invoked it
> creates a symlink in $PWD to the generated profile named
> .guix-environment. Future invocations of 'guix environment' would
> short-circuit all the daemon communication, package module loading,
> etc. and just apply the environment variables in the already built
> profile. Once we have that built, emacs-direnv integration is as
> simple as sourcing .guix-environment/etc/profile, or running 'guix
> environment' if the profile is not yet built.
>
> WDYT?
Like Carlo wrote, --root probably gets us halfway there already. Maybe
a new “--save” could be added and be equivalent to
“--root=$PWD/.guix-environment”. And then, indeed, “guix environment”
without any argument could simply source .guix-environment/etc/profile
and start from there.
How does that sound?
FWIW I’m interested in making ‘guix environment’ a drop-in replacement
for the “modules” command found on HPC clusters¹, which means startup
times below one second. A simple cache like what you write above could
get us very close to that.
¹ http://modules.sourceforge.net/
> (If time permits, I'd like to start contributing to Guix again by
> overhauling 'guix environment' to be more convenient for day-to-day
> development)
Your inspiration and hack power would be welcome! :-)
Ludo’.
^ permalink raw reply [flat|nested] 37+ messages in thread
end of thread, other threads:[~2017-08-30 10:16 UTC | newest]
Thread overview: 37+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <20170527105641.9426-1-mail@cbaines.net>
2017-05-27 11:31 ` Combining Guix, direnv and Emacs for environment customisation Christopher Baines
2017-05-30 15:03 ` Ludovic Courtès
2017-06-01 13:17 ` Roel Janssen
2017-06-03 13:59 ` Ludovic Courtès
2017-06-03 21:08 ` Roel Janssen
2017-06-04 21:15 ` Ludovic Courtès
2017-06-05 22:37 ` Roel Janssen
2017-06-07 10:35 ` Performance on NFS Ludovic Courtès
2017-06-07 13:06 ` Roel Janssen
2017-06-09 13:46 ` Ludovic Courtès
2017-06-12 8:45 ` Roel Janssen
2017-06-12 15:58 ` Ludovic Courtès
2017-06-16 15:23 ` Ludovic Courtès
2017-06-17 7:36 ` Roel Janssen
2017-06-17 22:40 ` Ludovic Courtès
2017-06-17 23:15 ` Roel Janssen
2017-06-18 8:43 ` Ricardo Wurmus
2017-06-19 8:01 ` RPC performance Ludovic Courtès
2017-06-19 8:15 ` Ludovic Courtès
2017-06-19 14:25 ` Ricardo Wurmus
2017-06-22 14:03 ` Andy Wingo
2017-06-22 16:05 ` Ludovic Courtès
2017-06-23 9:09 ` Andy Wingo
2017-06-23 9:24 ` Ludovic Courtès
2017-06-23 9:46 ` Andy Wingo
2017-06-26 11:54 ` Ludovic Courtès
2017-06-26 14:19 ` Andy Wingo
2017-06-19 21:25 ` Ludovic Courtès
2017-06-22 8:04 ` Ricardo Wurmus
2017-06-07 11:01 ` Combining Guix, direnv and Emacs for environment customisation Ricardo Wurmus
2017-06-07 12:25 ` Performance on NFS Ludovic Courtès
2017-06-07 12:59 ` Ricardo Wurmus
2017-08-29 19:02 ` Combining Guix, direnv and Emacs for environment customisation Thompson, David
2017-08-29 22:30 ` Maxim Cournoyer
2017-08-30 5:52 ` Carlo Zancanaro
2017-08-30 10:09 ` Ludovic Courtès
2017-08-30 10:16 ` Ludovic Courtès
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).