Best-practice to "develop" a system-config / service?

Best-practice to "develop" a system-config / service?

[Hartmut Goebel]

[-- Attachment #1: Type: text/plain, Size: 1754 bytes --]

Hi,

TL;DR: How do I get my in-development version of guix into the VM build
with this version?

I made KDE Plasma start up in a VM by adding tons of packages to the
system declaration (to be on the save side). Now for finishing and
providing a nice service declaration I need to strip this list of
packages down to the bare minimum.

Since a vm (sharing the host store) is unacceptable sloooooow, I've
build a vm-image using my development environment ("./pre-inst-env guix
system vm-image …"). Now within the VM-image there is a different
version of guix installed than the in-development one I used for
building the VM. Of course the former one does not know any of the
packages I've just added. So I can't not just run "guix system
reconfigure …".

After some luckless tries I assume the easiest solution would be to get
my in-development version of guix into the VM. How to achieve this?


What I've tried:

0) Using a VM sharing the host store: Here starting Plasma takes 2
minutes, which is not acceptable for series of tests.

1) My first idea was to share my development directory with the VM and
run "./pre-inst-env guix system reconfigure …" from within the VM. This
failed with

  ./pre-inst-env: …/scripts/guix: /gnu/store/…-profile/bin/guile: bad
interpreter: No such file or directory

Well, yes. within the VM, my external profile is not available :-(

2) Changing the patch in …/scripts/guix into
/run/current-system/profile/bin/guile failed with

    /gnu/store/…-libgcrypt-1.8.1/lib/libgcrypt

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |


[-- Attachment #2: 0xBF773B65.asc --]
[-- Type: application/pgp-keys, Size: 14855 bytes --]

Re: Best-practice to "develop" a system-config / service?

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 1824 bytes --]

Hartmut Goebel <h.goebel@crazy-compilers.com> writes:

> Hi,
>
> TL;DR: How do I get my in-development version of guix into the VM build
> with this version?
>
> I made KDE Plasma start up in a VM by adding tons of packages to the
> system declaration (to be on the save side). Now for finishing and
> providing a nice service declaration I need to strip this list of
> packages down to the bare minimum.
>
> Since a vm (sharing the host store) is unacceptable sloooooow, I've
> build a vm-image using my development environment ("./pre-inst-env guix
> system vm-image …"). Now within the VM-image there is a different
> version of guix installed than the in-development one I used for
> building the VM. Of course the former one does not know any of the
> packages I've just added. So I can't not just run "guix system
> reconfigure …".
>
> After some luckless tries I assume the easiest solution would be to get
> my in-development version of guix into the VM. How to achieve this?
>
>
> What I've tried:
>
> 0) Using a VM sharing the host store: Here starting Plasma takes 2
> minutes, which is not acceptable for series of tests.
>
> 1) My first idea was to share my development directory with the VM and
> run "./pre-inst-env guix system reconfigure …" from within the VM. This
> failed with
>
>   ./pre-inst-env: …/scripts/guix: /gnu/store/…-profile/bin/guile: bad
> interpreter: No such file or directory

Running `./configure` again inside the VM should make it pick up
available versions of Guile and other dependencies.  Will that work?

Another approach you might find useful is writing a "system test" that
starts a Plasma desktop.  Look in (gnu tests) for inspiration.  This has
the added benefit that we will notice early if KDE breaks.

Hope this helps!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Best-practice to "develop" a system-config / service?

[Ludovic Courtès]

Hi Hartmut,

Hartmut Goebel <h.goebel@crazy-compilers.com> skribis:

> I made KDE Plasma start up in a VM by adding tons of packages to the
> system declaration (to be on the save side). Now for finishing and
> providing a nice service declaration I need to strip this list of
> packages down to the bare minimum.
>
> Since a vm (sharing the host store) is unacceptable sloooooow,

Could you be more specific?  I understand from Chris Baines’ measurement
some time ago that it’s definitely slower than a bare-metal system, but
I don’t find it “unacceptably slow” when it comes to testing stuff like
this (it’s definitely faster than building a standalone image with ‘guix
system vm-image’ anyway!).

> I've build a vm-image using my development environment
> ("./pre-inst-env guix system vm-image …"). Now within the VM-image
> there is a different version of guix installed than the in-development
> one I used for building the VM. Of course the former one does not know
> any of the packages I've just added. So I can't not just run "guix
> system reconfigure …".
>
> After some luckless tries I assume the easiest solution would be to get
> my in-development version of guix into the VM. How to achieve this?

You probably don’t want to do that.

I think all you want, to test your KDE service, is to:

  1. Write an OS config that uses the service.

  2. Run “./pre-inst-env guix system vm that-config.scm”, run the VM,
     and check if it works.

That’s really all it takes to develop and test a system service.

You seemed to be willing to start a GuixSD VM that does *not* use the
service you want to test, and then to reconfigure that VM from the
inside so that it would run your service.  That’s super tedious, as you
wrote, and it’s strictly equivalent to the steps above anyway.

HTH!

Ludo’.

Speed of qemu VM sharing the store (was: Best-practice to "develop" a system-config / service?)

[Hartmut Goebel]

Am 11.11.2017 um 12:31 schrieb Ludovic Courtès:
>> Since a vm (sharing the host store) is unacceptable sloooooow,
> Could you be more specific?  I understand from Chris Baines’ measurement
> some time ago that it’s definitely slower than a bare-metal system, but
> I don’t find it “unacceptably slow” when it comes to testing stuff like
> this (it’s definitely faster than building a standalone image with ‘guix
> system vm-image’ anyway!).
>
See my posting at
<https://lists.gnu.org/archive/html/guix-devel/2017-10/msg00185.html>:

When sharing the host's store ("guix system vm …"), Plasma takes 80
seconds (minimum) to start up after log-in to X11, when clicking the
menu-button, it takes 36 sec. for the menu to appear. This plain renders
such a VM unusable even for testing.

YMMV, but I call this in-acceptable and this is why I'm using a vm-image
("guix system vm-image …", which is much faster: Plase start-up 25 sec.,
Menu appears immediately.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Best-practice to "develop" a system-config / service?

[Hartmut Goebel]

Am 11.11.2017 um 12:31 schrieb Ludovic Courtès:
> I think all you want, to test your KDE service, is to:
>
>   1. Write an OS config that uses the service.
>
>   2. Run “./pre-inst-env guix system vm that-config.scm”, run the VM,
>      and check if it works.
>
> That’s really all it takes to develop and test a system service.

Not in my case. Plasma heavily relies on plugins and such. I have added
about 50 package to the system configuration to make Plasma start. Now I
need [1] to iterately remove (and re-add) packages (not services!) to
learn what are actual the minimum requirements.

Creating a new VM for each iteration is *much* too time-consuming – no
matter if using "vm-image" or "vm" –, let alone since this required to
reboot the machine each time. Even if I would try to write a test-case
for this [2], each cycle would take too much time.

On a Fedora-like system I would simply 'rpm -e PACKAGENAME`.
Unfortunately guix is not able to uninstall a package it does not know
(see <https://lists.gnu.org/archive/html/guix-devel/2017-11/msg00160.html>).

[1] It "need" since I want to deliver a high-qualify service definition.
Otherwise I could live with these 50 packages.
[2] Which is hard, since I would need to figure out how to test "the
Plasma beast is running".

> You seemed to be willing to start a GuixSD VM that does *not* use the
> service you want to test, 

Nope. I want to start a GuixSD VM that *does* use the service, but I
need to remove and add *package* in the machine.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Speed of qemu VM sharing the store

[Ludovic Courtès]

Hartmut Goebel <h.goebel@crazy-compilers.com> skribis:

> Am 11.11.2017 um 12:31 schrieb Ludovic Courtès:
>>> Since a vm (sharing the host store) is unacceptable sloooooow,
>> Could you be more specific?  I understand from Chris Baines’ measurement
>> some time ago that it’s definitely slower than a bare-metal system, but
>> I don’t find it “unacceptably slow” when it comes to testing stuff like
>> this (it’s definitely faster than building a standalone image with ‘guix
>> system vm-image’ anyway!).
>>
> See my posting at
> <https://lists.gnu.org/archive/html/guix-devel/2017-10/msg00185.html>:
>
> When sharing the host's store ("guix system vm …"), Plasma takes 80
> seconds (minimum) to start up after log-in to X11, when clicking the
> menu-button, it takes 36 sec. for the menu to appear. This plain renders
> such a VM unusable even for testing.

OK, that’s terrible indeed.  So that’s with ‘-enable-kvm’ on the qemu
command line, right?

> YMMV, but I call this in-acceptable and this is why I'm using a vm-image
> ("guix system vm-image …", which is much faster: Plase start-up 25 sec.,
> Menu appears immediately.

25 seconds to start up is still a lot (in ‘guix system vm’ GNOME is
usable and much faster to start).  It would be nice to see if that comes
from I/O or something else.  Is there something we’re doing wrong
(independently of the VM) or is it expected for Plasma startup to be
this slow?

Ludo’.

Re: Best-practice to "develop" a system-config / service?

[Ludovic Courtès]

Hartmut Goebel <h.goebel@crazy-compilers.com> skribis:

> Am 11.11.2017 um 12:31 schrieb Ludovic Courtès:
>> I think all you want, to test your KDE service, is to:
>>
>>   1. Write an OS config that uses the service.
>>
>>   2. Run “./pre-inst-env guix system vm that-config.scm”, run the VM,
>>      and check if it works.
>>
>> That’s really all it takes to develop and test a system service.
>
> Not in my case. Plasma heavily relies on plugins and such. I have added
> about 50 package to the system configuration to make Plasma start. Now I
> need [1] to iterately remove (and re-add) packages (not services!) to
> learn what are actual the minimum requirements.
>
> Creating a new VM for each iteration is *much* too time-consuming – no
> matter if using "vm-image" or "vm" –, let alone since this required to
> reboot the machine each time. Even if I would try to write a test-case
> for this [2], each cycle would take too much time.

Well, I’ve always used the above approach to test system services as I
worked on them, and the 10–30 seconds it took for ‘guix system vm’ to
complete as I tweak the service were never that much of a hindrance to
me.

But yeah, I understand you’re using ‘vm-image’, so it’s a different
story since ‘vm-image’ takes ages.

Now, it seems to me that the workflow you describe has become this
complicated just because of the slowness of Plasma in the VM.  To me
that’s something worth investigating.  Granted a VM with a 9p-mounted
store is slower than a bare metal system, but it shouldn’t be this slow.
Something’s wrong.

> On a Fedora-like system I would simply 'rpm -e PACKAGENAME`.
> Unfortunately guix is not able to uninstall a package it does not know
> (see <https://lists.gnu.org/archive/html/guix-devel/2017-11/msg00160.html>).

It cannot uninstall a propagated package, if that’s what you mean.

Ludo’.

Re: Speed of qemu VM sharing the store

[Hartmut Goebel]

Am 11.11.2017 um 14:45 schrieb Ludovic Courtès:
> OK, that’s terrible indeed.  So that’s with ‘-enable-kvm’ on the qemu
> command line, right?

I verified this: These 80 seconds are with "-m 2048 -smp 2 -enable-kvm".

I have read/wrote-access to /dev/kvm. My $HOME is on a NFS-share, but
the cwd is /tmp (tmpfs) and the store is on a SSD. I'm sharing my
development-dir (which is on the NFS-share), changing run-vm.sh to share
/tmp instead did not speed up things.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Best-practice to "develop" a system-config / service?

[Hartmut Goebel]

Am 11.11.2017 um 14:51 schrieb Ludovic Courtès:
> Well, I’ve always used the above approach to test system services as I
> worked on them, and the 10–30 seconds it took for ‘guix system vm’ to
> complete as I tweak the service were never that much of a hindrance to
> me.

JFYI: Building a new VM takes more than 90 seconds on my system.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Best-practice to "develop" a system-config / service?

[Ludovic Courtès]

Hartmut Goebel <h.goebel@crazy-compilers.com> skribis:

> Am 11.11.2017 um 14:51 schrieb Ludovic Courtès:
>> Well, I’ve always used the above approach to test system services as I
>> worked on them, and the 10–30 seconds it took for ‘guix system vm’ to
>> complete as I tweak the service were never that much of a hindrance to
>> me.
>
> JFYI: Building a new VM takes more than 90 seconds on my system.

With ‘guix system vm’, right?  I guess it also depends on the number of
things that need to be built, but 90 seconds remains acceptable IMO.

Ludo’.

Re: Best-practice to "develop" a system-config / service?

[Hartmut Goebel]

Am 13.11.2017 um 11:44 schrieb Ludovic Courtès:
> With ‘guix system vm’, right?  I guess it also depends on the number of
> things that need to be built, but 90 seconds remains acceptable IMO.

90 seconds are fairly acceptable – as long as you have only a few
iterations.

But for Plasma I expect 50 and more iterations (since documentation is
very, very, terse), and in this case waiting 90 seconds to start the
next iteration s far too long :-(

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |