[bug#27401] [PATCH] doc: Suggest a QEMU image size large enough for the system examples.

[bug#27401] [PATCH] doc: Suggest a QEMU image size large enough for the system examples.

[Leo Famulari]

* doc/guix.texi (Installing GuixSD in a Virtual Machine): Increase suggested
image size from 5 GB to 50 GB.
---
 doc/guix.texi | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index db0e2fbd7..4933a98dd 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -7796,11 +7796,11 @@ Create a disk image that will hold the installed system.  To make a
 qcow2-formatted disk image, use the @command{qemu-img} command:
 
 @example
-qemu-img create -f qcow2 guixsd.img 5G
+qemu-img create -f qcow2 guixsd.img 50G
 @end example
 
-The resulting file will be much smaller than 5GB, but it will grow as
-the virtualized storage device is filled up.
+The resulting file will be much smaller than 50 GB (typically less than
+1 MB), but it will grow as the virtualized storage device is filled up.
 
 @item
 Boot the USB installation image in an VM:
-- 
2.13.1

[bug#27401] [PATCH] doc: Suggest a QEMU image size large enough for the system examples.

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 161 bytes --]

Leo Famulari <leo@famulari.name> writes:

> * doc/guix.texi (Installing GuixSD in a Virtual Machine): Increase suggested
> image size from 5 GB to 50 GB.

LGTM.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

[bug#27401] [PATCH] doc: Suggest a QEMU image size large enough for the system examples.

[Jan Nieuwenhuizen]

Leo Famulari writes:

> * doc/guix.texi (Installing GuixSD in a Virtual Machine): Increase suggested
> image size from 5 GB to 50 GB.

LGTM, however -- What about something like this patch too?  I seems that
the VMs we create (although they are of the ballooning cow2 kind), are
2GiB -- i.e., almost unusable for Real Stuff (TM)?

$ git diff
diff --git a/Makefile.am b/Makefile.am
index a3b3e557a..ff2ae120b 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -592,7 +592,7 @@ GUIXSD_VM_IMAGE_BASE = guixsd-vm-image-$(PACKAGE_VERSION)
 GUIXSD_INSTALLATION_IMAGE_SIZE ?= 950MiB
 
 # Size of the VM image (for x86_64 typically).
-GUIXSD_VM_IMAGE_SIZE ?= 2GiB
+GUIXSD_VM_IMAGE_SIZE ?= 50GiB
 
 # The release process works in several phases:
 #

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

[bug#27401] [PATCH] doc: Suggest a QEMU image size large enough for the system examples.

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 942 bytes --]

On Mon, Jun 19, 2017 at 07:55:03PM +0200, Jan Nieuwenhuizen wrote:
> Leo Famulari writes:
> 
> > * doc/guix.texi (Installing GuixSD in a Virtual Machine): Increase suggested
> > image size from 5 GB to 50 GB.
> 
> LGTM, however -- What about something like this patch too?  I seems that
> the VMs we create (although they are of the ballooning cow2 kind), are
> 2GiB -- i.e., almost unusable for Real Stuff (TM)?

I chose 2GiB for those images because it's pretty easy to grow a
partition and ext4 filesystem from a running system. On the other hand,
I couldn't find an easy way to shrink them.

We can't predict the size of the virtual block device storage provided
by a VPS hoster, so I chose this small size with these expectations:
1) 2GiB will never be too big
2) users will resize after booting

I didn't come up with a better solution for the problem of not knowing
the storage size in advance. What do you think?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[bug#27401] [PATCH] doc: Suggest a QEMU image size large enough for the system examples.

[Jan Nieuwenhuizen]

Leo Famulari writes:

>> LGTM, however -- What about something like this patch too?  I seems that
>> the VMs we create (although they are of the ballooning cow2 kind), are
>> 2GiB -- i.e., almost unusable for Real Stuff (TM)?
>
> I chose 2GiB for those images because it's pretty easy to grow a
> partition and ext4 filesystem from a running system. On the other hand,
> I couldn't find an easy way to shrink them.
>
> We can't predict the size of the virtual block device storage provided
> by a VPS hoster,

Ah, sure.  I missed that aspect.

> so I chose this small size with these expectations:
> 1) 2GiB will never be too big

Yes.

> 2) users will resize after booting

When I was asked to help with this, I was confronted with a 2GiB sda1
and right behind that an EFI sda2.  That was running in vmware (don't
ask ;-).  We hesitated, removed the sda2 and did the resize thing.

Sadly I have no further information, instead of testing the reboot we
decided to go for qemu.

> I didn't come up with a better solution for the problem of not knowing
> the storage size in advance. What do you think?

Yes, I think you made the right choice.

Greetings,
janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

[bug#27401] [PATCH] doc: Suggest a QEMU image size large enough for the system examples.

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 815 bytes --]

On Tue, Jun 20, 2017 at 08:03:17PM +0200, Jan Nieuwenhuizen wrote:
> Leo Famulari writes:
> > 2) users will resize after booting
> 
> When I was asked to help with this, I was confronted with a 2GiB sda1
> and right behind that an EFI sda2.  That was running in vmware (don't
> ask ;-).  We hesitated, removed the sda2 and did the resize thing.

Ah, I didn't plan for that sort of disk layout.

> Sadly I have no further information, instead of testing the reboot we
> decided to go for qemu.
> 
> > I didn't come up with a better solution for the problem of not knowing
> > the storage size in advance. What do you think?
> 
> Yes, I think you made the right choice.

Okay. I'd like to clarify and say that I think we could do something
more sophisticated, but I'm not sure what we would do :)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[bug#27401] Guessing the disk image size in 'guix system'

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 614 bytes --]

Hello!

Leo Famulari <leo@famulari.name> skribis:

> On Tue, Jun 20, 2017 at 08:03:17PM +0200, Jan Nieuwenhuizen wrote:

[...]

>> > I didn't come up with a better solution for the problem of not knowing
>> > the storage size in advance. What do you think?
>> 
>> Yes, I think you made the right choice.
>
> Okay. I'd like to clarify and say that I think we could do something
> more sophisticated, but I'm not sure what we would do :)

We can do “guix size $(guix system build foo.scm)” and add 10% or so to
that.

Maybe something like this mostly-untested patch would work?

Ludo’.


[-- Attachment #2: Type: text/x-patch, Size: 7861 bytes --]

diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 392737d07..15cc0d993 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2013, 2014, 2015, 2016 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2013, 2014, 2015, 2016, 2017 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2016 Christopher Allan Webber <cwebber@dustycloud.org>
 ;;; Copyright © 2016 Leo Famulari <leo@famulari.name>
 ;;; Copyright © 2017 Mathieu Othacehe <m.othacehe@gmail.com>
@@ -108,8 +108,7 @@
                                              (references-graphs #f)
                                              (memory-size 256)
                                              (disk-image-format "qcow2")
-                                             (disk-image-size
-                                              (* 100 (expt 2 20))))
+                                             (disk-image-size 'guess))
   "Evaluate EXP in a QEMU virtual machine running LINUX with INITRD (a
 derivation).  In the virtual machine, EXP has access to all its inputs from the
 store; it should put its output files in the `/xchg' directory, which is
@@ -138,19 +137,24 @@ made available under the /xchg CIFS share."
     (define builder
       ;; Code that launches the VM that evaluates EXP.
       (with-imported-modules (source-module-closure '((guix build utils)
+                                                      (guix build store-copy)
                                                       (gnu build vm)))
         #~(begin
             (use-modules (guix build utils)
+                         (guix build store-copy)
                          (gnu build vm))
 
-            (let ((inputs  '#$(list qemu coreutils))
-                  (linux   (string-append #$linux "/"
-                                          #$(system-linux-image-file-name)))
-                  (initrd  (string-append #$initrd "/initrd"))
-                  (loader  #$loader)
-                  (graphs  '#$(match references-graphs
-                                (((graph-files . _) ...) graph-files)
-                                (_ #f))))
+            (let* ((inputs  '#$(list qemu coreutils))
+                   (linux   (string-append #$linux "/"
+                                           #$(system-linux-image-file-name)))
+                   (initrd  (string-append #$initrd "/initrd"))
+                   (loader  #$loader)
+                   (graphs  '#$(match references-graphs
+                                 (((graph-files . _) ...) graph-files)
+                                 (_ #f)))
+                   (size    #$(if (eq? 'guess disk-image-size)
+                                  #~(round (* 1.1 (closure-size graphs)))
+                                  disk-image-size)))
 
               (set-path-environment-variable "PATH" '("bin") inputs)
 
@@ -160,7 +164,7 @@ made available under the /xchg CIFS share."
                                 #:memory-size #$memory-size
                                 #:make-disk-image? #$make-disk-image?
                                 #:disk-image-format #$disk-image-format
-                                #:disk-image-size #$disk-image-size
+                                #:disk-image-size size
                                 #:references-graphs graphs)))))
 
     (gexp->derivation name builder
@@ -198,10 +202,13 @@ the image."
   (expression->derivation-in-linux-vm
    name
    (with-imported-modules (source-module-closure '((gnu build vm)
+                                                   (guix build store-copy)
                                                    (guix build utils)))
      #~(begin
          (use-modules (gnu build vm)
-                      (guix build utils))
+                      (guix build store-copy)
+                      (guix build utils)
+                      (srfi srfi-26))
 
          (let ((inputs
                 '#$(append (list qemu parted e2fsprogs dosfstools)
@@ -227,9 +234,14 @@ the image."
                                #:copy-closures? #$copy-inputs?
                                #:register-closures? #$register-closures?
                                #:system-directory #$os-drv))
+                  (root-size  #$(if (eq? 'guess disk-image-size)
+                                    #~(closure-size
+                                       (map (cut string-append "/xchg/" <>)
+                                            graphs))
+                                    (- disk-image-size
+                                       (* 50 (expt 2 20)))))
                   (partitions (list (partition
-                                     (size #$(- disk-image-size
-                                                (* 50 (expt 2 20))))
+                                     (size root-size)
                                      (label #$file-system-label)
                                      (file-system #$file-system-type)
                                      (flags '(boot))
diff --git a/guix/build/store-copy.scm b/guix/build/store-copy.scm
index a296bdf78..fe2eb6f69 100644
--- a/guix/build/store-copy.scm
+++ b/guix/build/store-copy.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2013, 2014 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2013, 2014, 2017 Ludovic Courtès <ludo@gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -20,7 +20,9 @@
   #:use-module (guix build utils)
   #:use-module (srfi srfi-1)
   #:use-module (ice-9 rdelim)
+  #:use-module (ice-9 ftw)
   #:export (read-reference-graph
+            closure-size
             populate-store))
 
 ;;; Commentary:
@@ -46,6 +48,37 @@ The data at PORT is the format produced by #:references-graphs."
            (loop (read-line port)
                  result)))))
 
+(define (file-size file)
+  "Return the size of bytes of FILE, entering it if FILE is a directory."
+  (file-system-fold (const #t)
+                    (lambda (file stat result)    ;leaf
+                      (+ (stat:size stat) result))
+                    (lambda (directory stat result) ;down
+                      (+ (stat:size stat) result))
+                    (lambda (directory stat result) ;up
+                      result)
+                    (lambda (file stat result)    ;skip
+                      result)
+                    (lambda (file stat errno result)
+                      (format (current-error-port)
+                              "file-size: ~a: ~a~%" file
+                              (strerror errno))
+                      result)
+                    0
+                    file
+                    lstat))
+
+(define (closure-size reference-graphs)
+  "Return an estimate of the size of the closure described by
+REFERENCE-GRAPHS, a list of reference-graph files."
+  (define (graph-from-file file)
+    (call-with-input-file file read-reference-graph))
+
+  (define items
+    (delete-duplicates (append-map graph-from-file reference-graphs)))
+
+  (reduce + 0 (map file-size items)))
+
 (define* (populate-store reference-graphs target)
   "Populate the store under directory TARGET with the items specified in
 REFERENCE-GRAPHS, a list of reference-graph files."
diff --git a/guix/scripts/system.scm b/guix/scripts/system.scm
index 35675cc01..7e20b10da 100644
--- a/guix/scripts/system.scm
+++ b/guix/scripts/system.scm
@@ -854,7 +854,7 @@ Some ACTIONS support additional ARGS.\n"))
     (build-hook? . #t)
     (max-silent-time . 3600)
     (verbosity . 0)
-    (image-size . ,(* 900 (expt 2 20)))
+    (image-size . guess)
     (install-bootloader? . #t)))
 
 \f

[bug#27401] Guessing the disk image size in 'guix system'

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> We can do “guix size $(guix system build foo.scm)” and add 10% or so to
> that.
>
> Maybe something like this mostly-untested patch would work?

I’ve committed a variant of this patch as
a8ac4f081a9a679498ea42ccfe001f218bba3043.

I’ve tested it with several OS configs and it worked, except for
“lightweight-desktop.tmpl” where it failed with a confusing GRUB error
that might have been caused by ENOSPC.

Feedback and suggestion to improve the estimate welcome!

Ludo’.