unofficial mirror of bug-guix@gnu.org 
 help / color / mirror / code / Atom feed
* bug#52919: Hidden "disk-image-rw" files aren't deleted after use, filling $tmpdir
@ 2021-12-31 18:18 Leo Famulari
  2022-01-03 14:56 ` Mathieu Othacehe
  0 siblings, 1 reply; 9+ messages in thread
From: Leo Famulari @ 2021-12-31 18:18 UTC (permalink / raw)
  To: 52919

I noticed that files such as
".gk8hqaqb6z4q1gmk678sixjbc9r7kd9g-disk-image-rw" are filling up my
$TMPDIR when building ISO images with Guix.

This is new. I think the issue was introduced with recent work on (gnu
system image), perhaps from this commit:

https://git.savannah.gnu.org/cgit/guix.git/commit/?id=da924796744bbb7b035a986fe5d28d8d613ff6af




^ permalink raw reply	[flat|nested] 9+ messages in thread

* bug#52919: Hidden "disk-image-rw" files aren't deleted after use, filling $tmpdir
  2021-12-31 18:18 bug#52919: Hidden "disk-image-rw" files aren't deleted after use, filling $tmpdir Leo Famulari
@ 2022-01-03 14:56 ` Mathieu Othacehe
  2022-01-04  8:17   ` Ludovic Courtès
  0 siblings, 1 reply; 9+ messages in thread
From: Mathieu Othacehe @ 2022-01-03 14:56 UTC (permalink / raw)
  To: Leo Famulari; +Cc: 52919


Hello Leo,

> This is new. I think the issue was introduced with recent work on (gnu
> system image), perhaps from this commit:
>
> https://git.savannah.gnu.org/cgit/guix.git/commit/?id=da924796744bbb7b035a986fe5d28d8d613ff6af

Before that commit "guix system vm" was always producing volatile
images, regardless of the "--volatile" argument. The new behaviour is to
only produce volatile images if "--volatile" is passed.

This means that by default, "disk-image-rw" are created in /tmp so that
QEMU can operate on RW images. The side effect is that those big images
can fill /tmp pretty quickly as you noticed.

We could use the ~/.cache/guix directory instead to store those
images. We could also make "--volatile" the default and introduce a
"--persistent" argument instead. That would restore the previous "guix
system vm" behaviour.

WDYT?

Thanks,

Mathieu





^ permalink raw reply	[flat|nested] 9+ messages in thread

* bug#52919: Hidden "disk-image-rw" files aren't deleted after use, filling $tmpdir
  2022-01-03 14:56 ` Mathieu Othacehe
@ 2022-01-04  8:17   ` Ludovic Courtès
  2022-01-04 17:26     ` Leo Famulari
  0 siblings, 1 reply; 9+ messages in thread
From: Ludovic Courtès @ 2022-01-04  8:17 UTC (permalink / raw)
  To: Mathieu Othacehe; +Cc: 52919

Hi,

Mathieu Othacehe <othacehe@gnu.org> skribis:

>> This is new. I think the issue was introduced with recent work on (gnu
>> system image), perhaps from this commit:
>>
>> https://git.savannah.gnu.org/cgit/guix.git/commit/?id=da924796744bbb7b035a986fe5d28d8d613ff6af
>
> Before that commit "guix system vm" was always producing volatile
> images, regardless of the "--volatile" argument. The new behaviour is to
> only produce volatile images if "--volatile" is passed.

I had overlooked the change of default.

> This means that by default, "disk-image-rw" are created in /tmp so that
> QEMU can operate on RW images. The side effect is that those big images
> can fill /tmp pretty quickly as you noticed.
>
> We could use the ~/.cache/guix directory instead to store those
> images. We could also make "--volatile" the default and introduce a
> "--persistent" argument instead. That would restore the previous "guix
> system vm" behaviour.

Yes, reverting to the old behavior for ‘guix system vm’ and adding
‘--persistent’ sounds like a nice option to me.

As for the image backing storage, using ~/.cache/guix/images (with non
dot-prefixed file names) sounds cleaner and safer to me (no risk of /tmp
race or whatever).

Thanks,
Ludo’.




^ permalink raw reply	[flat|nested] 9+ messages in thread

* bug#52919: Hidden "disk-image-rw" files aren't deleted after use, filling $tmpdir
  2022-01-04  8:17   ` Ludovic Courtès
@ 2022-01-04 17:26     ` Leo Famulari
  2022-01-11 12:33       ` Mathieu Othacehe
  0 siblings, 1 reply; 9+ messages in thread
From: Leo Famulari @ 2022-01-04 17:26 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: Mathieu Othacehe, 52919

On Tue, Jan 04, 2022 at 09:17:53AM +0100, Ludovic Courtès wrote:
> As for the image backing storage, using ~/.cache/guix/images (with non
> dot-prefixed file names) sounds cleaner and safer to me (no risk of /tmp
> race or whatever).

I wonder if there are any other tools in Guix that tend to create
multi-gigabyte files in "$HOME"? It might be a bit surprising, but maybe
not a big deal.




^ permalink raw reply	[flat|nested] 9+ messages in thread

* bug#52919: Hidden "disk-image-rw" files aren't deleted after use, filling $tmpdir
  2022-01-04 17:26     ` Leo Famulari
@ 2022-01-11 12:33       ` Mathieu Othacehe
  2022-01-12 20:54         ` Ludovic Courtès
  0 siblings, 1 reply; 9+ messages in thread
From: Mathieu Othacehe @ 2022-01-11 12:33 UTC (permalink / raw)
  To: Leo Famulari; +Cc: 52919

[-- Attachment #1: Type: text/plain, Size: 767 bytes --]


Hey,

Here is a patch to store the rw images in the cache directory. Regarding
the volatile flag, things are a little more complex than what I thought
initially.

The "guix system" help menu states that:

--8<---------------cut here---------------start------------->8---
      --volatile         for 'image', make the root file system volatile
--8<---------------cut here---------------end--------------->8---

because for an image, it often makes sense to have the root file system
persistent. For the 'vm' command however, it's probably the other way
around as we would prefer not to copy the huge image files if possible.

I feel like creating a '--volatile-image' and a '--persistent-vm' option
is going to make things too much complex.

WDYT?

Thanks,

Mathieu

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-system-vm-Do-not-store-rw-image-in-the-tmp-directory.patch --]
[-- Type: text/x-patch, Size: 2073 bytes --]

From e4efaf2ffc6d7c699f9d1c41744cfff89be65c12 Mon Sep 17 00:00:00 2001
From: Mathieu Othacehe <othacehe@gnu.org>
Date: Tue, 11 Jan 2022 13:27:35 +0100
Subject: [PATCH 1/1] system: vm: Do not store rw image in the /tmp directory.

* gnu/system/vm.scm (system-qemu-image/shared-store-script): Store them in the
cache directory instead.
---
 gnu/system/vm.scm | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)

diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 0fc9fb57f4..bd21f9416c 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -283,7 +283,9 @@ (define kernel-arguments
               #+@(operating-system-kernel-arguments os "/dev/vda1")))
 
     (define rw-image
-      #~(format #f "/tmp/.~a-rw" (basename #$base-image)))
+      #~(format #f "~a/~a-rw"
+                #$(string-append (cache-directory) "/images")
+                (basename #$base-image)))
 
     (define qemu-exec
       #~(list #+(file-append qemu "/bin/"
@@ -310,10 +312,15 @@ (define builder
                     #+(file-append bash "/bin/sh"))
             (when (not #$volatile?)
               (format port "~a~%"
-                      #$(program-file "copy-image"
-                                      #~(unless (file-exists? #$rw-image)
-                                          (copy-file #$base-image #$rw-image)
-                                          (chmod #$rw-image #o640)))))
+                      #$(program-file
+                         "copy-image"
+                         (with-imported-modules '((guix build utils))
+                           #~(begin
+                               (use-modules (guix build utils))
+                               (unless (file-exists? #$rw-image)
+                                 (mkdir-p (dirname #$rw-image))
+                                 (copy-file #$base-image #$rw-image)
+                                 (chmod #$rw-image #o640)))))))
             (format port "exec ~a \"$@\"~%"
                     (string-join #$qemu-exec " "))
             (chmod port #o555))))
-- 
2.34.0


^ permalink raw reply	[flat|nested] 9+ messages in thread

* bug#52919: Hidden "disk-image-rw" files aren't deleted after use, filling $tmpdir
  2022-01-11 12:33       ` Mathieu Othacehe
@ 2022-01-12 20:54         ` Ludovic Courtès
  2022-01-13 10:55           ` Mathieu Othacehe
  0 siblings, 1 reply; 9+ messages in thread
From: Ludovic Courtès @ 2022-01-12 20:54 UTC (permalink / raw)
  To: Mathieu Othacehe; +Cc: 52919

Hi,

Mathieu Othacehe <othacehe@gnu.org> skribis:

> Here is a patch to store the rw images in the cache directory. Regarding
> the volatile flag, things are a little more complex than what I thought
> initially.
>
> The "guix system" help menu states that:
>
>       --volatile         for 'image', make the root file system volatile
>
> because for an image, it often makes sense to have the root file system
> persistent. For the 'vm' command however, it's probably the other way
> around as we would prefer not to copy the huge image files if possible.
>
> I feel like creating a '--volatile-image' and a '--persistent-vm' option
> is going to make things too much complex.
>
> WDYT?

Hmm.  Can we keep “image” persistent by default, and make ‘vm’ volatile
by default?  That way, ‘--volatile’ would only make sense for ‘image’,
and ‘--persistent’ would only make sense for ‘vm’.  (So we’d be adding
just one option: ‘--persistent’.)

WDYT?

> From e4efaf2ffc6d7c699f9d1c41744cfff89be65c12 Mon Sep 17 00:00:00 2001
> From: Mathieu Othacehe <othacehe@gnu.org>
> Date: Tue, 11 Jan 2022 13:27:35 +0100
> Subject: [PATCH 1/1] system: vm: Do not store rw image in the /tmp directory.
>
> * gnu/system/vm.scm (system-qemu-image/shared-store-script): Store them in the
> cache directory instead.

Leo had a point on the size of those files, and the fact that if they’re
in ~/.cache they’re not going to be removed automatically, which could
lead to user disks filling up.

So after all, it might be best to keep those images in ${TMPDIR:-/tmp}.

I would still ensure they have a name like “guix-image-$USER-XXX”, where
XXX is the store file basename.

Thoughts?

Apologies for the hesitations!

Ludo’.




^ permalink raw reply	[flat|nested] 9+ messages in thread

* bug#52919: Hidden "disk-image-rw" files aren't deleted after use, filling $tmpdir
  2022-01-12 20:54         ` Ludovic Courtès
@ 2022-01-13 10:55           ` Mathieu Othacehe
  2022-01-13 14:49             ` Ludovic Courtès
  0 siblings, 1 reply; 9+ messages in thread
From: Mathieu Othacehe @ 2022-01-13 10:55 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: 52919

[-- Attachment #1: Type: text/plain, Size: 664 bytes --]


Hey,

> Hmm.  Can we keep “image” persistent by default, and make ‘vm’ volatile
> by default?  That way, ‘--volatile’ would only make sense for ‘image’,
> and ‘--persistent’ would only make sense for ‘vm’.  (So we’d be adding
> just one option: ‘--persistent’.)
>
> WDYT?

I'm not fan of adding antithetic options: --x and --no-x. There's an
attached patch introducing --volatile-image and --persistent-vm options,
and documenting them. It's maybe not that bad after all.

> I would still ensure they have a name like “guix-image-$USER-XXX”, where
> XXX is the store file basename.

Sure.

Thanks,

Mathieu

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-scripts-system-Rationalize-persistency.patch --]
[-- Type: text/x-patch, Size: 6954 bytes --]

From b0c84a411f9f23f4f1a4155ba5efa68cac9004a2 Mon Sep 17 00:00:00 2001
From: Mathieu Othacehe <othacehe@gnu.org>
Date: Thu, 13 Jan 2022 11:35:40 +0100
Subject: [PATCH 1/2] scripts: system: Rationalize persistency.

Make sure that the images are created with a non volatile root by default and
the vm are created with a volatile root by default. Break the --volatile
option into --volatile-image and --persistent-vm options.

* guix/scripts/system.scm (perform-action): Turn volatile? argument into
volatile-vm-root?.
(show-help): Introduce --volatile-image and --persistent-vm options instead of
--volatile.
(%default-options): Adapt it.
(%options): Handle those options.
(process-action): Honor them.
* doc/guix.texi (Invoking guix system): Adapt it accordingly.
---
 doc/guix.texi           | 15 ++++++++++-----
 guix/scripts/system.scm | 25 +++++++++++++++++--------
 2 files changed, 27 insertions(+), 13 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index bc289bad7b..9f763bcfa7 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -35152,6 +35152,11 @@ $ $(guix system vm my-config.scm) -m 1024 -smp 2 -nic user,model=virtio-net-pci
 
 The VM shares its store with the host system.
 
+By default, the root file system of the VM is mounted volatile; the
+@option{--persistent-vm} option can be provided to make it persistent
+instead.  In that case, the VM disk-image file will be copied from the
+store to the @env{TMPDIR} directory to make it writable.
+
 Additional file systems can be shared between the host and the VM using
 the @option{--share} and @option{--expose} command-line options: the former
 specifies a directory to be shared with write access, while the latter
@@ -35189,14 +35194,14 @@ QEMU monitor and the VM.
 @cindex Creating system images in various formats
 @item image
 @cindex image, creating disk images
-The @code{image} command can produce various image types.  The
-image type can be selected using the @option{--image-type} option.  It
+The @code{image} command can produce various image types.  The image
+type can be selected using the @option{--image-type} option.  It
 defaults to @code{efi-raw}.  When its value is @code{iso9660}, the
 @option{--label} option can be used to specify a volume ID with
 @code{image}.  By default, the root file system of a disk image is
-mounted non-volatile; the @option{--volatile} option can be provided to
-make it volatile instead.  When using @code{image}, the bootloader
-installed on the generated image is taken from the provided
+mounted non-volatile; the @option{--volatile-image} option can be
+provided to make it volatile instead.  When using @code{image}, the
+bootloader installed on the generated image is taken from the provided
 @code{operating-system} definition.  The following example demonstrates
 how to generate an image that uses the @code{grub-efi-bootloader}
 bootloader and boot it with QEMU:
diff --git a/guix/scripts/system.scm b/guix/scripts/system.scm
index 98e788c657..3ca5592e34 100644
--- a/guix/scripts/system.scm
+++ b/guix/scripts/system.scm
@@ -772,7 +772,7 @@ (define* (perform-action action image
                          dry-run? derivations-only?
                          use-substitutes? target
                          full-boot?
-                         volatile?
+                         volatile-vm-root?
                          (graphic? #t)
                          container-shared-network?
                          (mappings '())
@@ -827,7 +827,8 @@ (define bootcfg
   (mlet* %store-monad
       ((sys       (system-derivation-for-action image action
                                                 #:full-boot? full-boot?
-                                                #:volatile? volatile?
+                                                #:volatile?
+                                                volatile-vm-root?
                                                 #:graphic? graphic?
                                                 #:container-shared-network? container-shared-network?
                                                 #:mappings mappings))
@@ -997,7 +998,9 @@ (define (show-help)
   (display (G_ "
       --no-bootloader    for 'init', do not install a bootloader"))
   (display (G_ "
-      --volatile         for 'image', make the root file system volatile"))
+      --volatile-image   for 'image', make the root file system volatile"))
+  (display (G_ "
+      --persistent-vm    for 'vm', make the root file system persistent"))
   (display (G_ "
       --label=LABEL      for 'image', label disk image with LABEL"))
   (display (G_ "
@@ -1078,9 +1081,12 @@ (define %options
          (option '("no-bootloader" "no-grub") #f #f
                  (lambda (opt name arg result)
                    (alist-cons 'install-bootloader? #f result)))
-         (option '("volatile") #f #f
+         (option '("volatile-image") #f #f
+                 (lambda (opt name arg result)
+                   (alist-cons 'volatile-image-root? #t result)))
+         (option '("persistent-vm") #f #f
                  (lambda (opt name arg result)
-                   (alist-cons 'volatile-root? #t result)))
+                   (alist-cons 'volatile-vm-root? #f result)))
          (option '("label") #t #f
                  (lambda (opt name arg result)
                    (alist-cons 'label arg result)))
@@ -1149,7 +1155,8 @@ (define %default-options
     (image-size . guess)
     (install-bootloader? . #t)
     (label . #f)
-    (volatile-root? . #f)
+    (volatile-image-root? . #f)
+    (volatile-vm-root? . #t)
     (graph-backend . "graphviz")))
 
 (define (verbosity-level opts)
@@ -1219,7 +1226,8 @@ (define save-provenance?
                                            ((docker-image) docker-image-type)
                                            (else image-type)))
                             (image-size (assoc-ref opts 'image-size))
-                            (volatile?  (assoc-ref opts 'volatile-root?))
+                            (volatile?
+                             (assoc-ref opts 'volatile-image-root?))
                             (shared-network?
                                (assoc-ref opts 'container-shared-network?))
                             (base-image (if (operating-system? obj)
@@ -1279,7 +1287,8 @@ (define (graph-backend)
                                #:validate-reconfigure
                                (assoc-ref opts 'validate-reconfigure)
                                #:full-boot? (assoc-ref opts 'full-boot?)
-                               #:volatile? (assoc-ref opts 'volatile-root?)
+                               #:volatile-vm-root?
+                               (assoc-ref opts 'volatile-vm-root?)
                                #:graphic? (not (assoc-ref opts 'no-graphic?))
                                #:container-shared-network?
                                (assoc-ref opts 'container-shared-network?)
-- 
2.34.0


^ permalink raw reply	[flat|nested] 9+ messages in thread

* bug#52919: Hidden "disk-image-rw" files aren't deleted after use, filling $tmpdir
  2022-01-13 10:55           ` Mathieu Othacehe
@ 2022-01-13 14:49             ` Ludovic Courtès
  2022-01-14 11:27               ` Mathieu Othacehe
  0 siblings, 1 reply; 9+ messages in thread
From: Ludovic Courtès @ 2022-01-13 14:49 UTC (permalink / raw)
  To: Mathieu Othacehe; +Cc: 52919

Hello,

Mathieu Othacehe <othacehe@gnu.org> skribis:

>> Hmm.  Can we keep “image” persistent by default, and make ‘vm’ volatile
>> by default?  That way, ‘--volatile’ would only make sense for ‘image’,
>> and ‘--persistent’ would only make sense for ‘vm’.  (So we’d be adding
>> just one option: ‘--persistent’.)
>>
>> WDYT?
>
> I'm not fan of adding antithetic options: --x and --no-x. There's an
> attached patch introducing --volatile-image and --persistent-vm options,
> and documenting them. It's maybe not that bad after all.

[...]

> From b0c84a411f9f23f4f1a4155ba5efa68cac9004a2 Mon Sep 17 00:00:00 2001
> From: Mathieu Othacehe <othacehe@gnu.org>
> Date: Thu, 13 Jan 2022 11:35:40 +0100
> Subject: [PATCH 1/2] scripts: system: Rationalize persistency.
>
> Make sure that the images are created with a non volatile root by default and
> the vm are created with a volatile root by default. Break the --volatile
> option into --volatile-image and --persistent-vm options.
>
> * guix/scripts/system.scm (perform-action): Turn volatile? argument into
> volatile-vm-root?.
> (show-help): Introduce --volatile-image and --persistent-vm options instead of
> --volatile.
> (%default-options): Adapt it.
> (%options): Handle those options.
> (process-action): Honor them.
> * doc/guix.texi (Invoking guix system): Adapt it accordingly.

It’s maybe not that important but I’m not convinced about the extra
“-image” and “-vm” suffixes; I don’t think it makes things clearer.


[...]

> -         (option '("volatile") #f #f
> +         (option '("volatile-image") #f #f
> +                 (lambda (opt name arg result)
> +                   (alist-cons 'volatile-image-root? #t result)))

As a rule of thumb, we should not remove an option without going through
a deprecation period.

So if we take that route, “volatile” should still be accepted, only with
deprecation warning emitted.  We can remove it entirely in 1.5.0 or so.

Thanks!

Ludo’.




^ permalink raw reply	[flat|nested] 9+ messages in thread

* bug#52919: Hidden "disk-image-rw" files aren't deleted after use, filling $tmpdir
  2022-01-13 14:49             ` Ludovic Courtès
@ 2022-01-14 11:27               ` Mathieu Othacehe
  0 siblings, 0 replies; 9+ messages in thread
From: Mathieu Othacehe @ 2022-01-14 11:27 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: 52919-done


Hey Ludo!

> It’s maybe not that important but I’m not convinced about the extra
> “-image” and “-vm” suffixes; I don’t think it makes things clearer.

Pushed without those suffixes!

Thanks for reviewing,

Mathieu




^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2022-01-14 11:30 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-12-31 18:18 bug#52919: Hidden "disk-image-rw" files aren't deleted after use, filling $tmpdir Leo Famulari
2022-01-03 14:56 ` Mathieu Othacehe
2022-01-04  8:17   ` Ludovic Courtès
2022-01-04 17:26     ` Leo Famulari
2022-01-11 12:33       ` Mathieu Othacehe
2022-01-12 20:54         ` Ludovic Courtès
2022-01-13 10:55           ` Mathieu Othacehe
2022-01-13 14:49             ` Ludovic Courtès
2022-01-14 11:27               ` Mathieu Othacehe

Code repositories for project(s) associated with this inbox:

	https://git.savannah.gnu.org/cgit/guix.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).