[bug#29100] [PATCH 0/3] Use overlayfs instead of unionfs.

[bug#29100] [PATCH 0/3] Use overlayfs instead of unionfs.

[Hartmut Goebel]

These patches replace the FUSE-based unionfs by the kernel-based overlayfs.
For the insallation image this reduces the size of the initrd by ca. 4.3%
(487K).

I verified this is working by successfiully running
   make check-system TESTS=installed-os.

One thing I did not change: In gnu/system/install.scm
(installation-os)[file-systems], /tmp is still mounted to tmpfs.  This was
introduced in ee03b75dfb3399f41002c38ac512473ab94afa74 for unionfs because
FUSE creates .fuse_hiddenXYZ' files for each open file, and this confuses
Guix's test suite, for instance (see <http://bugs.gnu.org/23056>).  This
should no longer be necessary when using overlayfs, since overlayfs uses a
separate workdir.  However I was not able to trigger bug 23056 while still
using unionfs, so I could not verify whether this mount is still needed with
overlayfs and I decided to keep it.

Hartmut Goebel (3):
  build: Use overlayfs instead of unionfs.
  install: Use overlayfs instead of unionfs.
  gnu: service: Update comment.

 gnu/build/linux-boot.scm    | 43 ++++++++++++---------------------------
 gnu/services/base.scm       |  2 +-
 gnu/system/install.scm      | 49 ++++++++++++++++++---------------------------
 gnu/system/linux-initrd.scm |  4 ++--
 4 files changed, 35 insertions(+), 63 deletions(-)

-- 
2.13.5

[bug#29100] [PATCH 1/3] build: Use overlayfs instead of unionfs.

[Hartmut Goebel]

Overlayfs is part of the kernel, while unionfs needs FUSE.  This also reduces
the size of the initrd by ca. 4.3% (487K).

* gnu/build/linux-boot.scm (mount-root-file-system): Remove optional parameter
  "unionfs"; mount using overlayfs instead of unionfs; new directory layout
  requied by overlayfs; update documentation;
  [mark-as-not-killable]: remove now unused function

* gnu/system/linux-intrd.scm (file-system-packages): Remove now unused
  packages "unionfs-fuse/static".
  (linux-modules): Replace "fuse" by "overlay".
---
 gnu/build/linux-boot.scm    | 43 +++++++++++++------------------------------
 gnu/system/linux-initrd.scm |  4 ++--
 2 files changed, 15 insertions(+), 32 deletions(-)

diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scm
index 3712abe91..7f07e8038 100644
--- a/gnu/build/linux-boot.scm
+++ b/gnu/build/linux-boot.scm
@@ -241,20 +241,10 @@ the last argument of `mknod'."
           (filter-map string->number (scandir "/proc")))))
 
 (define* (mount-root-file-system root type
-                                 #:key volatile-root? (unionfs "unionfs"))
+                                 #:key volatile-root?)
   "Mount the root file system of type TYPE at device ROOT.  If VOLATILE-ROOT?
-is true, mount ROOT read-only and make it a union with a writable tmpfs using
-UNIONFS."
-  (define (mark-as-not-killable pid)
-    ;; Tell the 'user-processes' shepherd service that PID must be kept alive
-    ;; when shutting down.
-    (mkdir-p "/root/etc/shepherd")
-    (let ((port (open-file "/root/etc/shepherd/do-not-kill" "a")))
-      (chmod port #o600)
-      (write pid port)
-      (newline port)
-      (close-port port)))
-
+is true, mount ROOT read-only and make it a overlay with a writable tmpfs
+using the kernel build-in overlayfs."
   (if volatile-root?
       (begin
         (mkdir-p "/real-root")
@@ -262,24 +252,17 @@ UNIONFS."
         (mkdir-p "/rw-root")
         (mount "none" "/rw-root" "tmpfs")
 
+        ;; Create the upperdir and the workdir of the overlayfs
+        (mkdir-p "/rw-root/upper")
+        (mkdir-p "/rw-root/work")
+
         ;; We want read-write /dev nodes.
-        (mkdir-p "/rw-root/dev")
-        (mount "none" "/rw-root/dev" "devtmpfs")
-
-        ;; Make /root a union of the tmpfs and the actual root.  Use
-        ;; 'max_files' to set a high RLIMIT_NOFILE for the unionfs process
-        ;; itself.  Failing to do that, we quickly run out of file
-        ;; descriptors; see <http://bugs.gnu.org/17827>.
-        (unless (zero? (system* unionfs "-o"
-                                "cow,allow_other,use_ino,suid,dev,max_files=65536"
-                                "/rw-root=RW:/real-root=RO"
-                                "/root"))
-          (error "unionfs failed"))
-
-        ;; Make sure unionfs remains alive till the end.  Because
-        ;; 'fuse_daemonize' doesn't tell the PID of the forked daemon, we
-        ;; have to resort to 'pidof' here.
-        (mark-as-not-killable (pidof unionfs)))
+        (mkdir-p "/rw-root/upper/dev")
+        (mount "none" "/rw-root/upper/dev" "devtmpfs")
+
+        ;; Make /root a overlay of the tmpfs and the actual root.
+        (mount "none" "/root" "overlay" 0
+               "lowerdir=/real-root,upperdir=/rw-root/upper,workdir=/rw-root/work"))
       (begin
         (check-file-system root type)
         (mount root "/root" type)))
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 948c543a1..4168582c4 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -236,7 +236,7 @@ FILE-SYSTEMS."
           (list btrfs-progs/static)
           '())
     ,@(if volatile-root?
-          (list unionfs-fuse/static)
+          (list ) ;; nothing requierd in this case
           '())))
 
 (define* (base-initrd file-systems
@@ -308,7 +308,7 @@ loaded at boot time in the order in which they appear."
             '("isofs")
             '())
       ,@(if volatile-root?
-            '("fuse")
+            '("overlay")
             '())
       ,@extra-modules))
 
-- 
2.13.5

[bug#29100] [PATCH 2/3] install: Use overlayfs instead of unionfs.

[Hartmut Goebel]

I dediced to keep adding /tmp as tmpfs since I was not able to trigger bug
while still using unionfs, so I could not verify whether this mount is still
needed with overlayfs.  Mapping /tmp to tmpfs does not harm, so we are on the
save side.

* gnu/system/install.scm (make-cow-store): Mount /gnu/store without additional
  read-only bind-mount, since in overlayfs the "lower" level is always
  read-only. Add work-dir required by overlayfs. No need to sleep anymore
  since now using the mount syscall. [unionfs]: remove now unused function.
  (%installation-services): Update comment.
  (installation-os)[file-systems]: Update comment.
---
 gnu/system/install.scm | 49 +++++++++++++++++++------------------------------
 1 file changed, 19 insertions(+), 30 deletions(-)

diff --git a/gnu/system/install.scm b/gnu/system/install.scm
index eb362f91a..0c771e81f 100644
--- a/gnu/system/install.scm
+++ b/gnu/system/install.scm
@@ -71,19 +71,6 @@ manual."
   "Return a gexp that makes the store copy-on-write, using TARGET as the
 backing store.  This is useful when TARGET is on a hard disk, whereas the
 current store is on a RAM disk."
-  (define (unionfs read-only read-write mount-point)
-    ;; Make MOUNT-POINT the union of READ-ONLY and READ-WRITE.
-
-    ;; Note: in the command below, READ-WRITE appears before READ-ONLY so that
-    ;; it is considered a "higher-level branch", as per unionfs-fuse(8),
-    ;; thereby allowing files existing on READ-ONLY to be copied over to
-    ;; READ-WRITE.
-    #~(fork+exec-command
-       (list (string-append #$unionfs-fuse "/bin/unionfs")
-             "-o"
-             "cow,allow_other,use_ino,max_files=65536,nonempty"
-             (string-append #$read-write "=RW:" #$read-only "=RO")
-             #$mount-point)))
 
   (define (set-store-permissions directory)
     ;; Set the right perms on DIRECTORY to use it as the store.
@@ -97,23 +84,23 @@ current store is on a RAM disk."
         (mkdir-p tmpdir)
         (mount tmpdir "/tmp" "none" MS_BIND))
 
-      (unless (file-exists? "/.ro-store")
-        (mkdir "/.ro-store")
-        (mount #$(%store-prefix) "/.ro-store" "none"
-               (logior MS_BIND MS_RDONLY)))
-
-      (let ((rw-dir (string-append target #$%backing-directory)))
+      (let* ((rw-dir (string-append target #$%backing-directory))
+             ;; FIXME: calculate work-dir from backing-directory:
+             ;; normpath(backing-directory + "../.overlayfs-workdir")
+             (work-dir (string-append target "/tmp/.overlayfs-workdir")))
         (mkdir-p rw-dir)
+        (mkdir-p work-dir)
         (mkdir-p "/.rw-store")
         #$(set-store-permissions #~rw-dir)
         #$(set-store-permissions "/.rw-store")
 
-        ;; Mount the union, then atomically make it the store.
-        (and #$(unionfs "/.ro-store" #~rw-dir "/.rw-store")
-             (begin
-               (sleep 1) ;XXX: wait for unionfs to be ready
-               (mount "/.rw-store" #$(%store-prefix) "" MS_MOVE)
-               (rmdir "/.rw-store"))))))
+        ;; Mount the overlay, then atomically make it the store.
+        (mount "none" "/.rw-store" "overlay" 0
+               (string-append "lowerdir=" #$(%store-prefix) ","
+                              "upperdir=" rw-dir ","
+                              "workdir=" work-dir))
+        (mount "/.rw-store" #$(%store-prefix) "" MS_MOVE)
+        (rmdir "/.rw-store"))))
 
 (define cow-store-service-type
   (shepherd-service-type
@@ -278,7 +265,7 @@ You have been warned.  Thanks for being so brave.
                     (allow-empty-passwords? #f)
                     (password-authentication? #t)))
 
-          ;; Since this is running on a USB stick with a unionfs as the root
+          ;; Since this is running on a USB stick with a overlayfs as the root
           ;; file system, use an appropriate cache configuration.
           (nscd-service (nscd-configuration
                          (caches %nscd-minimal-caches)))
@@ -317,10 +304,12 @@ Use Alt-F2 for documentation.
               (title 'label)
               (type "ext4"))
 
-            ;; Make /tmp a tmpfs instead of keeping the unionfs.  This is
-            ;; because FUSE creates '.fuse_hiddenXYZ' files for each open file,
-            ;; and this confuses Guix's test suite, for instance.  See
-            ;; <http://bugs.gnu.org/23056>.
+            ;; Make /tmp a tmpfs instead of keeping the overlayfs.  This
+            ;; originally was used for unionfs because FUSE creates
+            ;; '.fuse_hiddenXYZ' files for each open file, and this confuses
+            ;; Guix's test suite, for instance (see
+            ;; <http://bugs.gnu.org/23056>).  We keep this for overlayfs to be
+            ;; on the save side.
             (file-system
               (mount-point "/tmp")
               (device "none")
-- 
2.13.5

[bug#29100] [PATCH 2/3] install: Use overlayfs instead of unionfs.

[Danny Milosavljevic]

[...]
> -      (let ((rw-dir (string-append target #$%backing-directory)))
> +      (let* ((rw-dir (string-append target #$%backing-directory))
> +             ;; FIXME: calculate work-dir from backing-directory:
> +             ;; normpath(backing-directory + "../.overlayfs-workdir")
> +             (work-dir (string-append target "/tmp/.overlayfs-workdir")))

What about the FIXME?

[bug#29100] [PATCH 2/3] install: Use overlayfs instead of unionfs.

[Hartmut Goebel]

Am 02.11.2017 um 18:44 schrieb Danny Milosavljevic:
> [...]
>> -      (let ((rw-dir (string-append target #$%backing-directory)))
>> +      (let* ((rw-dir (string-append target #$%backing-directory))
>> +             ;; FIXME: calculate work-dir from backing-directory:
>> +             ;; normpath(backing-directory + "../.overlayfs-workdir")
>> +             (work-dir (string-append target "/tmp/.overlayfs-workdir")))
> What about the FIXME?

I'll change it into a TODO. Somebody with more knowledge in scheme as to
implement this.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

[bug#29100] [PATCH 2/3] install: Use overlayfs instead of unionfs.

[Ludovic Courtès]

Hi Hartmut,

Hartmut Goebel <h.goebel@crazy-compilers.com> skribis:

> Am 02.11.2017 um 18:44 schrieb Danny Milosavljevic:
>> [...]
>>> -      (let ((rw-dir (string-append target #$%backing-directory)))
>>> +      (let* ((rw-dir (string-append target #$%backing-directory))
>>> +             ;; FIXME: calculate work-dir from backing-directory:
>>> +             ;; normpath(backing-directory + "../.overlayfs-workdir")
>>> +             (work-dir (string-append target "/tmp/.overlayfs-workdir")))
>> What about the FIXME?
>
> I'll change it into a TODO. Somebody with more knowledge in scheme as to
> implement this.

What does this comment mean exactly?  The “has to” suggests that it’s
important.  :-)

Perhaps you mean:

  (canonicalize-path (string-append rw-dir "/../.overlayfs-workdir"))

?

Ludo’.

[bug#29100] [PATCH 2/3] install: Use overlayfs instead of unionfs.

[Hartmut Goebel]

Am 05.11.2017 um 22:05 schrieb Ludovic Courtès:
> Perhaps you mean:
>
>   (canonicalize-path (string-append rw-dir "/../.overlayfs-workdir"))

Exactly :-) Thanks.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

[bug#29100] [PATCH 3/3] gnu: service: Update comment.

[Hartmut Goebel]

* gnu/services/base.scm(guix-activation): Update comment.
---
 gnu/services/base.scm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/gnu/services/base.scm b/gnu/services/base.scm
index b605614ab..2ba655d94 100644
--- a/gnu/services/base.scm
+++ b/gnu/services/base.scm
@@ -1441,7 +1441,7 @@ failed to register hydra.gnu.org public key: ~a~%" status))))))))
   (match config
     (($ <guix-configuration> guix build-group build-accounts authorize-key? keys)
      ;; Assume that the store has BUILD-GROUP as its group.  We could
-     ;; otherwise call 'chown' here, but the problem is that on a COW unionfs,
+     ;; otherwise call 'chown' here, but the problem is that on a COW overlayfs,
      ;; chown leads to an entire copy of the tree, which is a bad idea.
 
      ;; Optionally authorize hydra.gnu.org's key.
-- 
2.13.5

[bug#29100] [PATCH 3/3] gnu: service: Update comment.

[Ludovic Courtès]

Hartmut Goebel <h.goebel@crazy-compilers.com> skribis:

> * gnu/services/base.scm(guix-activation): Update comment.
                         ^
Missing space.

Otherwise OK!

Ludo’.

[bug#29100] [PATCH 1/3] build: Use overlayfs instead of unionfs.

[Ludovic Courtès]

Heya,

Hartmut Goebel <h.goebel@crazy-compilers.com> skribis:

> Overlayfs is part of the kernel, while unionfs needs FUSE.  This also reduces
> the size of the initrd by ca. 4.3% (487K).

Awesome!  It’s definitely an improvement, and I’m happy that it fixes
<http://bugs.gnu.org/23056>.

> * gnu/build/linux-boot.scm (mount-root-file-system): Remove optional parameter
>   "unionfs"; mount using overlayfs instead of unionfs; new directory layout
>   requied by overlayfs; update documentation;
>   [mark-as-not-killable]: remove now unused function
>
> * gnu/system/linux-intrd.scm (file-system-packages): Remove now unused
                     ^^^
Typo.

>   packages "unionfs-fuse/static".
>   (linux-modules): Replace "fuse" by "overlay".

Nitpick: please capitalize sentences, add missing periods.

> +        ;; Make /root a overlay of the tmpfs and the actual root.

s/a/an/

> --- a/gnu/system/linux-initrd.scm
> +++ b/gnu/system/linux-initrd.scm
> @@ -236,7 +236,7 @@ FILE-SYSTEMS."
>            (list btrfs-progs/static)
>            '())
>      ,@(if volatile-root?
> -          (list unionfs-fuse/static)
> +          (list ) ;; nothing requierd in this case
>            '())))

You can remove the whole ‘if’ here.

OK with these changes, thank you!

Ludo’.

[bug#29100] [PATCH 1/3] build: Use overlayfs instead of unionfs.

[Hartmut Goebel]

Am 05.11.2017 um 22:08 schrieb Ludovic Courtès:
>>      ,@(if volatile-root?
>> -          (list unionfs-fuse/static)
>> +          (list ) ;; nothing requierd in this case
>>            '())))
> You can remove the whole ‘if’ here.

Okay. should I remove the then unused #:key volatile-root?, too? This
would change the function's signature and may require more changes at
other places.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

[bug#29100] [PATCH 1/3] build: Use overlayfs instead of unionfs.

[Ludovic Courtès]

Hartmut Goebel <h.goebel@crazy-compilers.com> skribis:

> Am 05.11.2017 um 22:08 schrieb Ludovic Courtès:
>>>      ,@(if volatile-root?
>>> -          (list unionfs-fuse/static)
>>> +          (list ) ;; nothing requierd in this case
>>>            '())))
>> You can remove the whole ‘if’ here.
>
> Okay. should I remove the then unused #:key volatile-root?, too?

No, I think you can keep it (and also the overlay.ko is added
conditionally based on that IIRC.)

Thanks,
Ludo’.

bug#29100: Updated, tested again and merged

[Hartmut Goebel]

Addressed all concerns and merged as
0af94ad52f65a5a62febd203e431f17642f07577.

I re-testet it, based on cd5c3979d (@master) using make check-system
TESTS=installed-os